Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Areas of Concentration Resources: Systematic Reviews for AoC

What is a Systematic Review?

systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question.  It  uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made.

The key characteristics of a systematic review are:

  • a clearly stated set of objectives with pre-defined eligibility criteria for studies
  • an explicit, reproducible methodology
  • a systematic search that attempts to identify all studies that would meet the eligibility criteria
  • an assessment of the validity of the findings of the included studies
  • a systematic presentation, and synthesis, of the characteristics and findings of the included studies

 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. Many systematic reviews contain meta-analyses. By combining information from all relevant studies, meta-analyses can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review. They also facilitate investigations of the consistency of evidence across studies, and the exploration of differences across studies.

What are the steps to performing a systematic review?


1. Formulate review question.

A focused, well-defined clinical question is essential before beginning any steps in the systematic review process. Starting with a well-articulated idea will help dictate the direction you will take in all the steps that follow.

  • Tools & Resources:

2. Find previous SRs.

It’s important to find out if there is already a recent review or protocol on your topic. A fresh point of view helps your work to be accepted by a scholarly journal. Will your article be redundant or is an update warranted?


3. Write & register the protocol. 

Systematic review standards, best practices, and guidelines, such as PRISMA, The Cochrane Handbook, and The National Academies of Sciences (previously Institute of Medicine) all require or recommend registering a protocol before performing a systematic review. The main purpose of registering a protocol is twofold: it aims to reduce unplanned duplication of systematic reviews and also provides transparency in the review process, therefore minimizing reporting bias. These protocols are registered with PROSPERO, an international prospective register of systematic reviews.

 

This is also where you will define your inclusion/exclusion criteria, a crucial step in the SR process that sets the boundaries for the systematic review. Inclusion criteria are characteristics that the articles must have if they are to be included in the study, while exclusion criteria are those characteristics that disqualify articles from inclusion in the study. These can include, but are not limited to: geographic location, participants, reported outcomes, exposure of interest, setting, study design, type of publication. 

  • Tools & Resources:
    • PRISMA-P is a 17-item checklist for elements considered essential in a protocol for a systematic review or meta-analysis.
    • PROSPERO template

Steps 4 - 6: Devise initial search strategy, Search, De-duplicate results


4. Devise initial search strategy.

A thorough and comprehensive search is the cornerstone to a systematic review. Systematic reviews aim to cast a wide net in order to find all available literature on a given topic. Therefore, systematic review searches can be complex and lengthy, often yielding thousands of results.

This means you’ll need to use a combination of controlled vocabulary (e.g., MeSH for MEDLINE, EMTREE for Embase)  and a comprehensive list of keywords.


5. Search.

Best practice states that a systematic review must search at least 3 different databases. Different databases cover different literature, so in order to be comprehensive several should be included in your systematic review. Three multidisciplinary databases that should at least be searched are:

 

Depending on your topic, you may want to search other discipline specific databases as well. Some that we subscribe to at WCM are:

  • PsycINFO - covers literature in psychology and related behavioral and social sciences, including psychiatry and sociology.
  • CINAHL - nursing & allied health. 
  • AgeLine - literature of social gerontology and includes aging-related content from the health sciences, psychology, sociology, social work.
  • Social Work Abstracts - focused on social work, human services, and related areas.
  • AMED - allied and complementary medicine. 

 

REMEMBER: Not all databases are searched in the same way. Different platforms have different search functionalities and controlled vocabulary. Once you’ve chosen your databases, be sure to look at their advanced search functions to see how to build your search in each database.


6. De-duplicate search results.

There is some overlap between any of the databases you will choose. These duplicates need to be removed before you begin your screening process. Once you’ve exported the results from each database, this can be done in a tool like EndNote, or in a systematic review screening tool like Covidence. After meeting with a faculty librarian for an initial consultation, you will be provided access to Covidence. 

Steps 7 - 9: Screen titles/abstracts, Obtain full text, Screen full text


7. Screen titles/abstracts.

At this first round of screening, you will be looking at just the titles and the abstracts against your inclusion/exclusion criteria to determine eligibility. In order to reduce selection bias, at least 2 reviewers must independently assess each record, with discrepancies resolved by consensus or a 3rd reviewer. This is where the bulk of the records will be screened out.


8. Obtain full text.

For those records that you’ve selected to go to the next stage of screening, you’ll need to track down the PDFs and attach them to the records in Covidence. If you have several hundred, there is a way to do this in bulk, but it requires EndNote desktop client.


9. Screen full text.

At this second round of screening you will be looking at the full article to determine eligibility; this is where you will decide which articles will be included in your review. In addition to inclusion/exclusion agreement, reviewers must also agree on the reason why a record was excluded. This information is reported in the PRISMA flow diagram.

Steps 10 - 12: Reference chasing, Extract data, Risk of bias assessment


10. Reference chasing. 

Once you’ve determined which articles will be included in your review, you’ll then pull the bibliographies and citing articles of those studies and perform another, final round of screening. Search for the article title in a database like Scopus, and you'll find reference cited lists (backward) as well as who has cited the paper (forward).

  • Tools & Resources:

11. Extract data.

According to your pre-defined data points, you’ll now begin pulling data from your final list of included articles. This should also be performed by 2 reviewers independently in order to reduce bias. This can be done with Excel or Covidence. 


12. Risk of bias assessment. 

Risk of bias assessment (sometimes called "quality assessment" or "critical appraisal") helps to establish transparency of evidence synthesis results and findings. A risk of bias assessment should be performed for each included study in your review. There are various tools to help with this process, often sorted by study type.

 

Steps 13 - 15: Re-check literature, Meta-analyze, Write review


13. Re-check literature.

As systematic reviews often take a year or more to complete, it’s necessary to make sure that your searches include the most current literature before you start writing your manuscript. If it has been over a year, searches should be re-run and any new records screened. Many journals require that SR searches be run within a year or even 6 months of submission.

 


14. Meta-analyze.

Meta-analysis is the statistical procedure for combining data from multiple studies. When the treatment effect (or effect size) is consistent from one study to the next, meta-analysis can be used to identify this common effect. When the effect varies from one study to the next, meta-analysis may be used to identify the reason for the variation. Not all systematic reviews can be meta-analyses; it depends on the data available in the studies. 


15. Write Review.

You’re now ready to begin writing your review. There are several guidelines and reporting checklists to help make sure your review is well reported and methodologically sound.