Skip to Main Content

Systematic Reviews: Risk of Bias

Collaborative support for researchers undertaking systematic reviews

What is Risk of Bias?

Risk of bias assessment (sometimes called "quality assessment," "validity assessment," or "critical appraisal") helps to establish transparency of evidence synthesis/systematic review results and findings. A risk of bias assessment is often performed for each included study in your review.  

 

Evidence syntheses/systematic reviews strive to eliminate bias in their findings.  Individual studies that are included in a synthesis may include biases in their results or conclusions, for example design flaws that raise questions about validity of findings or an overestimate of intervention effect.  

 

Risk of bias assessment generally is not required with evidence synthesis methods outside of systematic reviews. For example, scoping reviews typically don't include a risk of bias assessment. Because a scoping review is not aiming to make a clinical recommendation based on its findings, the included sources are typically not critically appraised. 

 

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018 Oct 2;169(7):467–73
 

How do I perform a Risk of Bias assessment?

There are several tools based on study type to assess risk of bias in your included studies. Remember, each study must be assessed individually - if you have 20 included studies, that's 20 risk of bias assessments. The tool you choose will depend on the types of studies you include in your review.

 

The LATITUDES Library is a searchable library of validity assessment tools designed for use in evidence syntheses. The website also provides access to training on the process of validity assessment. Below are some examples of assessment tools:

 

Randomized trials

RoB2 - Revised Cochrane Risk-of-Bias tool for randomized trials 

Non-randomized studies
ROBINS-I - Cochrane's Risk of Bias In Non-randomized Studies – of Interventions 
ROBINS-E - Cochrane's Risk of Bias In Non-randomized Studies - of Exposure
The Newcastle-Ottawa Scale (NOS) for assessing the quality of non-randomized studies
NIH Study Quality Assessment Tools - for Controlled Intervention, Observational Cohort and Cross-Sectional, Case-Control, Case Series Studies
QUADAS – Quality Assessment of Diagnostic Accuracy Studies
QUIPS – Quality in Prognosis Studies
The Joanna Briggs Institute Prevalence Critical Appraisal Tool - for studies reporting prevalence data
PROBAST - Prediction model Risk Of Bias ASsessment Tool
Randomized and Non-randomized studies
Downs and Black Checklist for Quality Assessment

Presenting Risk of Bias

Risk of bias assessments are usually presented in the manuscript as a table based on the findings from the risk of bias tool used. For each study, the individual domains assessed and overall rating are marked as "Low risk of bias" (green +); "Some concerns" (yellow -); or "High risk of bias" (red x).

 

 

robvis – creates publication quality risk-of-bias assessment figures.
Upload data from your risk of bias assessment into the app using robvis templates. Currently supports the following risk of bias assessment tools: RoB2, ROBINS-I, ROBINS-E, QUADAS, QIPS, and a generic form.