Planning and pilot-testing reduce the amount of disagreement between reviewers but they will not remove it completely. Often, disagreement can be resolved by discussion until the reviewers reach consensus. If that is not possible, the disagreement should be referred to another member of the team to make the final decision.

Second pass: Full text review

  1. The full text review examines in detail the studies that were not excluded at the first pass. There is no ‘maybe’ option for the reviewers at this stage: it’s crunch time.
When your studies move to the full text stage of your review, Covidence automatically adds the PDFs of open access articles. ?  To add the rest, there are two options:
  1. If the article is already saved to your device, simply drag and drop it into your review or use the ‘Upload full text’ button to choose the relevant file.
  2. If you don’t already have the articles, you can retrieve them using your reference manager and import them in bulk.
And if you’re not sure whether you have access to a particular article via your institution you can click the DOI link in Covidence to find out.
Figure 3: Use the DOI link to click through and display the access options
  Working through a list of records, reviewers vote either to include or exclude each of them, based on information contained in the full text of the study report. This means that the reviewer can look at the methods used in the study and check, for example, that the assignment of participants to the treatment and control groups was random. As before, reviewers cast their votes by considering whether or not the studies meet the inclusion and exclusion criteria. These can be conveniently displayed on screen above the list of references in Covidence. In this way, a reference can be excluded as soon as it fails to meet one of the criterion, for example:
  1. Study was not a randomised trial ⛔
  2. Study had the wrong comparator ⛔
  3. Study had the wrong patient population ⛔
Systematic reviewer pro-tip: Save time by listing exclusion criteria in order of importance and working down the list.
Covidence records one exclusion reason per study. If each reviewer gives a different reason, this sends the reference to the ‘Resolve conflicts’ list. Here the review team will implement its agreed conflict resolution process in order to reach agreement on the exclusion reason.  
Figure 4:Full text review in Covidence with inclusion and exclusion criteria displayed

Reaching consensus

Study screening is usually done by two reviewers working independently ??‍???‍?. If their votes agree, the reference either goes forward to data extraction (two ‘yes’ votes) or is excluded from the review (two ‘no’ votes). If the reviewers’ votes do not agree – and this can be on whether or not to include the study, or on the reason given for excluding it – the reference moves to the ‘Resolve conflicts’ list. Again, the reviewers must reach consensus. And again, if they cannot, the disagreement is referred to another member of the team, who has the final say. In Covidence, reviewers are blinded to the voting of others as they work through the references. This is to prevent their decisions being influenced by those of their colleagues. Once all the votes are in and the conflicts are resolved, Covidence automatically populates a flow diagram with the decision data. This documents the flow of studies through the review and follows the reporting standards for systematic reviews as set out in the PRISMA statement ✅. Covidence enables users to add tags and notes to references and to use these to filter and sort lists. If, for example, you have sent a request for information to several study authors, you can add a note to the relevant references to inform your colleagues ?. You could also add a tag to those records and use it to quickly produce a list of all the references for which additional information from study authors has been requested ?.

Measuring agreement

Covidence uses Cohen’s kappa to calculate the level of agreement among reviewers (known as the interrater reliability). Interrater reliability statistics are used to catch ‘coder drift’, the tendency for reviewers to deviate from the process as it becomes more familiar to them. Pilot testing of screening questions and training for reviewers can improve interrater reliability, which can be monitored throughout the screening process.

Extracting the data

The data in the selected studies now need to be extracted in a way that is structured, consistent and reliable. Covidence provides customisable data extraction forms to help you breeze through data extraction. Prefer to build your own form from scratch? That’s fine too. From here you can export data to a CSV file and from there to wherever you like to do your data analysis – the choice is yours ?.

In conclusion

A structured and disciplined approach is key to the success of screening. A large amount of material needs to be sifted systematically and the methods reported accurately and transparently. Covidence saves time on the repetitive tasks of screening and helps you complete your review quickly and efficiently. Sign up for a free trial of Covidence today!