What Are Systematic Reviews?

The best method we have for finding out whether new treatments are better than existing ones is the randomized controlled trial (RCT). In an RCT, people with a particular health condition are recruited and allocated to receive either the new treatment or the conventional treatment (or a placebo in the absence of a conventional treatment).

Crucially, this allocation is determined by chance, meaning that the people included in the two study groups will differ in terms of their characteristics, such as age, sex, or the severity of their condition, but these differences will be random in nature. We can leverage this randomness to learn about the effectiveness (or lack thereof) of the new treatment compared to the conventional treatment or placebo.

It is common for more than one RCT comparing the same two treatments, or the same new treatment to a placebo, for the same health condition to be performed. When this happens, it makes sense to consider the evidence provided by all of the RCTs in combination, rather than looking at each in isolation. We do this by carrying out a systematic review of the evidence. This involves identifying all of the pertinent RCTs and potentially combining their results to arrive at an overall conclusion about whether the treatment helps, harms, or does nothing at all.

Assessing the Quality of RCTs in Systematic Reviews

An important step in the systematic review process involves critically appraising the eligible RCTs. Traditionally, this meant examining published descriptions of the methods used to carry out the RCT, and making a judgment about whether those methods were adequate to reliably assess the effectiveness of the treatment. It is well-understood that design flaws can undermine random allocation in an RCT, and one such example is the lack of allocation concealment. If an investigator knows what the next allocation in the study will be, then this could influence recruitment decisions. For example, if the investigator knows that the next recruited participant will be allocated to the new treatment group, they may make a special effort to recruit an individual who they believe would respond well to the new treatment. In this way, the random nature of the assignment would be undermined by the beliefs and preferences of the investigator. Accordingly, when assessing the methodological rigor of an RCT, we look for descriptions of procedures to ensure the allocation sequence could not be known to people involved in recruiting study participants.

One example of a suitable procedure would be the use of a professional web-based randomization service, which only reveals the allocation to the investigator after the participant has been recruited to the RCT. By assessing various features of the eligible RCTs in this manner, the systematic reviewers can report not only the results of the trials but also their robustness.