Observational studies: answering real-life questions about healthcare practice

Part 4 – Interpreting the results of observational studies

In the last of our series on observational studies, this article addresses some of the issues involved in interpreting the results of such studies.

When interpreting or reviewing the results of any study or trial, there are important questions that should be kept in mind to ensure objective and balanced assessment of both the results and the possible implications for future healthcare.

Some of these questions apply to the conduct and design of the trial, for example:

• was the study conducted in an ethical manner?
• was the design of the study appropriate to answer the research questions that were asked?
• were the research questions clinically relevant and scientifically valuable?

However, bias, confounding factors, heterogeneity of the patient groups, and statistical power can all affect the interpretation and implication of the results. These potential influences need to be examined closely.

Bias
Bias occurs when preconceptions lead to incorrect conclusions about the effects of treatment. It is important to avoid bias in health research as it distorts outcomes – it could even result in an unsafe or inefficient treatment being licensed for use, or useful treatments being overlooked. Bias is avoided in RCTs by the process of randomisation, and in observational studies statistical analyses can minimise its effects.

Confounding factors
‘Confounding’ is when factors other than the treatment in question could influence the outcome. This can lead to erroneous conclusions, particularly in an observational study. For instance, patients with the worst prognosis may be systematically allocated to a particular treatment. It is possible to control for those confounding factors that are known to affect treatment outcomes, but it may not be possible to control for all confounding factors in an observational study.

Heterogeneity
Because enrolment in an observational study has few restrictions, the study patient population is usually more heterogeneous than that for an RCT. Statistical tests of heterogeneity are used to assess whether the observed variability in results is greater than that expected to occur by chance.

Statistical power
The statistical power is the ability of a study to demonstrate an association or causal relationship. If the statistical power of a study is low, the results will be questionable. By convention, 80% is an acceptable level of power. As with the design of an RCT, researchers must estimate the parameters needed to detect a difference between treatments in an observational study – for example, the numbers of patients and the length of follow-up.

A checklist can be helpful to determine whether these and other issues have been adequately addressed in the study report, and thus give a degree of confidence about the results. Some useful questions to consider are:

• has the measurement of important confounding factors been described, so that the reader can judge how they can be controlled?
• have all the subjects been accounted for in the study follow-up?
• has any issue of possible bias been addressed?
• is the statistical power of the study adequate to establish a difference between treatment groups?
• has ‘clinical significance’ been discussed so that it can be differentiated from ‘statistical significance’?
• do any measures of association have confidence intervals reported?
• has the issue of multiple comparisons been addressed?
• have the study results been placed in the context of existing findings, and any reasons for differences been discussed?

It is also vital, once you have reached the manuscript stage, to put the study into context alongside the evidence generated by other sources, including RCTs. Explaining any differences in findings is a crucial part of having the study results accepted as a significant contribution to the whole. We wish you luck in conducting these studies – we consider them an essential part of the evidence base for a therapy, and encourage you to consider them in your research planning.

If you want examples of some excellent observational studies in the industry, Lilly is a major contributor to this type of research. SOHO, ADORE, EDOS and EMBLEM are some of the acronyms of their studies. Google will pick these up if you add the word ‘study’ to the search term.

Read Observational Studies, Parts 1, 2 and 3.

Observational studies: answering real-life questions about healthcare practice

Part 3 – Conducting an observational study

Last month, we looked at the differences between observational studies and randomised controlled trials (RCTs). Here, we look at designing and conducting an observational study.

Observational studies should be conducted to high standards. You can use the following checklist when designing observational studies; all studies should be reviewed by a medical research committee.

• Non interventional: there must be no randomisation, and nothing in the study design that directs or interferes with the initial physician decision or subsequent care. No test results should be requested unless performed in the normal course of treatment. Data collection should be performed at normally scheduled visits. Additional patient questionnaires may be required.
• Scientifically valuable: the study must address a valid scientific question and have clear objectives and endpoints.
• Study design and size: The study design should clearly meet the research objectives. The study should be large enough to have adequate statistical power. As there is a high possibility of treatment switching, the study design needs experienced and thorough statistical input from the beginning
• Safety reporting: the study design should consider the need to collect safety data. Alternative mechanisms to monitor safety may be adequate, and adverse event reporting may not be required in non-interventional studies.
• Analysis plan: this should be finalised before the data become available, to minimise potential bias.
• Publication plan: this should be discussed with external investigators, and key publications should be agreed in principle. In observational studies in poorly researched areas, the baseline data can be of great value and interest to clinicians, so you should allow for some publications here as well as at later stages in the study.
• Patient consent: appropriate consent should be obtained. Typically, this is simpler than for an RCT, but is required where new patient data are collected (for example, quality of life questionnaires) and may be required before agreeing that existing data can be shared by the investigator.
• Ethical consent: appropriate ethics committee approval should be obtained.
• Appropriate reimbursement: investigators should be reimbursed for their time only, and not paid an amount that might be perceived as a subsidy for drug cost or act as an incentive. The last thing you want is for physicians’ decisions for treatment to be influenced.
• Where observation is concerned, chance favours only the prepared mind.

Here are the links to part 1 and part 2.

Observational studies: answering real-life questions about healthcare practice

By Ruth Whittington (ruth.whittington@rxcomms.com)

Part 2 – How do observational studies and randomised controlled trials differ?

Last month, I described observational studies and where they fit into the hierarchy of evidence. Randomised controlled trials (RCTs) provide some of the fundamental answers to important questions about a drug’s efficacy and safety, and so are considered top of the hierarchy of medical evidence.

It is easy to be persuaded into thinking (as some of our customers and colleagues do) that RCTs are the be-all and end-all of the evidence needed for a drug. However, looking at one treatment in isolation will not reflect the likely situation in real life clinical practice.

The first major difference between observational studies and RCTs is in how patients are enrolled. For an observational study the question of whether the patient is suitable for enrolment occurs after treatment has been decided, whereas for an RCT it occurs before treatment has been assigned.

Steps in patient recruitment

Once patients have been enrolled in the study, there are a number of ways to differentiate between randomised controlled trials and prospective observational studies. Compare and contrast these in the following table:

Differences between the conduction of a randomised controlled trial and the observational study:

Observational studies: answering real-life questions about healthcare practice

This is the first in a series of four articles about observational studies. It’s our impression that observational studies (also known as “naturalistic studies”) are becoming more and more important to healthcare authorities, and it behoves us all to understand more about the uses and abuses of this type of research.

So we have created this series to 1) describe what they are, 2) show how they differ from randomised controlled trials in the way data are collected, 3) have a brief look at how they should be conducted, and 4) describe how to interpret the results.

“Science is built up of facts, as a house is built of stones; but an accumulation of facts
is no more science than a heap of stones is a house.”
Henri Poincare (mathematician, 1854–1912) Science and Hypothesis, 1905

Part 1 – What are observational studies?

While randomised, controlled clinical trials (RCTs) are the cornerstone of the drug development process, they cannot replicate actual clinical practices. Observational studies help close the evidence gap by providing insights into real life situations, and thus aid our understanding of how both patients and their clinicians manage healthcare problems.

Observational studies are characterised by the lack of intervention when treatment decisions are made; the treatments are administered as they would be in normal clinical practice, and information is collected regarding the outcomes of those treatments. This means that switching therapies midway through the treatment can be common – patients are not restricted to a particular drug therapy. Some observational studies are carried out retrospectively using existing databases of patient data, but the most robust and useful type of study is carried out prospectively; i.e. the study design is decided, then the patients are enrolled.

Observational studies can also collect data on outcomes important to patients that may not be included in RCTs, for example:

• quality of life;

• satisfaction with treatment;

• out-of-pocket expenses;

• activities of daily living/independence;

• employment status/time off work;

• social inclusion;

• impact on careers; and

• longer term outcomes (can be over many years).

Patients and their concerns are central to the study, and unlike RCTs, patients are active partners in both their treatments and the study. Typically, patient reported outcomes (PROs) are integral to the study design.

The table below shows where observational studies tend to fit on the hierarchy of evidence established by Bandolier.

Levels of evidence

Level 1 is the highest; i.e. considered the most robust. Levels of evidence system adapted from Bandolier.

While RCTS answer questions such as “Is medicine efficacious?” or “Is a treatment safe and tolerable?” these answers are often provided in highly controlled settings, which may mean that the findings are not easily translated to actual clinical practice.

So while RCTs can provide definitive answers in specific circumstances and populations, other evidence is required to answer more far-reaching questions now posed by healthcare authorities, such as

• How will this treatment work in actual clinical practice?

• What happens if the patient has co-existing conditions?

• Will treatment be taken according to the dosage regimen prescribed?

Large, well-designed prospective observational studies help provide answers to these all-important questions about medicine use in the real world.