Evidence Based Medicine and Outcomes Analysis
– An Evaluation

Dr. Suman Bhusan Bhattacharyya

Abstract

Evidence based medicine is fast overtaking experience based medicine in the field of healthcare delivery. Outcomes analysis allows the assessment of the quality of care delivered and matches them against the resource costs. It is becoming increasingly imperative for the various stakeholders involved in the entire healthcare delivery process, that is, the care deliverers and clinical managers, are not only knowledgeable but also practice it on a regular and rigorous basis. This paper attempts to put both the knowledge and the practice methodologies in a simplified manner so that all the care deliverers can easily implement them in their workplaces.

Keywords: Evidence Based Medicine, Outcomes Analysis, Evidence Based Practice, Evidence Based Medicine Balance Sheets, Absolute Risk Reduction, Odds Ratio, Relative Risk, Numbers Needed to Treat, Numbers Needed to Harm, Meta-analysis Sensitivity, Specificity, Likelihood Ratio, Priori, Posteriori, Bayes' Rule

Introduction

A new paradigm for determining diagnosis and prognosis is evolving that is set to revolutionize the way patients are managed, thereby increasing effectivity without compromising productivity or quality of care provided. It is called 'practicing evidence based medicine'.

Clinical practice is changing due to rapid advances in technology and the clinicians need to change their practice methodology in order to continue to deliver high levels of specialized care. An average clinician today is faced with a multitude of problems. Ever increasing medical knowledge in the form of 27 Kg of guidelines, more than 3000 new papers per day, 1000 new Medline articles, 46 randomized clinical trials, and the number of biomedical journals alone doubling since 1970. Couple it with the average workload for a clinician of anything between 100 to 200 consultations a week resulting in 5000 to 10000 per year. Add to it the difficulty in being sure that one is doing the right things for all these cases, relying solely on experience while using 2 million pieces of information all stored in ones memory, ever increasing pressures to provide value-for-money services, raised patient demands and expectations, pressures due to a myriad of obtrusive and mostly confusing regulatory compliances, and rapidly altering business demands. It does not paint a happy picture by any stretch of the imagination.

Every encounter with a patient identifies appear more often than not to reveal some gaps in the understanding of the etiology, diagnosis, prognosis, or therapy of their illness on the part of most clinicians. Recent research reveals that even as seasoned clinicians generate about five knowledge 'needs' for every in-patient encounter, and two 'needs' for every three out-patients encounters. To bridge these gaps and fulfill the 'needs', it has become imperative to practice evidence based medicine. To evaluate the best evidence and make the necessary corrections, it has become necessary to perform outcomes analysis.

Evidence Based Medicine

The most commonly accepted definition of 'Evidence Based Medicine' is that it is the conscientious, explicit and judicious use of current best evidence in making clinical decisions about the care of individual patients.

Dr. David Eddy of Kaiser Permanente is widely credited for having coined the term 'Evidence Based Medicine'. He explains that when there is evidence that something works and is good and benefits the patient, one should do it. When one has evidence that there is no benefit and that it is ineffective, where it is going to harm the patient either directly or indirectly by stealing resources, one should not do it. When there is insufficient evidence to determine either way, one must be conservative, relying on individual clinician discretion. This means the integration of individual clinical experience with the best available external clinical evidence from systematic research.

Any practice that uses best evidence, while delivering healthcare, is called 'evidence based practice'. Such practice generally is the application of up-to-date information from relevant, valid research about the effects of different diagnostic tests and the predictive power of prognostic factors across the broad field of healthcare, including education, practice management and health economics.

A new approach for practicing evidence based medicine is to build evidence based balance sheets to integrate the best evidence with the possible outcomes to help in deciding the 'right' method of treatment for a patient. The purpose is to display this information as quantitative estimates of the effects of alternative treatments on all the important outcomes in a compact form so that all the stakeholders can make an informed choice when deciding on a treatment plan. These are especially useful for informed shared decision-making between clinicians and patients. The main strength of these balance sheets is that they summarize in one place the critical information needed to make decisions.

Evidence based healthcare is a newer term broadening the evidence based medicine techniques to other aspects of healthcare delivery. This is defined as the conscientious use of current best evidence in making decisions about the delivery of healthcare services.

Outcomes Analysis

Outcomes analysis is a non-prejudiced analysis of the outcome of an event, episode or encounter. It is used by the nurses as a measure for patient acuity, the hospital administrators and the health economists to perform cost-benefit analysis, and care providers to justify their clinical decisions regarding treatment plans as well as to validate the clinical protocols when followed. Not only are the outcomes of an event, but also the variances between different treatment methodologies are measured. In a clinical setting, it also allows one to find out how well a particular treatment method is faring.

The aim of outcomes analysis is not to find ways to reduce, but to put a cost on a given treatment regimen. It is entirely possible that the measure ends up recommending a treatment regimen that is costlier because it is more beneficial. It can also be used as a costing method to perform cost-benefit analysis and effort estimations.

The various parameters that are evaluated are as follows:

The Process

Evidence based medicine converts reading and appraising the information into using it to benefit individual patients while concurrently adding to the clinician's knowledge base. Instead of reading all the articles in a journal, it is better to focus on the ones that are related to specific problems. It is critical that one follows a constructive method of framing the pertinent question related to the problems on hand, and then searching for evidence related to that question. The aim is to keep one's knowledge at a more usefully productive level.

The seven 'A' methodology for practicing evidence based medicine are as follows:

Evidence based medicine requires some knowledge regarding the calculations and interpretations of relative risks, absolute and relative risk reductions, odds ratio, numbers needed to treat/harm, sensitivity, specificity, likelihood ratio, pre-test probabilities, etc.

Outcomes analysis is an inherent requirement for the total adoption of evidence based medicine. Without the results arrived at from analysis of outcomes being added to the knowledge repository for future reference, the internal expertise is not enriched, i.e. there is no value-add of the process for future patient with similar clinical picture demanding the answers to similar questions.

Calculations in Evidence Based Medicine

There two broad types of calculations that go into effective practice of evidence based medicine. These are evaluating the following:

Evidence regarding the efficacy of a certain treatment as opposed to another, including no treatment. Mostly results from randomized clinical trials are used. It is a type of prognostic assessment.

Evidence regarding a particular diagnostic test or patient finding. It is a type of diagnostic assessment.

For assessment of treatment protocols -

Absolute Risk Reduction (ARR) – This is the difference in the risk of the outcome between patients who have undergone a particular method of treatment (called experimental) and those who have not undergone that method (called control). This measure tells us the percentage of patients who were spared the adverse outcome as a result of having received the experimental rather than the control therapy. It is calculated as |EER – CER|

Relative Risk (RR) – This is the ratio of the risks in the experimental to the control groups and is represented as a percentage of the original risk. It is calculated as |EER – CER|/CER

Relative risk reduction (RRR) – This is the extent to which an experimental treatment reduces a risk, in comparison with the control, and assesses the effectiveness of a treatment. This is calculated by subtracting the RR from 1. If the RRR is 0, then the experimental treatment is no different from the control. The relative risk reduction is fundamentally an estimate of the percentage of baseline risk that is removed as a result of the experimental therapy. It is calculated as |CER-EER|/CER

Numbers Needed to Treat (NNT) – This is the most recently introduced measure of treatment efficacy, and is defined as the number of patients who need to be treated to achieve 1 additional good outcome. It the reciprocal of the ARR, and is measured if the outcome of the experimental treatment is positive. When the outcome is negative, numbers needed to treat (NNH) is measured. This is the number of patients who need to be treated with the experimental method to cause 1 additional patient being harmed as compared to those who are treated with the control method. The thumb rule is that if EER > CER, then calculate NNT, else calculate NNH. The numbers needed changes inversely in relation to the baseline risk. If the risk of an event doubles, one needs to treat only half as many patients to achieve the same results, and if the risk decreases by a factor of four, one needs to treat four times as many. It is calculated as 1/ARR

Odds Ratio (OR) – These are the odds of an event (usually adverse) occurring and is usually the measure of choice in the analysis of case-control studies. Generally, the odds ratio has certain optimal statistical properties that make it the fundamental measure of association in many types of studies. The statistical advantages become particularly important when data from several studies are combined, as in meta-analysis. Among such advantages, the comparison of risk represented by the odds ratio does not depend on whether the investigator chose to determine the risk of an event occurring (e.g., fatal) or not occurring (e.g., improvement). This is not true for relative risk where the definitions of experiment and control can alter the figures. In some situations the odds ratio and the relative risk will be close like in case ­control studies of a rare disease. The odds ratio is calculated by dividing the odds in the experimental group by the odds in the control group. It follows that efficacious treatments generate odds ratios that are less than 1, which is analogous to the relative risk for the adverse event (EER/CER) being less than 1.

Meta-analysis – It is a statistical procedure that integrates the results of several independent studies considered to be 'combinable' and should be viewed as an observational study of the evidence. Well conducted meta-analyses allow a more objective appraisal of the evidence than traditional narrative reviews, provide a more precise estimate of a treatment effect, and may explain heterogeneity between the results of individual studies. Ill conducted meta-analyses, on the other hand, may be biased owing to exclusion of relevant studies or inclusion of inadequate studies. Methods used for meta-analysis use a weighted average of the results, in which the larger trials have more influence than the smaller ones. Results from each trial are graphically displayed, together with their confidence intervals. Each study is represented by a black square and a horizontal line, which correspond to the point estimate and the 95% confidence intervals of the odds ratio. The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again. A solid vertical line is drawn that corresponds to no effect of treatment (odds ratio 1.0). When the confidence interval of any study includes 1 the difference in the effect of experimental and control treatment is not significant at conventional levels (p>0.05). An area made of black squares reflects the weight of the study in the meta-analysis. A diamond shape represents the combined odds ratio, calculated using a fixed effects model, with its 95% confidence interval. It should be noted here that a result that is meta-analytical in origin should be viewed with a higher degree of confidence than one that is not.

For assessment of findings/diagnostic tests -

Sensitivity (SnNouts – Sensitivity Outs) – This is the proportion of patients who have the target disorder and also test positive for the diagnostic test. When a sign, test or symptom has a high sensitivity, a negative result tends to rule out the diagnosis.

Specificity (SpPins – Specificity Ins) – This is the proportion of patients who do not have the target disorder and also test negative for the diagnostic test. When a sign, test or symptom has an extremely high specificity, a positive result tends to rule in the diagnosis.

Likelihood Ratio (LR) – This measures how likely the presence (or absence) of a finding (or diagnostic test) would result in ruling in (or out) a diagnosis. The ratio is used to assess how good a diagnostic test (or finding) is, to help in selecting an appropriate diagnostic test or a sequence thereof. It is better than sensitivity and specificity numbers because it is less likely to change with the prevalence of the disorder, can be calculated for several levels of signs and symptoms, can be used to combine the results of multiple diagnostic tests, and can be used to calculate the post-test probability for a target disorder. A likelihood ratio greater than 1 produces a post-test probability that is higher than the pre-test probability, while an LR lesser than 1 accomplishes the reverse, thereby altering the chances of finding the target disorder. A diagnostic test result with a very high LR (e.g. >10) would virtually rule in a disease when found positive, while one with a very low LR (e.g. <0.1) which would virtually rule out the chance that the patient has the disease.

Pre-test Probability (Priori of Bayes' Rule) – It is defined as the probability of the target disorder before a diagnostic test result is known. It is especially useful for (1) interpreting the results of a diagnostic test, (2) selecting one or more diagnostic tests, (3) choosing whether to start therapy without further testing (treatment threshold) or while awaiting further testing, (4) deciding whether it's worth testing at all (test threshold). The probability of the target disorder can be calculated as the proportion of patients with the target disorder, out of all the patients with the symptoms, both those with and without the disorder.

Post-test Probability (Posteriori of Bayes' Rule) – It is defined as the probability of the target disorder being present after the diagnostic test result is known.

When the test result is positive it is calculated as:

When the test result is negative it is calculated as:

Bayes' Rule – This is based on the theorem proposed in mid-nineteenth century by Rev. Thomas Bayes' and is on probability inference. It is a means of calculating the probability that it will occur in future trials from the number of times an event has occurred.

Examples Explaining The Above

Evaluation of treatment

Table for comparison of treatment methods

Randomized trial comparing treatment of condition X with method A and with method B

Calculations

These results indicate that the risk of fatality in method A is 31% and in method B is 46%, the relative risk of fatality after receiving treatment method A as compared to the treatment method B is 67%. That is, the risk of fatality after method A is only two-thirds as that of after method B, thereby indicating that method A is better than method B. The absolute reduction of risk is 15%. The relative reduction of risk is 33%. The odds of fatality after method A as compared to method B is 0.52 and as it is less than 1, the treatment is considered to be effective. The numbers needed to treat is 6.5, which means that 7 more patients need to be treated to decrease the adverse outcome by 1 when method A is used instead of B.

Evaluation of a diagnostic test

Presence of a target disorder when the diagnostic test value is positive

Suppose there is a patient with target disorder and a positive diagnostic test result. A systematic review provides the results that are summarized in the table below.

Calculations

These results indicate that 86% of the patients with the target disorder have a positive test result, while 76% of patients who do not have the disorder test negative. The likelihood ratio of finding a positive test result is 3.6, while the likelihood ratio of finding a negative test result is 0.19. The prevalence of the disease in the study is 42%. The post-test probability of finding a target disorder when the diagnostic test result is positive is 73%, while that of finding it when the result is negative is 12%. Since the likelihood ratio for positive test is more than 1, the post-test probability of finding the target disorder when the test is positive is more than the pre-test probability.

Outcomes Analysis

There two broad types of calculations that go into outcomes analysis. These are evaluating the following:

Outcomes of a particular treatment. It is a type of cost-benefit analysis.

Resource utilization in the course of a particular treatment, also known as patient acuity. It is a type of effort estimation.

There are no formal equations for this and are usually performed on a case-by-case or departmental/institutional basis.

Pros and Cons

Advantages of Evidence Based Medicine

Barriers to Evidence-Based Medicine

Discussion

Seeking an evidence base for medicine is as old as medicine itself. However, in the past decade the concept of evidence based medicine has done a sound job in focusing explicit attention on the application of evidence from valid clinical research to actual clinical practice. Although current clinical practice is often evidence based to an extent with new treatment methods being applied more often than not, there is still much to be gained. Important new evidence from research often takes a long time to be implemented in daily care, while established practices persist even if they have been proved to be ineffective or harmful. In the meantime, many clinicians struggle to apply the results of studies that do not seem to be that relevant to their daily practice.

Good clinicians should use both individual clinical expertise and the best available evidence from external sources. Neither alone is enough. Without clinical expertise, clinical practice risks becoming hostage to evidence and without current best evidence clinical practice risks becoming rapidly outmoded and outdated to the detriment of overall patient care. The evidence on its own is usually not conclusive but can help in supporting the process of patient care. Adopting all of these into clinical decisions enhances the chances of maximizing clinical outcomes and quality of life.

The practice of evidence based medicine is usually triggered by patient encounters which generate questions about the effects of therapy, utility (or futility) of diagnostic tests, prognosis of diseases, or etiology of the underlying disorders. It requires new skill-sets of the care provider and includes efficient literature-searching and application of formal rules of evidence in evaluating clinical literature, apart from the basic clinical skills of sharp observation, intelligent and logical inference from these. Careful application of intelligent and logical inference made from these is vital in justifying the practice of evidence based medicine.

Normally a clinician uses observational studies, logical intuition, personal experience and expert opinions. Most clinical care relies on a combination of informed guesswork, unsystematic observation, common sense, the consensus views of clinical experts, and the treatment and procedures used by most other clinicians in a local community – the standard and accepted practice. An assumption is made that a traditional knowledge of physiology, pathology, and common sense is sufficient to guide clinical practice and evaluate new treatments and diagnoses. However only 15-20% of medical practice is backed up by scientifically and statistically sound research .

Evidence-based practice aims to move beyond such anecdotal clinical experiences by bridging the gap between research and the practice of medicine. The aim is to use diagnostic tests and therapeutic interventions that are as accurate, as safe and as efficacious as possible. The clinical assessments are validated against the best evidence before they are applied to clinical care. A subsequent rigorous examination of the outcomes of different clinical actions through outcomes analysis an overall effort assessment is made to ensure the maintenance of high standards of clinical care.

The need for evidence based health care arises as a direct consequence of too many patients presenting with too many problems, while the information regarding the current treatment guideline exist in too many journals. The complexity of modern medicine exceeds the inherent limitations of the unaided human mind. A rich source of new evidence for clinical care has been generated as a consequence of application of modern research methods and statistical tools, and this very abundance of evidence has made the task of practicing evidence based medicine more difficult than ever for the individual clinician.

Evidence based medicine is here to stay and is being actively promoted by such institutions as NHS, etc. As increasing number of care providers begin to adopt and gain from this technology and better solutions that specially cater to its effective use are put in place, one can definitely look to providing improved care with lesser pain.

References

The material for this article has been extensively harvested from the references detailed above. Individual references as footnotes have deliberately been avoided to ensure ease of read and maintain continuity. The copyrights lie with their respective authors. Some web site addresses point to free online evidence based medicine resources.

Practice of Evidence-Based Medicine; By Dr Martin Dawes, University of Oxford

UK figures

Dr. David Sackett

Individual clinical expertise means the proficiency and judgment that individual clinicians acquire through clinical experience and clinical practice. Increased expertise is reflected in many ways, but especially in more effective and efficient diagnosis and in the more thoughtful identification and compassionate use of individual patients' predicaments, rights, and preferences in making clinical decisions about their care.

Best available external clinical evidence means clinically relevant research, often from the basic sciences of medicine, but especially from patient centered clinical research into the accuracy and precision of diagnostic tests (including the clinical examination), the power of prognostic markers, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens. Such evidence validates previously accepted diagnostic tests and treatments, and where necessary, replaces them with new ones that are more powerful, more accurate, more efficacious, and safer.

First Annual Nordic Workshop on how to critically appraise and use evidence in decisions about healthcare, National Institute of Public Health, Oslo, Norway, 1996.

These are used principally by the pharmaceutical companies where the treatment is the experiment and no treatment in the form of administration of a placebo is control.

Baseline risk is the risk of an adverse event among patient either in the control group or who are receiving the standard or inferior therapy.

Huque M F. Experiences with meta-analysis in NDA submissions. Proceedings of the Biopharmaceutical Section of the American Statistical Association: 1988;2:28-33.

All numbers are fictitious

This problem is peculiar to the GPs and not to the specialists whose words are taken more on face value

2003 figures

Late Archie Cochrane

Dr. David Eddy