L1 EBP EBP

Overview

This lecture works through a single contemporary clinical question — should eculizumab be funded for paroxysmal nocturnal haemoglobinuria (PNH) — as a worked example of the evidence-based practice (EBP) cycle. It moves from the clinical problem, through formulating a PICOT question, choosing where to look for evidence (levels of evidence, the EBHC pyramid, a systematic search), appraising the pivotal trial and its risk of bias, back to how PHARMAC actually used (and didn’t use) this evidence in its funding decisions, and what has changed since.

The clinical problem: PNH and eculizumab

• Paroxysmal nocturnal haemoglobinuria (PNH) is a rare acquired haemolytic anaemia caused by a stem cell mutation that causes a lack of a complement inhibitor on red blood cells.
• It causes haemolytic anaemia, free haemoglobin in the blood (haemoglobinuria, red urine), jaundice, fatigue, thrombosis, and other symptoms.
• Eculizumab (Soliris) is a monoclonal antibody that binds and inhibits complement activation. Effects: prevents haemolysis and haemolytic anaemia, may reduce thrombosis, may improve other symptoms. It may increase infection risk, so meningococcal vaccination is needed before starting.
• Current recommendation (UpToDate, 23/2/24): for symptomatic haemolytic PNH (e.g. thrombosis, organ dysfunction, pain) without severe bone marrow failure, treatment with a complement inhibitor (eculizumab or ravulizumab) is recommended over supportive care alone (Grade 1B). Eculizumab is currently the only one approved in NZ.
• A Grade 1B recommendation: Grade 1 = a strong recommendation, meaning that following it does more good than harm for most patients; clinicians should follow it unless there is a clear and compelling reason not to. Grade B means the best evidence for benefits/risks comes from RCTs with important limitations.

Slide 8 (a TV news screenshot) carries no on-slide text; its role illustrating the February 2013 media coverage is inferred from the surrounding slides, not stated on the slide itself.

• Real-world context: a 2013 newspaper story profiled Andrea Murphy, a Dunedin PNH patient receiving Soliris on compassionate grounds from Alexion (cost ~$500,000/year), while Pharmac’s advisory committee had recommended declining funding on cost/clinical-benefit grounds. Since starting Soliris she had no further blood clots, chest spasms or severe stomach pains; she had been diagnosed in 2009 after a stroke and brain haemorrhage. PNH Support Association noted Soliris was funded in 40+ countries, that about 1 in 3 sufferers die within 5 years of diagnosis, and about 8 people had been diagnosed with PNH in NZ. Blood clots and kidney failure are two significant causes of death in PNH.

PTAC’s initial review (Feb 2012)

• PHARMAC’s Pharmacology and Therapeutics Advisory Committee (PTAC) reviewed Alexion’s application to list eculizumab for PNH and recommended declining it, referring it instead to the Haematology Subcommittee.
• Decision criteria relevant to the recommendation: (i) health needs of all eligible people in NZ; (iii) availability/suitability of existing medicines and related products; (iv) clinical benefits and risks of the pharmaceutical; (v) cost-effectiveness of funding it versus other publicly funded health services; (vi) budgetary impact on the pharmaceutical and overall health budgets.
• PTAC noted the evidence for eculizumab was mainly from observational studies, prompting the question of what study type would give the best evidence of effectiveness.

Step 1 - Ask: formulating a focused (PICOT) question

• The EBP cycle (from ELM2): 1. Ask a focused question; 2. Acquire the evidence (systematic search); 3. Appraise the evidence (validity, clinical relevance, applicability); 4. Apply the findings to a clinical decision; 5. Audit performance/implementation.
• PICOT structure: Population, Intervention/Exposure, Comparison, Outcome, Time.
• Worked PICOT for this question: Population - patients with PNH; Intervention - eculizumab; Comparison - placebo; Outcome - reduce incidence of thrombosis; Time - long-term (about 5 years).

Step 2 - Acquire: levels of evidence and where to look

• To decide what evidence to seek, use the CEBM (Oxford Centre for Evidence-Based Medicine) revised Levels of Evidence table, which is organised by question type (e.g. diagnosis, prognosis, treatment benefits, treatment harms, screening).
• For “Does this intervention help? (Treatment Benefits)”: Level 1 = systematic review of randomised trials (or n-of-1 trials); Level 2 = randomised trial or observational study with a dramatic effect; Level 3 = non-randomised controlled cohort/follow-up study; Level 4 = case-series, case-control, or historically controlled studies; Level 5 = mechanism-based reasoning. A level can be graded down for poor study quality, imprecision, indirectness or inconsistency, or graded up for a large/very large effect size. A systematic review is generally better than an individual study.
• If there isn’t time for a full literature search, pre-appraised resources offer a quicker route to reliable information, organised as a pyramid (EBHC pyramid, bottom to top): 1. Studies - synopses/filtered views of individual studies; 2. Systematic reviews - synopses/filtered views; 3. Systematically derived recommendations (guidelines) - synthesis of multiple appraised guidelines; 4. Synthesised summaries for clinical reference - integrate appraisal of the three layers below; 5. Systems (decision support).
• Otago library resources by pyramid level:
  • Level 1 (study synopses): ACP Journal Club, Evidence-Based Medicine, Evidence-based complementary and alternative medicine, Evidence-based dentistry, Evidence-based mental health, Evidence-based nursing, International Journal of Evidence-Based Healthcare (all subscribed journals); The NNT (open access, uses only the highest-quality evidence, mostly Cochrane Reviews, no outside funding); Medscape (open access, free registration).
  • Level 3 (systematically derived recommendations/guidelines): NICE (UK, open access); NZ Primary Care Handbook 2012 (open access); Healthify He Puna Waiora, formerly Health Navigator (open access, NZ guidelines, often consensus- rather than evidence-based, also has patient information); ACCESSSS (open access, searches multiple evidence services, provides alerts); TRIP Database (open access, results sorted by evidence type); SUMSearch (open access, free meta-search engine).
  • Systematic reviews: Cochrane Library, a collection of databases of high-quality independent evidence.
  • Level 4 (summaries): DynaMed (subscribed - best comprehensive, transparent summary of evidence); BMJ Best Practice (subscribed - less transparent about its evidence base, similar to UpToDate); Cochrane Clinical Answers (open access and subscribed via Ovid - very compressed results from the Cochrane database). Note: the University’s UpToDate subscription ceased 30 September 2018; trainee interns (6th years) can still access it via Te Whatu Ora.
• Example of a level-4 summary in practice: DynaMed’s PNH management section lists complement inhibitors for classical PNH - first-line C5 inhibitors eculizumab or ravulizumab; crovalimab (Piasky, a C5 inhibitor) for adults/children ≥13 years and ≥40 kg; pegcetacoplan (a C3 inhibitor, inhibits both intravascular and extravascular haemolysis); iptacopan (Fabhalta, a factor B inhibitor); danicopan (Voydeya, a factor D inhibitor, add-on to ravulizumab/eculizumab for extravascular haemolysis). Complement inhibitor therapy carries a risk of life-threatening infections, particularly encapsulated bacteria such as Neisseria meningitidis, so vaccination is required ≥2 weeks before starting. Monitoring includes full blood count, reticulocyte count, LDH, biochemical profile, PNH flow cytometry, and direct antiglobulin test if haemolysis persists.
• DynaMed’s evidence synopsis on eculizumab efficacy: reduces need for transfusion and improves quality of life (TRIUMPH and SHEPHERD trials, Level 1 randomised trial evidence, NEJM 2006); reduces mortality (Level 2 retrospective cohort studies); may improve pulmonary hypertension signs/symptoms (Level 2 RCT, Br J Haematol 2010); may reduce thromboembolism rate (Level 2 cross-sectional study, Blood 2007).

Pre-appraised evidence has limits: the February 2013 case

• As of February 2013, evidence on eculizumab and thrombosis was limited: 1 small RCT, 2 open-label trials, 1 open-label extension study, and 3 cohort studies.
• Different sources interpreted this evidence differently:
  • UpToDate: across all PNH eculizumab studies to date there was a marked reduction in thromboembolic event rates on treatment compared with pre-treatment rates.
  • DARE (Database of Abstracts of Reviews of Effects, Cochrane Library): a pooled analysis of three studies suggested up to a seven-fold decrease in thromboembolism risk with eculizumab; the authors’ conclusion followed from the data, but was based on only one small RCT and two uncontrolled studies of unclear quality.
• Key messages: CEBM notes that pre-appraised sources (Clinical Evidence, NHS Clinical Knowledge Summaries, DynaMed, PIER, UpToDate) may be more comprehensive but risk relying on expert authority rather than independent appraisal. As with any evidence source, quality must be considered, and there may simply be very little information available on a given question.
• For eculizumab and thromboembolism specifically, information was limited in summaries, synopses of syntheses, and syntheses, and no systematic reviews of RCTs were identified. Where pre-appraised sources run out, the next step is to search for original journal articles directly.

Systematic search for primary evidence

• A medical librarian (Richard German) ran a systematic OvidSP search (Feb 2013) for original journal articles: “Hemoglobinuria, Paroxysmal/” (1040 results) OR “soliris.mp.” (7) OR “eculizumab.mp.” (289) combined (289), AND’d with the PNH term (131), then limited to therapy studies maximising specificity (6) and to randomised controlled trials (6). Note: adding thrombosis as a search term returned no results.
• Of the resulting six references, only one genuinely met the search criteria: Hillmen P, Young NS, Schubert J, et al. “The complement inhibitor eculizumab in paroxysmal nocturnal hemoglobinuria.” N Engl J Med. 2006;355(12):1233-43. (The other five were: an FDA report on eculizumab for PNH; a study on nitric oxide depletion/pulmonary hypertension; a study on kidney function; a study on anaemia improvement; a study on thrombin generation/inflammation markers.)

The NEJM abstract text on slide 33 was small in the source image; transcribed to the best of legibility, but a few words may be imprecise.

Step 3 - Appraise: the pivotal Hillmen et al. (2006) trial

• Rationale for critical appraisal: evidence-based guidelines cannot cover every clinical question or keep pace with practice, so individual practitioners need to be able to find and appraise evidence themselves (Glasziou, BMJ 2008).
• The trial’s PICOT: Population - PNH, age ≥18, transfusion dependent, recruited across 10 countries, 34 screened (note: this figure differs from the 115 screened shown in the study flow diagram); Intervention - IV eculizumab weekly for 4 weeks then every 2 weeks; Comparison - placebo infusions, double-blind; Outcome - primary: stable haemoglobin levels (no need for transfusion), plus multiple secondary outcomes; Time - 26 weeks. (The lecture notes students don’t need to remember the trial’s details.)
• Study flow: 115 patients screened, narrowing to 87 enrolled (source: PNH, age >18, transfusion-dependent, multiple sites). Of these, 43 received eculizumab and 44 received placebo (control). Outcome (stable Hb) at 26 weeks: eculizumab 21/43 = 49% “yes”; placebo 0/44 = 0% “yes”.
• Primary outcome results (Table 2): stabilised haemoglobin during treatment - 0% placebo vs 49% eculizumab (P<0.001). Packed red cell transfusion (units/patient) during treatment - median 10 (placebo) vs 0 (eculizumab), P<0.001; mean 11.0±0.8 (placebo) vs 3.0±0.7 (eculizumab); totals 482 units (placebo) vs 131 units (eculizumab).
• On thrombosis specifically: there was only one thrombosis event in the whole trial, and it occurred in the placebo group. The study is too small to answer the thrombosis question.

Assessing risk of bias: RAMBOMAN and the Cochrane RoB 2 tool

• RAMBOMAN is a framework for assessing risk of bias and confounding, covering: Recruitment, Allocation, Maintenance, Blind and Objective outcome Measures, ANalysis.
  • Recruitment: was it appropriate to the study’s goals? Were the setting and population described? Were participants similar to the eligible population?
  • Allocation: was randomisation done by an appropriate procedure? Was allocation concealed? Was there any unexpected difference between control and intervention groups?
  • Maintenance: completeness of follow-up; good adherence to intervention and control; low contamination/cross-over; other interventions balanced between groups; blinding maintained throughout.
  • Blind and Objective outcome Measures: were outcomes measured accurately, and was follow-up time suitable?
  • ANalysis: was it intention-to-treat, and were suitable analyses used (e.g. 95% CI of effect size given)?

Pages covering the last three RAMBOMAN components (Maintenance; Blind and Objective outcome Measures; ANalysis) lacked page-number footers in the source, so it could not be confirmed whether they sit on one slide or are split across several; the content above is transcribed in full and in the correct reading order regardless.

• Beyond RAMBOMAN, general appraisal should also consider representativeness: for descriptive studies, case-control studies, and cohort studies with external comparison, this is a question of bias (internal validity); for RCTs, it is a question of generalisability/external validity - can results be applied to patients not in the study?
• The Cochrane Collaboration’s Risk of Bias 2 (RoB 2) tool is another structured way to assess bias in RCTs.

The RoB 2 example flowchart for judging bias from the randomisation process (slide 46) is dense with overlapping arrows; the routing described here is a best-effort reading and individual arrow endpoints could not be verified with full confidence. As read: starting from "allocation sequence concealed?" - if yes/probably yes, proceed to "allocation sequence random?"; if no information, proceed to a "baseline imbalances suggest a problem?" check; if no/probably no, judged directly as high risk. From "allocation sequence random?": yes/probably yes/no information leads to the baseline-imbalance check (leading to low risk if no/probably no/no information, or some concerns if yes/probably yes); no/probably no leads directly to some concerns. From the baseline-imbalance check reached via the "no information" branch: no/probably no/no information leads to some concerns; yes/probably yes leads to high risk.

Back to PHARMAC: how the evidence was actually used

• Returning to the 2012 PTAC minutes: the committee considered the Hillmen (2006) trial supported eculizumab alleviating haemolysis and its sequelae, improving symptoms and quality of life, but noted the study could not address impact on life-threatening complications, since only one thrombosis occurred (in the placebo arm) and there were no deaths over the 6-month study period.
• Natural history evidence on survival was inconsistent: the supplier’s submission (Table 4) quoted median survival of 10-25 years, but PTAC noted this table omitted a French cohort study (de Latour et al., Blood 2008; 460 PNH patients) showing median survival of 22 years in the pre-eculizumab era, with a 76.3% 10-year survival rate overall and 92% 10-year survival in patients diagnosed after 1996.
• On thromboembolism, PTAC noted Hillmen et al. (Blood 2007) implied a reduction from 7.37 to 1.07 thromboembolic events per 100 patient-years after starting eculizumab, and that the authors argued this could increase life expectancy by reducing thrombosis (the leading cause of death in PNH). PTAC considered this data compelling but noted an alternative explanation: because thrombosis can itself lead to diagnosis of PNH, thrombosis events may simply cluster earlier in the disease course, confounding a simple before/after comparison.
• Timeline of decisions: August 2012 - the Haematology Subcommittee recommended listing eculizumab with low priority, limited to patients meeting certain criteria. March 2013 - PTAC maintained its recommendation not to fund. 11 December 2013 - PHARMAC formally declined funding, citing an unreasonable price: Alexion was asking $670,000perpatientperyear(oneofthemostexpensivedrugsPHARMAChadassessed),higherthanpricesintheUKorCanadaevenwithadiscount,andhigheronaper-patientbasisthancomparabledrugsforotherrareconditions(e.g.eltrombopag,anotherrareblooddisorderdrug,at$36,000/patient/year - about a 20th of eculizumab’s price). PHARMAC’s CEO (Steffan Crausaz) stated eculizumab could benefit up to 20 people in NZ at a cost of about $10 million/year, and that committing this funding could mean many other New Zealanders missing out on other new medicines.
• What happened since: a 2014 Cochrane systematic review found only one eligible study, and concluded there was an absence of evidence for eculizumab versus placebo on overall survival, nonfatal thrombotic events, or transformation to myelodysplastic syndrome, though evidence indicated eculizumab increases health-related quality of life and transfusion independence compared with placebo; the safety profile was unclear. A new search in 2025 found no further placebo-controlled RCTs, though some biosimilar drugs have since been compared with eculizumab. As of the lecture, eculizumab is still not funded by Pharmac.
• Underlying tension: patients with PNH want access to treatment, but PHARMAC must weigh the health needs of all New Zealanders against limited resources - a balancing act not always visible in media coverage, since the public doesn’t see the faces of people who might miss out on other treatments if eculizumab were funded. The price charged by the manufacturer is also central to the dispute. A further 2018 NZ Herald story (Rachel Paine) described PNH patients unable to return to NZ because the drug (costing ~$600,000/year) is not funded there.

Self-test

1. Define paroxysmal nocturnal haemoglobinuria (PNH) and describe its underlying cause.
2. State the components of a Grade 1B recommendation and explain what “Grade 1” and “Grade B” each mean.
3. Write out the PICOT question used in this lecture for eculizumab and thrombosis in PNH.
4. According to the CEBM levels of evidence, what type of study provides the best evidence for a treatment-benefit question, and what can cause a level to be graded down or up?
5. Describe the EBHC pyramid used to find pre-appraised evidence, from base to top.
6. Distinguish UpToDate’s and DARE’s interpretations of the February 2013 evidence on eculizumab and thrombosis, and explain why they differed.
7. Describe the steps of the systematic search for primary literature on eculizumab and thrombosis, and explain why only one of the six retrieved references met the search criteria.
8. Outline the PICOT and key results of the Hillmen et al. (2006) trial, and explain why it could not answer the thrombosis question.
9. List the five components of the RAMBOMAN framework, with one thing each component checks for.
10. Distinguish “bias” from “generalisability” as used when discussing representativeness of a study.
11. Explain the alternative explanation PTAC raised for the apparent reduction in thromboembolism rates in the Hillmen et al. (2007, Blood) cohort study.
12. Describe the sequence of PHARMAC funding decisions on eculizumab from 2012 to December 2013, and state the main reason PHARMAC gave for declining funding.
13. Summarise the findings of the 2014 Cochrane systematic review on eculizumab for PNH.
14. Integrative: using this case, explain how the type and quality of available evidence, and not just the treatment’s biological effect, can shape a funding decision.

Answers

Reveal answers

1. PNH is a rare acquired haemolytic anaemia caused by a stem cell mutation that leads to a lack of a complement inhibitor on red blood cells; this causes haemolytic anaemia, haemoglobinuria, jaundice, fatigue, thrombosis and other symptoms.
2. Grade 1 means it is a strong recommendation - following it does more good than harm for most patients, and clinicians should follow it unless there is a clear and compelling reason not to. Grade B means the best evidence for benefits and risks comes from randomised controlled trials with important limitations.
3. Population - patients with PNH; Intervention - eculizumab; Comparison - placebo; Outcome - reduced incidence of thrombosis; Time - long-term (about 5 years).
4. A systematic review of randomised trials (or n-of-1 trials) is Level 1, the best evidence for treatment benefit. A level can be graded down for poor study quality, imprecision, indirectness or inconsistency, or graded up for a large/very large effect size.
5. From bottom to top: 1. Studies (synopses/filtered views of individual studies); 2. Systematic reviews (synopses/filtered views); 3. Systematically derived recommendations/guidelines (synthesis of appraised guidelines); 4. Synthesised summaries for clinical reference (integrate the three layers below); 5. Systems (decision support).
6. UpToDate concluded there was a marked reduction in thromboembolic events on eculizumab compared with pre-treatment rates. DARE was more cautious, noting the pooled analysis suggesting up to a seven-fold reduction was based on only one small RCT and two uncontrolled studies of unclear quality. They differed because DARE weighted the poor quality/small size of the underlying evidence more heavily than UpToDate’s summary did.
7. A librarian searched OvidSP combining PNH and eculizumab/soliris terms, then limited to therapy studies and to RCTs, yielding six results. Only the Hillmen et al. (2006) NEJM trial genuinely met the search criteria (a placebo-controlled RCT of eculizumab in PNH); the other five were different study types or addressed different outcomes.
8. PICOT: adults with transfusion-dependent PNH, given IV eculizumab weekly then every 2 weeks vs placebo, double-blind, over 26 weeks, primary outcome stable haemoglobin. Results: 49% of eculizumab patients achieved stable Hb versus 0% of placebo patients (P<0.001), and transfusion units fell sharply on eculizumab. It could not answer the thrombosis question because only one thrombosis event occurred in the entire trial (in the placebo group), making the study too small to assess this outcome.
9. Recruitment (was the population/setting appropriate and representative?); Allocation (was randomisation appropriate and concealed?); Maintenance (was follow-up complete, adherence good, blinding maintained?); Blind and Objective outcome Measures (were outcomes measured accurately with suitable follow-up?); ANalysis (was it intention-to-treat with suitable statistics, e.g. 95% CI?).
10. Bias concerns whether a study’s results are internally valid (relevant to descriptive, case-control, and cohort-with-external-comparison studies). Generalisability concerns whether results from an RCT can be applied to patients outside the study (external validity).
11. PTAC noted that because thrombosis can itself be what leads to a PNH diagnosis, thrombosis events may cluster earlier in the disease course before diagnosis; this could produce an apparent “before vs after treatment” reduction in thromboembolism rates that isn’t purely due to the drug’s effect.
12. August 2012: Haematology Subcommittee recommended listing with low priority and restricted criteria. March 2013: PTAC maintained its recommendation not to fund. 11 December 2013: PHARMAC formally declined funding. The main reason given was price - Alexion’s asking price of $670,000/patient/year was considered unreasonable relative to prices overseas and to comparable drugs for other rare conditions.
13. The 2014 Cochrane review found only one eligible study and concluded there was an absence of evidence for eculizumab versus placebo on overall survival, nonfatal thrombotic events, and transformation to myelodysplastic syndrome, while evidence did support improved health-related quality of life and transfusion independence; the safety profile remained unclear.
14. The only RCT-level evidence (Hillmen 2006) was well-designed but too small and short to demonstrate reduced thrombosis or mortality, the outcomes that matter most for funding a very expensive drug; cohort and observational evidence on thrombosis and survival was more compelling but had potential confounding (e.g. thrombosis leading to earlier diagnosis) and lower certainty. Because the strength and type of evidence available did not clearly establish benefit on the hardest outcomes, PHARMAC could reasonably combine this evidential uncertainty with the drug’s very high price and its obligation to all New Zealanders to justify declining funding, even though eculizumab has a clear, biologically plausible and partially demonstrated benefit.