Background

Healthcare technology has developed rapidly in recent years, with escalating costs for new drugs and other diagnostic or therapeutic technologies. Demand for potentially beneficial healthcare has outstripped the supply, resulting in the need to ration limited resources. Where this occurs through informal mechanisms that restrict access to certain services, there may be overt distributive injustices due to differential access on the basis of socioeconomic, geographical or demographic factors. However, where government bodies, healthcare providers or other agencies develop evidence-based guidelines or policy, part of their role may be seen as ensuring that any rationing decisions are made on a just basis.

The requirements of distributive justice may be met by adherence to underlying substantive principles based, for example, on fair equality of opportunity,1 maximising cost benefit2 or a capability approach.3 Such principles may be in conflict, or may result in indeterminacy, and adherence to the principles of procedural justice has been suggested as an alternative or additional requirement. Commonly, the implementation of procedural justice refers to the four principles of accountability for reasonableness; transparency of the process, relevance of the decision criteria, potential for challenge and revision, and a regulatory framework.4 Substantive principles and procedural justice are not mutually exclusive but overlap, with claims of relevance usually involving reference to substantive principles.5

Whatever substantive or procedural principles are applied, the fairness and legitimacy of evidence-based guidance and policy rely on the accuracy and validity of the underlying evidence. In this paper, I argue that ‘evidence’, as it is understood and used in such processes, is not restricted to research-based knowledge that provides the best estimates of the benefits, risks and costs of treatment. Rather, it includes implicit or explicit value judgements, which are not necessarily in the domain of the expert advisory committees or other bodies that make such decisions. I highlight some of the systematic biases, distortions and epistemic injustices that are inherent in such evidence, and argue that evidential failings may undermine the claims to the legitimacy and fairness of the decision-making process.

Epistemic injustice and evidence

Fricker suggests that ‘there is a distinctively epistemic genus of injustice, in which someone is wronged specifically in their capacity as a knower’.6 Testimonial injustice has been described in healthcare where a patient may have unique knowledge of their experience of an illness or treatment, which is not given appropriate credibility.7

However, the potential to contribute to shared knowledge does not always take the form of conscious awareness in those who provide the information. We may ‘know’ that we have a feeling of nausea but be unaware of a fall in our haemoglobin level, without realising that both of these may have potential to add to the shared pool of knowledge about an illness or healthcare intervention. If we belong to a marginalised group that is excluded from a research study, then that exclusion may unjustly disadvantage ourselves and the group to which we belong, by distorting the results of such a study.

Epistemic contributions may occur on many different levels. We may take on the role of informant, providing knowledge of our experience of a condition, treatment or outcome, offering potential benefits to ourselves or others facing similar situations. To deny us credibility in this context seems to be a clear example of testimonial injustice. However, other situations are less clear cut; we may contribute to the generation of shared knowledge, without holding that knowledge ourselves. In some circumstances, we may offer ourselves as an object of research, allowing our blood parameters to be measured. Such ‘objectification’ may not be negative; to deny us the opportunity to contribute to shared knowledge in this way may be considered an injustice.

The term ‘evidence based’ carries with it an air of scientific authority, suggesting decisions that follow from a rigorous evaluation of what is known. However, examination of the evidence used in such decisions reveals that it includes information relating to values, expert judgements and alternative scenarios, the merits of which need to be considered by the decision-making body. Apart from those value judgements that inform the specific decision, the framework within which the decision is made embodies many assumptions, such as those about methodology and perspective.

Where value judgements are required for evidence-based decision-making, we are faced with a choice between using the values of different constituencies: empowered authorities, ‘expert’ advisors, patients, regional or national societal samples. The evidence required may include studies to quantify these values or develop consensus of opinion among these various groups, in order to justify a specific stance in relation to them. Many research methods that contribute to the evidence do not elicit what might conventionally be seen as ‘factual’ knowledge, but include preference-based studies, qualitative interviews or focus groups to elicit value judgements, or methods such as Delphi techniques to reach consensus among experts. This presentation of values and opinion in the form of scientific evidence can blur the boundary between those aspects of the evidence that relate to knowledge and those concerned with value judgements. It may, therefore, be more useful to think in terms of evidential injustices, rather than purely epistemic ones.

In a decision-making context, there is potential for injustices to occur through the way in which evidence is generated and reported, or how it is interpreted and used in the decision-making process. In the following sections, I explore the ways in which potential injustices may arise in each of the areas. I then go on to consider who is wronged by these evidential injustices and the steps that may be taken to try and ameliorate them.

Injustices in the generation of evidence

Extensive methodological development in recent years has focused on the evaluation of research evidence, with hierarchies of evidence and quality assessment frameworks that commonly place the systematic synthesis of the results from randomised controlled trials (RCT) as the pinnacle of evidence, regarding the outcomes of interventions. However, whatever research methodology is used, there remains a potential for biases to occur. The Centre for Evidence Based Practice in Oxford has created a ‘catalogue of bias’, which currently lists in detail 49 types of bias, and reports about 250 potential candidates for the catalogue (see https://catalogofbias.org/). In what follows, I focus on those biases that seem most likely to undermine distributive justice.

Potential injustices begin with the choice of subject areas for research. The decisions about which conditions, interventions and outcomes will be the subject of research are made by those who fund or commission the research and may thus serve the commercial or political ends of those with economic or political power. In the UK and elsewhere, about two-thirds of research funding is from the global pharmaceutical industry,8 largely aimed at the development of drugs and devices in particularly profitable sectors that have the potential for exclusivity through protected intellectual property.9 Even government and charity-funded research suffers from biases in the selection of topics, with funding distribution matching poorly to the sectors where there is most clinical need.10

The mismatch between patient need and research investment may partly reflect these commercial and political influences, but the growth of evidence-based practice has also altered the emphasis of research. RCTs may privilege certain conditions, treatments and outcomes that are more suitable for RCTs, whereas complex interventions, caring processes or person-centred approaches are less amenable to such research. Expert evidence of healthcare practitioners is seen as being at the lower end of the evidence hierarchy, so professionals working in certain disciplines may suffer a credibility deficit. Such disciplines include those where there is a less established tradition of positivist scientific research and greater dependence on interpersonal and qualitative aspects of healthcare.

This can become a self-perpetuating situation, with the research methodology becoming the de facto measure, not only of the scientific rigour of the research, but of the status of the researcher, leading to editorial boards, priority-setting committees and research commissioning boards being largely filled by academics who have a vested interest in such research.

The focus on RCTs leads to the potential for certain groups to be systematically excluded from the collection of evidence. This may result from the explicit exclusion criteria used in many clinical trials. Van Spall et al 11 show that many studies excluded the elderly, women, children or those with other diseases, without justification, and Cherubini et al highlight the biases caused by excluding older people from studies of heart failure.12

Some of the forms of epistemic exclusion may be less explicit. For example, methods used for establishing outcomes are often based on questionnaires that may only be accessible to those who are literate in a particular language.13 There may also be indirect forms of exclusion, such as those based on geography. On a national scale, large research studies tend to take place preferentially in areas associated with leading teaching hospitals, where practices, patient demographics, equipment and staffing may not match that available in other areas.14 For international trials, recruitment may be driven by the costs and regulatory arrangements in different jurisdictions and there is evidence that this has led to questionable practices in clinical trials.15

Other aspects of trial design may also lead to distortions. The comparator may be selected to maximise the expected benefit, trials may be placebo controlled or use a comparator that is not considered the best available in routine practice. Such comparator bias is likely to result in the exaggeration of any potential benefits of the new intervention.16

The selection of outcomes may also lead to biases. Important outcomes may be rare or delayed, in which case it may be impractical for studies to be large enough, or sufficiently prolonged, to accurately estimate their occurrence and sponsors are keen to minimise the cost, size and duration of clinical trials. This may lead to the use of composite outcomes, where several different events may be counted together, as in the use of major adverse cardiac events in cardiovascular trials.17 Another common adaptation is the use of ‘surrogate’ outcomes; where an outcome of interest is difficult to measure, due to the timescale or other factors, an alternative that is more ‘objective’, immediate or easy to measure may be found. For example, important clinical outcomes in cancer treatments may be the length and quality of survival, but these are difficult to measure and confounded by complex treatment pathways and the need for prolonged follow-up, so that measures such as progression-free survival, time to progression or tumour response rates are often used as surrogates.18

Further biases occur in the reporting of trial outcomes. It is estimated that about half of trials that take place never result in published or publicly available data19 and there is evidence of distortion or selective reporting in those that are published.20 21 This is compounded by editorial policies that make it more likely that positive results will be published and widely cited, leading to publication bias.22

Many studies are halted or abandoned and data from these may not be available, creating further distortions.23 In 2013, the restoring invisible and abandoned trials initiative promoted the reanalysis of data from such trials and demonstrated examples of trials where reanalysis challenged the original conclusions.24 One famous example of such distortions is the history of ‘Study 329’, a trial of paroxetine, which was reported as showing this to be an effective and well-tolerated treatment for depression. Subsequent investigation showed that GlaxoSmithKline (GSK) had failed to report safety data and in 2012 the US Department of Justice settled criminal and civil proceedings with a record US$3 billion fine for GSK, although it was estimated that they had made US$30 billion profit through the illegal promotion.25 In 2012, the British Medical Journal devoted an entire issue to the problem of hidden data from clinical trials, citing this and multiple other examples of potential distortions and misleading reports.26

One of the key aspects of the treatment of cancer from the patients’ perspective, and an essential aspect of the evaluation of cost effectiveness, is an assessment of quality-of-life outcomes. A review of RCT protocols and publications of 173 cancer trials27 showed that only 52% included quality-of-life outcomes in the protocol and, of these, only 39% reported the quality-of-life outcomes, 20% limited reports to other outcomes, and the results of 41% remained unreported. Submissions of evidence for the UK National Institute for Health and Care Excellence (NICE) appraisals show that companies sometimes fail to divulge the results of such measures, even when they are known to exist, preferring to use other methods that may produce more favourable estimates of cost effectiveness.¹

Injustices in the interpretation of evidence

Evidence is frequently incomplete or conflicting, so that it is necessary to use some degree of interpretation, extrapolation or make assumptions to fill any gaps. If this is done in such a way that certain groups or individuals are disadvantaged, then this is another area in which injustices may arise.

At this point, it seems relevant to return to the question of the role of the expert in the decision-making process. In discussing the relationship between healthcare professional and patient, Carel and Kidd discuss the appropriateness of assigning epistemic privilege to an expert.

… we claim that it is useful to distinguish between warranted and unwarranted epistemic privilege, e.g. healthcare professionals warrant epistemic privilege in their interpretation of a CT scan, but not in deciding where a patient should die (eg, in hospital or at home).7

In current decision-making processes, there are many areas of potentially relevant expertise: various clinical disciplines, pharmacology, medical science, health economics, utility analysis, decision analysis, medical ethics, statistics and public health. Specific aspects of evidence may warrant epistemic privilege for a particular group of experts, but it may be difficult to identify those areas of a complex technical argument where such privilege is unwarranted. An expert may be operating within a particular paradigm that presupposes some underlying set of values or assumptions, which are not directly within their area of expertise. Alternatively, the presentation of a rational and highly technical scientific argument may obscure the nature of some of the underlying value judgements.

This may be further clarified by a couple of examples from the processes used by NICE in making decisions on behalf of the NHS in England and Wales, which are likely to be representative of the type of decisions made in many other jurisdictions. The first example relates to the way in which NICE estimates utility for the cost–utility calculations that form the basis of many judgements.

Although NICE states in its documentation that it does not subscribe to a purely utilitarian approach to distributive justice,28 many of the decision-making processes that it applies are centred around a calculation of cost effectiveness and, although there are a number of flexibilities in the process, there is often a statement to the effect that the committee determined a ‘most plausible incremental cost effectiveness ratio (ICER)’. The ICER is a measure of cost effectiveness, usually presented as a cost per quality adjusted life year (QALY), calculated in line with their guidance on methodology.29

The calculation of the cost per QALY is based on a process of health economic modelling that involves a complex technical exercise, drawing together evidence from a variety of sources, the development of a computer model and choosing between a series of alternative assumptions and calculations. The ICER is only one of a variety of different parameters that may be relevant to decision-making, the QALY is only one of a number of measures of effectiveness and NICE methodology uses a specific way of calculating the QALY, each stage requiring a series of assumptions and value judgements.

The outcome measure which NICE advises for calculating the QALY is the EuroQol—a measure based on five ‘dimensions’ of health each of which is scored at three levels² and then converted to a utility weighting based on a tariff that is specific to England, known as the EuroQol, five-dimension score (EQ-5D).30 The relative value of these dimensions varies between individuals and has also been shown to vary systematically between different racial and ethnic groups.31 Furthermore, there are many aspects of healthcare that may be important to individuals or groups that are not captured at all in such measures. For example, studies have demonstrated that the EQ-5D is insensitive to levels of hearing loss32 and may not adequately capture the outcome of interventions that affect mental health,33 dementia34 or multiple sclerosis.35 Such measures also fail to include other widely valued aspects of healthcare such as processes of care,36 compassion, dignity and autonomy. It seems inevitable that rationing decisions based on evidence that fails to quantify areas such as mental health, disability or compassionate care will result in displacement of healthcare that provides benefits in these domains.

The result of NICE methodology is that experts, such as health economists and systematic reviewers, often acting on behalf of commercial sponsors, produce extensive and highly technical documentation, often running to many hundreds of pages. These generate numerical estimates of a parameter, the ICER, which drives decision-making. However, although these experts may warrant epistemic privilege in interpreting the research evidence and in the process of calculation, there may be unwarranted epistemic privilege deeply embedded and obscured within this apparently objective process, particularly relating to the weights and values assigned to the various aspects of risks and benefits.

A further example is that of discounting costs and benefits in health economic (and other public policy) evaluations.37 This is common practice and there are a number of different opinions and justifications for using particular rates that relate to economic growth, budgetary constraints and time preferences for expenditure and outcomes.38 This has recently been the subject of considerable debate among experts in the field of economics and health economics, often providing highly technical arguments for using particular rates39 or for and against the use of differential rates for costs and benefits.38

However, it is worth considering the overall effects of these differences on the outcome of the decision-making process. Tied up in this debate is a value judgement about our preferences, either as a society or as individuals, for immediate versus future gains and losses. A higher discount rate results in future costs and benefits becoming less important in any decision-making process. For example, higher discount rates will make screening and preventative medicine less attractive, due to the upfront costs and deferred benefits, or in a non-health example, would reduce the value of measures to address climate change that would benefit future generations.40 There is a danger that to defer to technical experts to advise on appropriate discount rates is to lose sight of the underlying value judgements that may not be a proper subject for such epistemic privilege. This is a particular concern if there is a risk of bias in the process for developing methodology, since industries with the resources to influence such decisions have a vested interest in minimising discount rates to maximise the price that can be justified for technologies with deferred and uncertain benefits.

These examples highlight the potential risks of increasing specialisation that relies on specialist expertise to generate or process evidence. This may require them to consciously or unwittingly provide judgements that may be outside their area of expertise and may disadvantage those who are subject to the decision, whose evidence is never sought or represented.

Vandana Shiva used the term ‘epistemological violence’ to describe the potential injustices that may be introduced between the scientific expert and non-expert

Here violence is inflicted on the subject socially through the sharp divide between the expert and the non-expert—a divide which converts the vast majority of non-experts into non-knowers even in those areas of life in which the responsibility of practice and action rests with them.

But even the expert is not spared: fragmentation of knowledge converts the expert into a non-knower in fields of knowledge other than his or her specialization.41

In this respect, it is not just the patients who are the ‘non-knowers’, but those healthcare professionals who, as consumers of the guidance, do not have the resources to re-examine or challenge any underlying data or assumptions.

Who is wronged by evidence-based guidance?

If the evidence underpinning rationing decisions is flawed, this may undermine efforts to ensure distributive justice. Injustices may occur either through denying access to technologies, which would have been available had unbiased evidence been available or, conversely, through approval of technologies that would have been rejected. In the former circumstances, the wronged parties are easily identified as those who may have benefited from incorrectly rejected technologies, but I would suggest that this is a less common situation. Since most evidence is generated, analysed, reported, and often interpreted, by an industry with commercial interest in the new technology, the direction of any bias is likely to exaggerate the benefits and underestimate the costs and risks of such treatments. Furthermore, such overt rationing decisions are transparent and open to challenge, the individuals who are wronged are identifiable, and those with a commercial interest in a positive recommendation have the financial, scientific and legal resources to challenge a negative decision.

It seems more likely that any evidential failings will result in inappropriately liberal guidance, in which case the wronged parties are less easily identifiable. In this situation, the injustice relates to opportunity costs and occurs through the diversion of limited resources away from aspects of healthcare which, given unbiased evidence, would have been seen to better serve the aim of distributive justice. Since the wronged parties are not easily identifiable or linked to the specific guidance, the aims of transparency and openness to challenge cannot be achieved, exacerbating potential injustices.

Some general points can be made about the areas and groups that are likely to be wronged. Disinvestment decisions rarely relate to specific technologies or are evaluated with the same rigour as those regarding new and costly technologies.42 Attempts to identify an appropriate threshold43 and studies of the effects of financial pressures44 suggest that displacement of resources is likely to occur in particular areas, potentially increasing waiting times, diluting care and limiting access for patients with chronic conditions, such as mental health, chronic physical conditions and disability.

Such wrongs are not necessarily limited to rationing decisions, but may be occur with any evidence-based guidance. Although other guidance is rarely mandatory, there may be performance indicators or incentives for healthcare professionals to adhere to such guidance, and the underlying biases and value judgements may not be transparent to individual professionals or patients relying on the guidance. For example, where trials have unjustifiably excluded elderly people or other specific groups, decision-makers face a choice between advising against providing a potentially beneficial treatment or recommending its use, based on the research from which these groups were deliberately excluded.

Addressing potential evidential injustices

Decisions regarding the allocation of limited healthcare resources will always be complex, requiring tradeoffs between competing priorities and recognition of practical limitations. However, regardless of the methods used to achieve distributive justice, the legitimacy of such procedures relies on a foundation of unbiased evidence. There are a number of practical steps that might be taken to improve the current situation.