The Risk to Population Health Equity Posed by Automated Decision Systems: A Narrative Review

01/18/2020 ∙ by Mitchell Burger, et al. ∙ UNSW 0

Artificial intelligence is already ubiquitous, and is increasingly being used to autonomously make ever more consequential decisions. However, there has been relatively little research into the consequences for equity of the use of narrow AI and automated decision systems in medicine and public health. A narrative review using a hermeneutic approach was undertaken to explore current and future uses of AI in medicine and public health, issues that have emerged, and longer-term implications for population health. Accounts in the literature reveal a tremendous expectation on AI to transform medical and public health practices, especially regarding precision medicine and precision public health. Automated decisions being made about disease detection, diagnosis, treatment, and health funding allocation have significant consequences for individual and population health and wellbeing. Meanwhile, it is evident that issues of bias, incontestability, and erosion of privacy have emerged in sensitive domains where narrow AI and automated decision systems are in common use. As the use of automated decision systems expands, it is probable that these same issues will manifest widely in medicine and public health applications. Bias, incontestability, and erosion of privacy are mechanisms by which existing social, economic and health disparities are perpetuated and amplified. The implication is that there is a significant risk that use of automated decision systems in health will exacerbate existing population health inequities. The industrial scale and rapidity with which automated decision systems can be applied to whole populations heightens the risk to population health equity. There is a need therefore to design and implement automated decision systems with care, monitor their impact over time, and develop capacities to respond to issues as they emerge.



There are no comments yet.


page 4

This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.

1 Introduction

There is tremendous hype surrounding the future of artificial intelligence111The UK Parliament’s House of Lords Select Committee on Artificial Intelligence *HouseLords18Ready wryly observed that “the debate around exactly what is, and is not, artificial intelligence, would merit a study of its own”. For the purposes of this review the practical definition of artificial intelligence specified by the House of Lords Select Committee is used: ‘Technologies with the ability to perform tasks that would otherwise require human intelligence, such as visual perception, speech recognition, and language translation.’ (p. 20) (AI) in health – particularly in medicine Coiera18,Maddox19, and also increasingly in public health Chowkwanyun18,TaylorRobinson18,WongZ19. The singular, global impact of AI on population health has been called out by the World Health Organization, who proclaim that “more human lives will be touched by health information technology than any other technology, ever.” WHO18BigData However, predictions vary about the extent to which AI will actually revolutionise medicine and public health practices in the shorter term [as discussed by, for example,][]Coiera18,Darcy16,Mukherjee17,TaylorRobinson18. In addition, there is growing concern about the impact of AI on equity and justice, and mounting evidence that AI systems can perpetuate, entrench and amplify existing discrimination and inequality [e.g.,][]AINow18YearReview,Campolo17,Crawford16Report,Eubanks17,UN18ReportSpecialRap.

It is important to stand back and observe that the hype and concern about the impact of artificial intelligence on health comes at a time of unprecedented global interest in AI. Amidst this interest, the potential future impacts of AI have been extensively analysed and discussed. For example, the United Nations Secretary-General has highlighted the potential of AI to advance human welfare, but also emphasised its potential to widen inequality and increase violence UN18SecGeneralStrat. It is common for the future risks of AI to dominate public discourse, especially those relating to automation [e.g.,][]Berg18,DawsonN18,Frontier18,Furman16,Morcom18,Schwab16,RoyalSociety18, autonomous weapons [e.g.,][]DawsonN18,Future15,Schneier19Data, and superintelligence [e.g.,][]Bostrom14,Bridle18,Brundage15,Pamlin15,Tegmark17.

However, while the idea of artificial intelligence may still conjure science fiction dreams and nightmares in the popular imagination, the reality is that AI is here already Campolo17,Rahwan19. Meredith Whittaker and colleagues at the AI Now Institute state: “The rapid deployment of AI and related systems in everyday life is not a concern for the future—it is already here, with no signs of slowing down.” Whittaker18Report Narrow artificial intelligence222‘Narrow’ AI (also referred to as ‘weak’ AI) generally refers to an agent that undertakes a specific, defined task; typically in a way that does not generalise to other tasks or domains Russell10. For context, higher grades of AI are: ‘artificial general intelligence’, which is also referred to as ‘strong AI’, and is taken to mean an agent performing tasks at least as well as humans across many or all domains Bughin17,DawsonN18. ‘Superintelligence’ is performance that exceeds human intelligence across all domains Bostrom14. and predictive algorithms suffuse society – they are woven into the fabric of our daily lives Amoore09,Mackenzie15,Zook17; mediating “our social, cultural, economic and political interactions” Rahwan19. In this way, AI is already ubiquitous Campolo17,Lavigne19,Schneier19Supply, often in very mundane forms in everyday technologies Mackenzie15 – smart phones, online advertising, social media, home assistants, recommendation engines for music and video, online dating, autopilots, and customer support chatbots.

Moreover, the use of systems that are underpinned by narrow AI to make consequential decisions autonomously

(referred to as ‘automated decision systems’) is already prevalent in a broad range of sectors. This includes loan and credit card applications, algorithmic trading, drone warfare, immigration, criminal justice, policing, job applications, education, university entry, utilities network management, and social welfare Angwin16,Campolo17,Knight17,Lecher18,Rahwan19,Reisman18AlgoImpactAssess. While it is difficult to define ‘automated decision systems’ because they are specific to context and purpose Reisman18AlgoImpactAssess, they are generally systems that use classifications and predictions produced by expert AI systems or machine learning algorithms to make recommendations and decisions. For example, Reisman and colleagues define predictive policing automated decision systems as “any systems, tools, or algorithms that attempt to predict crime trends and recommend the allocation of policing resources” (p. 13). A definition adapted for health could be:

any systems, tools, or algorithms that attempt to predict individual or population health trends or states, and recommend the allocation of health resources or specific interventions.

The use of automated decision systems by governments and corporations is rapidly expanding into ever more consequential and sensitive domains Rahwan19,Whittaker18Report. What is particularly insidious about the expanding use of automated decision making is the invisibility of the proliferation Campolo17. Crawford and colleagues write: “In many cases, people are unaware that a machine, and not a human process, is making life-defining decisions.” *[p. 23]Crawford16Report In addition, people tend to become rapidly habituated to advances in AI performance, leading to creeping normalisation. Contributing to this normalisation is the tendency for AI to have an ever-evolving definition as what is not yet possible Kurzweil05.333Indeed, Tesler’s theorem – attributed to the computer scientist Larry Tesler – is that: “AI is whatever hasn’t been done yet.” [p. 7]Bughin17. Pioneering AI researcher John McCarthy also had a similar saying: “As soon as it works, no one calls it AI anymore.” Meyer11. Mundanity, invisibility, and habituation are enabling automated decision systems to proliferate unseen.

As artificial intelligence takes on more and more responsibility for consequential decisions, fundamental questions of rights, fairness and equity arise [p. 6]Crawford16Report. This is prompting heated debate about where and when automated decision systems can be used Campolo17,Walsh17ABC. However, while the potential for AI to fundamentally reshape society in the future is well recognised, and high-level warnings about medium and long-term risks and societal impacts are widespread, the effects of the unseen proliferation of narrow AI and automated decision systems in sensitive domains have only recently started to be scrutinised. Compared to domains such as criminal justice, policing and autonomous warfare, there has been relatively little research into the existing and possible consequences of the use of narrow AI and automated decision systems in medicine, and even less in public health.

2 Method

This review aims to begin to address this evidence gap by answering the following research questions:

  1. Broadly, what is the current state of adoption of narrow AI and automated decision systems in medicine and public health? And how are they expected to be used in the future?

  2. What issues have emerged in the application of narrow AI and automated decision making in other sensitive domains? Is there evidence of these issues emerging in medicine and/or public health applications?

  3. What are the possible implications for public health? In particular, what are the longer-term prospects and risks for population health and equity?

To address these questions, a narrative review Ferrari2015 using a hermeneutic approach was undertaken. A hermeneutic review involves an iterative process of developing understanding through cycles of search and acquisition of literature, together with iterative analysis and interpretation Boell2014. It is an approach that it is suitable for questions requiring clarification and insight which cover diverse and dynamic bodies of scholarly and grey literature Boell2014,Greenhalgh18.

Literatures at the intersection of artificial intelligence, big data, and public health were sourced from Scopus, PubMed, Web of Science, and IEEE Xplore databases, and arXiv, medRxiv, and bioRxiv pre-print servers. Search terms and topics used included ‘artificial intelligence’, ‘machine learning’, ‘big data’, together with ‘public health’, and ‘epidemiology’. Grey literature was sourced using Google Scholar, Hacker News444

, WHO IRIS, United Nations Official Document System, and World Economic Forum Reports. Citation tracking was used to identify additional references. Mapping, classifying and selection was initially undertaken in EndNote X9 (see Figure 

1 for examples). Additional classification and thematic analysis was undertaken using NVivo 12 Pro qualitative analysis software.

Figure 1: Example classifications in EndNote (count of references)

3 Results

3.1 Current state of adoption in health, and expected future uses

While the use of narrow artificial intelligence and automated decision systems is already widespread in sectors such as finance, policing and criminal justice; the health sector has a comparatively low - but rapidly expanding - level of adoption Bughin17,Lancet19,Zandi19. In 2017, an American independent scientific advisory group JASON555See Federation of American Scientists *FederationJASON19 described the state of adoption of AI in the health sector generally as being at an exploratory phase: “AI is beginning to play a growing role in transformative changes now underway in both health and health care, in and out of the clinical setting. At present the extent of the opportunities and limitations is just being explored.” [p. 1]Derrington17 In the ensuing years the state of adoption of AI in health has advanced markedly, with increasing reports that AI is now starting to replace doctors [e.g.][writing for New Scientist]LePage19. The most active areas of application are diagnostic support, for example in medical imaging, and multivariate risk prediction. Challen and colleagues *Challen19 plotted the current state of adoption of machine learning666Machine learning is a sub-branch of artificial intelligence research and practice - for definitions of machine learning and deep learning see for example Brooks *Brooks17Machine, Russell and Norvig *Russell10, and LeCun et al. *LeCun15

in healthcare as at 2019, and projected future applications in terms of increasing levels of automation. Similarly, Ching and colleagues *Ching18 provide an excellent and comprehensive survey of the state of adoption of deep learning in medicine.

In medicine, there have been prominent and increasingly frequent demonstrations of the capability of AI - in particular machine learning - to perform diagnostics using medical images with the same performance levels as experienced clinicians Coiera18,Derrington17,Yasaka18,LiuInPressLancetDH. Examples of high-profile publications include: automated detection of key trauma and stroke indications in head CT scans [in The Lancet]Chilamkurthy18; automated classification of skin cancer at dermatologist-level [in Nature]Esteva17; detection of diabetic retinopathy [in JAMA]Gulshan16; automated classification of abnormalities in chest radiographs [in PLOS Medicine]Rajpurkar18; and identifying breast cancer using screening mammograms [arXiv pre-print]Wu19DeepNeuralNetworks. A systematic review published in The Lancet Digital Health found the “diagnostic performance of deep learning models to be equivalent to that of health-care professionals”, although concerns were raised about the prevalence of poor reporting in deep learning studies LiuInPressLancetDH. As a further indicator of the rapid advance of AI adoption, there has been an acceleration since 2018 in US Food and Drug Administration (FDA) approvals of AI algorithms Mesko19. Future applications include “using artificial intelligence and machine learning to support the integration of genomic information into health care systems” [p. 22]Williamson18, so as to enable personalised drug protocols, precision prevention Meagher17, and early diagnosis of rare childhood diseases Wright18.

In public health accounts, AI is typically cautiously regarded as having the potential to re-envisage and transform public health practices [e.g.,][]Chowkwanyun18,Rubens14,Lancet19,WongZ19. Zandi and colleagues *Zandi19 capture the promissory potential in their call for papers on ethical challenges of AI in public health:

These technologies promise great benefits to the practice of medicine and to the health of populations. This is especially true in epidemiology and the tracking of outbreaks of infectious diseases, behavioural science, precision medicine and the modelling and treatment of rare and/or chronic diseases.

In conjunction with big data, AI approaches are expected to offer new opportunities to measure the impact of upstream determinants of health over the lifecourse Krieger17,Meagher17,StephensonForthcoming. This would be a way of quantifying and revealing the “structured chances” that “drive population distributions of health, disease, and well-being” Krieger12. This opportunity arises because deep learning in particular offers novel capabilities to deal with complex, high-dimensional data and relatively small sample sizes Ching18,LeCun15. The most aspirational accounts predict these new approaches will be able to facilitate action on social and environmental determinants of health, and thereby reduce health disparities Katsis17,Meagher17,StephensonForthcoming,Weeramanthri18.

However, despite this potential being recognised, public health has been comparatively slow to broadly adopt AI in practice Lavigne19,Panch19AIOppRisks,Sadilek18,Lancet19. It is also apparent that the applications of AI are very similar to the uses of big data in public health, a good overview of which is provided by Dolley *Dolley18. Emerging applications of AI in public health are:

  • population health surveillance and disease detection [e.g.,][]BenAmmar18,Lake19,Muller19,Perlman17,Sadilek18,Salathe16,Subramani18,Thorpe17,Xiong18;

  • predicting infectious disease outbreaks [e.g.,][]Bates17,Lim17,Park18,WongZ19;

  • primary and secondary prevention of disease [e.g.,][]Barrett13,Chatelan18,Contreras18,Meagher17,Potash15

  • environmental health [e.g.,][]Dong18,Gonzalez-Jimenez18,KamelBoulos19,Li17DeepAirPollution,Sincak14,Weichenthal19

  • disease screening [e.g.,][]McKinney20,Tran18,WongT16; and

  • risk factor intervention and treatment adherence [e.g.,][]Deb18,Huang18,Labovitz17,Thompson19.

An illustrative example is McKinney and colleagues *McKinney20 demonstrating material reductions in the rates of false positives and false negatives using an AI system for breast cancer screening, highlighting AI’s potential to improve the efficacy and cost-effectiveness of breast cancer screening programs. There have also been a number of demonstrations of the capability of AI to achieve accurate risk prediction [e.g.,][]Attiga18,Chandir18,Harrington18,Khourdifi18,Kwon18,Miotto16,Nadkarni19,Pergialiotis18,Prelot18,Rajiwall17,Rajiwall18,WalshC17Predicting,Wiens16. Weng and colleagues *Weng17 exemplified this capability by demonstrating that machine learning approaches could use routine clinical data to significantly improve the accuracy of cardiovascular risk prediction, compared to an established algorithm.

Risk prediction algorithms are also increasingly being used as a first line of automated triage in advance of primary care appointments [p. 23]Derrington17. For example, UK-based company Babylon Health777 has a partnership with the UK’s National Health Service (NHS) called ‘GP at hand’ to provide online general practice consultations, with over 35,000 registered members as at January 2019 Babylon19Presentation. Babylon Health uses a digital symptom checker underpinned by AI to triage patients – this is an example of an automated decision system. Although concerns have been raised about the safety of digital symptom checkers Fraser18, in mid-2019 Babylon Health was able to raise an additional US$550m in investment capital in order to enable the company to expand into the United States and develop the capability of its AI to diagnose more serious conditions Lunden19. Another application is automated prescribing of contraceptives. A small-scale study published in September 2019 in the New England Journal of Medicine evaluated the safety of telecontraception, which involves the automated prescribing of contraceptives with or without clinicians in the loop. The study found that telecontraception may increase the accessibility of contraception, and also promote better adherence to treatment guidelines compared to in-person clinics Jain19.

AI-based risk stratification is also being used to enable automated, risk-adjusted, per capita funding allocation for health services and primary care. This means that the amount of money allocated to people for primary care services for a period of time is assigned based on their health status and algorithmic predictions of risk. For example, a commercial algorithm is used by a number of Accountable Care Organisations in America to make healthcare resourcing decisions for over 70 million people Obermeyer19,Obermeyer19FAT. As another example, the Australian Government is currently trialling risk-adjusted funding for primary care through the Health Care Homes initiative.888 The amount of funding provided to participating general practitioners to coordinate the care of individual patients will be decided using a predictive risk algorithm. The algorithm – developed by the CSIRO999 – factors in more than 50 variables, including demographics, a proxy for social determinants (the Australian Bureau of Statistics’ SEIFA indices for social and economic status101010See Australian Bureau of Statistics *ABS18SEIFA), physiology, medicines, conditions, pathology results, and lifestyle factors Hibbert18.

In summary, accounts in literature and the media reveal a tremendous expectation on AI to transform medical and public health practices. There have been prominent demonstrations of successful narrow AI capability in medical and public health applications – particularly in diagnostic decision making, risk prediction and disease surveillance. These demonstrations reinforce the hype and expectation surrounding AI, and stimulate its rapidly expanding adoption in medicine and public health.

3.2 Emerging issues

As the adoption of narrow AI and automated decision making in sensitive domains expands, this review has found that significant evidence of emerging issues has been gathered, including in health. Indeed, Whittaker and colleagues [p. 42]Whittaker18Report contend that the harms and biases in AI systems are now beyond question. “That debate has been settled,” they write, “the evidence has mounted beyond doubt in the last year.” They point to a growing consensus – citing a string of high-profile examples – that AI systems are perpetuating and amplifying inequities AINow18YearReview,Whittaker18Report. This review focuses on three key issues: 1) bias, 2) opacity and incontestability, and 3) erosion of privacy – as they appear to be materialising in medical and public health applications of AI, and also because of the potential for these issues to entrench and amplify existing inequities, with possible downstream implications for population health.

3.2.1 Bias

Of the issues that have emerged in the application of AI and automated decision making, perhaps the most prominent is bias. Defining bias is difficult because the term has specific meanings in fields such as statistics, epidemiology, and psychology, and these are often confusingly contradictory Campolo17. Whittaker and colleagues *Whittaker18Report distinguish between two types of bias arising from automated decision systems: allocative – where resources or opportunities are unfairly distributed; and representational – where harmful stereotypes and categorisations are reproduced and amplified.

The hope that AI will assist to overcome biases in human decision making [e.g.,][]Baur17,Lavanchy18,Luckin17 has been used as a justification for the use of automated decision systems [e.g.,][]Lecher18. However, there have been glaring examples of racial, gender and socioeconomic biases evident in AI and automated decision making used in a number of sensitive domains:

  • criminal justice Angwin16,EPIC17,Eubanks17,Lapowsky18,Lum16;

  • policing BennettMoses18,RichardsonForthcoming,Sentas17,Stanley18,Vogt18Part1,Vogt18Part2;

  • hiring practice Dastin18,Mann16;

  • university admissions Schwartz19;

  • online advertising Bolukbasi16,Campolo17,Lambrecht18;

  • education AINow18Litigating,Campolo17,Feathers19,Madnani17;

  • immigration Whittaker18Report; and

  • facial recognition Buolamwini18,Concerned19Letter,Garvie16,Lohr18,Singer18,Vincent19.

There is also emerging evidence of the harmful impact of biases in the context of algorithmic censorship Binns17,Cobbe19. For example, there is racial bias in how hate speech is moderated Sap19, gender bias in how nudity is censored on Instagram Cook19,Toor16, and censorship of marginalised communities through overly-restrictive automated filtering of LGBTQ content on YouTube, Tumblr and Twitter Allen18.

Generally, algorithmic biases can arise in two main ways: in the upfront design (specification) of an algorithm, and in the data that are used to train algorithms, for example by being unrepresentative, or encoding existing systemic biases Ankeny17,Crawford16Report,DawsonN18,McGoey17. Bughin and colleagues *[p.37]Bughin17 explain how bias can be caused by data: “Since the real world is racist, sexist, and biased in many other ways, real-world data that feeds algorithms will also have these features—and when AI algorithms learn from biased training data, they internalize the biases, exacerbating those problems.” As bias can arise unintentionally from data used to train the algorithms, it can be very difficult to detect and measure Barocas16,Campolo17,Lecher18,Reisman18AlgoImpactAssess.

Algorithmic bias – especially undetected bias – can lead to inaccurate and inappropriate generalisation Brooks17Machine,Khoury14,Maddox19,Muller19. Generalisation is a key issue in machine learning theory and practice Mackenzie15. The general rigidity and brittleness of machine learning models means that models built for a specific purpose cannot be readily transferred to other applications, nor are they robust to changes over time Brooks17Machine. Barocas and Selbst *Barocas16 make the crucial point that inappropriate generalisation is typically a result of careless reliance on “statistically sound inferences that are nevertheless inaccurate” (p. 688) – rather than deliberate prejudice. Again, that the disparate impact is inadvertent, makes it wickedly difficult to detect. And moreover, inappropriate generalisation can have a performative111111Performativity, in this context, is the act of making a prediction having the effect of contributing to the predicted outcome coming into being. (i.e. self-fulfilling) impact DawsonD19,Mackenzie15, where inaccurate predictions actively contribute to produce discriminatory outcomes. This is especially evident in criminal justice and predictive policing implementations of automated decision systems Angwin16,BennettMoses18,Lum16,RichardsonForthcoming,Stanley18. Inaccurate generalisation also stems from AI’s inherent reliance on data, and the axiomatic tension between over-fitting to past data and predictive accuracy. Writing for Computerworld, George Nott quotes Genevieve Bell: "Humans can sometimes fear their choices are being "prescribed by their past" by these algorithms, which by their nature work on retrospective data" Nott17. The reliance on past data is a key reason why there is a risk that automated decision systems will perpetuate inequities, particularly where the systems rely on data that either reflects past systemic inequalities, or does not adequately encode social and environmental determinants Chowkwanyun18.

As with other high-stakes domains, bias has been called out as a key issue that will need to be addressed before AI can be trusted and more widely adopted in health Campolo17,Challen19. The use of biased data is known to reproduce and amplify discrimination and injustice Barocas16,Crawford16Report,Jasanoff17. And many scholars have highlighted the lack of diversity, inclusiveness, and representativeness in health and other datasets [e.g.,][]Barocas16,Campolo17,Dolley18,Lavigne19,LePage19,Meagher17,Panch19InconvientTruth,Prainsack19,Walsh19ACOLA,Whittaker18Report. The paucity of environmental and social exposure data has also been identified [e.g.,][]AIHW18,Derrington17, however the potential for this to lead to biases in narrow AI and automated decision systems needs to be further explored.

In public health too, it is well-recognised that skewed and unrepresentative data can bias the results of traditional epidemiological and population health analyses such as disease surveillance, leading to inaccurate estimates and inference for diverse populations Bates17,Krieger12,Thorpe17. Exemplifying how data quality can affect automated decision systems, flawed data was blamed for the failures of Idaho’s automated decision system to equitably allocate home care funding Stanley17. And in a study that has striking similarities to

ProPublica’s revelatory investigative reporting into racially biased crime risk prediction Angwin16, Obermeyer and colleagues *Obermeyer19 detected significant racial bias in a commercial algorithm used by Accountable Care Organisations in America and applied to an estimated 200 million people each year. Their analysis revealed that White patients were given the same risk score as Black patients who were considerable sicker, inadvertently leading to Black patients having unequal access to care. The authors estimated that resolving this disparity would have more than doubled the proportion of Black patients receiving additional assistance (from 17.7% to 46.5%).

3.2.2 Opacity and incontestability

Another key issue is the opacity of artificial intelligence, and the ensuing incontestability of automated decisions. Algorithms and AI are opaque and invisible processes, often characterised as ‘black boxes’ DawsonN18,Knight17,Pasquale15,Rahwan19,Salathe18. Once an AI algorithm has been trained – particularly one based on deep learning – it is not clear how it is making decisions Knight17,Waldrop19. Findings from the Pew Research Centre reveal confusion amongst the general public about the inner workings of algorithms, and wariness about inscruatble algorithmic processes that have delegated responsibility for high-stakes decisions SmithA19.

A consequence of this opacity is the difficulty of questioning and contesting automated decisions. Whittaker and colleagues at the AI Now Institute observe that when automated decision systems make errors, “the ability to question, contest, and remedy these is often difficult or impossible” Whittaker18Report. This is exemplified in the United States criminal justice system, where “Defendants rarely have an opportunity to challenge their [algorithmic] assessments” Angwin16. Furthermore, the ability of humans to intervene, override or even explain decisions is severely limited, rendering frontline workers disempowered intermediaries Whittaker18Report. Early reports suggest that issues of incontestability have emerged in the use of automated decision systems in health. This powerlessness is evident in Colin Lecher’s article for The Verge *Lecher18 about the case of a women with cerebral palsy in Arkansas who had her health services funding cut in half by an automated algorithmic decision. When an attorney began to investigate complaints about the algorithm, he found: “No one seemed able to answer basic questions about the process. The nurses said, ‘It’s not me; it’s the computer’.”

For people who are the subjects of automated decisions, there is even more of a sense of powerlessness. Regarding the American Civil Liberties Union (ACLU) legal challenge to Idaho’s use of an algorithmic decision system to allocate home care funding [see also][]Stanley17, Lecher *Lecher18 writes: “Most importantly, when Idaho’s system went haywire, it was impossible for the average person to understand or challenge”. Incontestability can therefore lead to aggregation of power, limited opportunities for redress, and an unwillingness and inability of vulnerable people to contest their own treatment, thereby perpetuating and exacerbating existing inequities and discriminatory dynamics Crawford16Report,Eubanks17.

The tendency to blindly trust complex statistical methodologies both fortifies the inscrutability of automated decisions, and also intensifies the performativity of prediction. In the field of public health, concern has been raised about overconfidence in big data and complex statistical techniques Chiolero18,Lavigne19. Salathé *Salathe16 refers to this as “big-data hubris”. Similarly, Krieger *Krieger17, quoting prescient statistician Lancelot Hogben, cautions against hiding “behind an impressive façade of flawless algebra”. Artificial intelligence systems, because they are considered ‘intelligent technology’, are particularly prone to going uncontested [pp. 6-7]Crawford16Report. Underlying this misplaced trust is a reductionist belief in the neutrality of data – a belief in data being beyond reproach. Sheila Jasanoff *Jasanoff17 captures this eloquently:

…in modernity, information, along with its close correlate data, has been taken for granted as a set of truth claims about the way the world is. Information, as conventionally understood, quite simply is what is: it consists of valid observations about what the world is like. Data represents a specific form of information, a compilation of particular types of facts designed to shed light on identifiable issues or problems. As representations of reality, both public information and public data were seen until recently as lying to some extent outside the normal domains of political inquiry. (p. 5, emphasis in the original)

But data are not neutral Barrowman18,boyd12,Zook17. Jasanoff *Jasanoff17 goes on to say: “But as scholarship on science and technology has repeatedly shown, information is a social construct — not a mirror of the world but a human-made representation of matters in that world.”

What follows from overconfidence in complex statistical techniques and belief in the neutrality of data is a misplaced trust in the ability of automated decision systems to make correct, unbiased decisions. Virginia Eubanks is quoted by Lecher *Lecher18 as saying that “there is a “natural trust” that computer-based systems will produce unbiased, neutral results.” Likewise, Campolo and colleagues *Campolo17, citing Sandra Mayson’s work on algorithmic risk assessment in setting bail, point out the potential of risk assessment to “legitimize and entrench” problematic reliance on statistical correlation, and to "[lend such assessments] the aura of scientific reliability.” And similarly, Barocas and Selbst *Barocas16 identify the “imprimatur of impartiality” conferred on the decisions taken by algorithmic systems. This is important because it gives rise to a false confidence in the superiority of automated decisions.

Through their complexity, invisibility, and inhumanity, algorithmic decisions are achieving incontestability. And thus, when predictions are made, they verge on acts of creation, of magic Chun11. Will Knight wrote in 2017: “As the technology advances, we might soon cross some threshold beyond which using AI requires a leap of faith.” Considering how “indecipherable” algorithmic systems can be Rahwan19, and how they can be “beyond the understanding even of the people using them” Lecher18, it should therefore not be surprising that the use of automated decision systems creates a growing “accountability gap" Whittaker18Report. This is perpetuated by trade secrecy and intellectual property provisions that enable proprietary systems to be shielded from scrutiny, even in the face of legal challenge Angwin16,Campolo17,DawsonD19,Muller19,Obermeyer19,Rahwan19,Salathe18,Whittaker18Report. The research of the AI Now Institute has uncovered “black boxes stacked on black boxes: not just at the algorithmic level, but also trade secret law and untraceable supply chains.” AINow18YearReview In this way, trade secrecy reinforces the incontestability of automated decisions Whittaker18Report, which heightens the risk that existing biases and disparities are perpetuated and amplified.

3.2.3 Privacy erosion

The use of artificial intelligence is also having a significant impact on human rights to privacy, freedom of expression, and access to information OVIC18,PrivacyIntl16,PrivacyIntl18,Santow18. While privacy issues are not limited to AI, three issues – risk of re-identification, intrusive data extraction and capitalisation, and invasive surveillance – stand out in relation to AI. Firstly, many scholars and institutions have highlighted the risk of re-identification and compromising individual privacy which arise from big data analytics and AI [e.g.,][]DawsonD19,Dolley18,Mittelstadt2018,Ohm10,Rocher19,Zook17. Secondly, expanding use of AI in surveillance – for example use of facial recognition in policing DawsonD19, and monitoring of employees’ emotional state for performance evaluation and retention decisions Campolo17 – is eroding privacy and amplifying discriminatory dynamics DawsonD19,Whittaker18Report,Zuboff19. And thirdly, as a result of the recognition of and the use of AI to exploit the economic value of personal data Leonelli19,WEF11, systems for data extraction, ‘datafication’ and capitalisation are becoming increasingly intrusive and pervasive Jasanoff17,Newell15,Parry17,Sadowski19,Schneier19Data,West17,Zuboff19. This intrusiveness and exploitation has resulted in growing community wariness of data sharing, and erosion of social licence for use of individual data IPSOS16,Richmond19,SmithA19.

Reports about erosion of privacy are also already prevalent in health applications of narrow AI. Like other domains, the value of personal health data has long been recognised Leonelli19,McMorrow14,WEF11. The extraction of data will be driven more and more by commercialisation and productisation of data as a tradable asset Newell15,Parry17,Sadowski19,West17,Zuboff19. In health, this has seen the emergence of specialist data brokers, such as Explorys121212, which was purchased by IBM in 2015. An example brokerage is Memorial Sloan Kettering (MSK) Cancer Center entering into a licencing agreement with AI start-up Paige.AI131313 to “grant exclusive access to MSK’s intellectual property in computational pathology, including access to MSK’s 25 million pathology slides.” MedTech18. However, the lure of using personal health data in AI and big data applications is driving an erosion of rights to privacy, confidentiality, and data ownership Balthazar18,Bogle19,Sharon18. For example, there are increasing privacy concerns expressed in the media and literature about health apps’ lack of transparency around data sharing and use Grundy19. Another example is data sharing between the UK National Health Service and Google DeepMind being considered a betrayal of public trust Hern18,Lomas18,Revell17. Google has also faced media criticism for gathering personal health information on millions of people in the United States as part of ’Project Nightingale’ Anonymous19Guardian,Copeland19,Fussell19, as has Memorial Sloan Kettering health service for its data sharing arrangement with Paige.AI Ornstein18. Campolo and colleagues *Campolo17 point out that in domains like health, because of AI’s reliance on large amounts of data, the privacy rights of vulnerable populations are particularly at risk due to lack of informed consent and due processes mechanisms.

4 Discussion

It is clear from the results of this review that there is significant evidence of and concern about issues that have emerged in the use of narrow AI and automated decision making in sensitive domains, including growing evidence in health. Meanwhile, increasingly frequent and prominent demonstrations of narrow AI capability in medicine and public health are stimulating a rapid expansion in adoption.

The examples given in section 3.1 of automated decision systems that allocate health services funding, illustrate how decisions which are automatically made based on the results of an algorithm can be consequential for population health. In the example of the Health Care Homes program in Australia, the decisions have consequences for individual and population health and wellbeing, for the livelihood of general practitioners, and for the sustainability of the primary care tier of the Australian health system. Importantly, in this context, ‘population health’ is taken to mean the ‘collective health’ [p. 96]Rose08 of populations. Rose wrote that “healthiness is a characteristic of the population as a whole and not simply of its individual members.” (p. 95) Here, ‘populations’ are relational constructs: “dynamic beings constituted by intrinsic relationships both among their members and with the other populations that together produce their existence” Krieger12. This is distinct from the “dominant view that populations are (statistical) entities composed of component parts defined by innate attributes” Krieger12.

As the use of narrow AI-based automated decision systems and algorithmic prediction in health continues to expand, it is probable that the same issues – bias, incontestability, and erosion of privacy – which have unequivocally emerged in other sensitive domains, will also manifest widely in medicine and public health applications. The implication of these issues manifesting widely is that there is a significant risk that use of automated decision systems in health will amplify and entrench existing population health inequities, and also potentially create new inequities. Examples of population health inequities that may be affected include the persistent life expectancy gap afflicting Aboriginal and Torres Strait Islander peoples in Australia Holland18,Lowitja19, and the stark socio-economic gradient in health outcomes evident in many countries CoSD08,Marmot05,NRCandIOM2013. In addition, because of the strong influence of social determinants of health CoSD08, the use of automated decision systems in other domains such as welfare and immigration, will undoubtedly also have a downstream effect on population health [see for example][]Medhora19.

There are two key reasons why it is highly probably that issues such as bias, incontestability and erosion of privacy will manifest widely in health. Firstly, early reports of issues associated with the use of automated decision systems and predictive algorithms in health have already surfaced, as outlined in the results section. Secondly, the same circumstances and drivers which have compelled adoption of automated decision systems and given rise to issues in other high-stakes domains, also exist in medicine and public health – indicating the adoption trajectory and consequences are likely to be similar. Key amongst these drivers are cost and capacity pressures facing health services, and the commercial imperative to capitalise on growing health data assets using AI approaches.

The same imperatives to constrain costs and capitalise on data exist in other sensitive domains – such as education, policing, criminal justice, and immigration – and this has incentivised the adoption of automated decision systems by public agencies and corporations in those domains Reisman18AlgoImpactAssess,Whittaker18Report. A process of learning and emulation akin to policy transfer141414Policy transfer can be defined as a process of learning and emulation by policy makers in which “knowledge about policies, administrative arrangements, institutions etc. in one time and/or place is used in the development of policies, administrative arrangements and institutions in another time and/or place.” Dolowitz96 will likely ensure the adoption trajectory of automated decision systems will be similar in the health domain. This emulation and diffusion of innovation occurs because jurisdictions and agencies “face common problems” and they look to other communities for lessons and solutions Hadjiisky17. The process is accelerated by a futures industry whose purpose is to market ideas and trade on promises and expectation. Hadjiisky and colleagues write:

…an entire global marketplace of ideas and recommendations on ‘best practices’ has emerged, including international organizations, commissions, donor groups, consultants, think tanks, institutes, networks, partnerships, and various gatherings of the great and the good such as Davos. They may not use the terminology of ‘policy transfer’ but that, in essence, is what they are debating and selling. (p. 2-3)

Indeed, there is tremendous expectation heaped upon AI and precision health approaches to increase the efficiency of healthcare services and systems as a means of containing costs Lecher18,Panch19InconvientTruth,Prainsack19,Whittaker18Report. Dolley *Dolley18 captures this in relation to public health:

Precision public health is exciting. Today’s public health programs can achieve new levels of speed and accuracy not plausible a decade ago. Adding precision to many parts of public health engagement has led and will lead to tangible benefits. Precision can enable public health programs to maintain the same efficacy while decreasing costs, or hold costs constant while delivering better, smarter, faster, and different education, cures and interventions, saving lives. (p. 6)

The drive to rapidly adopt AI in health is given urgency by the oft-cited pressures facing health systems around the world, including population ageing, workforce shortages, increased prevalence and incidence of noncommunicable diseases, and variability in service quality and clinical outcomes ACSQHC18,Britnell19,CSIRO18,Lancet19. These pressures are especially acute in low-income countries, where health resources are particularly scare Lancet19. Similarly, long-standing problems with current diagnostic approaches, such as invasiveness, cost, accessibility, and low precision, as well as the limitations of traditional analytic approaches, are driving interest in improved AI-enabled methods Derrington17,Maddox19.

In addition, the drive to adopt AI in health follows closely on the heels of the imperative in medicine and public health to capitalise on big data. Much like AI more recently, big data has commonly been expected to transform medicine and public health practice [e.g.,][]Bates17,Choucair15,Dolley18,Khoury14,Mattick14,Salathe18,Thorpe17,Weeramanthri18,Williamson18. AI – particularly deep learning – promises the ability to finally exploit big, complex, noisy, highly-dimensional health data that health organisations have been accumulating Katsis17,Lavigne19,Ogino19,Salathe18,Thorpe17,WongZ19. For example, an editorial in The Lancet Public Health *Lancet19 states: “The ability of artificial intelligence and machine learning algorithms to analyse these multiple and rich data types at a scale not previously possible could bring a step change in public health and epidemiology.” There is a convergence between the accumulation of data Sadowski19 and hyper-enthusiasm about AI. The drive to make use of and assetize data in health Tarkkala19 will drive adoption of AI and automated decision systems. Typifying this drive is a call by the Chief Executive of a new United Kingdom National health Service agency, NHSX, to capitalise on big data and AI SmithR19. Underscoring the financial imperative pressuring health services and agencies to adopt AI, the economic opportunity has been identified not only by corporations, but by governments and public agencies [e.g. in Australia:][]AlphaBeta18,CSIRO18,SenateSelectComm16.

What the drive to address health system pressures and capitalise on data portends is that the adoption of AI systems will be substantiated on the basis of health service efficiency and productivity, and not necessarily on population health impact. Obermeyer and colleagues’ *Obermeyer19 analysis already exhibits the perverse outcomes resulting from optimisation of AI prediction based on health service cost as a goal function, and not population health outcomes. It is probable therefore, that the main goals of implementation will be to improve efficiency and productivity. Meanwhile, because of this focus on efficiency, issues such as bias, incontestability, and erosion of privacy may well go overlooked. And critically, these issues are mechanisms by which existing social, economic and health disparities are perpetuated and amplified, plausibly leading to a longer-term risk that population health inequities will be exacerbated.

Previous scholarly warnings regarding the risk that ‘precision’ health approaches have the potential to exacerbate health inequities lends credence to the existence of the risk to population health equity posed by narrow AI and automated decision systems. There have been strong warnings from within public health (albeit with little empirical evidence to date) that ‘precision medicine’, as well as emerging ‘precision public health’ and ‘precision prevention’ approaches (which employ narrow AI, automated decision systems and algorithmic risk prediction) have the potential to produce disparate impacts, amplify existing prejudices, and propagate health inequities Jasanoff17,Khoury16,Meagher17,Panch19AIOppRisks,Prainsack17,Prainsack19,TaylorRobinson18,Lancet19. Elucidating this, Lavigne and colleagues *Lavigne19 write:

…particularly when applying these approaches to decision-making or predictions at a population level, attention must be paid to the potential for these approaches to produce health inequities, either through the use of biased data or through uneven access to the technology. Predictions and models based on non-representative or biased data can propagate underlying biases and exacerbate health inequities at a population level if sufficient care is not taken to mitigate these issues. (p. 176)

Moreover, socioeconomic gradients in access to, as well as the means and resources to best utilise new precision health tools – for example genomic risk prediction – have the potential to widen inequalities further Meagher17,Prainsack19,Vayena15. And the focus on individual risk factors promoted by precision approaches can also reinforce the notion of individual responsibility, prolonging the use of individualist, behaviourist interventions, which tend to entrench and exacerbate socioeconomic disparities in health outcomes Baum14,Dolley18,Meagher17,Prainsack19,StephensonForthcoming,TaylorRobinson18. The focus on individual risk factors also undermines the rationale and societal propensity to act on structural, upstream determinants of health inequities, thereby permitting inequities in population health to persist Chowkwanyun18,Khoury16,Meagher17,Panch19AIOppRisks. Meagher and colleagues succinctly capture this idea in relation to genomic data: “genomic explanations for health disparities can distract and even exculpate society from taking responsibility for the structural determinants of those inequities, undermining the political momentum of those seeking justice” (p. 11).

5 Conclusion

Amid unprecedented global interest in artificial intelligence, there are tremendous expectations that AI will transform medicine and public health practice. And while it may go largely unremarked, narrow AI and predictive algorithms are already ubiquitous; woven into the fabric of our daily lives. Decisions which are being made automatically about disease detection, diagnosis, treatment and funding allocation have significant consequences for individual and population health and wellbeing.

The evidence collated in this review makes it clear that issues have emerged in sensitive domains like criminal justice where narrow AI and automated decision systems are already in common use. As their use in health rapidly expands, it is probable that the same issues – bias, incontestability, and erosion of privacy – will also manifest widely in medicine and public health applications. Reports of this happening are already appearing. Moreover, the combination of hype, the drive to adopt automated decision systems to address cost pressures, and the commercial imperative to capitalise on health data assets, may conspire to obscure issues - as has occurred in other domains.

Crucially, bias, incontestability, and erosion of privacy are mechanisms by which existing social, economic and health disparities are perpetuated and amplified by automated decision systems. Therefore there is a significant risk that the use of automated decision systems in health will exacerbate existing population health inequities and potentially create new ones. Medical and public health interventions have obviously produced disparate outcomes in the past; what makes the risk with narrow AI and automated decision systems different is the industrial scale and rapidity with which they can be applied to whole populations, combined with the incontestability of decisions. This means negative consequences can quickly escalate.

While it is too soon to say whether the issues emerging in health applications of narrow AI and automated decision systems have actually led to worsened population health inequity, it is incumbent on health practitioners and policy makers to explore and be mindful of the potential implications of using automated decision systems, so as to ensure the use of AI promotes population health and equity. There is a need to design and implement automated decision systems with care, monitor their impact over time (especially longer-term effects on population health), and take responsibility for responding to issues as they emerge – even if this is long after a system has first been introduced. To finish, Obermeyer and colleagues *Obermeyer19 set a very positive example. After uncovering inadvertent racial bias in an automated decision system allocating health assistance funding, they approached the algorithm manufacturer, who was able to independent replicate the results to confirm the existence of bias. The researchers and the algorithm manufacturer are now collaborating on developing solutions to address this bias. This is a fine example to emulate.