Encouraging Flu Vaccination Among High-Risk Patients Identified by ML
Encouraging Flu Vaccination Among High-Risk Patients Identified by ML
The purpose of the current study is to test different interventions to determine the most
 effective way to promote flu vaccine uptake in a high-risk population identified by an
 "artificial intelligence" (AI) or machine learning (ML) algorithm. The specific aims are:

 1. Evaluate the effect on flu vaccination rates of informing health-system patients who are
 identified by an ML analysis of EHR data to be at high risk for flu complications that
 they are at high risk with either (a) no additional explanation, (b) an explanation that
 this determination comes from an analysis of their medical records, and (c) the
 additional explanation that an AI or ML algorithm made this determination.

 2. Evaluate the effects of the same three interventions on diagnoses of flu in the same
 patients.
Background

 On average, 8% of the US population gets sick from flu each flu season (Tokars et al. 2018).
 Since 2010, the annual disease burden of influenza has included 9-45 million illnesses,
 140,000-810,000 hospitalizations, and 12,000-61,000 deaths (CDC 2020). The CDC recommends the
 flu vaccination to everyone aged 6+ months, with rare exception; almost anyone can benefit
 from the vaccine, which can reduce illnesses, missed work, hospitalizations, and death (CDC
 2019a). Flu vaccination will be especially important for high-risk patients during the
 COVID-19 pandemic so that flu cases are reduced and resources conserved.

 While most recover from influenza without treatment, the elderly, those with comorbidities,
 and other high-risk individuals can experience complications such as pneumonia, other
 respiratory illness, and death. Geisinger, a large health system in Pennsylvania and New
 Jersey, has partnered with Medial EarlySign (Medial; www.earlysign.com) to develop a machine
 learning (ML) algorithm to identify patients at risk for serious (moderate to severe)
 flu-associated complications on the basis of their existing electronic health record (EHR)
 data. Geisinger will deploy this system during the 2020-21 flu season and contact the
 identified patients with special messages (in addition to standard efforts made by the health
 system every flu season) to encourage vaccination. Flu vaccination will be especially
 important for high-risk patients during the COVID-19 pandemic so that flu cases are reduced
 and resources conserved.

 Published results suggest Medial's ML systems identify high-risk patients in other contexts
 (Goshen et al., 2018; Zack et al., 2019). However, there is little evidence about (a) whether
 informing patients they are at high risk makes them more likely to receive vaccination; (b)
 how patients react to being told their risk status is the result of an analysis of their
 health records; and (c) whether informing patients their risk status has been determined by
 an "algorithm," by "machine learning," and/or by "artificial intelligence" will increase or
 decrease their likelihood of getting vaccinated. This study will address these gaps in the
 literature, which are especially important in light of the anticipated future growth of AI/ML
 system use throughout healthcare.

 Medial's algorithm is an example of how interoperable health information exchange (HIE)-the
 ability for health information technology to share patient data-can improve the efficiency
 and effectiveness of healthcare. However, patients may not appreciate these benefits or the
 fact that healthcare has become substantially more integrated and collaborative. A systematic
 review of patient privacy concerns about HIE found that 15-74% of patients expressed privacy
 concerns, depending on the study, and concluded that patient perspectives remain poorly
 understood. A flu outreach message that explicitly references a review of patient medical
 records might backfire as patients react badly to a sense they have lost control of their
 health records.

 There is conflicting evidence on how people respond to advice or information that comes from
 an algorithm or machine. Dietvorst et al. (2015) documented a pattern of "algorithm
 aversion," in which people choose inferior human over superior algorithmic forecasts,
 especially after they observed the algorithm make an error. In contrast, Logg et al. (2018)
 described "algorithm appreciation," in which people followed advice more when they thought it
 came from algorithms than when they thought it came from human beings. Finally, Bigman and
 Gray (2019) found aversion to algorithms that make "moral decisions," including a
 (fictitious) medical decision of choosing whether or not to operate on a high-risk patient.
 In the current setting, the algorithm is merely advising patients on taking an action (an
 annual flu shot) that is already the standard of care, and there is no opportunity to observe
 an erroneous recommendation, so the hypothesis is that "algorithm appreciation" will cause
 people to react positively to being informed of the algorithm's role. Thus, this study will
 address two important research questions:

 1. Does informing patients that they are at high risk for flu complications (a) increase
 the likelihood that they will receive flu vaccine; and (b) decrease the likelihood that
 they receive diagnoses of flu and/or flu-like symptoms in the ensuing flu season?

 2. Does informing patients that their high-risk status was determined (a) by analyzing
 their medical records (vs. by no specified method); and (b) by an AI/ML algorithm*
 analyzing their medical records (as opposed to via unspecified methods or human medical
 records analysis) affect the likelihood that they receive the flu vaccine and/or
 diagnoses of flu and/or flu-like symptoms in the ensuing flu season?

 Our specific aims are:

 1. Evaluate the effect on flu vaccination rates of informing health-system patients who are
 identified by an ML analysis of EHR data to be at high-risk for flu complications that
 they are at high risk with either (a) no additional explanation, (b) an explanation that
 this determination comes from an analysis of their medical records, and (c) the
 additional explanation that an AI or ML algorithm made this determination.

 2. Evaluate the effects of the same three interventions on diagnoses of flu in the same
 patients.

 Research Strategy

 Included in the study will be current Geisinger patients 17+ years of age with one or more
 visits to a Geisinger primary care physician (PCP) between January 1, 2008 and January 30,
 2020 and no contraindications for flu vaccine. Medial will provide flu-complication risk
 scores from their ML algorithm (based on coded EHR data), on the basis of which the top 10%
 of patients at highest risk will be included. Based on prior behavior and other predictors in
 a second ML model, Medial will also provide the likelihood each patient will get vaccinated
 during the study flu season; these values and the primary risk scores will be used as
 covariates in exploratory data analyses. The anticipated number of patients in the top 10% of
 risk is 56,000.

 On average in the last 3 flu seasons, 55% of Geisinger patients aged 65+ are vaccinated each
 season, so we will use this as a proxy base rate for a control condition in our power
 analysis. The study will have 92% power to detect a 2% absolute difference or greater in the
 vaccination outcome between conditions (55% vs 57%, two-tailed alpha of .05), on the
 assumption that each condition will have 56,000/4=14,000 patients. For the rarer outcome of
 flu diagnosis, we have 95% power to detect a 0.8% absolute difference or greater-from an
 estimated 3.9% rate in this high-risk population (based on the CDC estimate for people age
 65+ [Tokars et al., 2018]) to a 3.1% rate.

 The primary study outcomes will be the rates of flu vaccination and flu diagnoses during the
 2020-21 season (September-March) by targeted patients. Secondary, exploratory outcomes will
 also be measured: Rates of flu vaccination and diagnoses by fellow household members of
 targeted patients; rates of flu vaccination and diagnoses by non-targeted patients who were
 assigned a risk score that fell just below the cutoff of targeted patients ("sub-threshold
 risk"); rates of flu complications and flu-like symptoms among targeted patients, household
 members, and those at sub-threshold risk; and rates of other relevant healthcare utilization
 outcomes such as ER visits and hospitalizations.

 Generalized linear mixed models (GLMMs) will examine the primary study outcomes as a function
 of the study arms (between-subjects), with patient-visited PCPs and/or clinics included as
 random effects variables, assuming high intraclass correlation coefficients. GLMMs will
 specify a binary distribution and log-link function in the case of dichotomous outcome
 variables (e.g., flu vaccination, flu diagnosis), and a negative binomial distribution and
 log-link function in the case of any highly positively skewed count variables such as ER
 visits and hospitalizations (where over-dispersion typically remains in the case of a Poisson
 distribution model). For these exploratory analyses, within-patient change (from the same
 period one year earlier) will also be analyzed. Also, each patient will receive the same type
 of communication (a/b/c/d) via up to three modalities-printed letter to their mailing
 address, SMS to their mobile phone, and/or secure message via Geisinger's patient
 portal-depending on what information is on file for each patient. The communication channels
 used for each patient will be covariates in later analyses.

 *Note: The study will not necessarily use the terms "AI," "ML," or "algorithm" in the
 messages to groups b, c, and d; instead, these messages will be designed to be readable and
 comprehensible by the patient audience while still including the key concepts that
 differentiate the interventions from one another.
Inclusion Criteria:

 -  Current Geisinger patient at the time of study

 -  Falls in the top 10% of patients at highest risk, as identified by the
 flu-complication risk scores of Medial's machine learning algorithm (which operates on
 coded EHR data)

 -  May limit inclusion to patients that are under Geisinger primary care, depending on
 algorithm performance of patients who have non-Geisinger PCPs

 Exclusion Criteria:

 -  Has contraindications for flu vaccination

 -  Has opted out of receiving communications from Geisinger via all of the modalities
 being tested