Welcome to the third of an ongoing series of roundtable discussions among Chartis consulting leaders around the emerging reality of artificial intelligence (AI) in healthcare.
Concern is growing around how leveraging AI could not only entrench but amplify inequities in healthcare. Studies have highlighted how bias—often present in medical and demographic data and then the algorithms and models that use this data to inform diagnoses and treatments—frequently results in inequitable and unfavorable outcomes for historically underrepresented populations. This roundtable focuses on what’s at stake for health equity as the healthcare industry further adopts AI.
Join Tom Kiesau, Chartis Chief Innovation Officer and Head of Chartis Digital; Andrew Resnick, MD, Chartis Chief Medical and Quality Officer; Duane Reynolds, Chartis Chief Health Equity Officer; Julie Massey, MD, Chartis Clinical IT Practice Leader; and Jon Freedman, Partner in Chartis Digital, as they discuss AI, what Chartis is seeing in real time, and what they think is coming next.
Tom Kiesau: Welcome, everyone. The potential for AI to negatively impact health equity is becoming increasingly clear. What are some of the greatest risks of AI to exacerbate health inequities?
JULIE MASSEY, MD:
One of the biggest challenges is the data that is used to train the AI algorithms. There are challenges around the quality of existing data—which goes back to problems with data governance and how the information was captured. But we also know there are biases on which we are training our algorithms that will perpetuate or even worsen some existing disparities.
A striking case in point is a recent effort to better identify sepsis in children. The developers were meticulous about creating an algorithm that would flag sepsis equally well across races and ethnicities. But then they discovered that the data was biased because of widespread delayed blood testing practices that inaccurately implied that Hispanic children develop sepsis more slowly.
ANDREW RESNICK, MD:
An even simpler example is the glomerular filtration rate (GFR) calculation issue that became well known in the last few years—the racial bias was unrecognized for decades. This formula to calculate kidney function is incredibly straightforward and basic in comparison to AI algorithms. But it hurt millions of Black patients, preventing them from getting transplants and other needed care. It’s incredible how harmful AI could be without understanding all the data sources.
JON FREEDMAN:
Despite good intentions and intentionality, there’s still a lack of understanding around what biased data looks like and how to identify it. Without the right people and tools, many organizations don’t know what to look for.
JULIE:
For AI in use today, the clinicians don’t have transparency about how or why an algorithm came to the recommendation it is making. There’s a black box. And given the high potential for systemic bias, that’s a major problem.
Tom: Unlike troubleshooting code, it’s not a straightforward path to identify if there is a bias, or where the bias in AI is coming from. What’s so dangerous is we know bias in healthcare delivery exists, but we have a limited ability to explore it. That said, what are some ways organizations can mitigate these risks?
JON:
First, don’t oversimplify it. The solution isn’t as simple as just removing all racial and ethnic data from the data sets. A recent study published in JAMA about risk prediction in colorectal cancer recurrence suggests that this approach may actually result in worse outcomes for the very populations that these data adjustments were trying to protect.
DUANE REYNOLDS:
Collecting demographic data isn’t bad in and of itself. It’s the social constructs around it that lead to disparate outcomes. In fact, true representation of different demographics is critical. Certain diseases manifest themselves differently in different populations. You have to have the right data so gaps aren’t dictating bias.
If you’re not looking at things like race, ethnicity, sexual orientation, socioeconomic status, and geography, for instance, you may be marginalizing patients who aren’t in the majority. You may need to partner with other organizations to help ensure you have representative data of patients in subpopulations.
ANDREW:
At the same time, you shouldn’t depend on limited demographic data in constructing algorithms for care. The demographics alone of even very large studies can lead to very false conclusions and imply incorrect insights about discrete demographic groups. People are much more complex than the demographic buckets we put them in. You can’t assume certain demographics have a propensity for certain things—that’s racialized medicine. And demographics should not be confused with genetics, which are an accurate basis of care—that’s personalized medicine.
But you do need to collect demographic data so you can continually evaluate whether you are providing equitable care and ensure you’re not designing poor systems that result in inequitable care.
Tom: If you’re only looking at the observable characteristics of patients, that’s the tip of the iceberg, and you’re missing all the underlying complexity. Health systems will need a constantly growing data set that includes things like genomics and behavioral data—things that are not part of the current data constructs.
What other mitigating factors do leaders need to consider?
DUANE:
It’s imperative to have clinicians, data managers, patients, and others at the table who represent different lenses and lived experience. These individuals can help you interpret and understand where disparate impact may occur as you’re creating, testing, and analyzing these algorithms.
Part of the reason the issue with GFR became so well known is because Naomi Nkinsi, a Black female medical student, started questioning the diagnostic tool and pushing back. She was looking at it from a very different perspective and asking herself, “Why would there be a different calculation if someone is Black?” Representation matters. If that didn't happen, we wouldn’t be changing the GFR calculation at a national scale now.
JON:
Transparency is another important factor to ensure you’re not building bias into your AI. How is the model constructed? What are the inputs? Invite open challenging of your model. Undergo peer review processes and create transparency in laypeople’s terms so the general public understands what’s going into your model and how it arrives at various conclusions. Establish policy around every stage of the AI “life cycle.”
Tom: Finally, let’s talk about the ways AI might be able to proactively improve health equity.
ANDREW:
Near-term opportunities include increased health literacy and other patient-facing communications. Generative AI can quickly change the reading level and even the dialect of text, making communications more accessible and effective. Using AI in two-way communications with patients also could enable greater understanding of how patients are doing and improve outcomes related to post-discharge care, for instance.
DUANE:
If you create algorithms that detect clinical variants, layered with demographic information, you could potentially discern how certain decisions might be made in a biased way. Those decisions lead to downstream care pathways that ultimately impact outcomes for specific populations. Identifying some of those clinical variants can help solve some of the disparity issues.
ANDREW:
With AI, we have a much greater ability now to predict who needs more healthcare-related resources and support, whether because of social determinants of health, biological characteristics, or other contributors. This gives us an amazing power as healthcare leaders and society to give support to the people we know have the greatest needs—because everyone deserves the same opportunities to achieve optimal health.
JON:
There’s also an opportunity to improve access, particularly for people who live in remote areas without access to necessary care. AI has the potential to help speed access to diagnosis and treatment.
