Professor James Zou and colleagues have published a perspective article in the journal Science, "Implications of predicting race variables from medical images."

Abstract from Science: There are now more than 500 US Food and Drug Administration (FDA)-approved medical artificial intelligence (AI) devices, and AI is used in diverse medical tasks such as assessing the risk of heart failure and diagnosing cancer from images. Beyond predicting standard diagnoses, AI models can infer a substantial number of patient features from medical images in ways that humans cannot. For example, several studies have demonstrated that some AI models can infer race variables (crude simplistic categories) directly from medical images such as chest x-rays and cardiac ultrasounds, even though there are no known human-readable race correlates in these images. This has raised concerns about the possibility of AI systems to discriminate. At the same time, AI predictions of demographic attributes, including race variables, also have the potential to help assess and monitor health care disparities and generate new insights into risk factors.

From Stanford Human-Centered Artificial Intelligence (HAI): "Hidden Racial Variables? How AI Inferences of Race in Medical Images Can Improve—or Worsen—Health Care Disparities.”