siddvohHearsay

Hearsay

Vision-language medical diagnoses without an image.

Asked to describe a medical image that was never attached, frontier vision-language models can return a diagnosis anyway. Which diagnosis is not random: it is structured by who the patient is said to be.

Read the paper The extended study

The paper · TrustVLM Workshop · ACM ICMR 2026

Hearsay: Vision-Language Medical Diagnoses Without an Image

Peer-reviewed and presented at the TrustVLM workshop at ACM ICMR 2026 in Amsterdam. The workshop is non-archival, so the arXiv preprint is the version to cite.

Audits frontier vision-language models under missing-image medical prompts and identifies structured diagnostic confabulation, demographic sensitivity, and structured-output failure modes.

This page is a plain-language walkthrough. The paper is the source to cite, and its reference and BibTeX are at the end of the page.

01

The paper's abstract

When asked to describe a medical image that was never attached, frontier vision-language models do not abstain: they confabulate a diagnosis. We show that this confabulation is not random. It is structured by who the patient is said to be.

Across chest X-ray, brain MRI, and dermatology, Claude Opus-4.7, GPT-5.4, and Gemini-3.1-Pro are each queried with only a demographic descriptor and no image, and changing the descriptor systematically shifts the diagnosis returned. Claude concentrates sharply: a 65-year-old white man asking about a skin mole receives Melanoma in nearly every response, and a 32-year-old Black woman asking about her chest X-ray receives a Sarcoidosis diagnosis whose reasoning reads “suspected, based on demographics and classic pattern.” GPT-5.4’s effect is broader, fabricating across every demographic cell we test, most conspicuously naming Sarcoidosis for young Black patients on chest X-ray.

Two structural findings sharpen the problem. A hedged regime appears in which the prose acknowledges the missing image while the structured diagnosis field nevertheless names a disease, a dissociation invisible to prose-only audits. And Claude’s dermatology effect collapses entirely when ‘skin mole’ is swapped for ‘skin lesion’ while GPT-5.4’s is preserved, indicating that mirage is a family of distinct failure modes rather than a single phenomenon.

Trustworthy VLM deployment in clinical pipelines requires auditing the structured output channel directly, and probe-word sensitivity should be treated as a first-class evaluation dimension.

02

Where it was reported

CMU Robotics Institute · 20 July 2026

Healthcare Blind Spots: AI Models Prone To Fabricating Diagnoses

Robotics Institute feature on the Hearsay study: asked to describe a medical image that was never attached, Claude, GPT-5 and Gemini fabricated a diagnosis in 18% of about 11,700 responses, shaped by the patient's stated age, gender and race.

The National · 27 July 2026

Patients warned off using AI chatbots for self-diagnosis as flaws revealed

Report on the same study: AI chatbots fabricated a diagnosis in 18 per cent of cases when the medical image was left out, and a diagnosis can change when only race, age or gender is written into the prompt.

The Business Journals · 30 September 2026

Companies grapple with how to manage workers and AI errors

Workplace feature that opens with the study and quotes Vohra on fixing the structure around an AI tool when it makes a mistake.

03

The study that followed

Hearsay was the first paper in this line of work. The extended study, Absent-Byte Diagnoses, tests five hosted models and one local model on the same failure, runs 288 direct comparisons across age, race and sex wording, and adds a client-side check that blocked every violation it was tested against.

Read the Absent Byte walkthrough, which shows the failure with an interactive example, the full grid of results, and the fix.

04

The paper

TrustVLM Workshop · ACM ICMR 2026

Hearsay: Vision-Language Medical Diagnoses Without an Image

Siddharth Vohra

1st Workshop on Toward Trustworthy Vision-Language Models in the Wild (TrustVLM) at ACM ICMR 2026 · Amsterdam, The Netherlands · June 2026

Peer-reviewed and presented at the TrustVLM workshop at ACM ICMR 2026 in Amsterdam. The workshop is non-archival, so the arXiv preprint is the version to cite.

Three models, three imaging domains, and the finding that started the project.

Read the paper

If this is useful in your work, feel free to cite it as: Vohra, S. (2026). Hearsay: Vision-Language Medical Diagnoses Without an Image. arXiv:2607.26886.

Cite this
Download BibTeX as .bib
@misc{vohra2026hearsay,
  title         = {Hearsay: Vision-Language Medical Diagnoses Without an Image},
  author        = {Vohra, Siddharth},
  year          = {2026},
  eprint        = {2607.26886},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  note          = {Presented at the 1st Workshop on Toward Trustworthy
                   Vision-Language Models in the Wild (TrustVLM), ACM ICMR 2026,
                   Amsterdam. Non-archival},
  url           = {https://arxiv.org/abs/2607.26886}
}

No patient data is involved anywhere in this work. The patient descriptions are made up and match no real person, and the diagnoses shown are things a model said, not medical advice. Work by Siddharth Vohra here is unrelated to his position at Amazon Web Services.