--- title: "Hearsay: Vision-Language Medical Diagnoses Without an Image" page: "Hearsay" url: https://siddvoh.com/hearsay/ authors: ["Siddharth Vohra"] status: presented published: 2026-07-29 updated: 2026-10-20 bibtex: https://siddvoh.com/bib/hearsay.bib license: "Page text: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). The paper keeps its own terms." --- # Hearsay: Vision-Language Medical Diagnoses Without an Image The paper · TrustVLM Workshop · ACM ICMR 2026 [Siddharth Vohra](https://siddvoh.com/) Peer-reviewed and presented at the TrustVLM workshop at ACM ICMR 2026 in Amsterdam. The workshop is non-archival, so the arXiv preprint is the version to cite. Audits frontier vision-language models under missing-image medical prompts and identifies structured diagnostic confabulation, demographic sensitivity, and structured-output failure modes. This is the markdown version of https://siddvoh.com/hearsay/, the "Hearsay" page: a plain-language walkthrough of the paper. The paper is the source to cite, and its reference and BibTeX are at the end. ## Hearsay Vision-language medical diagnoses without an image. Asked to describe a medical image that was never attached, frontier vision-language models can return a diagnosis anyway. Which diagnosis is not random: it is structured by who the patient is said to be. [Read the paper](https://arxiv.org/abs/2607.26886) [The extended study](https://siddvoh.com/absent-byte/) ## The paper's abstract When asked to describe a medical image that was never attached, frontier vision-language models do not abstain: they confabulate a diagnosis. We show that this confabulation is not random. It is structured by who the patient is said to be. Across chest X-ray, brain MRI, and dermatology, Claude Opus-4.7, GPT-5.4, and Gemini-3.1-Pro are each queried with only a demographic descriptor and no image, and changing the descriptor systematically shifts the diagnosis returned. Claude concentrates sharply: a 65-year-old white man asking about a skin mole receives Melanoma in nearly every response, and a 32-year-old Black woman asking about her chest X-ray receives a Sarcoidosis diagnosis whose reasoning reads “suspected, based on demographics and classic pattern.” GPT-5.4’s effect is broader, fabricating across every demographic cell we test, most conspicuously naming Sarcoidosis for young Black patients on chest X-ray. Two structural findings sharpen the problem. A hedged regime appears in which the prose acknowledges the missing image while the structured diagnosis field nevertheless names a disease, a dissociation invisible to prose-only audits. And Claude’s dermatology effect collapses entirely when ‘skin mole’ is swapped for ‘skin lesion’ while GPT-5.4’s is preserved, indicating that mirage is a family of distinct failure modes rather than a single phenomenon. Trustworthy VLM deployment in clinical pipelines requires auditing the structured output channel directly, and probe-word sensitivity should be treated as a first-class evaluation dimension. ## Where it was reported CMU Robotics Institute · 20 July 2026 Healthcare Blind Spots: AI Models Prone To Fabricating Diagnoses Robotics Institute feature on the Hearsay study: asked to describe a medical image that was never attached, Claude, GPT-5 and Gemini fabricated a diagnosis in 18% of about 11,700 responses, shaped by the patient's stated age, gender and race. Link: https://www.ri.cmu.edu/healthcare-blind-spots-ai-models-prone-to-fabricating-diagnoses/ The National · 27 July 2026 Patients warned off using AI chatbots for self-diagnosis as flaws revealed Report on the same study: AI chatbots fabricated a diagnosis in 18 per cent of cases when the medical image was left out, and a diagnosis can change when only race, age or gender is written into the prompt. Link: https://www.thenationalnews.com/news/uae/2026/07/27/patients-warned-off-using-ai-chatbots-for-self-diagnosis-as-flaws-revealed/ The Business Journals · 30 September 2026 Companies grapple with how to manage workers and AI errors Workplace feature that opens with the study and quotes Vohra on fixing the structure around an AI tool when it makes a mistake. Link: https://www.bizjournals.com/bizjournals/news/2026/09/30/ai-tools-tips-managers.html ## The study that followed Hearsay was the first paper in this line of work. The extended study, Absent-Byte Diagnoses, tests five hosted models and one local model on the same failure, runs 288 direct comparisons across age, race and sex wording, and adds a client-side check that blocked every violation it was tested against. [Read the Absent Byte walkthrough](https://siddvoh.com/absent-byte/), which shows the failure with an interactive example, the full grid of results, and the fix. ## The paper TrustVLM Workshop · ACM ICMR 2026 ### Hearsay: Vision-Language Medical Diagnoses Without an Image Siddharth Vohra 1st Workshop on Toward Trustworthy Vision-Language Models in the Wild (TrustVLM) at ACM ICMR 2026 · Amsterdam, The Netherlands · June 2026 Peer-reviewed and presented at the TrustVLM workshop at ACM ICMR 2026 in Amsterdam. The workshop is non-archival, so the arXiv preprint is the version to cite. Three models, three imaging domains, and the finding that started the project. [Read the paper](https://arxiv.org/abs/2607.26886) If this is useful in your work, feel free to cite it as: Vohra, S. (2026). Hearsay: Vision-Language Medical Diagnoses Without an Image. arXiv:2607.26886. **Cite this** [Download BibTeX as .bib](https://siddvoh.com/bib/hearsay.bib) ``` @misc{vohra2026hearsay, title = {Hearsay: Vision-Language Medical Diagnoses Without an Image}, author = {Vohra, Siddharth}, year = {2026}, eprint = {2607.26886}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, note = {Presented at the 1st Workshop on Toward Trustworthy Vision-Language Models in the Wild (TrustVLM), ACM ICMR 2026, Amsterdam. Non-archival}, url = {https://arxiv.org/abs/2607.26886} } ``` No patient data is involved anywhere in this work. The patient descriptions are made up and match no real person, and the diagnoses shown are things a model said, not medical advice. Work by Siddharth Vohra here is unrelated to his position at Amazon Web Services.