Mutual Colormap Dependence: How Recoloring Biomedical Images Changes Vision-Language Model Answers

Vohra, Siddharth; Jiang, Runmin; Xu, Min

Mutual Colormap Dependence

Show a vision-language model the same medical image in another colormap, and it can change its answer.

A model never sees the intensities a scan measured, only the colors a colormap gave them. We showed four frontier models the same images in six colormaps, told them to ignore the color, and asked again about the plain gray image as a control. On chest X-rays, all four changed their answers with the colormap.

Read the paper Code and data

The radiograph in the gray colormap — One radiograph labeled positive · Claude Sonnet 4.6

The radiograph in the viridis colormap — One radiograph labeled positive · Claude Sonnet 4.6

The paper · FABRIC Workshop · WACV 2027

Mutual Colormap Dependence: How Recoloring Biomedical Images Changes Vision-Language Model Answers

Siddharth Vohra, Runmin Jiang, Min Xu

Accepted to the FABRIC workshop at WACV 2027, to be presented in January 2027 in Orlando, Florida.

Four vision-language models answered questions about biomedical images shown in six colormaps, each told to ignore the color. On 64 chest X-rays, all four changed their answers with the colormap. For Claude, a colored map overturned the majority answer on 29 radiographs, where asking again in gray overturned about 4.

Abstract

Vision-language models (VLMs) are being built to answer questions about biomedical images, yet they never see the measured intensities, only the colors that a colormap has assigned to them. For questions about image content, the colormap should not change the answer. We call the extent to which it does Mutual Colormap Dependence (MCD), the mutual information between the colormap and a model's answer, which is zero exactly when every colormap yields the same answer distribution. Because VLMs sample their answers, we compare recoloring with a control that uses the same number of calls to ask again about the unchanged gray image, so that ordinary resampling is not mistaken for an effect of the colormap. We test GPT-5.4, Gemini 3.1 Pro, Claude Sonnet 4.6, and Qwen3-VL-235B under six luminance-monotonic colormaps (gray, viridis, plasma, inferno, magma, and cividis) on 50 medical visual-question-answering items from 16 SLAKE images, 64 ChestX-ray14 chest radiographs, and 200 BBBC039 fluorescence-microscopy images for exact nucleus counting. Every prompt tells the model to ignore the colormap. Even so, with three answers per colormap on the chest radiographs, the answers of all four models depend on the colormap (all Holm-adjusted p ≤ 0.002). For Claude, some colored map overturns the majority answer on 29 of the 64 radiographs, where resampling gray alone overturns about 4, and Qwen's accuracy falls from 0.54 under gray to between 0.32 and 0.41 under the colored maps, mostly because it abstains. With one answer per colormap, recoloring adds disagreement beyond repeated sampling in 5 of 12 model-panel cells under per-cell 95% intervals, and repeating the instruction to ignore color does not reliably reduce it. The colormap is therefore part of what a VLM sees, and it can change the answer.

This page is a plain-language walkthrough. The paper is the source to cite, and its reference and BibTeX are at the end of the page.

Same image, six colormaps

A chest X-ray or a micrograph starts as a grid of measured intensities. A colormap turns those numbers into colors, and the colors are all a model receives. Plot a radiograph with Matplotlib’s defaults, paste it into a chatbot, and it arrives in viridis.

We rendered each image in gray, viridis, plasma, inferno, magma and cividis. All six get brighter as intensity rises and are the perceptually ordered kind that visualization research recommends, so none adds the false edges of a rainbow map. The questions asked about anatomy, structure or a count, never about color, so the right answer is the same in every rendering.

Four models answered: GPT-5.4, Gemini 3.1 Pro, Claude Sonnet 4.6 and the open-weight Qwen3-VL-235B. Every prompt told them to ignore the colormap.

From every medical prompt

Ignore the colormap or hue; only anatomy, structure, texture, and relative intensity patterns matter.

Three public datasets

Reference answers come from each dataset’s own release.

SLAKE: 50 closed questions about 16 CT, MRI and X-ray images.
ChestX-ray14: 64 radiographs, each with one yes-or-no question about a finding, scored against the dataset’s labels, which were extracted by natural language processing.
BBBC039: 200 fluorescence micrographs. The model had to give the exact number of nuclei.

frontier models from four developers.

colormaps, all brightening as intensity rises.

20,880

answers, every one given under the instruction to ignore the colormap.

Asking again is the control

These models sample their answers. Ask the same question twice about the same image and you can get two answers, so a different answer under magma might be nothing more than a second draw.

So recoloring gets a twin. Beside one answer under each of the six colormaps, we ask six times about the unchanged gray image. Both arms use the same number of calls, and recoloring counts only when it adds disagreement beyond what asking again already produces.

The measure is Mutual Colormap Dependence, the mutual information between the colormap and the answer. It is zero exactly when every colormap gives the same spread of answers, it counts a switch between two wrong answers or from answering to declining, and it needs no labels.

The control matters most for exact nucleus counts. GPT and Qwen give mostly different counts for six repeats of the identical gray image, so without the control these would look like the most colormap-sensitive results. With it, recoloring adds no detectable disagreement for either.

On chest X-rays, every model depends on the colormap

On the 64 chest X-rays we asked three times under each colormap. A permutation test finds that all four models’ answers depend on the colormap, each at a Holm-adjusted p of 0.002 or less.

Claude shows it most cleanly. Under any one colormap its answers barely vary, yet they change from one colormap to the next. The radiograph at the top of this page is one of 20 where Claude gives a fixed answer under each colormap that differs across colormaps, and on all 20 its six gray repeats agree.

Radiographs, of 64, where a colored map overturned the gray majority

Recoloring Asking about gray again, 95% range

Claude Sonnet 4.6

29 against 2 to 6, 3.9 on average

GPT-5.4

16 against 2 to 7, 4.6 on average

Gemini 3.1 Pro

14 against 7 to 15

The range comes from drawing the six majorities from the nine gray answers of each radiograph instead, with replacement, 2,000 times.

Many of these changes overturn an answer that agreed with the label. For Claude, 17 of the 29 are among the 37 radiographs where its gray majority matches the label. For GPT, 9 of the 45. Asking again would overturn about 3 of those for either model. Gemini’s 14 sit inside the 7 to 15 that asking again produces, yet the three-answer test still detects its dependence.

Each model fails its own way

Qwen3-VL-235B on 64 chest X-rays, three answers under each colormap

Correct

Declined

gray 54% 8%

viridis 38% 34%

plasma 32% 38%

inferno 36% 31%

magma 37% 28%

cividis 41% 27%

Under gray, Qwen answers correctly 54% of the time and declines 8%. Under every colored map it declines between 27% and 38% of the time, and its accuracy falls to between 32% and 41%.

Most of Qwen’s changes on chest X-rays are declines to answer. Averaged over the five colored maps, its rate of declining on chest X-rays rises 23.6 percentage points over gray, and 83 of its 91 changed majorities are abstentions. Its accuracy falls 16.8 points on chest X-rays and 10.8 on SLAKE. Declining is not a wrong diagnosis, but the display decides whether Qwen answers at all.

On chest X-rays, GPT keeps answering but says yes 13.0 points less often under color. Claude rarely declines and answers more consistently under color, so its changes there are shifts between consistent answers. Gemini becomes noisier under color.

Saying it twice does not fix it

In a separate arm, the prompt kept its instruction and added one more sentence.

The added reminder

The image may be pseudocolored arbitrarily; treat the chosen colormap as presentation only and not as evidence.

It changed disagreement detectably in two of eight cells, in opposite directions: down for Claude on SLAKE, up for GPT on SLAKE. Of the eight accuracy changes, only Qwen’s on chest X-rays was detectable, a rise of 5.5 points.

Nor does it look like a split between gray and color. Leaving out gray, the only colorless map, changes disagreement no more detectably than leaving out a colored map in ten of twelve model and dataset cells.

What to take from it

Report the colormap

It belongs in the description of any VLM evaluation on scalar images, next to windowing and resolution.

Control for resampling

A perturbation test on a model that samples its answers needs a resampling control with the same number of calls.

Do not count on the instruction

Telling the model to ignore color did not prevent the dependence, and repeating it did not reliably reduce it.

MCD measures behavior. It shows that the answers depend on the display, not why, and it does not by itself measure clinical harm, which would need expert-adjudicated labels.

The paper

FABRIC Workshop · WACV 2027

Mutual Colormap Dependence: How Recoloring Biomedical Images Changes Vision-Language Model Answers

Siddharth Vohra, Runmin Jiang, Min Xu

Workshop on Foundation AI for Biomedical Reasoning, Imaging, and Cognition (FABRIC) at WACV 2027 · Orlando, Florida, USA · January 2027

Accepted to the FABRIC workshop at WACV 2027, to be presented in January 2027 in Orlando, Florida.

The full method, the matched screen in all twelve model and dataset cells, and the error-signal analysis.

Read the paper

If this is useful in your work, feel free to cite it as: Vohra, S., Jiang, R., and Xu, M. (2027). Mutual Colormap Dependence: How Recoloring Biomedical Images Changes Vision-Language Model Answers. Workshop on Foundation AI for Biomedical Reasoning, Imaging, and Cognition (FABRIC) at WACV 2027 (to appear).

Cite this

.bib

@inproceedings{vohra2027colormap,
  title     = {Mutual Colormap Dependence: How Recoloring Biomedical Images
               Changes Vision-Language Model Answers},
  author    = {Vohra, Siddharth and Jiang, Runmin and Xu, Min},
  booktitle = {Workshop on Foundation AI for Biomedical Reasoning, Imaging, and
               Cognition (FABRIC) at WACV 2027},
  year      = {2027},
  address   = {Orlando, Florida, USA},
  note      = {To appear},
  url       = {https://siddvoh.com/mutual-colormap-dependence/}
}

Code and data on GitHub All 20,880 answers, the prompts and settings, and the analysis that rebuilds every result above.

The radiograph shown comes from ChestX-ray14, released by the NIH Clinical Center at https://nihcc.app.box.com/v/ChestXray-NIHCC (Wang et al., CVPR 2017). Every result on this page is in the paper and rebuilds from the released answers. Model answers were collected for research and are not medical advice. Work by Siddharth Vohra here is unrelated to his position at Amazon Web Services.