Research Agents Feed Doubts, Not Beliefs: Auditing Belief Framing in Live-Web Search

We started with 48 health claims that experts had already fact-checked. Sixteen are settled as false, sixteen are settled as true, and sixteen are contested, which means the evidence is mixed or too thin to call.

Every claim went to four AI agents: Claude Sonnet 5, GPT-5.6 Terra, Gemini 3.6 Flash and the open-weights GLM-5.2. Each agent had one tool, a search box. It had to complete between four and eight searches, and it was told to keep its report to 250 words or fewer and to cite its sources.

We asked about each claim three ways. The neutral request gives the claim and states no belief. The believer request says the user thinks the claim is true. The skeptic request says the user thinks it is false. The believer and skeptic requests differ by a single word. The neutral request also ends differently: it asks for a conclusion with sources, where the other two ask for the evidence behind the user’s answer. The claim itself is identical in all three.

The search box ran real web searches, with one safeguard. Every search went through a cache, so the same search came back with the same eight results however many times it was run. That matters more than it sounds. Without it, two runs could differ only because the search engine’s results had changed in between, and that would look exactly like an effect of how the question was asked.

The three groups of claims were assigned by two independent model-assisted screens, which we do not present as human annotation. They agreed on 42 of the 48 claims, and the six disagreements were settled against sources before any agent ran.

Every run we analysed ends in a written report. To compare more than a thousand of them, each report has to become a number.

An AI model that we call the judge reads the claim and the report and gives the report’s conclusion one of five labels. The judge is told to label only what the report concludes, not whether it agrees.

The number we care about is the change. Take one agent and one claim, and compare the scores of its reports under two different requests. The comparison fixed in advance as the main one sets the believer request against the skeptic request. Comparing each of them with the neutral request, as in the example above, is a secondary breakdown, and it is the one that shows which side moves. If the stated belief made no difference, all of these changes would average out to zero.

The judge is Gemini 3.6 Flash, the same model as one of the four agents. As a check, a model from a different company than the judge, GPT-5.6 Luna, labelled a fixed sample of 298 reports, and the two agreed on 89.9 percent of them. Agreement was highest, at 96.0 percent, on the reports the Gemini agent wrote, which is the opposite of what a shared blind spot would produce.

Start with the main comparison, believer against skeptic. Pooled across the four agents and the 48 claims, the same agent’s score differs by 0.26 points between the two requests, with a 95% interval of 0.15 to 0.40 from 307 pairs of reports. The interval is the range the true average plausibly falls in. In 54 pairs the score moved the user’s way, and in 6 it moved against. To see which side does the moving, compare each request with the neutral one. If these agents simply agreed with whoever was asking, both stated beliefs would move the score. Only one moves it by an amount we can detect.

When the user says they believe the claim, the score moves by two hundredths of a point on average, and the 95% interval around that sits across zero. That is no shift we can detect. The reports that did change split almost evenly: 26 moved toward endorsing and 29 toward rejecting.

When the user says they doubt the claim, the score falls by 0.26 on average. Of the 68 reports that changed, 60 moved toward rejecting. All four agents moved that way, though for one of them the interval touches zero.

Most of this is a nudge and not a reversal. A fall of 0.26 is about a quarter of the distance between two neighbouring labels. A reversal would be the same agent landing on opposite sides of the scale for the believer and for the skeptic on the same claim, one endorsing and the other rejecting, with each side taken from the average of its runs. That happened in 4 of the 167 agent and claim combinations that could be compared, which is 2.4 percent, against none in 133 when the neutral request was simply repeated. The planned paired test did not find that difference significant. So the finding is the smaller one. Doubt shifts how far the agent leans, and it rarely turns an endorsement into a rejection.

Counting report by report in the released runs gives the same picture. Of the 60 reports that moved toward rejecting under the skeptic request, 37 moved by a single label. Sorted by where they ended up, 22 stayed on the same side and changed only in strength, 11 went from an endorsing label to hedged, 20 went from hedged to a rejecting label, and 7 crossed from an endorsing label to a rejecting one.

Could this be chance? We also ran the neutral request twice. Across the three agents this baseline covers, the second report got a different score in 13.5 percent of pairs, so some movement happens with no stated belief at all. But those changes went both ways and averaged minus 0.04, close to zero. What a stated doubt adds is a direction. The repeat for the fourth agent was added after the first round of review and is reported separately in the paper.

The shift is not spread evenly across the three groups of claims. This chart takes the gap between the believer request and the skeptic request and splits it by group.

On contested claims the gap is roughly six times what it is on the settled ones. One reading is that this is fine: where the evidence is mixed, a hedged answer is the appropriate one.

The standard we hold the agents to is narrower than being right. Whatever the fair summary of mixed evidence is, it should not depend on what the person asking happens to believe. The other reading is that the agent is leaning with the user, and the neutral repeats are what separate the two. Plain hedging has no way to know which way the user leans, so its changes would go both ways, as the neutral repeats do. Under a stated doubt they do not: 60 of the 68 reports that changed moved toward rejecting.

On settled claims, a report that contradicts the verified answer outright is rare (the example at the top of this page is one of the exceptions), and this study cannot establish that a stated belief makes it more or less common. The score on claims settled as true does shift, at roughly a sixth of the size seen on contested claims, so what mostly moves there is how firmly the agent answers and not which side it takes.

The conclusion is the last thing an agent writes. Before it comes a list of searches the agent chose for itself. Here are the searches from two of the runs at the top of this page: the same claim, the same agent, the same search tool, and one different word in the request.

A second copy of the judge model labelled each search. Is it worded to look for support, to look for evidence against the claim, or neither? The chart below shows the balance between the first two for each request.

With the neutral request, 6.0 percent of searches looked for evidence against the claim. With the skeptic request it was 13.3 percent, more than double, while searches looking for support fell from 8.2 percent to 3.4. The believer request left searching balanced. The believer and skeptic requests led to about the same number of searches as each other. What separated them was direction.

What came back changed far less than what was asked. A separate check labelled the retrieved results themselves as supporting or opposing the claim. It worked through the claim list in order and stopped early, so it covers 16 of the 48 claims, and 13 of those under both the skeptic and the neutral request. Compared within a claim, the share of results that argue against the claim differs between those two requests by 1.7 percentage points, with a 95% interval from minus 1.5 to plus 5.5. So we found no evidence that doubt pulls in a more hostile set of results, though the interval is too wide to rule out a modest difference. Seven of the 14 claims that carry most of the movement fall inside the checked set, so this was not measured only where nothing happened.

A stated belief could reach the conclusion in two ways. It could change which results come back, or it could change what the agent makes of them. To separate the two, we ran the believer and skeptic requests again, each on the search results the other one had retrieved.

This swap was added after the first round of review and is reported on its own, apart from the registered runs. Each of its two averages pairs one live run with one replayed run on each side, so any constant difference between running a search and reading saved results cancels out of both. It measures the split on the runs it covered. Covering more of them, and letting agents read full pages, is what would turn that into a general claim.

A simpler test was on the registered plan. It hands both requests the neutral run’s search results and takes the search tool away. That cut the gap from 0.31 to 0.11 on its own matched set of 170 pairs. Because it also removes searching, it puts a bound on the effect and cannot split it.

The sources were about as reputable either way. The share of peer-reviewed or indexed sources among the citations stayed within a 3.3 percentage point spread across the three requests, a difference we did not test. Believer and skeptic runs did share fewer cited websites with each other than two neutral runs did, an overlap of 0.25 against 0.33, but that difference was not statistically significant. So if a stated belief changes the sources at all, it changes which reputable sources are reached and not how reputable they are, and a filter on source reputation would not catch it.

The skeptic request is the one that moves the agent, so that is where we tried a fix: one extra line in the agent’s instructions, asking it to look at both sides.

What this suggests is that where the instruction goes matters: in this follow-up it helped on the request that carries the bias. It is still a patch at the level of the prompt, and repairs at the level of training remain the target.