--- title: "Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines" page: "Lost in Handoff" url: https://siddvoh.com/identity-handoffs/ authors: ["Siddharth Vohra", "Runmin Jiang", "Xiaomo Li", "Min Xu"] status: accepted published: 2026-09-04 updated: 2026-10-20 bibtex: https://siddvoh.com/bib/rop.bib license: "Page text: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). The paper keeps its own terms." --- # Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines The paper · GroundLM Workshop · EMNLP 2026 [Siddharth Vohra](https://siddvoh.com/), Runmin Jiang, Xiaomo Li, Min Xu Accepted to the GroundLM workshop at EMNLP 2026, to be presented on 29 October 2026 in Budapest. Until the proceedings appear, the arXiv preprint is the version to cite. Traces whether an item selected early in a grounded QA pipeline survives retrieval and reaches the answer model. On 1,463 aligned HybridQA records, body-only BM25 dropped it out of the top five 26.6% of the time; hybrid reranking cut that to 1.0%. **Abstract** Grounded language-model pipelines can be divided into three stages: selecting an object, retrieving passages for it, and using that evidence to answer. If the selected object must reach the reader, losing it breaks the handoff. Benchmark recall checks the dataset-linked object, which can differ. We audit 600 HybridQA questions across three selector families. On 1,463 resolvable records where the selected object matches the dataset-traced passage, exact key lookup and exact title matching return the object every time. With every ranked rule given the same decoded selected title, body-only BM25 omits it on 389 records (26.6%) at cutoff five, while hybrid retrieval with reranking omits it on 14 (1.0%). The two identities differ on 329 of 1,792 resolvable records. With original-question rankings, their top-five checks disagree on 106 records (5.9%). Frozen reader comparisons associate the aligned object's presence with 28.6 to 31.0 points higher exact match. In a deliberately selected 64-item cohort, removing that passage sharply lowers exact match, while removing a similar-length comparison passage does not reproduce the drop. We release the Returned-Object Profile (ROP), an executable record of the target, returned-ID field, cutoff, membership rule, and complete expected population, with data and an offline replay. This is the markdown version of https://siddvoh.com/identity-handoffs/, the "Lost in Handoff" page: a plain-language walkthrough of the paper. The paper is the source to cite, and its reference and BibTeX are at the end. ## Lost in Handoff The page a system picks does not always reach the model that answers. A lot of question-answering software works in three moves. Something picks a page. Something else searches for it and returns a short list of text. A language model reads that list and writes the answer. The model never sees the picked page unless the search step puts it in the list, so a good pick can still end up counting for nothing. [Read the paper](https://arxiv.org/abs/2609.04579) [Code and data](https://github.com/siddvoh/rop) 01 Pick a page is chosen 02 Search five passages come back 03 Answer a model writes the answer Looked up by its key 0 of 1,463 lost the page Found by keyword search 389 of 1,463 lost the page Same page, same three steps. All that changes is how step two was asked to fetch it. ## Three steps, and a gap between the first two Picture a person asking a question about a table on a web page. A first step reads the table and picks one linked page. A second step turns that pick into a search and returns the five pages it ranks highest. A third step reads those five pages and writes the answer. The model is handed the question, the table the question came from, and those five passages. It never sees the picked page unless the second step put it in the list. So a pipeline can fail in three different places: the pick was wrong, the pick was right but got lost on the way, or the pick arrived and was ignored. One score for “did the right page show up” cannot tell those apart, and they need three different repairs. **600** questions, each written by a person, each from a different table. **1,800** runs: three families of model, every question, every one recorded. **1,463** of those runs where the picked page and the benchmark's page are the same page. ## Two pages can both be called the right one Benchmarks come with an answer key that links one page per question. A system picks its own. They are often the same page, and sometimes they are not, and then the score depends entirely on which one you asked about. Switch the search rule below and watch the two swap places. Someone asked The 6th oldest Catholic university in the United States gave what reason for discontinuing its football program in 1973? Fetch it with - Keyword search (shown) - BGE-M3 1 St. Francis Xavier X-Women women's ice hockey 2 Cintas Center 3 Xavier Musketeers football the page the benchmark links 4 St. Francis Xavier X-Men 5 St. Francis Xavier athletic teams the model sees nothing past here 6 Xavier University the page the system picked Only the benchmark's page arrives. Across 1,792 records where both pages exist in the corpus, the two disagree on 329, drawn from 160 of the 600 questions. Searching with the plain question, the two give different verdicts on 106 records, or 5.9 percent. Searching with words taken from the picked page’s own title, they differ on 213. Which number you report depends on which page you scored and what you typed into the search box, so a report needs to say both. In the checks the paper runs, swapping the target never reorders the five search stacks, for any model family or cutoff. What changes is the number you are entitled to report. ## Nine ways to hand a page over Take only the 1,463 records where both pages agree, so there is nothing to argue about. Hold that page fixed. Give the seven search rules the same words to search with. Then count how often the page is missing from the five that reach the model. How the page was fetched - Look the page up by its key: 0 of 1,463 records lost - Match the exact page title: 0 of 1,463 records lost - Hybrid search, then re-rank: 14 of 1,463 records lost - ColBERTv2: 20 of 1,463 records lost - BGE-M3: 65 of 1,463 records lost - Keyword search on titles: 66 of 1,463 records lost - Hybrid search: 148 of 1,463 records lost - Keyword search on titles and text: 284 of 1,463 records lost - Keyword search on page text: 389 of 1,463 records lost (shown) Lost before the model saw it 389 of 1,463 · 26.6% reached the model dropped One square is one record. The page is the same page in every row of the list, and the seven search rules were all handed the same words to search with. On these records, looking the page up by its key never loses it, and neither does matching its exact title. The search rules lose between 14 and 389 of the same 1,463. Of the 14 that the best of those seven loses, six sit just below the line and a wider window would catch them; the other eight are not in the top twenty at all. ## What the answer looks like when the page is gone Recorded runs that differ in whether the shared page was among the five tell a consistent story. How often the answer was exactly right page missing page present Page found by its ID, against the plain question 832 pairs 50.6 79.2 a gap of 28.6 points Page found by its ID, against a rewritten query 200 pairs 45.0 76.0 a gap of 31.0 points Rewritten query, against the plain question 903 pairs 52.6 81.9 a gap of 29.3 points These pairs also differ in which other passages were present, their order and their length, so this is a pattern in recorded runs, not proof of a cause. The test below changes one thing only. ## Removing one page on purpose 64 questions were chosen before any new run happened: cases where the model had been right with the shared page and wrong without it, in both of the recorded comparisons. Each was asked again three times with nothing removed, and three times with the picked page removed. 59 of the 64 also got a third pass with a different page removed instead. 64 questions chosen in advance, then run again - Nothing removed The five passages as the system built them. (shown) - A different page removed A passage of similar length that does not hold the answer. - The picked page removed Everything else left exactly as it was. What the model is handed the question the table it came from never removed a passage a passage the picked page the one that holds the answer a passage a similar passage Exactly right, over 64 questions 92.7 63 of the 64 questions came out exact on at least two of the three tries, and 51 on all three. Every one of the 24 calls that failed outright counts as wrong. Even if all of them had gone the other way, the gap stays at least 73.4 points. The comparison in the middle is the one that matters. On the 59 questions that have both, cutting a passage of similar length that does not hold the answer leaves the score where it was, and the interval on that difference includes zero. Cutting the picked page collapses it. The table the question came from is still in the prompt either way. ## A record you can run, instead of a number you have to trust The released instrument is a file rather than a table. It names the page being checked, the field the returned IDs come from, the cutoff, the rule for what counts as returned, and the full list of records the run is supposed to contain. A reference program reads that file alongside the run and gives one verdict per record and one status for the whole run. **What a pass rate hides** Suppose a run is meant to produce two records and both pass. Now one goes missing, from a timeout or a parser error. The pass rate over what survived is still perfect. - **Page found, inside the cutoff**: pass - **Page found, outside the cutoff**: fail - **The list came back malformed**: cannot be checked - **A record is duplicated or unplanned**: the run is void - **A planned record never came back**: named as missing, not quietly dropped Each of those verdicts points somewhere different: a page found but outside the cutoff is the retrieval step, a missing or malformed list is the logging, and a duplicated or unplanned record is how the run was assembled. A single percentage merges them. The released package carries one such record per result, each checked against the frozen rankings and answer records it was built from, and the check runs with no network connection and no API key. ## What this does not show One corpus: English Wikipedia tables and the pages they link. One recorded output per model and question, so nothing here says how much a picker’s choice wobbles if you run it again. Counts pooled across the three model families are not three independent samples of the same questions. The 329 records where the two pages disagree carry no judgement about which page would have supported a correct answer. Nobody read them. And the deletion result holds for the 64 questions it was run on, which were chosen precisely because they were clear-cut. The released checker confirms that the bytes it was given match their hashes and that the records it saw match the list it was promised. It does not establish who produced a log or when. That needs controls outside any evaluator. ## If you are building one of these **Say which page your number is about** The page your system picked and the page the benchmark links are not always the same. One score cannot describe both. **Write down the words you searched with** The same pair of pages disagreed on 106 records under one query and 213 under another. **Reserve a slot for the picked page** Look it up directly and fill the rest of the list from search. Exact lookup returned it on all 1,463 records; keyword search on page text missed 389. **Check membership at the cutoff you actually use** Not at twenty, if the model only ever sees five. **Count every record you planned to run** A pass rate over the ones that came back cannot tell you something is missing. ## The paper GroundLM Workshop · EMNLP 2026 ### Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines Siddharth Vohra, Runmin Jiang, Xiaomo Li, Min Xu Grounding Language Models: Learning Faithfully and Efficiently (GroundLM) Workshop at EMNLP 2026 · Budapest, Hungary · 29 October 2026 Accepted to the GroundLM workshop at EMNLP 2026, to be presented on 29 October 2026 in Budapest. Until the proceedings appear, the arXiv preprint is the version to cite. The full grid across four query forms, five search stacks and three cutoffs, the record format, and the reference checker. [Read the paper](https://arxiv.org/abs/2609.04579) If this is useful in your work, feel free to cite it as: Vohra, S., Jiang, R., Li, X., and Xu, M. (2026). Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines. Grounding Language Models: Learning Faithfully and Efficiently (GroundLM) Workshop at EMNLP 2026 (to appear). arXiv:2609.04579. **Cite this** [Download BibTeX as .bib](https://siddvoh.com/bib/rop.bib) ``` @inproceedings{vohra2026rop, title = {Does the Selected Object Reach the Reader? Auditing Identity Handoffs in Grounded Language-Model Pipelines}, author = {Vohra, Siddharth and Jiang, Runmin and Li, Xiaomo and Xu, Min}, booktitle = {Grounding Language Models: Learning Faithfully and Efficiently (GroundLM) Workshop at EMNLP 2026}, year = {2026}, address = {Budapest, Hungary}, note = {To appear}, eprint = {2609.04579}, archivePrefix = {arXiv}, url = {https://arxiv.org/abs/2609.04579} } ``` [Code and data on GitHub Every answer we recorded, the analysis that produced the numbers above, and the check itself.](https://github.com/siddvoh/rop) The questions and tables come from the public HybridQA release and keep their own terms. Every figure here is drawn from the released records, and the released checks run without a network connection or an API key. Work by Siddharth Vohra here is unrelated to his position at Amazon Web Services.