Why a research paper defeats naive ingestion
Two column text is not the only thing that breaks, and it does not break the way the
tutorials say. On this corpus the geometric sort splices both columns into
95.9% of its output lines, while
content-stream order reads the columns correctly and destroys the tables instead.
Which naive extractor you picked decides which failure you get.
Index built 2026-08-10T07:23:47. 1822 passages from 90 arXiv papers and 12 synthetic internal notes. Embeddings: Snowflake/snowflake-arctic-embed-s (ONNX int8, Apache-2.0), loaded from Demo 1. One embedding stack across both demonstrations. Demo 1 harness live model configured
Retrieval, extraction, every statistical test and the backlog are model-free. Only the synthesis paragraph on screen 2 uses a model, and it degrades to extraction.
Retrieval, extraction, every statistical test and the backlog are model-free. Only the synthesis paragraph on screen 2 uses a model, and it degrades to extraction.
The page
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach — arXiv:2407.16833, page 4, CC BY 4.0. abs page. Column gutter detected at x=300.0 pt.
This page was not chosen by eye. Every page of every full-text paper in the corpus was scanned (117 pages, 57 of them two-column) and the page with the highest measured splice rate is the one shown. Choosing the most persuasive example by hand is the habit this demonstration exists to argue against.
Three extractors, same page
Naive A — geometric sort interleaved
Avg Narr Qasp Mult Hotp 2Wiki Musi Sum En.QA En.MC 1-1 LC 49.70 32.76 47.83 52.33 61.85 62.96 40.22 20.73 43.08 85.57 1-2 RAG 37.33 22.54 44.68 49.53 48.36 54.24 26.56 19.51 19.46 51.09 1-3 S ELF -R OUTE 46.41 28.32 45.23 51.47 55.18 62.68 40.66 19.77 37.51 76.86 Gemini-1.5-Pro 1-4 answerable % 76.78 73.00 85.00 96.67 84.50 81.00 58.50 93.50 56.41 62.45 1-5 token % 38.39 23.07 49.93 36.88 32.97 53.49 56.14 17.96 42.25 32.84 2-1 LC 48.67 32.78 44.54 55.28 62.42 70.69 41.65 21.92 32.36 76.42 2-2 RAG 32.60 18.05 46.02 50.74 36.86 50.21 16.09 19.97 14.43 41.05 GPT-4O 2-3 S ELF -R OUTE 48.89 31.36 47.99 53.17 62.14 70.14 41.69 21.31 34.95 77.29 2-4 answerable % 57.36 44.00 67.50 94.00 52.50 62.00 30.00 92.00 27.07 47.16 2-5 token % 61.40 66.40 72.25 39.65 65.79 77.05 85.00 20.26 73.01 53.21 3-1 LC 32.07 23.34 42.96 49.19 45.33 41.04 17.92 19.61 14.73 34.50 3-2 RAG 30.33 18.22 38.15 49.21 37.84 35.16 16.41 18.94 15.39 43.67 GPT-3.5-Turbo 3-3 S ELF -R OUTE 35.32 24.06 38.65 52.07 47.28 44.62 34.44 19.88 22.03 44.54 3-4 answerable % 74.10 71.50 80.00 91.33 68.50 69.00 47.00 93.50 50.43 95.63 3-5 token % 38.85 20.56 55.08 35.29 48.70 65.91 65.08 16.40 38.17 4.50 Table 1: Results of Gemini-1.5-Pro, GPT-3.5-Turbo, and GPT-4O using the Contriever retriever. LC consistently outperforms RAG, while S ELF -R OUTE achieves performance comparable to LC using much less tokens. most queries, RAG scores and LC scores are highly “Write unanswerable if the query can not similar. In fact, for 63% queries, the model pre- be answered based on the provided text” . dictions are exactly identical; and for 70% queries, For the queries deemed answerable, we accept the the score difference is less than 10 (absolute value). RAG prediction as the final answer. For the queries Interestingly, the identical predictions are not nec- deemed unanswerable, we proceed to the second essarily correct, as shown by the varying colors rep- step, providing the full context to the long-context resenting the average score, i.e., (S RAG + S LC )/2. LLMs to obtain the final prediction (i.e., LC). This observation suggests that RAG and LC tend As our results will demonstrate, most queries can to make not only the same correct predictions but be solved by the first RAG-and-Route step (e.g., also similar errors. 82% for Gemini-1.5-Pro), with only a small por- This finding motivates us to leverage RAG for tion requiring the following long-context prediction the majority of queries, reserving computationally step. Since the RAG-and-Route step only needs more expensive LC for a small subset of queries the retrieved chunks (e.g., 1.5k tokens) as input, where it truly excels. By doing so, RAG can signif- which is significantly shorter than the full contexts icantly reduce computational costs without sacrific- (e.g., 10k - 100k tokens), the overall computation ing overall performance. cost is substantially reduced. Detailed token count analysis will be provided in the results. 4.2 Self-Route 4.3 Results Based on the above motivation, we propose S ELF - R OUTE , a simple yet effective method combining Rows *-3 to *-5 in Tab. 1 present the results of our RAG and LC to reduce cost while maintaining a method, utilizing the three LLMs. Rows *-3 report performance comparable to LC. S ELF -R OUTE uti- the performance. Rows *-4 show the percentage lizes LLM itself to route queries based on self- of answerable queries, as predicted in the RAG- reflection, under the assumption that LLMs are and-Route step. Rows *-5 display the percentage well-calibrated in predicting whether a query is of tokens used by our method, compared to that answerable given provided context. of LC. In terms of performance (rows *-3), S ELF - Concretely, our method consists of two steps: a R OUTE significantly outperforms RAG, achieving RAG-and-Route step and a long-context prediction results comparable to LC. Across all three models, step. In the first step, we provide the query and S ELF -R OUTE surpasses RAG (rows *-2) by over the retrieved chunks to the LLM, and prompt it to 5%. Compared to LC (rows *-1), there is a slight predict whether the query is answerable and, if so, performance drop for GPT-4O (-0.2%) and G
Naive B — content-stream order reading order intact
Avg Narr Qasp Mult Hotp 2Wiki Musi Sum En.QA En.MC 1-1 LC 49.70 32.76 47.83 52.33 61.85 62.96 40.22 20.73 43.08 85.57 Gemini-1.5-Pro 1-2 RAG 37.33 22.54 44.68 49.53 48.36 54.24 26.56 19.51 19.46 51.09 1-3 SELF-ROUTE 46.41 28.32 45.23 51.47 55.18 62.68 40.66 19.77 37.51 76.86 1-4 answerable % 76.78 73.00 85.00 96.67 84.50 81.00 58.50 93.50 56.41 62.45 1-5 token % 38.39 23.07 49.93 36.88 32.97 53.49 56.14 17.96 42.25 32.84 GPT-4O 2-1 LC 48.67 32.78 44.54 55.28 62.42 70.69 41.65 21.92 32.36 76.42 2-2 RAG 32.60 18.05 46.02 50.74 36.86 50.21 16.09 19.97 14.43 41.05 2-3 SELF-ROUTE 48.89 31.36 47.99 53.17 62.14 70.14 41.69 21.31 34.95 77.29 2-4 answerable % 57.36 44.00 67.50 94.00 52.50 62.00 30.00 92.00 27.07 47.16 2-5 token % 61.40 66.40 72.25 39.65 65.79 77.05 85.00 20.26 73.01 53.21 GPT-3.5-Turbo 3-1 LC 32.07 23.34 42.96 49.19 45.33 41.04 17.92 19.61 14.73 34.50 3-2 RAG 30.33 18.22 38.15 49.21 37.84 35.16 16.41 18.94 15.39 43.67 3-3 SELF-ROUTE 35.32 24.06 38.65 52.07 47.28 44.62 34.44 19.88 22.03 44.54 3-4 answerable % 74.10 71.50 80.00 91.33 68.50 69.00 47.00 93.50 50.43 95.63 3-5 token % 38.85 20.56 55.08 35.29 48.70 65.91 65.08 16.40 38.17 4.50 Table 1: Results of Gemini-1.5-Pro, GPT-3.5-Turbo, and GPT-4O using the Contriever retriever. LC consistently outperforms RAG, while SELF-ROUTE achieves performance comparable to LC using much less tokens. most queries, RAG scores and LC scores are highly similar. In fact, for 63% queries, the model predictions are exactly identical; and for 70% queries, the score difference is less than 10 (absolute value). Interestingly, the identical predictions are not necessarily correct, as shown by the varying colors representing the average score, i.e., (SRAG + SLC)/2. This observation suggests that RAG and LC tend to make not only the same correct predictions but also similar errors. This finding motivates us to leverage RAG for the majority of queries, reserving computationally more expensive LC for a small subset of queries where it truly excels. By doing so, RAG can significantly reduce computational costs without sacrificing overall performance. 4.2 Self-Route Based on the above motivation, we propose SELFROUTE, a simple yet effective method combining RAG and LC to reduce cost while maintaining a performance comparable to LC. SELF-ROUTE utilizes LLM itself to route queries based on selfreflection, under the assumption that LLMs are well-calibrated in predicting whether a query is answerable given provided context. Concretely, our method consists of two steps: a RAG-and-Route step and a long-context prediction step. In the first step, we provide the query and the retrieved chunks to the LLM, and prompt it to predict whether the query is answerable and, if so, generate the answer. This is similar to standard RAG, with one key difference: the LLM is given the option to decline answering with the prompt “Write unanswerable if the query can not be answered based on the provided text”. For the queries deemed answerable, we accept the RAG prediction as the final answer. For the queries deemed unanswerable, we proceed to the second step, providing the full context to the long-context LLMs to obtain the final prediction (i.e., LC). As our results will demonstrate, most queries can be solved by the first RAG-and-Route step (e.g., 82% for Gemini-1.5-Pro), with only a small portion requiring the following long-context prediction step. Since the RAG-and-Route step only needs the retrieved chunks (e.g., 1.5k tokens) as input, which is significantly shorter than the full contexts (e.g., 10k - 100k tokens), the overall computation cost is substantially reduced. Detailed token count analysis will be provided in the results. 4.3 Results Rows *-3 to *-5 in Tab. 1 present the results of our method, utilizing the three LLMs. Rows *-3 report the performance. Rows *-4 show the percentage of answerable queries, as predicted in the RAGand-Route step. Rows *-5 display the percentage of tokens used by our method, compared to that of LC. In terms of performance (rows *-3), SELFROUTE significantly outperforms RAG, achieving results comp
Structure-aware columns, tables, captions
Avg Narr Qasp 1-1 LC 49.70 32.76 47.83 1-2 RAG 37.33 22.54 44.68 1-3 S ELF -R OUTE 46.41 28.32 45.23 Gemini-1.5-Pro 1-4 answerable % 76.78 73.00 85.00 1-5 token % 38.39 23.07 49.93 2-1 LC 48.67 32.78 44.54 2-2 RAG 32.60 18.05 46.02 GPT-4O 2-3 S ELF -R OUTE 48.89 31.36 47.99 2-4 answerable % 57.36 44.00 67.50 2-5 token % 61.40 66.40 72.25 3-1 LC 32.07 23.34 42.96 3-2 RAG 30.33 18.22 38.15 GPT-3.5-Turbo 3-3 S ELF -R OUTE 35.32 24.06 38.65 3-4 answerable % 74.10 71.50 80.00 3-5 token % 38.85 20.56 55.08 Mult Hotp 2Wiki Musi Sum En.QA En.MC 52.33 61.85 62.96 40.22 20.73 43.08 85.57 49.53 48.36 54.24 26.56 19.51 19.46 51.09 51.47 55.18 62.68 40.66 19.77 37.51 76.86 96.67 84.50 81.00 58.50 93.50 56.41 62.45 36.88 32.97 53.49 56.14 17.96 42.25 32.84 55.28 62.42 70.69 41.65 21.92 32.36 76.42 50.74 36.86 50.21 16.09 19.97 14.43 41.05 53.17 62.14 70.14 41.69 21.31 34.95 77.29 94.00 52.50 62.00 30.00 92.00 27.07 47.16 39.65 65.79 77.05 85.00 20.26 73.01 53.21 49.19 45.33 41.04 17.92 19.61 14.73 34.50 49.21 37.84 35.16 16.41 18.94 15.39 43.67 52.07 47.28 44.62 34.44 19.88 22.03 44.54 91.33 68.50 69.00 47.00 93.50 50.43 95.63 35.29 48.70 65.91 65.08 16.40 38.17 4.50 Table 1: Results of Gemini-1.5-Pro, GPT-3.5-Turbo, and GPT-4O using the Contriever retriever. LC consistently outperforms RAG, while S ELF -R OUTE achieves performance comparable to LC using much less tokens. most queries, RAG scores and LC scores are highly similar. In fact, for 63% queries, the model predictions are exactly identical; and for 70% queries, the score difference is less than 10 (absolute value). Interestingly, the identical predictions are not necessarily correct, as shown by the varying colors representing the average score, i.e., (S RAG + S LC )/2. This observation suggests that RAG and LC tend to make not only the same correct predictions but also similar errors. This finding motivates us to leverage RAG for the majority of queries, reserving computationally more expensive LC for a small subset of queries where it truly excels. By doing so, RAG can significantly reduce computational costs without sacrificing overall performance. ## 4.2 Self-Route Based on the above motivation, we propose S ELF - R OUTE , a simple yet effective method combining RAG and LC to reduce cost while maintaining a performance comparable to LC. S ELF -R OUTE utilizes LLM itself to route queries based on selfreflection, under the assumption that LLMs are well-calibrated in predicting whether a query is answerable given provided context. Concretely, our method consists of two steps: a RAG-and-Route step and a long-context prediction step. In the first step, we provide the query and the retrieved chunks to the LLM, and prompt it to predict whether the query is answerable and, if so, generate the answer. This is similar to standard RAG, with one key difference: the LLM is given the option to decline answering with the prompt “Write unanswerable if the query can not be answered based on the provided text” . For the queries deemed answerable, we accept the RAG prediction as the final answer. For the queries deemed unanswerable, we proceed to the second step, providing the full context to the long-context LLMs to obtain the final prediction (i.e., LC). As our results will demonstrate, most queries can be solved by the first RAG-and-Route step (e.g., 82% for Gemini-1.5-Pro), with only a small portion requiring the following long-context prediction step. Since the RAG-and-Route step only needs the retrieved chunks (e.g., 1.5k tokens) as input, which is significantly shorter than the full contexts (e.g., 10k - 100k tokens), the overall computation cost is substantially reduced. Detailed token count analysis will be provided in the results. ## 4.3 Results Rows *-3 to *-5 in Tab. 1 present the results of our method, utilizing the three LLMs. Rows *-3 report the performance. Rows *-4 show the percentage of answerable queries, as predicted in the RAGand-Route step. Rows *-5 display the percentage of tokens used by our method, compared to that of LC. In terms of performance (rows *-3), S ELF - R OUTE significantly outperforms RAG, achieving results comparable to LC. Across all three models,
Naive A sorts every word by (y, x) and joins — the most common home-grown extractor.
Naive B is pypdfium2 page text, which is what a "chat with your PDF"
tutorial does. The third arm is pdfplumber word boxes plus explicit
column, heading, caption and table logic.
Measured, not asserted
| Measure | Naive A geometric |
Naive B content stream |
|---|---|---|
| Output lines containing text from both columns | 95.9% | 24.6% |
| Column switches between consecutive lines | 100.0% | 2.1% |
| Sentences recovered intact (of 25) | 8.0% | 48.0% |
| Word-pair adjacency preserved | 87.6% | 88.7% |
"Sentences recovered intact" takes every sentence the structure-aware arm found and asks whether that exact word sequence appears contiguously in the naive output. End-of-line hyphenation is normalised away first, so this measures reading order rather than the dehyphenator.
Spliced lines, verbatim
Avg Narr Qasp Mult Hotp 2Wiki Musi Sum En.QA En.MC1-1 LC 49.70 32.76 47.83 52.33 61.85 62.96 40.22 20.73 43.08 85.571-2 RAG 37.33 22.54 44.68 49.53 48.36 54.24 26.56 19.51 19.46 51.091-3 S ELF -R OUTE 46.41 28.32 45.23 51.47 55.18 62.68 40.66 19.77 37.51 76.86
Each of these is one line of the geometric extractor's output. The left half comes from the left column and the right half from the right column, and an embedding model will happily encode the result as if it were a sentence.
Across the whole corpus
Splice rate per two-column page, both naive arms. The finding that matters for a build decision: content-stream order is close to zero on almost every page, because these are LaTeX-produced PDFs whose content stream is already in logical order. That is a property of the producer, not a guarantee — the same extractor on a Word-produced two-column paper behaves like the geometric arm.
What content-stream order still loses
Reading order is one of four failure modes. On this page the structure-aware arm additionally recovered 1 table(s) as grids, 1 figure or table caption(s) separated from the body, and 0 display formula line(s) kept out of the prose stream. A table flattened into a run of numbers reads plausibly and answers questions wrongly, and no reading-order metric detects it.