For each criterion discussed in Section 1 , we first go through the definition and examples of the criterion, then propose its delegate measure:

Beyond Memorization.

LLMs are known to memorize large amounts of training data (Tirumala et al., 2022 ; Delétang et al., 2023 ; Sutskever, 2023 ) . For popular texts ( e.g. , widely read books), models may recall content or associated analysis from pretraining, bypassing the need for actual comprehension. As training datasets continue to expand, this issue becomes increasingly problematic. Therefore, a benchmark should be designed such that it cannot be solved purely through memorized knowledge, ensuring that the full length and structure of the context remain necessary for reasoning.

$\bullet$ Measurement: We first evaluate the memorization performance of GPT-4o by having it answer questions without using RAG. We then compute its quantile within the range defined by human and random performance. This measurement is illustrated in Figure 4 .

Global Dependency.

The task should require aggregating information from multiple pieces of evidence that are scattered across the context or exhibit global dependencies. Otherwise, it reduces to a short-context problem solvable by retrieving the relevant passage for a given question. In such cases, the task becomes more about improving retrieval quality than long-context understanding ability.

$\bullet$ Measurement: This dimension differs slightly from the previous one in how the performance interval is defined. To evaluate how much better a model performs compared to using a single document, the reference point should be the model’s performance with sufficient evidence, rather than human performance. To approximate this, we retrieve the top-20 documents using both the question and the answer, and treat this as the upper bound of performance with sufficient evidence. We then ask GPT-4o to select the best supporting document from the top-20 retrieved using both the question and answer, treating it as the strongest single piece of evidence . We compute the quantile of the model’s RAG performance using only this selected document, relative to the interval defined by the QA-top-20 RAG performance and the random baseline.

Depth of Reasoning.

By nature, long-context reasoning involves synthesizing multiple pieces of evidence across the input. The complexity of reasoning—especially multi-step deduction—is closely tied to task difficulty. Many existing benchmarks emphasize shallow reasoning, primarily requiring decomposition or enumeration (e.g., counting mentions of a fact (Xu & Ma, 2025 ) , or multi-hop QA where the questions are often constructed as sequences of relatively simple sub-questions (Yang et al., 2018 ) . This limits the need for global reasoning and makes tasks easier than intended.

$\bullet$ Measurement: Similar to the previous dimension, the key in this measurement lies in identifying a representative performance interval and a suitable reference model for computing the quantile. We choose QwQ-32B as a strong reasoning model and treat its RAG performance using the top-20 documents retrieved with both the question and answer (as defined in the previous measurement) as the upper bound. We then compute the quantile of Qwen2.5-7B under the same RAG setting. This gap reflects the potential for improvement attributable to stronger reasoning capabilities. Thus, the deeper the reasoning required by a dataset, the larger this gap is expected to be.

Note that this proxy becomes less informative if the dataset is so challenging that neither model significantly outperforms the random baseline. However, as shown in Table 7 , this is not currently the case. Therefore, this measurement remains a meaningful indicator.

Beyond Summarization/Salience.

This often-overlooked criterion is crucial: Tasks that can be resolved simply by generating a summary of a long input are less challenging and may no longer probe deep understanding, especially given recent advances in summarization ( e.g. , hierarchical or iterative methods (Chang et al., 2023 ) ). In such cases, the challenge of long-context understanding is effectively reduced to a summarization task over short segments, creating a shortcut. A high-quality benchmark should instead require attention to fine-grained details that go beyond high-level abstraction, or it can be reduced to a relatively easier task of summarization thus is less meaningful.

$\bullet$ Measurement: This dimension is measured by comparing the performance of RAG using original text chunks against RAG using chunk summaries of the same input length. We do not use quantiles here but instead directly report the relative improvement, as some datasets actually perform worse when using the original texts, indicating that the task questions primarily target salient events, making summarization sufficient for answering.

Human-Machine Gap.

To highlight essential capabilities that general-purpose intelligent systems ought to possess, a meaningful benchmark should show a significant gap between humans and machines. At the same time, poor human accuracy or inter-annotator agreement typically signals low annotation quality or high subjectivity — both detrimental to robust evaluation.

$\bullet$ Measurement: We identify the best machine performance from the evaluations in the Beyond Memorization and Beyond Summarization dimensions and compute its quantile within the interval defined by human performance and the random baseline.

Compared Datasets.

We compare our PRELUDE with representative benchmarks from prior work, including (Kočiskỳ et al., 2018 ; Karpinska et al., 2024 ; Zhao et al., 2024 ; Xu et al., 2025b ) . These widely used datasets span multiple genres such as classic and detective novels, support both English and Chinese languages, and cover a range of task formats including free-form QA, multiple choice, and true-or-false questions. For a fair comparison, we sample $\sim$ 100 questions from each benchmark: InfBench.MC (25 books), DetectiveQA (10 books), and NarrativeQA (10 books from the development set), ensuring a similar number of examples across datasets.

We simple prompt the LLMs with no additional context to generate the prequels for human annotation. Figure 8 demonstrates the English translation of our prompt.

We apply LoRA (Hu et al., 2021 ) to all available target modules for in-domain training with RAG and non-RAG settings. The training is conducted on the dataset excluding the human-study subset, resulting in a total of 695 instances. Since the average token length in the RAG setting is above 20k, we perform LoRA tuning on both settings using 360-LLaMA-Factory (Haosheng Zou & Zhang, 2024 ) . We fine-tune each model for 2 epochs with a batch size of 1 on a single machine with 8 H20 GPUs. The dimension of LoRA’s attention layer is set to 64, while $\alpha$ is configured to 128. In addition, the learning rate and warmup ratio are set to 1e-4 and 0.1 respectively.

We use the prompt template in Figure 9 and 11 for the RAG experiment, depending on whether the instance originates from a Chinese book or an English book. For the “No Context” experiment, we remove the original excerpt part both in the ICL examples and the actual instance. The exact content of ICL in both languages is in Figure 10 and 12 .

Subjective Example – Fuzzy Logic : Example • ‣ D.1 shows an example that our annotators felt difficult to arrive an agreement due to the fuzzy reasoning. This setting results in the fact that the character appears in the story at around 120 years old. The novel does feature several characters in their 100s who remain healthy and skilled in combat. Still, as the most advanced martial artist in the story, whether 120 is a reasonable age is a matter of fuzzy logic.

Example I. An example of disagreement among human annotators due to fuzzy reasoning.

Subjective Example – Different Interpretation of Persona (Example • ‣ D.1 ) : In the story, the Sweeper Monk is not a formal disciple of Shaolin; he is more like a hermit. So the disagreement arises that when others mention that many Shaolin disciples had learned Xiaowuxiang Gong, whether the Sweeper Monk would interpret that as referring to himself? The annotators would project themselves into the character’s perspective when making predictions, which introduces subjectivity.

Example II. An example of disagreement among human annotators due to subjective interpretation of character persona.

Example that RAG Performs Worse : Example • ‣ D.1 gives an example in which the vanilla LLM can predict correct answer but fails when equipped with RAG. The prequel is consistent because it does not violate any part of the story. Specifically, in a flashback scene from the original novel, it is revealed that Murong Bo’s mother subjected him to harsh training in order to raise him as a future monarch for national restoration. When the RAG system fails to recognize that this passage describes Murong Bo’s childhood, it tends to rely solely on the more prominent information in the input (e.g., Document 0) to make its judgment.

Example III. An example that RAG makes the performance worse.

Example with Correct Answer but Incorrect Reasoning : In Example D.1 , the model correctly answers the question, but bases on incorrect reasons. The actual basis for judging this as a contradiction lies in the fact that both characters are portrayed as emphasizing personal martial prowess, with no demonstration of strategic ability or any depiction of possessing military treatises in the original novel. The act of gifting The Six Secret Teachings (Liu Tao)—a text regarded in Chinese tradition as one of the most profound and authoritative works on military strategy—stands in sharp contrast to their characterization.

The model’s explanation is invalid because it fails to grasp that Wu Anguo was representing Beihai’s forces (under Kong Rong) in the battle at Hulao Pass. Instead, it superficially interprets the text and incorrectly assumes that Wu Anguo’s sphere of activity was limited to Hulao Pass.

This reflects the same underlying issue as in the RAG failure example: LLMs are heavily influenced by the provided context and tend to overanalyze non-informative clues, often becoming overly literal and fixating on minor wording differences.