How the lab works, how it was evaluated, and where it falls short

• Demo text. Chapters 1, 2 and 12 of Statistics by Barbara Illowsky and Susan Dean (OpenStax, Rice University, CC BY 4.0), fetched from the openstax/osbooks-statistics repository at a pinned commit (7dea80a) and converted to plain text with exercises, homework, calculator notes and labs removed: 19 sections and 345 passages. Mathematics is rendered as plain text.
• Your uploads. PDF, TXT and Markdown files are parsed, chunked and embedded in your browser and stored only in its IndexedDB. Passages retrieved from them are sent to a model provider only when you generate with your own key; in the Library's whole-document mode a passage is a whole file, and Generate and Ask say so first.
• Gold set. 40 questions written from the demo text, each with verbatim evidence spans, and 480 pooled relevance judgements: 119 answer-bearing and 135 related passages. The file is web/src/data/eval/gold-set.json.
• The 2024 database held private tutoring notes. It stays in the repository for the record and is never read by the app or deployed.

1. Chunk. Text is split into sentences and packed into passages of at most 900 characters, carrying up to 180 characters of trailing sentences into the next passage (DR-002).
2. Embed. all-MiniLM-L6-v2, the 2024 model, runs in a Web Worker. Each passage is embedded once and cached; the demo text's vectors are precomputed.
3. Retrieve. By default every passage is scored by cosine similarity, exactly as the 2024 retriever did. BM25 keyword ranking and a hybrid (reciprocal rank fusion of the dense and BM25 top 100, k = 60) are options on Retrieve.
4. Generate. With your own key, the numbered passages and your topic go to the model with fixed grounding rules: use only the passages, cite them as [n], and say when they do not cover something. Output is JSON validated against a schema; citations outside the retrieved passages are removed and flagged.
5. Review. Automatic checks are shown next to each item. You accept, edit or reject each one, and the decision is recorded against the call in the AI log. Only accepted items are exported.

Full results, per-question rankings and downloads are on Evaluation; the reasoning is in DR-003.

• Retrieval gold set. Student-style questions across six types, verbatim evidence spans for answer-bearing passages, and a depth-10 pool of all three rankers judged blind to the ranker before any metric was computed. Unjudged passages count as not relevant. The chunking comparison scores sections, so every section either indexing unit ranks in its top 3 was also read for every question (128 question-section pairs, added after review; none held a further answer).
• Metrics. nDCG@10 is the primary metric, declared in the analysis code before the first run (in the same session, so not independently time-stamped). MRR@10, Recall@5 and @10, and Success@1, 3, 5 and 10 are secondary.
• Uncertainty. The unit is the question. Means get percentile bootstrap intervals (10,000 resamples, seed 20241009); hit rates get Wilson intervals. Rankers are compared on the same resampled questions, with Cohen's d_z and a sign-flip permutation test; Holm's adjustment covers the three primary comparisons. Differences in hit rates use Newcombe's paired score interval with McNemar's exact test.
• Headline. Hybrid minus dense on nDCG@10 is +0.048 (0.008 to 0.089), Holm-adjusted p = 0.084: a positive point estimate that is not significant after Holm's adjustment.
• Generation. A bring-your-own-key harness runs eight fixed topics through your model, applies automatic checks (citations, numbers, word overlap, answerability, multiple-choice form, distractors, duplicates) and collects your rubric ratings. Rates get Wilson intervals with the sample size divided by a design effect estimated from how much they vary between topics. The automatic checks stay hidden until you rate an item as faithful or not, and only those blind ratings enter the raw agreement and Cohen's kappa between the check and your judgement. A topic whose output is refused, cut off or invalid counts as a failure of the model, and evaluation calls do not use the provider's refusal fallback.
• Verification. Every published retrieval metric, interval and test, including the chunking comparison and the breakdown by type, is recomputed in Python with numpy, scipy, statsmodels and pytrec_eval (dense and section rankings from the stored vectors), the paired score intervals are checked in R with contingencytables, and the report is a test snapshot. BM25 and hybrid rankings come from the app's own code, and a test checks they match what Retrieve shows.

• The 40 questions stand in for the questions students ask about this text, and each question is an independent draw from that population, which is what the bootstrap and permutation tests resample.
• A passage holding a verbatim evidence span answers the question, and a passage outside the pooled top 10 of all three rankers would not change the conclusions. The chunking comparison does not need this assumption: every section either indexing unit ranked in its top 3 was read.
• The textbook is correct: passages are treated as ground truth, and faithfulness means agreeing with them.
• Browser queries are embedded by the 8-bit model, while the evaluation used full-precision Python query vectors. The vectors agree to a cosine of at least 0.996 for queries and 0.991 for chunks, but agreement of the resulting rankings on the gold set has not been measured; the evaluation shows the same ranking code given the same query vector.
• For generation, the items of one pack are correlated, so their intervals use a design effect estimated across packs (topics), and kappa's interval resamples whole packs.

• Model-drafted labels. Claude Opus 5.5 drafted the questions and relevance judgements under the owner's direction, and no person has reviewed them yet, so the retrieval results are provisional.
• The questions were written while reading the evidence, which favours word overlap and so BM25.
• Forty questions give wide intervals: a true difference of 0.03 in nDCG@10 is well within the noise.
• One corpus, a statistics textbook. Lecture slides, with their fragments and bullet points, may behave differently.
• No generation scores are published, because there is no budget for a reference run; the automatic checks are proxies, and their agreement with a person is measured only when someone rates a run.
• Mathematics is plain text after conversion, which hurts both embeddings and readability.

• Have a second person label the pools independently and report agreement on relevance.
• Use questions written by students who have not seen the passages, or past exam questions.
• Grow the set to about 63 questions (80% power for the observed effect, d_z = 0.36, at the 5% level) or about 84 at the first Holm threshold.
• Compare chunk sizes and a stronger small embedder on the gold set before changing the default ranker.
• Tighten the Content-Security-Policy to nonce-based scripts so 'unsafe-inline' can go, and add an optional per-session spending cap.
• With a small budget, publish one labelled reference run of the generation evaluation.