Errata.
spend $——.———— · the homebuyer’s history
Memory that keeps its corrections — every answer cited, every revision kept, refusal when the history never says.

results · Naive top-k RAG (k=10) · rerunC-nothink

single-session-user 77.3

51 of 66 rows right — 22 questions × 3 seeds (11/22/33)the corpus’s own question_type, over all 450 rows — a gold-abstention row is right iff the arm abstained, every other row iff the judge said CORRECTsame arm, by ability: OverallInformation extractionMulti-sessionTemporalKnowledge updateAbstentionsame arm, by question_type: single-session-assistantsingle-session-preferencemulti-sessiontemporal-reasoningknowledge-updatethe same cut, other arms: Errata 100.0Full-context baseline 100.0
  1. ABSTAINED577d4d32single-session-userasked 2023/05/30 (Tue) 23:544,837 ctx tok

    What time do I stop checking work emails and messages?

    naive
    ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions avoiding screens before bedtime and limiting stimulating activities, but does not specify a time for stopping work emails.INSUFFICIENT_INFORMATION: The history mentions avoiding stimulating activities like exercise, playing video games, or watching exciting movies at least 2-3 hours before bedtime, but does not specify a time for stopping work emails.
    gold
    7 pm
    judge
    Candidate failed to provide the 7 pm answer
    errata 7 pmfull-context 7 pm
  2. ABSTAINED60d45044single-session-userasked 2023/05/30 (Tue) 20:465,036 ctx tok

    What type of rice is my favorite?

    naive
    ABSTAINED — not in the history
    gold
    Japanese short-grain rice
    judge
    Candidate declines to answer, doesn't state Japanese short-grain rice as favorite
    errata Japanese short-grain ricefull-context Japanese short-grain rice
  3. ABSTAINEDa82c026esingle-session-userasked 2023/05/30 (Tue) 21:264,224 ctx tok

    What game did I finally beat last weekend?

    naive
    ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions playing Dark Souls 3 and enjoying the sense of accomplishment when defeating a tough boss, but does not specify which game was beaten last weekend.INSUFFICIENT_INFORMATION: The history mentions playing 8 games of soccer and discussing Dark Souls 3, but does not state which game was beaten last weekend.
    gold
    Dark Souls 3 DLC
    judge
    Candidate declines to answer instead of stating Dark Souls 3 DLC
    errata Dark Souls 3 DLCfull-context Dark Souls 3 DLC
  4. ABSTAINEDc19f7a0bsingle-session-userasked 2023/05/30 (Tue) 22:194,754 ctx tok

    What time do I usually get home from work on weeknights?

    naive
    ABSTAINED — not in the history
    gold
    6:30 pm
    judge
    Candidate declines to answer; reference states 6:30 pm
    errata around 6:30 pmfull-context 6:30 pm
  5. ABSTAINEDe47becbasingle-session-userasked 2023/05/30 (Tue) 23:404,290 ctx tok

    What degree did I graduate with?

    naive
    ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The chat history mentions taking an online course on diversity and inclusion but does not specify a degree.INSUFFICIENT_INFORMATION: The history mentions taking an online course on diversity and inclusion but does not state what degree you graduated with.
    gold
    Business Administration
    judge
    Candidate declines to answer, omits Business Administration
    errata Business Administrationfull-context Business Administration
  6. CORRECT0862e8bfsingle-session-userasked 2023/05/30 (Tue) 12:184,454 ctx tok

    What is the name of my cat?

    naive
    Luna
    gold
    Luna
    judge
    Candidate states the cat's name is Luna, matching reference.
    errata Lunafull-context Luna
  7. ABSTAINED0862e8bf_abssingle-session-userasked 2023/05/30 (Tue) 21:394,520 ctx tok

    What is the name of my hamster?

    naive
    ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions a cat named Luna, but does not mention a hamster.INSUFFICIENT_INFORMATION: The history mentions a cat named Luna.
    gold
    ABSTAIN — the history never says
    errata abstainedfull-context abstained
  8. ABSTAINED15745da0_abssingle-session-userasked 2023/05/30 (Tue) 23:023,940 ctx tok

    How long have I been collecting vintage films?

    naive
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    errata abstainedfull-context abstained
  9. CORRECT19b5f2b3single-session-userasked 2023/05/30 (Tue) 20:505,080 ctx tok

    How long was I in Japan for?

    naive
    two weeks
    gold
    two weeks
    judge
    Matches reference: two weeks
    errata two weeksfull-context Two weeks
  10. ABSTAINED19b5f2b3_abssingle-session-userasked 2023/05/30 (Tue) 17:145,133 ctx tok

    How long was I in Korea for?

    naive
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    errata abstainedfull-context abstained
  11. CORRECT21436231single-session-userasked 2023/05/30 (Tue) 19:254,990 ctx tok

    How many largemouth bass did I catch on my fishing trip to Lake Michigan?

    naive
    12
    gold
    12
    judge
    Matches the reference quantity of 12
    errata 12full-context 12
  12. CORRECT29f2956bsingle-session-userasked 2023/05/30 (Tue) 20:454,497 ctx tok

    How much time do I dedicate to practicing guitar every day?

    naive
    30 minutes daily
    gold
    30 minutes
    judge
    Both state 30 minutes daily
    errata 30 minutes dailyfull-context 30 minutes
  13. ABSTAINED29f2956b_abssingle-session-userasked 2023/05/30 (Tue) 20:284,779 ctx tok

    How much time do I dedicate to practicing violin every day?

    naive
    ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions playing an instrument as a hobby but does not specify violin or daily practice time.INSUFFICIENT_INFORMATION: The chat history mentions playing an instrument as a productive hobby but does not specify violin or daily practice time.
    gold
    ABSTAIN — the history never says
    errata abstainedfull-context abstained
  14. CORRECT36580ce8single-session-userasked 2023/05/30 (Tue) 22:094,457 ctx tok

    What health issue did I initially think was just a cold?

    naive
    bronchitis
    gold
    bronchitis
    judge
    Matches reference fact: bronchitis
    errata bronchitisfull-context bronchitis
  15. CORRECT6ade9755single-session-userasked 2023/05/30 (Tue) 21:044,151 ctx tok

    Where do I take yoga classes?

    naive
    Serenity Yoga
    gold
    Serenity Yoga
    judge
    Matches reference entity: Serenity Yoga
    errata Serenity Yogafull-context Serenity Yoga
  16. CORRECT6b168ec8single-session-userasked 2023/05/30 (Tue) 20:244,742 ctx tok

    How many bikes do I own?

    naive
    3
    gold
    three
    judge
    3 equals three
    errata threefull-context You own three bikes.
  17. CORRECT8550ddaesingle-session-userasked 2023/05/30 (Tue) 23:514,737 ctx tok

    What type of cocktail recipe did I try last weekend?

    naive
    lavender gin fizz
    gold
    lavender gin fizz
    judge
    Exact match: lavender gin fizz
    errata lavender gin fizzfull-context Lavender gin fizz
  18. CORRECT8ebdbe50single-session-userasked 2023/05/30 (Tue) 18:444,500 ctx tok

    What certification did I complete last month?

    naive
    Data Scienceother seeds: Data Science certificationData Science
    gold
    Data Science
    judge
    Matches reference fact: Data Science certification
    errata Data Science certificationfull-context Data Science
  19. CORRECTad7109d1single-session-userasked 2023/05/30 (Tue) 21:563,651 ctx tok

    What speed is my new internet plan?

    naive
    500 Mbps
    gold
    500 Mbps
    judge
    Matches reference value of 500 Mbps
    errata 500 Mbpsfull-context 500 Mbps
  20. ABSTAINEDbc8a6e93_abssingle-session-userasked 2023/05/30 (Tue) 15:265,108 ctx tok

    What did I bake for my uncle's birthday party?

    naive
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    errata abstainedfull-context abstained
  21. ABSTAINEDf4f1d8a4_abssingle-session-userasked 2023/05/30 (Tue) 21:514,613 ctx tok

    What did my dad gave me as a birthday gift?

    naive
    ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions a silver necklace from a local boutique store that the user bought for their sister, and a dress the user decided to buy for themselves.INSUFFICIENT_INFORMATION: The history mentions a silver necklace bought for the user's sister and a dress the user decided to buy for themselves, but does not mention any gift from their dad.
    gold
    ABSTAIN — the history never says
    errata abstainedfull-context abstained
  22. CORRECTf8c5f88bsingle-session-userasked 2023/05/30 (Tue) 23:394,852 ctx tok

    Where did I buy my new tennis racket from?

    naive
    a sports store downtown
    gold
    the sports store downtown
    judge
    Same fact: sports store downtown, minor article difference only
    errata a sports store downtownfull-context a sports store downtown