Errata.
spend $——.———— · the homebuyer’s history
Memory that keeps its corrections — every answer cited, every revision kept, refusal when the history never says.

results · Errata · rerunJ-arith

single-session-user 100.0

66 of 66 rows right — 22 questions × 3 seeds (11/22/33)the corpus’s own question_type, over all 450 rows — a gold-abstention row is right iff the arm abstained, every other row iff the judge said CORRECTsame arm, by ability: OverallInformation extractionMulti-sessionTemporalKnowledge updateAbstentionsame arm, by question_type: single-session-assistantsingle-session-preferencemulti-sessiontemporal-reasoningknowledge-updatethe same cut, other arms: Full-context baseline 100.0Naive top-k RAG (k=10) 77.3
  1. CORRECT0862e8bfsingle-session-userasked 2023/05/30 (Tue) 12:181,797 ctx tokE 0.670

    What is the name of my cat?

    errata
    Luna
    gold
    Luna
    judge
    Candidate states 'Luna', matching reference answer exactly.
    3 citations — session · turn · the span it was read from
    • answer_c6fd8ebd:t10“my cat's name is Luna”
    • answer_c6fd8ebd:t10“my cat's name is Luna”
    • 6c172949_1:t4“I think the resort was called Oakwood Hills Resort”
    full-context Lunanaive Luna
  2. ABSTAINED0862e8bf_abssingle-session-userasked 2023/05/30 (Tue) 21:392,026 ctx tokE 0.520

    What is the name of my hamster?

    errata
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    full-context abstainednaive abstained
  3. ABSTAINED15745da0_abssingle-session-userasked 2023/05/30 (Tue) 23:023,121 ctx tokE 0.681

    How long have I been collecting vintage films?

    errata
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    full-context abstainednaive abstained
  4. CORRECT19b5f2b3single-session-userasked 2023/05/30 (Tue) 20:502,871 ctx tokE 0.511

    How long was I in Japan for?

    errata
    two weeks
    gold
    two weeks
    judge
    Matches reference: two weeks
    3 citations — session · turn · the span it was read from
    • answer_5ff494b9:t2“I spent two weeks traveling solo around the country and it was an incredible experience.”
    • answer_5ff494b9:t4“I stayed in small hostels and guesthouses during my solo trip to Japan”
    • answer_5ff494b9:t2“I spent two weeks traveling solo around the country and it was an incredible experience.”
    full-context Two weeksnaive two weeks
  5. ABSTAINED19b5f2b3_abssingle-session-userasked 2023/05/30 (Tue) 17:143,263 ctx tokE 0.589

    How long was I in Korea for?

    errata
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    full-context abstainednaive abstained
  6. CORRECT21436231single-session-userasked 2023/05/30 (Tue) 19:251,820 ctx tokE 0.541

    How many largemouth bass did I catch on my fishing trip to Lake Michigan?

    errata
    12
    gold
    12
    judge
    Matches quantity 12
    3 citations — session · turn · the span it was read from
    • answer_1e6d4567:t2“I caught 12 largemouth bass on my last trip there”
    • answer_1e6d4567:t2“I caught 12 largemouth bass on my last trip there”
    • answer_1e6d4567:t0“I'm planning a fishing trip to Lake Michigan with my friends”
    full-context 12naive 12
  7. CORRECT29f2956bsingle-session-userasked 2023/05/30 (Tue) 20:453,333 ctx tokE 0.521

    How much time do I dedicate to practicing guitar every day?

    errata
    30 minutes daily
    gold
    30 minutes
    judge
    Both state 30 minutes daily practice
    3 citations — session · turn · the span it was read from
    • answer_7cc5362f:t0“I've been practicing guitar for 30 minutes daily, and it's been helping me progress nicely.”
    • answer_7cc5362f:t0“I've been practicing guitar for 30 minutes daily, and it's been helping me progress nicely.”
    • 00842aad_2:t0“I've been trying to wake up at 7:30 am every weekday”
    full-context 30 minutesnaive 30 minutes daily
  8. ABSTAINED29f2956b_abssingle-session-userasked 2023/05/30 (Tue) 20:282,988 ctx tokE 0.472

    How much time do I dedicate to practicing violin every day?

    errata
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    full-context abstainednaive abstained
  9. CORRECT36580ce8single-session-userasked 2023/05/30 (Tue) 22:092,980 ctx tokE 0.503

    What health issue did I initially think was just a cold?

    errata
    bronchitis
    gold
    bronchitis
    judge
    Matches reference fact exactly: bronchitis
    3 citations — session · turn · the span it was read from
    • answer_93e1bd22:t4“I actually recently had a bad case of bronchitis that I initially thought was just a cold”
    • answer_93e1bd22:t4“I actually recently had a bad case of bronchitis that I initially thought was just a cold”
    • answer_93e1bd22:t0“I'm trying to get back on track with my fitness routine after a few months of dealing with health issues.”
    full-context bronchitisnaive bronchitis
  10. CORRECT577d4d32single-session-userasked 2023/05/30 (Tue) 23:542,087 ctx tokE 0.541

    What time do I stop checking work emails and messages?

    errata
    7 pm
    gold
    7 pm
    judge
    Matches reference fact of 7 pm
    3 citations — session · turn · the span it was read from
    • answer_0dd4d99a:t4“I've been trying to establish a better evening routine, stopping work emails and messages by 7 pm to separate my work and personal life.”
    • answer_0dd4d99a:t4“stopping work emails and messages by 7 pm to separate my work and personal life”
    • b283f9eb_1:t10“I'll make sure to check the data types, handle duplicates, and align the data correctly.”
    full-context 7 pmnaive abstained
  11. CORRECT60d45044single-session-userasked 2023/05/30 (Tue) 20:461,869 ctx tokE 0.628

    What type of rice is my favorite?

    errata
    Japanese short-grain rice
    gold
    Japanese short-grain rice
    judge
    Exact match: Japanese short-grain rice
    3 citations — session · turn · the span it was read from
    • answer_9cddca88:t4“my favorite Japanese short-grain rice”
    • answer_9cddca88:t4“my favorite Japanese short-grain rice”
    • 35a61ccd_1:t0“I'm thinking of renovating an old farmhouse on a plot of land I inherited from my grandfather.”
    full-context Japanese short-grain ricenaive abstained
  12. CORRECT6ade9755single-session-userasked 2023/05/30 (Tue) 21:041,929 ctx tokE 0.696

    Where do I take yoga classes?

    errata
    Serenity Yoga
    gold
    Serenity Yoga
    judge
    Matches reference entity exactly: Serenity Yoga
    3 citations — session · turn · the span it was read from
    • answer_9398da02:t6“I usually stick to the vinyasa flow classes, they're my favorite.”
    • answer_9398da02:t6“I usually stick to the vinyasa flow classes, they're my favorite.”
    • answer_9398da02:t10“We've even started a Sunday morning yoga and brunch routine”
    full-context Serenity Yoganaive Serenity Yoga
  13. CORRECT6b168ec8single-session-userasked 2023/05/30 (Tue) 20:241,769 ctx tokE 0.485

    How many bikes do I own?

    errata
    three
    gold
    three
    judge
    Both state three bikes
    3 citations — session · turn · the span it was read from
    • answer_e623ae87:t10“I think I'll go with the Abus SmartX bike lock with GPS tracking”
    • answer_e623ae87:t4“I've got three of them - a road bike, a mountain bike, and a commuter bike”
    • answer_e623ae87:t2“I'm actually thinking of taking my Trek Emonda for the ride”
    full-context You own three bikes.naive 3
  14. CORRECT8550ddaesingle-session-userasked 2023/05/30 (Tue) 23:512,858 ctx tokE 0.667

    What type of cocktail recipe did I try last weekend?

    errata
    lavender gin fizz
    gold
    lavender gin fizz
    judge
    Matches exactly: lavender gin fizz
    3 citations — session · turn · the span it was read from
    • answer_c8354ae9:t2“I tried a lavender gin fizz recipe last weekend, but it didn't quite turn out as expected”
    • 8ec23b2c:t4“after playing it for 10 hrs last weekend.”
    • 8ec23b2c:t4“after playing it for 10 hrs last weekend”
    full-context Lavender gin fizznaive lavender gin fizz
  15. CORRECT8ebdbe50single-session-userasked 2023/05/30 (Tue) 18:442,922 ctx tokE 0.660

    What certification did I complete last month?

    errata
    Data Science certification
    gold
    Data Science
    judge
    States Data Science certification, matching reference fact
    3 citations — session · turn · the span it was read from
    • answer_8ad8a34f:t2“I need to add my latest certification in Data Science, which I completed last month, to my profile.”
    • answer_8ad8a34f:t2“my latest certification in Data Science, which I completed last month”
    • d4f7a065_2:t0“grilled salmon with roasted vegetables on Saturday”
    full-context Data Sciencenaive Data Science
  16. CORRECTa82c026esingle-session-userasked 2023/05/30 (Tue) 21:262,829 ctx tokE 0.408

    What game did I finally beat last weekend?

    errata
    Dark Souls 3 DLC
    gold
    Dark Souls 3 DLC
    judge
    Matches exactly: Dark Souls 3 DLC
    3 citations — session · turn · the span it was read from
    • answer_787e6a6d:t2“I finally beat that last boss in the Dark Souls 3 DLC last weekend, after weeks of trying.”
    • answer_787e6a6d:t2“I finally beat that last boss in the Dark Souls 3 DLC last weekend, after weeks of trying.”
    • ccaabf0b_1:t0“I've played 8 games of soccer”
    full-context Dark Souls 3 DLCnaive abstained
  17. CORRECTad7109d1single-session-userasked 2023/05/30 (Tue) 21:562,067 ctx tokE 0.651

    What speed is my new internet plan?

    errata
    500 Mbps
    gold
    500 Mbps
    judge
    Matches reference value exactly: 500 Mbps
    3 citations — session · turn · the span it was read from
    • answer_679840f8:t4“I upgraded to 500 Mbps about three weeks ago, and it's made a huge difference.”
    • answer_679840f8:t4“I upgraded to 500 Mbps about three weeks ago”
    • 7ad0bd25_1:t6“I'm thinking of trying out a new energy bar for my upcoming cycling event.”
    full-context 500 Mbpsnaive 500 Mbps
  18. ABSTAINEDbc8a6e93_abssingle-session-userasked 2023/05/30 (Tue) 15:261,898 ctx tokE 0.403

    What did I bake for my uncle's birthday party?

    errata
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    full-context abstainednaive abstained
  19. CORRECTc19f7a0bsingle-session-userasked 2023/05/30 (Tue) 22:191,791 ctx tokE 0.706

    What time do I usually get home from work on weeknights?

    errata
    around 6:30 pm
    gold
    6:30 pm
    judge
    Same time, 6:30 pm, just with 'around'
    3 citations — session · turn · the span it was read from
    • answer_f442ccbe:t8“I usually get home from work around 6:30 pm on weekdays”
    • answer_f442ccbe:t8“I usually get home from work around 6:30 pm on weekdays”
    • answer_f442ccbe:t2“I usually have about an hour to spare between getting home from work and starting dinner”
    full-context 6:30 pmnaive abstained
  20. CORRECTe47becbasingle-session-userasked 2023/05/30 (Tue) 23:401,925 ctx tokE 0.468

    What degree did I graduate with?

    errata
    Business Administration
    gold
    Business Administration
    judge
    Matches exactly: Business Administration
    3 citations — session · turn · the span it was read from
    • answer_280352e9:t4“I graduated with a degree in Business Administration”
    • answer_280352e9:t4“I graduated with a degree in Business Administration”
    • 4fd76316:t10“I think I'll call Smokey's to check their wait times.”
    full-context Business Administrationnaive abstained
  21. ABSTAINEDf4f1d8a4_abssingle-session-userasked 2023/05/30 (Tue) 21:511,792 ctx tokE 0.520

    What did my dad gave me as a birthday gift?

    errata
    ABSTAINED — not in the history
    gold
    ABSTAIN — the history never says
    full-context abstainednaive abstained
  22. CORRECTf8c5f88bsingle-session-userasked 2023/05/30 (Tue) 23:391,579 ctx tokE 0.668

    Where did I buy my new tennis racket from?

    errata
    a sports store downtown
    gold
    the sports store downtown
    judge
    Same fact: sports store downtown, article difference only
    3 citations — session · turn · the span it was read from
    • answer_c3567066:t4“I'm really happy with my new tennis racket, which I got from a sports store downtown.”
    • answer_c3567066:t4“It's been performing really well, and I feel like it's improved my game.”
    • answer_c3567066:t4“I'm really happy with my new tennis racket, which I got from a sports store downtown.”
    full-context a sports store downtownnaive a sports store downtown