results · Errata · rerunJ-arith
Knowledge update 94.4
51 of 54 rows right — 18 questions × 3 seeds (11/22/33)accuracy over the non-abstention questions of this ability; mean ± sd across seeds 11/22/33same arm, by ability: OverallInformation extractionMulti-sessionTemporalAbstentionsame arm, by question_type: single-session-usersingle-session-assistantsingle-session-preferencemulti-sessiontemporal-reasoningknowledge-updatethe same cut, other arms: Full-context baseline 61.1Naive top-k RAG (k=10) 83.3
- INCORRECTa2f3aa27knowledge-updateasked 2023/06/15 (Thu) 23:261,814 ctx tokE 0.595
How many followers do I have on Instagram now?
- errata
- 1250
- gold
- 1300
- judge
- Reference states 1300 followers, candidate states 1250
3 citations — session · turn · the span it was read from
- answer_5126c02d_1:t2“I've got 1250 followers on Instagram now”
- answer_5126c02d_1:t2“I've got 1250 followers on Instagram now”
- answer_5126c02d_2:t0“I think I'm close to 1300 now.”
- CORRECT031748aeknowledge-updateasked 2023/11/10 (Fri) 04:202,860 ctx tokE 0.632
How many engineers do I lead when I just started my new role as Senior Software Engineer? How many engineers do I lead now?
- errata
- 4 engineers; five engineers
- gold
- When you just started your new role as Senior Software Engineer, you led 4 engineers. Now, you lead 5 engineers
- judge
- 4 engineers initially, 5 (five) engineers now — matches reference
3 citations — session · turn · the span it was read from
- answer_8748f791_1:t8“how many engineers do I lead in my new role as Senior Software Engineer?”
- answer_8748f791_1:t10“I lead a team of 4 engineers in my new role as Senior Software Engineer.”
- answer_8748f791_2:t2“I've been enjoying my role as Senior Software Engineer for a while”
- CORRECT0ddfec37knowledge-updateasked 2024/01/08 (Mon) 09:272,863 ctx tokE 0.445
How many autographed baseballs have I added to my collection in the first three months of collection?
- errata
- 15
- gold
- 15
- judge
- Matches the reference quantity of 15
3 citations — session · turn · the span it was read from
- answer_a22b654d_2:t8“I'm thinking of selling some of my autographed baseballs to make room for new additions to my collection.”
- answer_a22b654d_2:t0“I've added 20 autographed baseballs to my collection in the past few months”
- answer_a22b654d_2:t0“I've added 20 autographed baseballs to my collection in the past few months”
- CORRECT0f05491aknowledge-updateasked 2023/08/01 (Tue) 22:292,061 ctx tokE 0.637
How many stars do I need to reach the gold level on my Starbucks Rewards app?
- errata
- 120 stars
- gold
- 120
- judge
- Both state 120 stars for gold level
3 citations — session · turn · the span it was read from
- answer_d6d2eba8_1:t11“to reach the Gold level on the Starbucks Rewards app, you need to earn a total of 125 stars within a 12-month period.”
- answer_d6d2eba8_1:t10“I actually need **125 stars** to reach the Gold level on the Starbucks Rewards app, not 400.”
- answer_d6d2eba8_2:t6“Actually, I need 120 stars to reach the gold level, not 300.”
- CORRECT10e09553knowledge-updateasked 2023/12/06 (Wed) 11:182,607 ctx tokE 0.610
How many largemouth bass did I catch with Alex on the earlier fishing trip to Lake Michigan before the 7/22 trip?
- errata
- 7
- gold
- 7
- judge
- Matches quantity of 7 largemouth bass
3 citations — session · turn · the span it was read from
- answer_67be2c38_1:t2“I caught 7 largemouth bass on my trip to Lake Michigan with Alex on 7/10”
- answer_67be2c38_1:t2“I caught 7 largemouth bass on my trip to Lake Michigan with Alex on 7/10”
- answer_67be2c38_1:t2“I caught 7 largemouth bass on my trip to Lake Michigan with Alex on 7/10”
- CORRECT26bdc477knowledge-updateasked 2023/06/06 (Tue) 18:271,818 ctx tokE 0.638
How many trips have I taken my Canon EOS 80D camera on?
- errata
- five trips
- gold
- five
- judge
- Both state five trips
3 citations — session · turn · the span it was read from
- answer_f762ad8d_2:t0“I've had my Canon EOS 80D with me on five trips now, and it's been a beast!”
- answer_f762ad8d_1:t2“including three trips to Yellowstone, Yosemite, and the Grand Canyon”
- answer_f762ad8d_2:t0“I've had my Canon EOS 80D with me on five trips now”
- CORRECT3ba21379knowledge-updateasked 2023/06/01 (Thu) 05:093,044 ctx tokE 0.538
What type of vehicle model am I currently working on?
- errata
- Ford F-150 pickup truck
- gold
- Ford F-150 pickup truck
- judge
- Exact match: Ford F-150 pickup truck
3 citations — session · turn · the span it was read from
- answer_cd345582_2:t2“I have just wrapped up a model and switched to a Ford F-150 pickup truck.”
- answer_cd345582_2:t6“I'm thinking of adding some weathering effects to the underside of my Ford F-150 pickup truck model”
- answer_cd345582_1:t4“my current Ford Mustang Shelby GT350R project”
- CORRECT41698283knowledge-updateasked 2023/09/21 (Thu) 08:411,836 ctx tokE 0.555
What type of camera lens did I purchase most recently?
- errata
- 70-200mm zoom lens
- gold
- a 70-200mm zoom lens
- judge
- Same fact: 70-200mm zoom lens
3 citations — session · turn · the span it was read from
- answer_c7ddc051_1:t2“I'm mostly using my camera for portrait and low-light photography”
- answer_c7ddc051_2:t8“I recently took some great shots with my 70-200mm zoom lens at a local park”
- answer_c7ddc051_1:t2“I recently got a new 50mm prime lens, which has been working out great.”
- CORRECT4b24c848knowledge-updateasked 2023/10/20 (Fri) 08:092,991 ctx tokE 0.452
How many tops have I bought from H&M so far?
- errata
- five tops
- gold
- five
- judge
- Same quantity, five tops matches five
3 citations — session · turn · the span it was read from
- answer_2cec623b_1:t4“I've already bought three tops from H&M”
- answer_2cec623b_2:t0“I've already got five tops from H&M so far”
- answer_2cec623b_2:t0“I've already got five tops from H&M so far”
- CORRECT6aeb4375knowledge-updateasked 2023/10/22 (Sun) 15:381,913 ctx tokE 0.634
How many Korean restaurants have I tried in my city?
- errata
- four
- gold
- four
- judge
- Matches quantity 'four' as stated in reference.
3 citations — session · turn · the span it was read from
- answer_3f9693b7_2:t2“I've tried four different ones so far”
- answer_3f9693b7_2:t2“I've tried four different ones so far”
- answer_3f9693b7_1:t8“I've tried three different ones recently, and each has its own unique flavor and style.”
- CORRECT7e974930knowledge-updateasked 2023/10/26 (Thu) 11:443,202 ctx tokE 0.482
How much did I earn at the Downtown Farmers Market on my most recent visit?
- errata
- $420
- gold
- $420
- judge
- Matches the $420 figure exactly.
3 citations — session · turn · the span it was read from
- answer_c9f5693c_2:t6“I earned $420 at the Downtown Farmers Market on my most recent visit.”
- answer_c9f5693c_2:t6“I earned $420 at the Downtown Farmers Market on my most recent visit.”
- answer_c9f5693c_1:t3“So, your total earnings from all the markets and events you've participated in so far this year are $1280.”
- CORRECT945e3d21knowledge-updateasked 2023/12/14 (Thu) 06:321,892 ctx tokE 0.595
How often do I attend yoga classes to help with my anxiety?
- errata
- three times a week
- gold
- Three times a week.
- judge
- Matches reference: three times a week.
3 citations — session · turn · the span it was read from
- answer_6a4f8626_2:t2“attend yoga classes, which is three times a week”
- answer_6a4f8626_2:t2“yoga classes, which is three times a week”
- answer_6a4f8626_1:t4“I've been doing yoga twice a week, which has really been helping me relax and focus.”
- CORRECT9ea5eabcknowledge-updateasked 2023/06/08 (Thu) 10:333,101 ctx tokE 0.731
Where did I go on my most recent family trip?
- errata
- Paris
- gold
- Paris
- judge
- Candidate states Paris, matching the reference fact
3 citations — session · turn · the span it was read from
- answer_02e66dec_1:t2“we had a great time snorkeling together”
- answer_02e66dec_1:t2“I was thinking about my recent family trip to Hawaii”
- answer_02e66dec_2:t6“since I recently went to Paris with my family”
- CORRECTb01defabknowledge-updateasked 2023/04/12 (Wed) 06:371,948 ctx tokE 0.663
Did I finish reading 'The Nightingale' by Kristin Hannah?
- errata
- Yes
- gold
- Yes
- judge
- Both affirm 'Yes'
3 citations — session · turn · the span it was read from
- answer_8c0712af_2:t8“"The Nightingale" by Kristin Hannah, which I finished reading recently.”
- answer_8c0712af_2:t0“I just finished reading "The Seven Husbands of Evelyn Hugo". I also recently finished "The Nightingale" by Kristin Hannah”
- answer_8c0712af_2:t0“I also recently finished "The Nightingale" by Kristin Hannah”
- CORRECTc4ea545cknowledge-updateasked 2023/09/01 (Fri) 19:151,862 ctx tokE 0.596
Do I go to the gym more frequently than I did previously?
- errata
- Yes, the user now works out four times a week, whereas previously they went to the gym on Tuesdays, Thursdays, and Saturdays (three days a week).
- gold
- Yes
- judge
- Both affirm yes, with matching increased frequency detail
3 citations — session · turn · the span it was read from
- answer_d3bf812b_1:t4“I go to the gym on Tuesdays, Thursdays, and Saturdays.”
- answer_d3bf812b_1:t4“I go to the gym on Tuesdays, Thursdays, and Saturdays.”
- answer_d3bf812b_2:t0“I've been consistent with my gym routine - four times a week, actually.”
- CORRECTcc5ded98knowledge-updateasked 2023/06/13 (Tue) 01:303,277 ctx tokE 0.646
How much time do I dedicate to coding exercises each day?
- errata
- about two hours per day
- gold
- about two hours
- judge
- Same quantity: about two hours per day
3 citations — session · turn · the span it was read from
- answer_a5b68517_2:t0“I've been dedicating about two hours each day to coding exercises”
- answer_a5b68517_2:t0“I've been dedicating about two hours each day to coding exercises”
- answer_a5b68517_2:t6“I've been dedicating about two hours each day to coding exercises”
- CORRECTdad224aaknowledge-updateasked 2023/06/24 (Sat) 03:581,992 ctx tokE 0.640
What time do I wake up on Saturday mornings?
- errata
- 7:30 am
- gold
- 7:30 am
- judge
- Matches reference: 7:30 am
3 citations — session · turn · the span it was read from
- answer_4a97ae40_1:t2“I've been waking up around 8:30 am on Saturdays”
- answer_4a97ae40_2:t2“I like to wake up at 7:30 am on Saturdays”
- answer_4a97ae40_1:t4“I'm already waking up at 8:30 am on Saturdays”
- CORRECTeace081bknowledge-updateasked 2023/06/06 (Tue) 04:181,571 ctx tokE 0.623
Where am I planning to stay for my birthday trip to Hawaii?
- errata
- Oahu
- gold
- Oahu
- judge
- Matches reference exactly: Oahu
3 citations — session · turn · the span it was read from
- answer_8a791264_2:t0“I'm planning a birthday trip to Hawaii in October”
- answer_8a791264_2:t0“I'm planning a birthday trip to Hawaii in October”
- 4d04b866:t4“stopped at 5 different hotels along the way”