results · Errata · rerunJ-arith
Multi-session 67.7
63 of 93 rows right — 31 questions × 3 seeds (11/22/33)accuracy over the non-abstention questions of this ability; mean ± sd across seeds 11/22/33same arm, by ability: OverallInformation extractionTemporalKnowledge updateAbstentionsame arm, by question_type: single-session-usersingle-session-assistantsingle-session-preferencemulti-sessiontemporal-reasoningknowledge-updatethe same cut, other arms: Full-context baseline 35.5Naive top-k RAG (k=10) 29.0
- INCORRECT0a995998multi-sessionasked 2023/02/15 (Wed) 23:501,870 ctx tokE 0.649
How many items of clothing do I need to pick up or return from a store?
- errata
- 1
- gold
- 3
- judge
- Reference states 3 items, candidate states 1
3 citations — session · turn · the span it was read from
- answer_afa9873b_3:t6“I got them on February 5th, but they were too small, so I exchanged them for a larger size.”
- answer_afa9873b_2:t0“I still need to pick up my dry cleaning for the navy blue blazer I wore to a meeting a few weeks ago.”
- answer_afa9873b_2:t0“I still need to pick up my dry cleaning for the navy blue blazer I wore to a meeting a few weeks ago.”
- INCORRECT2788b940multi-sessionasked 2023/05/30 (Tue) 20:002,013 ctx tokE 0.507
How many fitness classes do I attend in a typical week?
- errata
- 4
- gold
- 5
- judge
- Reference states 5 classes, candidate states 4
3 citations — session · turn · the span it was read from
- answer_8f6b938d_2:t6“I attend Hip Hop Abs on Saturdays at 10:00 AM”
- answer_8f6b938d_1:t10“I try to mix up my workout routine by attending different classes like Zumba, Hip Hop Abs, yoga, and BodyPump.”
- answer_8f6b938d_4:t0“I need something to motivate me during my weightlifting classes, like BodyPump on Mondays at 6:30 PM.”
- INCORRECT2b8f3739multi-sessionasked 2023/06/01 (Thu) 23:372,231 ctx tokE 0.470
What is the total amount of money I earned from selling my products at the markets?
- errata
- $465
- gold
- $495
- judge
- Candidate states $465, reference states $495
3 citations — session · turn · the span it was read from
- answer_23759615_2:t0“earning $225.”
- answer_23759615_1:t0“sold 12 bunches of fresh organic herbs from my backyard garden at the farmers' market on May 15th, earning a total of $120.”
- answer_23759615_2:t0“I just sold 15 jars of my homemade jam at the Homemade and Handmade Market on May 29th, earning $225.”
- INCORRECT2e6d26dcmulti-sessionasked 2023/05/13 (Sat) 23:552,731 ctx tokE 0.529
How many babies were born to friends and family members in the last few months?
- errata
- 4
- gold
- 5
- judge
- Reference states 5 babies, candidate states 4
3 citations — session · turn · the span it was read from
- answer_fa526fc0_4:t0“my friend from college, David, had a baby boy named Jasper a few weeks ago”
- answer_fa526fc0_1:t0“I'm trying to plan a baby gift for a friend's upcoming baby shower. I was thinking of getting a personalized blanket”
- answer_fa526fc0_1:t0“I'm trying to plan a baby gift for a friend's upcoming baby shower.”
- ABSTAINED60159905multi-sessionasked 2023/05/30 (Tue) 14:401,907 ctx tokE 0.543
How many dinner parties have I attended in the past month?
- errata
- ABSTAINED — not in the history
- gold
- three
- judge
- No answer provided
- ABSTAINED7024f17cmulti-sessionasked 2023/05/30 (Tue) 21:242,735 ctx tokE 0.507
How many hours of jogging and yoga did I do last week?
- errata
- ABSTAINED — not in the history
- gold
- 0.5 hours
- judge
- Candidate answer is empty, no value provided
- ABSTAINED92a0aa75multi-sessionasked 2023/05/30 (Tue) 21:252,875 ctx tokE 0.552
How long have I been working in my current role?
- errata
- ABSTAINED — not in the history
- gold
- 1 year and 5 months
- judge
- No answer provided
- ABSTAINEDa1cc6108multi-sessionasked 2023/05/30 (Tue) 20:233,004 ctx tokE 0.652
How old was I when Alex was born?
- errata
- ABSTAINED — not in the history
- gold
- 11
- judge
- Candidate provided no answer
- INCORRECTbc149d6bmulti-sessionasked 2023/05/30 (Tue) 22:021,976 ctx tokE 0.538
What is the total weight of the new feed I purchased in the past two months?
- errata
- 50 pounds
- gold
- 70 pounds
- judge
- Reference states 70 pounds, candidate states 50 pounds
3 citations — session · turn · the span it was read from
- answer_92147866_1:t0“I got a 50-pound batch”
- 86935301_2:t8“I've been listening to the Hamilton soundtrack nonstop on my daily commute for the past two weeks.”
- 86935301_2:t8“I've been listening to the Hamilton soundtrack nonstop on my daily commute for the past two weeks.”
- INCORRECTgpt4_59c863d7multi-sessionasked 2023/05/30 (Tue) 20:571,801 ctx tokE 0.534
How many model kits have I worked on or bought?
- errata
- 6
- gold
- I have worked on or bought five model kits. The scales of the models are: Revell F-15 Eagle (scale not mentioned), Tamiya 1/48 scale Spitfire Mk.V, 1/16 scale German Tiger I tank, 1/72 scale B-29 bomber, and 1/24 scale '69 Camaro.
- judge
- Reference states five model kits, candidate says six
3 citations — session · turn · the span it was read from
- answer_593bdffd_4:t0“I just got this kit and a 1/24 scale '69 Camaro at a model show last weekend.”
- answer_593bdffd_4:t10“I'm planning to start working on the engine wiring and fuel lines for my '69 Camaro model.”
- answer_593bdffd_4:t8“I've been thinking about adding some wire details to my 1/24 scale '69 Camaro model”
- CORRECT078150f1multi-sessionasked 2023/05/30 (Tue) 23:363,049 ctx tokE 0.543
How much more money did I raise than my initial goal in the charity cycling event?
- errata
- $50
- gold
- $50
- judge
- Matches the $50 amount stated in reference
3 citations — session · turn · the span it was read from
- answer_254d8b09_1:t8“I initially aimed to raise $200 in donations for the local children's hospital.”
- answer_254d8b09_1:t8“I initially aimed to raise $200 in donations for the local children's hospital.”
- answer_254d8b09_1:t8“I recently participated in a charity cycling event”
- CORRECT099778bbmulti-sessionasked 2023/05/30 (Tue) 22:262,111 ctx tokE 0.600
What percentage of leadership positions do women hold in the my company?
- errata
- 20%
- gold
- 20%
- judge
- Matches reference value of 20%
3 citations — session · turn · the span it was read from
- answer_80d6d664_1:t0“we have a total of 100 leadership positions across the company”
- answer_80d6d664_2:t2“I was impressed to learn that women occupy 20 of the leadership positions in our company.”
- answer_80d6d664_1:t7“4. **Korn Ferry's Global Leadership Forecast**: Korn Ferry's report includes data on the representation of women in leadership positions”
- CORRECT129d1232multi-sessionasked 2023/05/01 (Mon) 23:452,303 ctx tokE 0.511
How much money did I raise in total through all the charity events I participated in?
- errata
- $5,850
- gold
- $5,850
- judge
- Matches exact amount $5,850
3 citations — session · turn · the span it was read from
- answer_1de862d6_1:t2“I recently participated in a charity walk and managed to raise $250 through sponsors”
- answer_1de862d6_2:t2“I recently participated in a Bike-a-Thon for Cancer Research”
- answer_1de862d6_1:t2“managed to raise $250 through sponsors”
- CORRECT21d02d0dmulti-sessionasked 2023/04/26 (Wed) 23:582,086 ctx tokE 0.602
How many fun runs did I miss in March due to work commitments?
- errata
- 2
- gold
- 2
- judge
- Matches the reference quantity of 2 fun runs missed.
3 citations — session · turn · the span it was read from
- answer_2c637141_1:t0“except for the run on March 5th when I had to miss due to work commitments”
- answer_2c637141_1:t0“I had to miss due to work commitments.”
- answer_2c637141_2:t0“I've been pretty busy with work lately and missed a few events, including a 5K fun run on March 26th.”
- CORRECT3a704032multi-sessionasked 2023/05/30 (Tue) 21:512,853 ctx tokE 0.533
How many plants did I acquire in the last month?
- errata
- 3
- gold
- 3
- judge
- Matches quantity of 3 plants
3 citations — session · turn · the span it was read from
- answer_c2204106_1:t4“my snake plant, which I got from my sister last month”
- aa41ac47:t8“The last show I watched on it was "Game of Thrones"”
- answer_c2204106_1:t4“my snake plant, which I got from my sister last month”
- CORRECT55241a1fmulti-sessionasked 2023/05/30 (Tue) 21:571,802 ctx tokE 0.626
What is the total number of comments on my recent Facebook Live session and my most popular YouTube video?
- errata
- 33
- gold
- 33
- judge
- Matches the reference total of 33 comments
3 citations — session · turn · the span it was read from
- answer_fa08bf49_2:t4“My most popular video has 21 comments”
- answer_fa08bf49_1:t0“my recent Facebook Live session about cooking vegan recipes”
- answer_fa08bf49_1:t0“which got 12 comments”
- CORRECT60bf93edmulti-sessionasked 2023/01/24 (Tue) 23:562,659 ctx tokE 0.514
How many days did it take for my laptop backpack to arrive after I bought it?
- errata
- 5
- gold
- 5 days. 6 days (including the last day) is also acceptable.
- judge
- Candidate states 5 days, matching reference
3 citations — session · turn · the span it was read from
- 7d8eecd9:t1“Two witnesses prophesy for 1,260 days, calling people to repentance, before being killed and resurrected.”
- answer_e0956e0a_1:t2“I've been using it to carry my work laptop to and from the office every day.”
- answer_e0956e0a_1:t2“I've been using it to carry my work laptop to and from the office every day.”
- CORRECT77eafa52multi-sessionasked 2023/05/30 (Tue) 20:072,968 ctx tokE 0.491
How much more did I have to pay for the trip after the initial quote?
- errata
- $300
- gold
- $300
- judge
- Matches the $300 additional payment fact exactly
3 citations — session · turn · the span it was read from
- answer_33c251f0_1:t2“They initially quoted me $2,500 for the entire trip”
- answer_33c251f0_1:t2“They initially quoted me $2,500 for the entire trip”
- answer_33c251f0_2:t0“the corrected price for the entire trip was $2,800”
- CORRECT80ec1f4fmulti-sessionasked 2023/03/03 (Fri) 23:251,869 ctx tokE 0.578
How many different museums or galleries did I visit in the month of February?
- errata
- 2
- gold
- 2
- judge
- Matches quantity of 2 museums/galleries
3 citations — session · turn · the span it was read from
- answer_990c8992_1:t0“I took my niece to the Natural History Museum on 2/8 and she loved the dinosaur exhibit!”
- answer_990c8992_1:t0“I took my niece to the Natural History Museum on 2/8 and she loved the dinosaur exhibit!”
- answer_990c8992_1:t8“I recently visited the Natural History Museum with my niece”
- CORRECT85fa3a3fmulti-sessionasked 2023/05/30 (Tue) 19:552,293 ctx tokE 0.516
What is the total cost of the new food bowl, measuring cup, dental chews, and flea and tick collar I got for Max?
- errata
- $50
- gold
- $50
- judge
- Matches reference value of $50
3 citations — session · turn · the span it was read from
- answer_dcb18b9b_2:t6“I also got a flea and tick collar for Max recently, which was $20, but that's also a one-time expense.”
- answer_dcb18b9b_2:t6“a flea and tick collar for Max recently, which was $20”
- answer_dcb18b9b_2:t4“I buy the dental chews every month, one pack at a time.”
- CORRECTa08a253fmulti-sessionasked 2023/05/30 (Tue) 23:163,164 ctx tokE 0.596
How many days a week do I attend fitness classes?
- errata
- 4
- gold
- 4 days.
- judge
- 4 matches the reference quantity of 4 days
3 citations — session · turn · the span it was read from
- answer_47152166_1:t0“I attend Zumba classes on Tuesdays and Thursdays at 6:30 pm, and a weightlifting class on Saturdays at 10 am.”
- answer_47152166_2:t10“I'm looking forward to seeing how they can help with my flexibility and overall fitness.”
- b1d9eb66_4:t2“I'm really interested in the triathlon option.”
- CORRECTa346bb18multi-sessionasked 2023/05/30 (Tue) 22:301,691 ctx tokE 0.664
How many minutes did I exceed my target time by in the marathon?
- errata
- 12
- gold
- 12
- judge
- Matches reference value of 12 minutes
3 citations — session · turn · the span it was read from
- answer_4934b2d7_2:t0“my target time for the marathon was 4 hours and 10 minutes”
- answer_4934b2d7_2:t0“my target time for the marathon was 4 hours and 10 minutes”
- answer_4934b2d7_1:t0“I just completed my first full marathon in 4h 22min”
- CORRECTa9f6b44cmulti-sessionasked 2023/03/20 (Mon) 23:571,820 ctx tokE 0.662
How many bikes did I service or plan to service in March?
- errata
- 2
- gold
- 2
- judge
- Matches the quantity of 2 bikes
3 citations — session · turn · the span it was read from
- answer_cc021f81_1:t6“I got my road bike serviced at Pedal Power on March 10th”
- answer_cc021f81_1:t6“I got my road bike serviced at Pedal Power on March 10th”
- answer_cc021f81_1:t0“getting my road bike serviced at Pedal Power on March 10th”
- CORRECTb3c15d39multi-sessionasked 2023/03/01 (Wed) 23:483,055 ctx tokE 0.588
How many days did it take for me to receive the new remote shutter release after I ordered it?
- errata
- 5
- gold
- 5 days. 6 days (including the last day) is also acceptable.
- judge
- 5 days matches accepted reference value
3 citations — session · turn · the span it was read from
- answer_05d808e6_2:t0“I just got a new remote shutter release that arrived on February 10th”
- answer_05d808e6_2:t10“I got a new remote shutter release that arrived on February 10th”
- answer_05d808e6_2:t10“my new remote shutter release that arrived on February 10th”
- CORRECTd3ab962emulti-sessionasked 2022/09/24 (Sat) 23:491,911 ctx tokE 0.397
What is the total distance of the hikes I did on two consecutive weekends?
- errata
- 8 miles
- gold
- 8 miles
- judge
- Matches reference value of 8 miles exactly
3 citations — session · turn · the span it was read from
- answer_5237bb0b_1:t0“I just got back from an amazing 5-mile hike at Red Rock Canyon two weekends ago”
- answer_5237bb0b_1:t0“I just got back from an amazing 5-mile hike at Red Rock Canyon two weekends ago”
- answer_5237bb0b_2:t0“just did a 3-mile loop trail at Valley of Fire State Park last weekend.”
- CORRECTd851d5bamulti-sessionasked 2023/03/20 (Mon) 23:591,787 ctx tokE 0.578
How much money did I raise for charity in total?
- errata
- $3,750
- gold
- $3,750
- judge
- Matches the $3,750 figure exactly
3 citations — session · turn · the span it was read from
- answer_5cdf9bd2_2:t0“we raised $1,000 for the local children's hospital!”
- answer_5cdf9bd2_1:t6“I'm thinking of raising money for a local food bank again”
- answer_5cdf9bd2_4:t0“I helped raise $2,000 for a local animal shelter on January 20th”
- CORRECTe25c3b8dmulti-sessionasked 2023/05/30 (Tue) 15:021,864 ctx tokE 0.540
How much did I save on the designer handbag at TK Maxx?
- errata
- 300 USD
- gold
- $300
- judge
- 300 USD matches $300
3 citations — session · turn · the span it was read from
- answer_6702277b_2:t6“that designer handbag I got for $200”
- answer_6702277b_2:t6“I've had luck finding great deals at TK Maxx before, like that designer handbag I got for $200.”
- answer_6702277b_2:t0“I've been really loving the handbag I got from TK Maxx, it's been my go-to bag lately.”
- CORRECTe831120cmulti-sessionasked 2023/05/30 (Tue) 18:413,397 ctx tokE 0.575
How many weeks did it take me to watch all the Marvel Cinematic Universe movies and the main Star Wars films?
- errata
- 3.5
- gold
- 3.5 weeks
- judge
- 3.5 matches 3.5 weeks, same quantity
3 citations — session · turn · the span it was read from
- answer_86c505e7_1:t0“I've had some crazy movie binges lately, like when I watched all 22 Marvel Cinematic Universe movies in two weeks.”
- answer_86c505e7_2:t2“so add those too”
- answer_86c505e7_2:t0“I just finished a Star Wars marathon, watched all the main films in a week and a half”
- CORRECTef9cf60amulti-sessionasked 2023/05/30 (Tue) 22:482,168 ctx tokE 0.558
How much did I spend on gifts for my sister?
- errata
- $300
- gold
- $300
- judge
- Matches the reference value of $300
3 citations — session · turn · the span it was read from
- answer_87e3a1cb_1:t6“I recently got my sister a silver necklace with a small pendant from Tiffany's that cost around $200”
- answer_87e3a1cb_1:t0“Can you give me some ideas for a nice gift in the $200 range?”
- answer_87e3a1cb_2:t0“I'm trying to get some ideas for my sister's birthday gift next year... I was thinking of getting her a spa day.”
- CORRECTefc3f7c2multi-sessionasked 2023/05/30 (Tue) 16:242,775 ctx tokE 0.574
How much earlier do I wake up on Fridays compared to other weekdays?
- errata
- 30 minutes
- gold
- 30 minutes
- judge
- Matches reference value of 30 minutes
3 citations — session · turn · the span it was read from
- answer_fc81d1a2_1:t0“I usually do them right after waking up at 6:30 AM on weekdays.”
- answer_fc81d1a2_2:t0“on Fridays, I like to get a head start, so I wake up at 6:00 AM.”
- answer_fc81d1a2_2:t0“By the way, on Fridays, I like to get a head start, so I wake up at 6:00 AM.”
- CORRECTgpt4_2f91af09multi-sessionasked 2023/05/30 (Tue) 17:142,711 ctx tokE 0.536
How many total pieces of writing have I completed since I started writing again three weeks ago, including short stories, poems, and pieces for the writing challenge?
- errata
- 23
- gold
- 23
- judge
- Candidate states 23, matching reference quantity.
3 citations — session · turn · the span it was read from
- answer_669318cf_3:t0“Last week's prompt was "forgotten memories", and I wrote a piece titled "The Smell of Old Books".”
- answer_669318cf_3:t0“I've also just started participating in a writing challenge on a social media platform, where we're given a prompt every week”
- answer_669318cf_3:t6“I'll start by setting up notebooks in Evernote for each of my short stories and poetry collections”