results · Errata · rerunJ-arith
temporal-reasoning 59.0
69 of 117 rows right — 39 questions × 3 seeds (11/22/33)the corpus’s own question_type, over all 450 rows — a gold-abstention row is right iff the arm abstained, every other row iff the judge said CORRECTsame arm, by ability: OverallInformation extractionMulti-sessionTemporalKnowledge updateAbstentionsame arm, by question_type: single-session-usersingle-session-assistantsingle-session-preferencemulti-sessionknowledge-updatethe same cut, other arms: Full-context baseline 23.9Naive top-k RAG (k=10) 33.3
- INCORRECT0bc8ad93temporal-reasoningasked 2023/03/11 (Sat) 05:281,925 ctx tokE 0.511
I mentioned visiting a museum two months ago. Did I visit with a friend or not?
- errata
- You visited with a friend.
- gold
- No, you did not visit with a friend.
- judge
- Reference says no friend, candidate says yes friend
3 citations — session · turn · the span it was read from
- answer_f4ea84fc_1:t0“I attended a guided tour at the Natural History Museum yesterday with my dad”
- answer_f4ea84fc_3:t0“my recent visit to the Science Museum ... during the behind-the-scenes tour I just took with a friend who's a chemistry professor”
- answer_f4ea84fc_3:t6“my recent visits to various museums in the past few months”
- INCORRECT370a8ff4temporal-reasoningasked 2023/10/15 (Sun) 17:532,812 ctx tokE 0.472
How many weeks had passed since I recovered from the flu when I went on my 10th jog outdoors?
- errata
- 10
- gold
- 15
- judge
- Reference says 15 weeks, candidate says 10
3 citations — session · turn · the span it was read from
- answer_61d1be50_1:t0“I'm feeling much better now that I finally recovered from the flu today”
- answer_61d1be50_2:t0“I went on my 10th jog outdoors today”
- answer_61d1be50_2:t0“I went on my 10th jog outdoors today”
- INCORRECT6e984301temporal-reasoningasked 2023/04/01 (Sat) 18:553,094 ctx tokE 0.677
How many weeks have I been taking sculpting classes when I invested in my own set of sculpting tools?
- errata
- The material does not contain the information needed to answer.
- gold
- 3
- judge
- Candidate declined to answer instead of stating 3 weeks
3 citations — session · turn · the span it was read from
- answer_88841f26_2:t2“I actually got my own set of sculpting tools today, including a modeling tool set, a wire cutter, and a sculpting mat.”
- answer_88841f26_1:t0“I just started taking sculpting classes at a local art studio today, every Saturday morning from 10 am to 1 pm”
- answer_88841f26_2:t2“I actually got my own set of sculpting tools today, including a modeling tool set, a wire cutter, and a sculpting mat”
- INCORRECT982b5123temporal-reasoningasked 2023/05/21 (Sun) 10:303,089 ctx tokE 0.572
How many months ago did I book the Airbnb in San Francisco?
- errata
- three months
- gold
- Five months ago
- judge
- Reference states five months, candidate says three months
3 citations — session · turn · the span it was read from
- answer_ab603dd5_2:t0“I've been to SF before, exactly two months ago, for my best friend's wedding - it was a 5-day trip”
- 07ba9acd_2:t8“on my flight to San Francisco two weeks ago when I got stuck with a delayed flight... in the airport for hours”
- answer_ab603dd5_1:t0“had to book three months in advance”
- ABSTAINED9a707b81temporal-reasoningasked 2022/04/15 (Fri) 18:463,111 ctx tokE 0.530
How many days ago did I attend a baking class at a local culinary school when I made my friend's birthday cake?
- errata
- ABSTAINED — not in the history
- gold
- 21 days. 22 days (including the last day) is also acceptable.
- judge
- No answer provided
- INCORRECTa3838d2btemporal-reasoningasked 2023/11/29 (Wed) 03:313,276 ctx tokE 0.698
How many charity events did I participate in before the 'Run for the Cure' event?
- errata
- 2
- gold
- 4
- judge
- Reference states 4 events, candidate states 2
3 citations — session · turn · the span it was read from
- answer_4ffa04a2_1:t0“I just ran 5 kilometers in the "Run for the Cure" event on October 15th”
- answer_4ffa04a2_1:t0“I just ran 5 kilometers in the "Run for the Cure" event on October 15th”
- answer_4ffa04a2_1:t6“I just got back from the "Run for the Cure" event on October 15th, where I ran 5 kilometers to raise money for breast cancer research.”
- ABSTAINEDcc6d1ec1temporal-reasoningasked 2023/05/21 (Sun) 19:312,829 ctx tokE 0.784
How long had I been bird watching when I attended the bird watching workshop?
- errata
- ABSTAINED — not in the history
- gold
- Two months
- judge
- No answer provided
- ABSTAINEDeac54addtemporal-reasoningasked 2023/03/28 (Tue) 20:351,658 ctx tokE 0.440
What was the significant buisiness milestone I mentioned four weeks ago?
- errata
- ABSTAINED — not in the history
- gold
- I signed a contract with my first client.
- judge
- Candidate answer is empty, missing the stated fact
- ABSTAINEDgpt4_1d80365etemporal-reasoningasked 2023/06/01 (Thu) 17:173,003 ctx tokE 0.474
How many days did I spend on my solo camping trip to Yosemite National Park?
- errata
- ABSTAINED — not in the history
- gold
- 2 days. 3 days (including the last day) is also acceptable.
- judge
- No answer provided
- ABSTAINEDgpt4_468eb063temporal-reasoningasked 2023/04/20 (Thu) 10:123,249 ctx tokE 0.595
How many days ago did I meet Emma?
- errata
- ABSTAINED — not in the history
- gold
- 9 days ago. 10 days (including the last day) is also acceptable.
- judge
- Candidate answer is empty, provides no fact
- ABSTAINEDgpt4_4929293btemporal-reasoningasked 2023/06/22 (Thu) 18:332,096 ctx tokE 0.494
What was the the life event of one of my relatives that I participated in a week ago?
- errata
- ABSTAINED — not in the history
- gold
- my cousin's wedding
- judge
- Candidate provided no answer
- INCORRECTgpt4_59149c78temporal-reasoningasked 2023/02/01 (Wed) 08:411,808 ctx tokE 0.485
I mentioned that I participated in an art-related event two weeks ago. Where was that event held at?
- errata
- City Art Museum
- gold
- The Metropolitan Museum of Art.
- judge
- Different venue name: City Art Museum vs Metropolitan Museum of Art
3 citations — session · turn · the span it was read from
- 23754665:t6“I remember attending the "Impressionist Masterpieces" exhibition at the City Art Museum on a Saturday”
- 23754665:t11“Register for the museum's newsletter or email list to receive updates on upcoming exhibitions, events, and special programs.”
- 74bcc8f4_1:t4“I love roasting a big batch on Sundays to munch on throughout the week.”
- ABSTAINEDgpt4_61e13b3ctemporal-reasoningasked 2023/04/01 (Sat) 00:573,002 ctx tokE 0.581
How many weeks passed between the time I sold homemade baked goods at the Farmers' Market for the last time and the time I participated in the Spring Fling Market?
- errata
- ABSTAINED — not in the history
- gold
- 3 weeks
- judge
- Candidate provided no answer
- INCORRECTgpt4_7abb270ctemporal-reasoningasked 2023/03/10 (Fri) 08:153,093 ctx tokE 0.521
What is the order of the six museums I visited from earliest to latest?
- errata
- Science Museum, Metropolitan Museum of Art, Modern Art Museum, Museum of History, Natural History Museum, Museum of Contemporary Art
- gold
- Science Museum, Museum of Contemporary Art, Metropolitan Museum of Art, Museum of History, Modern Art Museum, Natural History Museum
- judge
- Order of museums differs from reference sequence
3 citations — session · turn · the span it was read from
- answer_7093d898_5:t6“I recently saw some amazing ancient Egyptian artifacts at the Metropolitan Museum of Art”
- answer_7093d898_5:t6“I recently saw some amazing ancient Egyptian artifacts at the Metropolitan Museum of Art... I took some photos”
- answer_7093d898_3:t0“I saw it in person today at the Metropolitan Museum of Art's "Ancient Egyptian Artifacts" exhibition”
- ABSTAINEDgpt4_d6585ce9temporal-reasoningasked 2023/04/22 (Sat) 08:013,119 ctx tokE 0.589
Who did I go with to the music event last Saturday?
- errata
- ABSTAINED — not in the history
- gold
- my parents
- judge
- No answer provided
- ABSTAINEDgpt4_e072b769temporal-reasoningasked 2023/05/06 (Sat) 09:183,060 ctx tokE 0.695
How many weeks ago did I start using the cashback app 'Ibotta'?
- errata
- ABSTAINED — not in the history
- gold
- 3 weeks ago
- judge
- No answer provided
- CORRECT0bc8ad92temporal-reasoningasked 2023/03/25 (Sat) 17:182,924 ctx tokE 0.621
How many months have passed since I last visited a museum with a friend?
- errata
- 5
- gold
- 5
- judge
- Both state 5 months
3 citations — session · turn · the span it was read from
- bcc37468_2:t0“over the past month, I've finished three audiobooks: "The 7 Habits of Highly Effective People", "Daring Greatly", and "The Power of Now"”
- answer_f4ea84fb_3:t4“I've been trying to take advantage of the many museum events and exhibits in my area.”
- answer_f4ea84fb_3:t4“I've been trying to take advantage of the many museum events and exhibits in my area.”
- CORRECT2a1811e2temporal-reasoningasked 2023/03/26 (Sun) 12:053,067 ctx tokE 0.560
How many days had passed between the Hindu festival of Holi and the Sunday mass at St. Mary's Church?
- errata
- 21
- gold
- 21 days. 22 days (including the last day) is also acceptable.
- judge
- Matches reference value of 21 days
3 citations — session · turn · the span it was read from
- answer_1cc3cd0c_2:t2“I sat in the third pew from the front at St. Mary's Church”
- answer_1cc3cd0c_1:t4“I had a lot of fun throwing colors with my colleague Rohan and his family”
- answer_1cc3cd0c_1:t0“had a blast throwing colors and learning about its significance”
- CORRECT8077ef71temporal-reasoningasked 2022/04/04 (Mon) 21:032,876 ctx tokE 0.603
How many days ago did I attend a networking event?
- errata
- 26
- gold
- 26 days. 27 days (including the last day) is also acceptable.
- judge
- 26 matches the reference value
3 citations — session · turn · the span it was read from
- answer_0dd54b7c_1:t0“I just got back from a networking event that ran from 6 PM to 8 PM today”
- 76703db4_3:t0“I just came back from our annual family picnic in June, and it was a blast!”
- f7ffddbc:t0“I'm still feeling motivated from that charity yoga event I did back in April - it was a lot of fun!”
- ABSTAINED982b5123_abstemporal-reasoningasked 2023/05/27 (Sat) 17:103,249 ctx tokE 0.670
When did I book the Airbnb in Sacramento?
- errata
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- ABSTAINEDc8090214_abstemporal-reasoningasked 2023/05/30 (Tue) 05:173,426 ctx tokE 0.543
How many days before I bought my iPad did I attend the Holiday Market?
- errata
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- CORRECTe4e14d04temporal-reasoningasked 2023/05/28 (Sun) 15:463,195 ctx tokE 0.716
How long had I been a member of 'Book Lovers Unite' when I attended the meetup?
- errata
- Two weeks
- gold
- Two weeks
- judge
- Matches reference fact: two weeks
3 citations — session · turn · the span it was read from
- answer_cf425855_2:t0“I attended a meetup organized by Book Lovers Unite last week where we discussed the book”
- answer_cf425855_2:t10“attended a meetup organized by a group last week, where we discussed the book”
- answer_cf425855_2:t0“I attended a meetup organized by Book Lovers Unite last week”
- CORRECTgpt4_21adecb5temporal-reasoningasked 2023/07/01 (Sat) 08:162,925 ctx tokE 0.532
How many months passed between the completion of my undergraduate degree and the submission of my master's thesis?
- errata
- 6
- gold
- 6 months
- judge
- Candidate states 6, matching the 6 months reference
3 citations — session · turn · the span it was read from
- answer_1e2369c9_1:t0“I just completed my undergraduate degree in computer science”
- answer_1e2369c9_1:t0“I just completed my undergraduate degree in computer science”
- answer_1e2369c9_2:t0“I just submitted my master's thesis on computer science today”
- CORRECTgpt4_2312f94ctemporal-reasoningasked 2023/03/15 (Wed) 04:452,770 ctx tokE 0.565
Which device did I got first, the Samsung Galaxy S22 or the Dell XPS 13?
- errata
- Samsung Galaxy S22
- gold
- Samsung Galaxy S22
- judge
- Matches reference: Samsung Galaxy S22
3 citations — session · turn · the span it was read from
- e831a29f_2:t8“offer a limited-time promotion of 15% off the bundle for first-time customers”
- ultrachat_437795:t3“1. Use energy-efficient devices: Look for devices that have high energy efficiency ratings.”
- ultrachat_437795:t5“1. look for devices with energy star ratings: this certification shows that a device meets high standards for energy efficiency.”
- CORRECTgpt4_2655b836temporal-reasoningasked 2023/04/10 (Mon) 10:152,596 ctx tokE 0.775
What was the first issue I had with my new car after its first service?
- errata
- The car's GPS system had an issue on 2023-03-22.
- gold
- GPS system not functioning correctly
- judge
- Both state GPS system malfunction as the issue
3 citations — session · turn · the span it was read from
- answer_4be1b6b4_2:t0“I just got my car serviced for the first time on March 15th, and it was a great experience.”
- answer_4be1b6b4_2:t0“I just got my car serviced for the first time on March 15th, and it was a great experience.”
- answer_4be1b6b4_2:t0“I just got my car serviced for the first time on March 15th, and it was a great experience.”
- CORRECTgpt4_4edbafa2temporal-reasoningasked 2023/07/01 (Sat) 02:363,218 ctx tokE 0.653
What was the date on which I attended the first BBQ event in June?
- errata
- June 3, 2023
- gold
- June 3rd
- judge
- June 3rd matches; year added as extra detail
3 citations — session · turn · the span it was read from
- answer_0a00c163_2:t8“like the one I made on the 10th of June.”
- 22be78fc_2:t0“I just did my first 5K run at the local park on October 10th”
- answer_0a00c163_2:t0“I attended a backyard BBQ party at my colleague's house on the 3rd of June”
- CORRECTgpt4_4fc4f797temporal-reasoningasked 2023/06/01 (Thu) 21:223,072 ctx tokE 0.507
How many days passed between the day I received feedback about my car's suspension and the day I tested my new suspension setup?
- errata
- 38
- gold
- 38 days. 39 days (including the last day) is also acceptable.
- judge
- 38 matches reference answer's accepted value
3 citations — session · turn · the span it was read from
- answer_be07688f_1:t10“I've been getting feedback from judges that my car's suspension was too soft, affecting my handling, especially in the fast corners.”
- answer_be07688f_1:t4“I'm also planning to test my car's new suspension setup during an open track day at VIRginia International Raceway next month”
- answer_be07688f_2:t2“I want to make sure my car's new suspension setup is dialed in.”
- CORRECTgpt4_6dc9b45btemporal-reasoningasked 2021/10/02 (Sat) 03:562,874 ctx tokE 0.637
How many months ago did I attend the Seattle International Film Festival?
- errata
- 4
- gold
- 4 months ago
- judge
- 4 matches the reference's 4 months ago
3 citations — session · turn · the span it was read from
- answer_c4df007f_1:t0“I attended SIFF for a week”
- answer_c4df007f_1:t8“As someone who's attended several festivals, including SIFF, I've seen firsthand the importance of showcasing diverse voices and stories.”
- answer_c4df007f_1:t6“As someone who's attended several festivals, including SIFF, I've seen firsthand how these events can launch careers.”
- ABSTAINEDgpt4_70e84552_abstemporal-reasoningasked 2023/05/21 (Sun) 10:403,528 ctx tokE 0.496
Which task did I complete first, fixing the fence or purchasing three cows from Peter?
- errata
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- CORRECTgpt4_7ddcf75ftemporal-reasoningasked 2023/06/20 (Tue) 16:303,076 ctx tokE 0.400
How many days ago did I go on a whitewater rafting trip in the Oregon mountains?
- errata
- 3
- gold
- 3 days ago. 4 days (including the last day) is also acceptable.
- judge
- Matches reference value of 3 days
3 citations — session · turn · the span it was read from
- answer_ad8a76cb_1:t0“I just got back from an amazing whitewater rafting trip in the Oregon mountains ... today”
- answer_ad8a76cb_1:t0“I just got back from an amazing whitewater rafting trip in the Oregon mountains with a group of coworkers today”
- 8c652eb0_2:t8“I think I'll be visiting the Everglades National Park during my day trip.”
- CORRECTgpt4_8279ba02temporal-reasoningasked 2023/03/25 (Sat) 02:462,935 ctx tokE 0.404
How many days ago did I buy a smoker?
- errata
- 10
- gold
- 10 days ago. 11 days (including the last day) is also acceptable.
- judge
- 10 matches reference value of 10 days ago
3 citations — session · turn · the span it was read from
- e1c807f9:t2“I took my first class on a Thursday evening back in February at Rhythm and Soul”
- b6018747_2:t10“my collection of 57 rare records”
- b6018747_2:t8“I just got a new vinyl record that I'm excited to add to my collection.”
- ABSTAINEDgpt4_93159ced_abstemporal-reasoningasked 2023/05/30 (Tue) 18:542,916 ctx tokE 0.579
How long have I been working before I started my current job at Google?
- errata
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- CORRECTgpt4_98f46fc6temporal-reasoningasked 2023/05/01 (Mon) 12:153,433 ctx tokE 0.493
Which event did I participate in first, the charity gala or the charity bake sale?
- errata
- charity bake sale
- gold
- I participated in the charity bake sale first.
- judge
- States charity bake sale, matching reference fact.
3 citations — session · turn · the span it was read from
- answer_5850de18_2:t0“I help organize a charity bake sale at my office today”
- answer_5850de18_2:t0“I help organize a charity bake sale at my office today”
- answer_5850de18_1:t0“I've been doing a lot lately, and I want to keep the momentum going.”
- CORRECTgpt4_9a159967temporal-reasoningasked 2023/04/27 (Thu) 05:281,797 ctx tokE 0.656
Which airline did I fly with the most in March and April?
- errata
- United Airlines
- gold
- United Airlines
- judge
- Matches reference: United Airlines
3 citations — session · turn · the span it was read from
- answer_8a42fedf_1:t2“In March, I took a business trip to Chicago with United Airlines, flying from my hometown to Chicago on the 10th and returning on the 12th, with two flights eac”
- answer_8a42fedf_1:t0“I'm leaning towards United since I've accumulated more miles with them.”
- answer_8a42fedf_1:t2“flying from my hometown to Chicago on the 10th and returning on the 12th, with two flights each way”
- CORRECTgpt4_a2d1d1f6temporal-reasoningasked 2023/04/18 (Tue) 01:483,178 ctx tokE 0.698
How many days ago did I harvest my first batch of fresh herbs from the herb garden kit?
- errata
- 3
- gold
- 3 days ago. 4 days (including the last day) is also acceptable.
- judge
- Matches 3 days ago as stated in reference
3 citations — session · turn · the span it was read from
- answer_f6d6e33f_1:t0“I just harvested my first batch of fresh herbs from the herb garden kit today”
- answer_f6d6e33f_1:t0“I just harvested my first batch of fresh herbs from the herb garden kit today”
- answer_f6d6e33f_1:t8“I've been harvesting a lot of thyme from my garden and I want to use it up before it goes bad.”
- CORRECTgpt4_b0863698temporal-reasoningasked 2023/03/26 (Sun) 04:132,969 ctx tokE 0.523
How many days ago did I participate in the 5K charity run?
- errata
- 7
- gold
- 7 days ago. 8 days (including the last day) is also acceptable.
- judge
- Matches reference value of 7 days ago
3 citations — session · turn · the span it was read from
- 4a79d078_2:t2“I helped with registration and handed out water to the participants.”
- answer_550bb2d1_1:t0“I just got back into running and did a 5K charity run today”
- answer_550bb2d1_1:t0“did a 5K charity run today”
- ABSTAINEDgpt4_c27434e8_abstemporal-reasoningasked 2023/05/23 (Tue) 11:263,247 ctx tokE 0.528
Which project did I start first, the Ferrari model or the Porsche 991 Turbo S model?
- errata
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- CORRECTgpt4_e414231ftemporal-reasoningasked 2023/03/21 (Tue) 21:431,897 ctx tokE 0.491
Which bike did I fixed or serviced the past weekend?
- errata
- road bike
- gold
- road bike
- judge
- Matches reference: road bike
3 citations — session · turn · the span it was read from
- answer_e28c1f0e_1:t0“I finally got around to fixing that flat tire on my mountain bike today - replaced the inner tube, cleaned the chain, and it's been running smoothly.”
- answer_e28c1f0e_2:t8“ask the mechanic to check the tire pressure and make any necessary adjustments”
- answer_e28c1f0e_2:t2“the brakes on my road bike feeling a bit spongy lately”
- ABSTAINEDgpt4_fe651585_abstemporal-reasoningasked 2023/03/17 (Fri) 03:093,283 ctx tokE 0.422
Who became a parent first, Tom or Alex?
- errata
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says