results · Naive top-k RAG (k=10) · rerunC-nothink
temporal-reasoning 33.3
39 of 117 rows right — 39 questions × 3 seeds (11/22/33)the corpus’s own question_type, over all 450 rows — a gold-abstention row is right iff the arm abstained, every other row iff the judge said CORRECTsame arm, by ability: OverallInformation extractionMulti-sessionTemporalKnowledge updateAbstentionsame arm, by question_type: single-session-usersingle-session-assistantsingle-session-preferencemulti-sessionknowledge-updatethe same cut, other arms: Errata 59.0Full-context baseline 23.9
- ABSTAINED0bc8ad92temporal-reasoningasked 2023/03/25 (Sat) 17:184,261 ctx tok
How many months have passed since I last visited a museum with a friend?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions a behind-the-scenes tour of the Science Museum with a chemistry professor friend, but does not specify when this visit occurred relative to today's date.INSUFFICIENT_INFORMATION: The history mentions a visit to the Science Museum with a chemistry professor friend but does not provide a specific date for that visit.
- gold
- 5
- judge
- Candidate declines to answer, doesn't state 5 months
- INCORRECT0bc8ad93temporal-reasoningasked 2023/03/11 (Sat) 05:283,585 ctx tok
I mentioned visiting a museum two months ago. Did I visit with a friend or not?
- naive
- Yes, you visited with a friend who is a chemistry professor.other seeds: You visited with a friend who is a chemistry professor.You visited with a friend who is a chemistry professor.
- gold
- No, you did not visit with a friend.
- judge
- Reference says no friend; candidate claims yes with a friend.
- ABSTAINED2a1811e2temporal-reasoningasked 2023/03/26 (Sun) 12:054,669 ctx tok
How many days had passed between the Hindu festival of Holi and the Sunday mass at St. Mary's Church?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions attending Holi on February 26th and a sermon at St. Mary's Church last Sunday, but does not provide the specific date of the Sunday mass to calculate the days passed.INSUFFICIENT_INFORMATION: The history mentions attending Holi on February 26th and a sermon at St. Mary's Church last Sunday, but does not provide the specific date of the Sunday mass to calculate the days passed.
- gold
- 21 days. 22 days (including the last day) is also acceptable.
- judge
- Candidate declines to answer, no day count given
- ABSTAINED370a8ff4temporal-reasoningasked 2023/10/15 (Sun) 17:534,745 ctx tok
How many weeks had passed since I recovered from the flu when I went on my 10th jog outdoors?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions recovering from the flu on 2023/01/19 and going on the 10th jog outdoors on 2023/04/10, but does not explicitly state the number of weeks that passed between these two events.INSUFFICIENT_INFORMATION: The history mentions recovering from the flu on 2023/01/19 and going on the 10th jog outdoors on 2023/04/10, but does not explicitly state the number of weeks that passed between these two events.
- gold
- 15
- judge
- Candidate declines to answer instead of giving 15 weeks
- INCORRECT6e984301temporal-reasoningasked 2023/04/01 (Sat) 18:554,663 ctx tok
How many weeks have I been taking sculpting classes when I invested in my own set of sculpting tools?
- naive
- 6
- gold
- 3
- judge
- Reference says 3 weeks, candidate says 6
- INCORRECT982b5123temporal-reasoningasked 2023/05/21 (Sun) 10:304,892 ctx tok
How many months ago did I book the Airbnb in San Francisco?
- naive
- three
- gold
- Five months ago
- judge
- Reference says five months, candidate says three
- ABSTAINED9a707b81temporal-reasoningasked 2022/04/15 (Fri) 18:464,784 ctx tok
How many days ago did I attend a baking class at a local culinary school when I made my friend's birthday cake?
- naive
- ABSTAINED — not in the history
- gold
- 21 days. 22 days (including the last day) is also acceptable.
- judge
- Candidate gives no specific day count, declines to answer
- ABSTAINEDa3838d2btemporal-reasoningasked 2023/11/29 (Wed) 03:314,605 ctx tok
How many charity events did I participate in before the 'Run for the Cure' event?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions the "Run for the Cure" event, a charity swim event the user is considering, and a cycling event they plan to register for, but does not state how many events were participated in prior to the 'Run for the Cure' event.INSUFFICIENT_INFORMATION: The history mentions the "Run for the Cure" event, a charity swim event the user is considering, and volunteering at the "Food for Thought" charity gala, but does not specify how many events were participated in prior to the 'Run for the Cure' event.
- gold
- 4
- judge
- Candidate declined to answer, reference states 4 events
- ABSTAINEDcc6d1ec1temporal-reasoningasked 2023/05/21 (Sun) 19:314,383 ctx tok
How long had I been bird watching when I attended the bird watching workshop?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions a guided walk at a local nature preserve and keeping a life list of 35 species, but does not specify the duration of time spent bird watching before attending.INSUFFICIENT_INFORMATION: The history mentions a life list of 35 species and attending a guided walk, but does not state the duration of time spent bird watching.
- gold
- Two months
- judge
- Candidate declines to answer, omits 'two months' duration
- ABSTAINEDe4e14d04temporal-reasoningasked 2023/05/28 (Sun) 15:464,502 ctx tok
How long had I been a member of 'Book Lovers Unite' when I attended the meetup?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions the Facebook group "Book Lovers Unite" and attending a meetup, but does not state how long you had been a member of that specific group.INSUFFICIENT_INFORMATION: The history mentions the Facebook group "Book Lovers Unite" and attending a meetup, but does not specify how long you had been a member of that group.
- gold
- Two weeks
- judge
- Candidate declines to answer, missing 'two weeks' fact
- ABSTAINEDeac54addtemporal-reasoningasked 2023/03/28 (Tue) 20:354,128 ctx tok
What was the significant buisiness milestone I mentioned four weeks ago?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions a deadline for wrapping up the Johnson account in two weeks from March 1, 2023.INSUFFICIENT_INFORMATION: The history mentions wrapping up the Johnson account, onboarding a new team, and mentoring Alex.
- gold
- I signed a contract with my first client.
- judge
- Candidate declines to answer, omits signing first client contract
- ABSTAINEDgpt4_1d80365etemporal-reasoningasked 2023/06/01 (Thu) 17:174,844 ctx tok
How many days did I spend on my solo camping trip to Yosemite National Park?
- naive
- ABSTAINED — not in the history
- gold
- 2 days. 3 days (including the last day) is also acceptable.
- judge
- Candidate declines to answer, doesn't state 2-3 days
- ABSTAINEDgpt4_21adecb5temporal-reasoningasked 2023/07/01 (Sat) 08:164,277 ctx tok
How many months passed between the completion of my undergraduate degree and the submission of my master's thesis?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history does not mention the user's undergraduate degree or master's thesis submission dates.INSUFFICIENT_INFORMATION: The history does not mention the user's undergraduate degree or master's thesis submission dates.
- gold
- 6 months
- judge
- Candidate declines to answer instead of stating 6 months
- INCORRECTgpt4_2312f94ctemporal-reasoningasked 2023/03/15 (Wed) 04:454,976 ctx tok
Which device did I got first, the Samsung Galaxy S22 or the Dell XPS 13?
- naive
- Dell XPS 13
- gold
- Samsung Galaxy S22
- judge
- Candidate names Dell XPS 13, reference says Samsung Galaxy S22
- INCORRECTgpt4_2655b836temporal-reasoningasked 2023/04/10 (Mon) 10:154,474 ctx tok
What was the first issue I had with my new car after its first service?
- naive
- The material does not mention any issue occurring after the first service on March 15th; it only states that the user had a GPS issue on 3/22 which was resolved by the dealership.
- gold
- GPS system not functioning correctly
- judge
- Candidate denies GPS issue was after service, contradicts reference
- ABSTAINEDgpt4_468eb063temporal-reasoningasked 2023/04/20 (Thu) 10:124,800 ctx tok
How many days ago did I meet Emma?
- naive
- ABSTAINED — not in the history
- gold
- 9 days ago. 10 days (including the last day) is also acceptable.
- judge
- Candidate claims no Emma mention; reference gives 9 days ago
- INCORRECTgpt4_4929293btemporal-reasoningasked 2023/06/22 (Thu) 18:334,111 ctx tok
What was the the life event of one of my relatives that I participated in a week ago?
- naive
- The user's niece had her kindergarten graduation ceremony last week.
- gold
- my cousin's wedding
- judge
- Different relative and event: niece's graduation vs cousin's wedding
- ABSTAINEDgpt4_4fc4f797temporal-reasoningasked 2023/06/01 (Thu) 21:224,882 ctx tok
How many days passed between the day I received feedback about my car's suspension and the day I tested my new suspension setup?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions receiving feedback on March 17, 2023, and planning for a track day on April 24, 2023, but does not confirm that the test actually occurred on that date.INSUFFICIENT_INFORMATION: The history mentions receiving feedback on March 17, 2023, and planning for a track day on April 24, 2023, but does not confirm that the test actually occurred on that specific date.
- gold
- 38 days. 39 days (including the last day) is also acceptable.
- judge
- Candidate declines to answer, doesn't give 38/39 days
- ABSTAINEDgpt4_59149c78temporal-reasoningasked 2023/02/01 (Wed) 08:414,745 ctx tok
I mentioned that I participated in an art-related event two weeks ago. Where was that event held at?
- naive
- ABSTAINED — not in the historyother seeds: City Art MuseumCity Art Museum
- gold
- The Metropolitan Museum of Art.
- judge
- Candidate names City Art Museum, not Metropolitan Museum of Art
- ABSTAINEDgpt4_61e13b3ctemporal-reasoningasked 2023/04/01 (Sat) 00:574,946 ctx tok
How many weeks passed between the time I sold homemade baked goods at the Farmers' Market for the last time and the time I participated in the Spring Fling Market?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions selling baked goods at the Farmers' Market on 2023/02/26 and participating in the Spring Fling Market on 2023/03/21, but does not state that the February date was the last time you sold there.INSUFFICIENT_INFORMATION: The history mentions selling baked goods at the Farmers' Market on 2023/02/26 and participating in the Spring Fling Market on 2023/03/21, but does not state that the February date was the last time you sold there.
- gold
- 3 weeks
- judge
- Candidate declines to answer instead of stating 3 weeks
- ABSTAINEDgpt4_7abb270ctemporal-reasoningasked 2023/03/10 (Fri) 08:154,503 ctx tok
What is the order of the six museums I visited from earliest to latest?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions visits to the Science Museum, the Metropolitan Museum of Art, and the Modern Art Museum, but does not provide a complete list of six museums or their specific visit dates.INSUFFICIENT_INFORMATION: The history mentions visits to the Science Museum, the Metropolitan Museum of Art, and the Modern Art Museum, but does not provide dates for all six museums or a complete list of six visited museums.
- gold
- Science Museum, Museum of Contemporary Art, Metropolitan Museum of Art, Museum of History, Modern Art Museum, Natural History Museum
- judge
- Candidate declined to answer, missing the full ordered list
- ABSTAINEDgpt4_98f46fc6temporal-reasoningasked 2023/05/01 (Mon) 12:154,425 ctx tok
Which event did I participate in first, the charity gala or the charity bake sale?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history mentions a charity bake sale, a charity walk for cancer research, and a charity gala, but does not provide dates to determine which occurred first.INSUFFICIENT_INFORMATION: The history mentions a charity bake sale, a charity walk for cancer research, and a charity gala, but does not provide dates to determine which occurred first.
- gold
- I participated in the charity bake sale first.
- judge
- Candidate declined to answer instead of stating bake sale was first
- ABSTAINEDgpt4_9a159967temporal-reasoningasked 2023/04/27 (Thu) 05:284,812 ctx tok
Which airline did I fly with the most in March and April?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: the history mentions flights in March but does not provide enough information to determine which airline was flown with the most in both March and AprilINSUFFICIENT_INFORMATION: The history mentions flights in March but does not provide enough information to determine which airline was flown with the most across both March and April.
- gold
- United Airlines
- judge
- Candidate declines to answer; reference specifies United Airlines
- ABSTAINEDgpt4_d6585ce9temporal-reasoningasked 2023/04/22 (Sat) 08:014,878 ctx tok
Who did I go with to the music event last Saturday?
- naive
- ABSTAINED — not in the history
- gold
- my parents
- judge
- Reference says parents; candidate says friends and declines to answer
- ABSTAINEDgpt4_e072b769temporal-reasoningasked 2023/05/06 (Sat) 09:184,704 ctx tok
How many weeks ago did I start using the cashback app 'Ibotta'?
- naive
- ABSTAINED — not in the history
- gold
- 3 weeks ago
- judge
- Candidate declines to answer instead of stating 3 weeks ago
- INCORRECTgpt4_e414231ftemporal-reasoningasked 2023/03/21 (Tue) 21:434,810 ctx tok
Which bike did I fixed or serviced the past weekend?
- naive
- The user fixed a flat tire on their mountain bike.other seeds: The user fixed a flat tire on their mountain bike on 2023/03/15, which was the previous Wednesday.The user fixed a flat tire on their mountain bike on 2023/03/15.
- gold
- road bike
- judge
- Candidate says mountain bike, reference says road bike
- CORRECT8077ef71temporal-reasoningasked 2022/04/04 (Mon) 21:034,658 ctx tok
How many days ago did I attend a networking event?
- naive
- 26
- gold
- 26 days. 27 days (including the last day) is also acceptable.
- judge
- Matches accepted value of 26 days
- ABSTAINED982b5123_abstemporal-reasoningasked 2023/05/27 (Sat) 17:104,968 ctx tok
When did I book the Airbnb in Sacramento?
- naive
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- ABSTAINEDc8090214_abstemporal-reasoningasked 2023/05/30 (Tue) 05:173,781 ctx tok
How many days before I bought my iPad did I attend the Holiday Market?
- naive
- ABSTAINED — not in the historyother seeds: INSUFFICIENT_INFORMATION: The history does not mention buying an iPad or attending a Holiday Market.INSUFFICIENT_INFORMATION: The history does not mention buying an iPad or attending a Holiday Market.
- gold
- ABSTAIN — the history never says
- CORRECTgpt4_4edbafa2temporal-reasoningasked 2023/07/01 (Sat) 02:364,897 ctx tok
What was the date on which I attended the first BBQ event in June?
- naive
- 3rd of June
- gold
- June 3rd
- judge
- Same date, June 3rd, just reworded
- CORRECTgpt4_6dc9b45btemporal-reasoningasked 2021/10/02 (Sat) 03:564,233 ctx tok
How many months ago did I attend the Seattle International Film Festival?
- naive
- 4
- gold
- 4 months ago
- judge
- 4 matches 4 months ago
- ABSTAINEDgpt4_70e84552_abstemporal-reasoningasked 2023/05/21 (Sun) 10:404,448 ctx tok
Which task did I complete first, fixing the fence or purchasing three cows from Peter?
- naive
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- CORRECTgpt4_7ddcf75ftemporal-reasoningasked 2023/06/20 (Tue) 16:304,773 ctx tok
How many days ago did I go on a whitewater rafting trip in the Oregon mountains?
- naive
- 3
- gold
- 3 days ago. 4 days (including the last day) is also acceptable.
- judge
- Candidate states 3 days, matching reference answer.
- CORRECTgpt4_8279ba02temporal-reasoningasked 2023/03/25 (Sat) 02:464,815 ctx tok
How many days ago did I buy a smoker?
- naive
- 10
- gold
- 10 days ago. 11 days (including the last day) is also acceptable.
- judge
- Matches the 10 days ago fact stated in reference
- ABSTAINEDgpt4_93159ced_abstemporal-reasoningasked 2023/05/30 (Tue) 18:544,156 ctx tok
How long have I been working before I started my current job at Google?
- naive
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- CORRECTgpt4_a2d1d1f6temporal-reasoningasked 2023/04/18 (Tue) 01:484,881 ctx tok
How many days ago did I harvest my first batch of fresh herbs from the herb garden kit?
- naive
- 3
- gold
- 3 days ago. 4 days (including the last day) is also acceptable.
- judge
- 3 days matches reference
- CORRECTgpt4_b0863698temporal-reasoningasked 2023/03/26 (Sun) 04:134,813 ctx tok
How many days ago did I participate in the 5K charity run?
- naive
- 7
- gold
- 7 days ago. 8 days (including the last day) is also acceptable.
- judge
- Candidate states 7, matching reference answer of 7 days
- ABSTAINEDgpt4_c27434e8_abstemporal-reasoningasked 2023/05/23 (Tue) 11:264,640 ctx tok
Which project did I start first, the Ferrari model or the Porsche 991 Turbo S model?
- naive
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says
- ABSTAINEDgpt4_fe651585_abstemporal-reasoningasked 2023/03/17 (Fri) 03:094,078 ctx tok
Who became a parent first, Tom or Alex?
- naive
- ABSTAINED — not in the history
- gold
- ABSTAIN — the history never says