TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models (2024)

1 Introduction

Time flies over us, but leaves its shadow behind.Understanding time is a crucial part of human comprehension of the world.Envision the blossoming of flowers, and you’ll associate it with the arrival of spring.The ponder within it encompasses the intricate interplay of world knowledge, causality, and event temporal relationships.Temporal reasoning, in contrast to reasoning of a singular nature, comes with inherent complexity, encompassing implicit arithmetic, logical implications, and world knowledge. It is a form of integrated reasoning built upon foundational reasoning like mathematical and logical reasoning(Cobbe etal., 2021; Mishra etal., 2022; Yu etal., 2020).Recently, large language models (LLMs) have demonstrated remarkable performance in complex reasoning(Hendrycks etal., 2021; Srivastava etal., 2022; Brown etal., 2020; Chowdhery etal., 2023; OpenAI, 2023; Touvron etal., 2023),but their performance in temporal reasoning has not yet been extensively explored.

Recent research for temporal reasoning typically focuses only on a few aspects, such as temporal commonsense or temporal question answering(Zhou etal., 2019; Chen etal., 2021; Dhingra etal., 2022; Wang and Zhao, 2023).Due to the inherent complexity of temporal reasoning, it is challenging to accurately measure models’ temporal reasoning capabilities based on limited aspects.

To address this issue, we propose TimeBench, a comprehensive and hierarchical temporal reasoning benchmark.Specifically, drawing inspiration from the human cognitive process of transitioning from abstraction and concreteness to integration(Barsalou etal., 2018), we categorize temporal reasoning into three levels: symbolic temporal reasoning, commonsense temporal reasoning, and event temporal reasoning.These levels respectively represent understanding abstract time expression, grasping concrete world knowledge, and integrating and applying this knowledge in real-world scenarios.TimeBench comprises 10 tasks with 16 sub-tasks, covering a broad spectrum of temporal reasoning phenomena.Besides, prior work typically features only a single task form, too simplistic to capture the model’s performance.In contrast, we incorporate four distinct task forms, offering a more realistic simulation of challenges.

To quantify the temporal reasoning capabilities of contemporary LLMs, we extensively assess widely-used LLMs, including proprietary models such as ChatGPT(Ouyang etal., 2022) and GPT-4(OpenAI, 2023), as well as open-source like LLaMA2(Touvron etal., 2023), Vicuna-1.5(Chiang etal., 2023), Mistral(Jiang etal., 2023), Baichuan2(Yang etal., 2023a), ChatGLM3Zeng etal. (2023) and FLAN-T5(Chung etal., 2022).We conduct experiments under zero-shot and few-shot settings, combining commonly used reasoning techniques, chain-of-thought prompting(Kojima etal., 2022; Wei etal., 2022).The experimental results suggest that GPT-4 outperforms other models, showcasing strong temporal reasoning capabilities, as shown in Figure1.Nevertheless, there is still a considerable gap between the strongest models and humans.On the contrary, open-source models show inferior performance in temporal reasoning, attributed to shortcomings in abstract time understanding, temporal relations modeling, and a lack of temporal commonsense.In addition, we also observe that chain-of-thought prompting does not yield a consistent improvement in performance.These findings indicate that there is still significant room for improvement in models’ temporal reasoning capabilities.Moreover, we have conducted a thorough analysis of the deficiencies and obstacles faced by models in temporal reasoning.

2 TimeBench Benchmark

3 Methodology

We perform evaluations using the prompt-based approach, including standard prompting and chain-of-thought prompting.Experiments are conducted under both zero-shot and few-shot settings.

4 Experimental Setup

5 Experimental Results

5.2 Zero-shot Results

Experimental results of alignment models under zero-shot settings are shown in Table5.In zero-shot settings, GPT-4 and GPT-3.5 rank first and second, respectively,and they significantly outperform all open-source models by a large margin.It is noteworthy that open-source models exhibit a larger performance decline compared to proprietary models when transitioning from few-shot to zero-shot scenarios.GPT, Baichuan2 and LLaMA2 suffer drops of 5.6%, 14.6% and 27.2%, respectively.We attribute this performance decline to the quality of alignment.Restricted by their limited instruction-following capability,open-source models struggle to fully unleash their performance solely through instructions.Therefore, few-shot prompting is a better approach for stimulating their temporal reasoning abilities.

5.3 Chain-of-Thought in Temporal Reasoning

Previous research has found that chain-of-thought prompting can enhance the model’s reasoning ability(Wei etal., 2022; Kojima etal., 2022).We aim to explore the following questions:Does CoT prompting bring consistent improvement in temporal reasoning?Due to the diversity of temporal reasoning, the above question has not yet been definitively answered.To investigate this, we select several popular LLMs and analyze their performance affected by chain-of-thought prompting.

Chain-of-thought reasoning is not consistently effective.

As illustrated in Figure2, introducing zero-shot CoT prompting results in consistent declines, with an overall decrease of 7.4%.In the few-shot scenario, CoT prompting also fails to yield consistent improvements,varying depending on the task.There is a 10.8% improvement in symbolic reasoning, while a significant decline of 15.2% in commonsense reasoning.In event temporal reasoning, there is a slight improvement of 1.3%.Next, we will conduct a more detailed analysis of the impact of CoT on specific tasks.

Impact of CoT prompting across tasks.

In order to explore the impact of CoT on various tasks thoroughly, we delve into the performance changes of each model across specific tasks within each category, as illustrated in Figure3.In the zero-shot setting, open-source models achieve a slight improvement in event temporal reasoning with chain-of-thought prompting, while in other cases, they face performance degradation.While in the few-shot setting, almost all models exhibit significant improvement in symbolic temporal reasoning, with a concurrent prevalent decline in commonsense temporal reasoning.We attribute this to the knowledge sensitivity inherent in commonsense reasoning,where step-by-step reasoning cannot compensate for the lack of knowledge.In event temporal reasoning, improvements mainly stem from datasets involving implicit multi-step reasoning (MenatQA and TempReason), indicating that CoT is more effective for multi-hop questions.In summary, zero-shot CoT consistently has a negative impact on temporal reasoning.While in few-shot scenario, CoT has a positive impact on symbolic and complex tasks, while negatively affecting knowledge-sensitive tasks.

6 Analysis and Discussion

6.1 Scaling Effect of Model Size

We investigate how the scale of models affects temporal reasoning capabilities.The trend is illustrated in Figure4.As the model scale increases, there is a notable improvement in performance.When the parameter size expands from 7B to 13B, LLaMA2 and Baichuan2 show improvements of 13.0% and 10.5%, respectively.Furthermore, when LLaMA scales up to 70B, the trend of performance improvement continues without stopping.The overall improvement follows a log-linear relationship with scale.There are no significant performance differences among LLaMA2, Baichuan2, and ChatGLM3 under similar parameter specifications,while Mistral demonstrates impressive prowess, outperforming all other 13B models with nearly half the number of parameters.

6.2 Challenges in Temporal Reasoning

Model	Order	Duration	Freq.	Stationarity	Typical	Avg.
GPT-4	76.4$\downarrow$	92.8$\uparrow$	83.3$\uparrow$	71.4$\downarrow$	54.5$\downarrow$	77.5
GPT-3.5	50.5$\uparrow$	39.8$\downarrow$	55.2$\uparrow$	48.4$\uparrow$	28.7$\downarrow$	43.5
Baichuan2${}^{\dagger}_{\text{13}b}$	40.5$\downarrow$	51.8$\uparrow$	43.7$\uparrow$	46.2$\uparrow$	29.8$\downarrow$	42.5
LLaMA2${}^{\dagger}_{\text{70}b}$	65.2$\uparrow$	72.1$\uparrow$	66.3$\uparrow$	36.3$\downarrow$	52.7$\downarrow$	63.0
Mistral${}^{\dagger}_{\text{7}b}$	27.0$\downarrow$	44.4$\uparrow$	58.3$\uparrow$	38.5$\downarrow$	38.3$\downarrow$	42.5

LLMs underperform in (multi-hop) symbolic reasoning

Except for GPT-4, the performance of all other models in symbolic temporal reasoning is unsatisfactory.A noticeable decrease is observed in duration-conversion task compared to other atomic tasks (25% in GPT-4 and 27% in LLaMA2${}_{\text{70}b}$).This is because the duration-conversion task (s3) necessitates a two-step reasoning process.It first unifies time units, and subsequently engages in numerical comparison.In contrast, other atomic tasks (s1, s2 and arithmetic) can be completed with a single reasoning step.In summary, LLMs perform poorly in symbolic temporal reasoning and exhibit more pronounced declines when encountering multi-step reasoning.

Mastery of commonsense knowledge varies in LLMs

We analyze models’ performance across various commonsense aspects, as shown in Table6.We regard the model’s average performance in commonsense reasoning tasks as the baseline.If the model outperforms the baseline in a specific aspect, it suggests greater proficiency in this type of knowledge, and vice versa.The findings indicate that LLMs generally demonstrate good knowledge of event duration and frequency.However, their comprehension of event order and typical events is relatively weakerThe uneven mastery of commonsense knowledge significantly affects the model’s reasoning performance,especially when dealing with complex questions that involve multiple types of knowledge.Retrieval-augmented reasoning presents a promising avenue for mitigating the model’s knowledge scarcity.

LLMs exhibit poor implicit temporal reasoning capabilities.

When comparing explicit and implicit event temporal reasoning, specifically TimeQA-explicit versus others, we observe a significant performance decrease in implicit reasoning.Additionally, on TRACIE with numerous implied events, most models only surpass a random baseline (50.0).Even GPT-4, despite its advanced capabilities, achieves only a 66.4% accuracy, suggesting that the LLM struggles with modeling implicit temporal relationships.We consider it helpful to explicitly model the temporal relationships between events and time expressions, for instance constructing timelines or temporal graphs.

LLMs are good factual reasoners rather than factual extractors

When humans engage in temporal reasoning, it generally involves two steps: first, extracting time-fact pairs from the context, and then performing fact-based reasoning.TempReason provides extracted facts for conducting fact-based reasoning.By comparing the model’s performance in context-based (TimeQA) against fact-based (TempReason) reasoning,we identify the bottleneck in event temporal reasoning.LLMs excel in TempReason, which signifies their strong capability in fact-based reasoning.However, their performance in context-based reasoning is significantly weaker compared to their performance in fact-based reasoning.This implies that errors could arise during the extraction of time-sensitive facts from the context.We attribute this performance gap to the model’s deficiency in factual extraction capabilitiesThus, we consider LLMs to be strong factual reasoners rather than factual extractors in event temporal reasoning.

6.3 Alignment Impairs Temporal Reasoning

In the experiments mentioned earlier (Table5), we observe a sharp decline in zero-shot performance of alignment models.To investigate whether alignment is the cause of the decline in temporal reasoning, we conducted experiments on alignment models under few-shot settings.Figure5 illustrates the overall performance decline after alignment.With the exception of Baichuan2, all other models are severely impaired, experiencing a significant drop of up to 22%.Through manual analysis of error cases, we have summarized two reasons:(1) Alignment reduces the model’s usability, causing it to tend towards refusal to answer when confronted with knowledge-sensitive questions.(2) Alignment damages the model’s in-context learning capability, resulting in situations where the model deviates from the demonstrations.Furthermore, we believe that the lack of temporal reasoning-related training data in alignment exacerbates this issue, leading to disparities between different reasoning capabilities, such as mathematical and temporal reasoning.

7 Related Work

8 Conclusion

Temporal reasoning entails inherent diversity and complexity.The lack of a comprehensive benchmark makes it challenging to quantify LLMs’ temporal reasoning capabilities.In this work, we present TimeBench, a comprehensive and hierarchical benchmark for LLM temporal reasoning, tailored to mirror temporal reasoning in complex scenarios.We conduct extensive experiments on state-of-the-art LLMs to investigate their temporal reasoning capabilities.Our findings indicate a substantial gap between state-of-the-art LLMs and human performance, emphasizing the need for further research in this area.Moreover, we provide a meticulous analysis and discussion, outlining the current challenges that models face and suggesting potential directions for improvement.

Limitations

TimeBench is a comprehensive benchmark to quantify the temporal reasoning capabilities of LLMs.While we have taken various factors into account, there are a few limitations.Firstly, our evaluation only applied prompt-based method under zero-shot and few-shot setting, lacking evaluations specifically tailored for models fine-tuned on the temporal domain.Secondly, the instructions and demonstrations were manually crafted,which may potentially lead to discrepancies in prompts interpretation among different LLMs.Thirdly, the dataset constituting the benchmark includes data from past years and a portion sourced from Wikipedia, which may contaminate the training corpus of LLMs.

Acknowledgements

The research in this article is supported by the National Key Research and Development Project (2021YFF0901602), the National Science Foundation of China (U22B2059, 62276083), and Shenzhen Foundational Research Funding (JCYJ20200109113441941), Major Key Project of PCL (PCL2021A06).Ming Liu is the corresponding author.

References

• Ainslie etal. (2023)Joshua Ainslie, James Lee-Thorp, Michiel deJong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. 2023.GQA: training generalized multi-query transformer models from multi-head checkpoints.In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4895–4901. Association for Computational Linguistics.
• Banerjee and Lavie (2005)Satanjeev Banerjee and Alon Lavie. 2005.Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
• Barsalou etal. (2018)LawrenceW Barsalou, Léo Dutriaux, and Christoph Scheepers. 2018.Moving beyond the distinction between concrete and abstract concepts.Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1752):20170144.
• Beltagy etal. (2020)IzBeltagy, MatthewE Peters, and Arman Cohan. 2020.Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150.
• Brown etal. (2020)TomB. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, DanielM. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.Language models are few-shot learners.In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
• Chen etal. (2021)Wenhu Chen, Xinyi Wang, and WilliamYang Wang. 2021.A dataset for answering time-sensitive questions.In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual.
• Chen etal. (2023)Ziqiang Chen, Shaojuan Wu, Xiaowang Zhang, and Zhiyong Feng. 2023.TML: A temporal-aware multitask learning framework for time-sensitive question answering.In Companion Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, pages 200–203. ACM.
• Chiang etal. (2023)Wei-Lin Chiang, Zhuohan Li, ZiLin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, JosephE. Gonzalez, Ion Stoica, and EricP. Xing. 2023.Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
• Chowdhery etal. (2023)Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, HyungWon Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, YiTay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, AndrewM. Dai, ThanumalayanSankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.2023.Palm: Scaling language modeling with pathways.J. Mach. Learn. Res., 24:240:1–240:113.
• Chu etal. (2024)Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. 2024.Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future.In The 62nd Annual Meeting of the Association for Computational Linguistics: ACL 2024, Bangkok, Thailand, August 11–16, 2024. Association for Computational Linguistics.
• Chu etal. (2023)Zheng Chu, Zekun Wang, Jiafeng Liang, Ming Liu, and Bing Qin. 2023.MTGER: multi-view temporal graph enhanced temporal reasoning over time-involved document.In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 15218–15233. Association for Computational Linguistics.
• Chung etal. (2022)HyungWon Chung, LeHou, Shayne Longpre, Barret Zoph, YiTay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, ShixiangShane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, EdH. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, QuocV. Le, and Jason Wei. 2022.Scaling instruction-finetuned language models.Preprint, arXiv:2210.11416.
• Cobbe etal. (2021)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.Training verifiers to solve math word problems.CoRR, abs/2110.14168.
• Dhingra etal. (2022)Bhuwan Dhingra, JeremyR. Cole, JulianMartin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and WilliamW. Cohen. 2022.Time-aware language models as temporal knowledge bases.Trans. Assoc. Comput. Linguistics, 10:257–273.
• Gupta etal. (2023)Vivek Gupta, Pranshu Kandoi, MahekBhavesh Vora, Shuo Zhang, Yujie He, Ridho Reinanda, and Vivek Srikumar. 2023.Temptabqa: Temporal question answering for semi-structured tables.In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2431–2453. Association for Computational Linguistics.
• Han etal. (2021)Rujun Han, Xiang Ren, and Nanyun Peng. 2021.ECONET: effective continual pretraining of language models for event temporal reasoning.In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5367–5380. Association for Computational Linguistics.
• Hendrycks etal. (2021)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.Measuring massive multitask language understanding.In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
• Jain etal. (2023)Raghav Jain, Daivik Sojitra, Arkadeep Acharya, Sriparna Saha, Adam Jatowt, and Sandipan Dandapat. 2023.Do language models have a common sense regarding time? revisiting temporal commonsense reasoning in the era of large language models.In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6750–6774, Singapore. Association for Computational Linguistics.
• Jiang etal. (2023)AlbertQ. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, DevendraSingh Chaplot, Diego deLasCasas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, LélioRenard Lavaud, Marie-Anne Lachaux, Pierre Stock, TevenLe Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and WilliamEl Sayed. 2023.Mistral 7b.CoRR, abs/2310.06825.
• Ko etal. (2023)Dohwan Ko, JiSoo Lee, Woo-Young Kang, Byungseok Roh, and Hyunwoo Kim. 2023.Large language models are temporal and causal reasoners for video question answering.In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 4300–4316. Association for Computational Linguistics.
• Kojima etal. (2022)Takeshi Kojima, ShixiangShane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.Large language models are zero-shot reasoners.In NeurIPS.
• Lin (2004)Chin-Yew Lin. 2004.Rouge: A package for automatic evaluation of summaries.In Text summarization branches out, pages 74–81.
• Liu etal. (2023)Hanmeng Liu, Zhiyang Teng, Ruoxi Ning, Jian Liu, Qiji Zhou, and Yue Zhang. 2023.Glore: Evaluating logical reasoning of large language models.CoRR, abs/2310.09107.
• Llorens etal. (2015)Hector Llorens, Nathanael Chambers, Naushad UzZaman, Nasrin Mostafazadeh, JamesF. Allen, and James Pustejovsky. 2015.Semeval-2015 task 5: QA tempeval - evaluating temporal information understanding with question answering.In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015, pages 792–800. The Association for Computer Linguistics.
• Mathur etal. (2021)Puneet Mathur, Rajiv Jain, Franck Dernoncourt, Vlad Morariu, QuanHung Tran, and Dinesh Manocha. 2021.TIMERS: Document-level temporal relation extraction.In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 524–533, Online. Association for Computational Linguistics.
• Miller etal. (2015)Timothy Miller, Steven Bethard, Dmitriy Dligach, Chen Lin, and Guergana Savova. 2015.Extracting time expressions from clinical text.In Proceedings of BioNLP 15, pages 81–91, Beijing, China. Association for Computational Linguistics.
• Mishra etal. (2022)Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022.LILA: A unified benchmark for mathematical reasoning.In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5807–5832. Association for Computational Linguistics.
• OpenAI (2023)OpenAI. 2023.GPT-4 technical report.CoRR, abs/2303.08774.
• Ouyang etal. (2022)Long Ouyang, Jeffrey Wu, XuJiang, Diogo Almeida, CarrollL. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, PaulF. Christiano, Jan Leike, and Ryan Lowe. 2022.Training language models to follow instructions with human feedback.In NeurIPS.
• Papineni etal. (2002)Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.Bleu: a method for automatic evaluation of machine translation.In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
• Pustejovsky etal. (2003)James Pustejovsky, JoséM. Castaño, Robert Ingria, Roser Saurí, RobertJ. Gaizauskas, Andrea Setzer, Graham Katz, and DragomirR. Radev. 2003.Timeml: Robust specification of event and temporal expressions in text.In New Directions in Question Answering, Papers from 2003 AAAI Spring Symposium, Stanford University, Stanford, CA, USA, pages 28–34. AAAI Press.
• Qin etal. (2021)Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021.TIMEDIAL: temporal commonsense reasoning in dialog.In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7066–7076. Association for Computational Linguistics.
• Qiu etal. (2023)Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, EdoardoM. Ponti, and ShayB. Cohen. 2023.Are large language models temporally grounded?CoRR, abs/2311.08398.
• Raffel etal. (2020)Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and PeterJ. Liu. 2020.Exploring the limits of transfer learning with a unified text-to-text transformer.J. Mach. Learn. Res., 21:140:1–140:67.
• Son and Oh (2023)Jungbin Son and Alice Oh. 2023.Time-aware representation learning for time-sensitive question answering.In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 70–77. Association for Computational Linguistics.
• Srivastava etal. (2022)Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu AwalMd Shoeb, Abubakar Abid, Adam Fisch, AdamR. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, AlexanderW. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, AnantharamanS. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, AndrewM. Dai, Andrew La, AndrewK. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and etal. 2022.Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.CoRR, abs/2206.04615.
• Su etal. (2023)Xin Su, Phillip Howard, Nagib Hakim, and Steven Bethard. 2023.Fusing temporal graphs into transformers for time-sensitive question answering.In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 948–966. Association for Computational Linguistics.
• Tan etal. (2023)Qingyu Tan, HweeTou Ng, and Lidong Bing. 2023.Towards benchmarking and improving the temporal reasoning capability of large language models.In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14820–14835. Association for Computational Linguistics.
• Thukral etal. (2021)Shivin Thukral, Kunal Kukreja, and Christian Kavouras. 2021.Probing language models for understanding of temporal expressions.In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2021, Punta Cana, Dominican Republic, November 11, 2021, pages 396–406. Association for Computational Linguistics.
• Touvron etal. (2023)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, CristianCanton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, PunitSingh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, EricMichael Smith, Ranjan Subramanian, XiaoqingEllen Tan, Binh Tang, Ross Taylor, Adina Williams, JianXiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and ThomasScialom. 2023.Llama 2: Open foundation and fine-tuned chat models.Preprint, arXiv:2307.09288.
• UzZaman etal. (2013)Naushad UzZaman, Hector Llorens, Leon Derczynski, JamesF. Allen, Marc Verhagen, and James Pustejovsky. 2013.Semeval-2013 task 1: Tempeval-3: Evaluating time expressions, events, and temporal relations.In Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013, pages 1–9. The Association for Computer Linguistics.
• Vashishtha etal. (2019)Siddharth Vashishtha, Benjamin VanDurme, and AaronSteven White. 2019.Fine-grained temporal relation extraction.In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2906–2919, Florence, Italy. Association for Computational Linguistics.
• Vedantam etal. (2015)Ramakrishna Vedantam, C.Lawrence Zitnick, and Devi Parikh. 2015.Cider: Consensus-based image description evaluation.In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 4566–4575. IEEE Computer Society.
• Verhagen etal. (2007)Marc Verhagen, RobertJ. Gaizauskas, Frank Schilder, Mark Hepple, Graham Katz, and James Pustejovsky. 2007.Semeval-2007 task 15: Tempeval temporal relation identification.In Proceedings of the 4th International Workshop on Semantic Evaluations, SemEval@ACL 2007, Prague, Czech Republic, June 23-24, 2007, pages 75–80. The Association for Computer Linguistics.
• Verhagen etal. (2010)Marc Verhagen, Roser Saurí, Tommaso Caselli, and James Pustejovsky. 2010.Semeval-2010 task 13: Tempeval-2.In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval@ACL 2010, Uppsala University, Uppsala, Sweden, July 15-16, 2010, pages 57–62. The Association for Computer Linguistics.
• Virgo etal. (2022)FelixGiovanni Virgo, Fei Cheng, and Sadao Kurohashi. 2022.Improving event duration question answering by leveraging existing temporal information extraction data.In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC 2022, Marseille, France, 20-25 June 2022, pages 4451–4457. European Language Resources Association.
• Wang and Zhao (2023)Yuqing Wang and Yun Zhao. 2023.TRAM: benchmarking temporal reasoning for large language models.CoRR, abs/2310.00835.
• Wei etal. (2022)Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, EdH. Chi, QuocV. Le, and Denny Zhou. 2022.Chain-of-thought prompting elicits reasoning in large language models.In NeurIPS.
• Wei etal. (2023)Yifan Wei, Yisong Su, Huanhuan Ma, Xiaoyan Yu, Fangyu Lei, Yuanzhe Zhang, Jun Zhao, and Kang Liu. 2023.Menatqa: A new dataset for testing the temporal comprehension and reasoning abilities of large language models.In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 1434–1447. Association for Computational Linguistics.
• Yang etal. (2023a)Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, CeBian, Chao Yin, Chenxu Lv, DaPan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. 2023a.Baichuan 2: Open large-scale language models.CoRR, abs/2309.10305.
• Yang etal. (2023b)Sen Yang, Xin Li, Lidong Bing, and Wai Lam. 2023b.Once upon a time in graph: Relative-time pretraining for complex temporal reasoning.In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 11879–11895. Association for Computational Linguistics.
• Yu etal. (2020)Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020.Reclor: A reading comprehension dataset requiring logical reasoning.In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
• Zeng etal. (2023)Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, WengLam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023.GLM-130B: an open bilingual pre-trained model.In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
• Zhang and Wan (2023)Yunxiang Zhang and Xiaojun Wan. 2023.Situatedgen: Incorporating geographical and temporal contexts into generative commonsense reasoning.In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.
• Zhang etal. (2023)Zhuosheng Zhang, Yao Yao, Aston Zhang, Xiangru Tang, Xinbei Ma, Zhiwei He, Yiming Wang, Mark Gerstein, Rui Wang, Gongshen Liu, and Hai Zhao. 2023.Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents.CoRR, abs/2311.11797.
• Zhao etal. (2023)WayneXin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.A survey of large language models.Preprint, arXiv:2303.18223.
• Zhou etal. (2019)Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019."going on a vacation" takes longer than "going for a walk": A study of temporal commonsense understanding.In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3361–3367. Association for Computational Linguistics.
• Zhou etal. (2021)Ben Zhou, Kyle Richardson, Qiang Ning, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2021.Temporal reasoning on implicit events from distant supervision.In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 1361–1371. Association for Computational Linguistics.
• Zhu etal. (2023)Xinyu Zhu, Cheng Yang, Bei Chen, Siheng Li, Jian-Guang Lou, and Yujiu Yang. 2023.Question answering as programming for solving time-sensitive questions.In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 12775–12790. Association for Computational Linguistics.

Appendix A TimeBench Details

TimeBench features 3 major categories, 10 tasks and 15 subtasks, each with distinct challenges, totaling 19,000 instances.Detailed statistics are available in Figure7 and Table7.

A.4 Evaluation Metrics

Accuracy is used for NLI and date arithmetic tasks.M-S tasks are evaluated using option-level EM and F1.FRC tasks (excluding date arithmetic) are assessed with token-level EM and F1.For CTG task, we take the average of multiple generation metrics, which are outlined as follows.

Metrics for SituatedGen

Following SituatedGen(Zhang and Wan, 2023), we use BLEU-4(Papineni etal., 2002), METEOR(Banerjee and Lavie, 2005), ROUGE-L(Lin, 2004), CIDEr(Vedantam etal., 2015), and MATCH(Zhang and Wan, 2023) scores to metric the results of CTG.³³3We utilize pycocoevalcap package to calucate BLEU-4, METEOR, ROUGE-L, CIDEr.

The overall score is calculated as the sum of the above scores. We set the weight of CIDEr to $1/10$ for balancing when summation.

$$S=\textrm{BLEU-4}+\textrm{METEOR}+\textrm{ROUGE-L}\\+\textrm{CIDER}/10+\textrm{MATCH}$$

As the overall score $S$ does not represent a percentile, we proceed to normalize the models’ scores to align with humans’ relative performance levels.

Appendix B Supplemental Materials