llm-logic

Supplementary Material for "Conditional and Modal Reasoning in Large Language Models"

Paper by Wesley H. Holliday (wesholliday@berkeley.edu), Matthew Mandelkern (mandelkern@nyu.edu), and Cedegao E. Zhang (cedzhang@mit.edu) available at https://arxiv.org/abs/2401.17169, forthcoming in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (https://2024.emnlp.org).

Contents

• llm-logic.ipynb: The Jupyter notebook containing all the code used to run the experiments and generate the graphs presented in the paper.

• prompts/: Includes all of the inference pattern questions used in the experiments.

• data/: Stores the LLM responses to the inference pattern questions.

• external_data/: Stores the GSM8K and MMLU scores for the tested models

• graphs/ and results/: These directories are generated upon running the "llm-logic" notebook and contain the graphs and results of experiments, respectively.

Usage

To replicate the experiments or explore the results, open and run the "llm-logic.ipynb" Jupyter notebook.

Contact

For any queries related to the paper or the supplementary material, please write to wesholliday@berkeley.edu.