Skip to content

Latest commit

 

History

History
88 lines (66 loc) · 4.98 KB

README.md

File metadata and controls

88 lines (66 loc) · 4.98 KB
Raw

economic_games_chatgpt

example workflow

Setting up the virtual environment

We use Python version 3.10.8 to create a virtual environment to be used in this project

python3.10 -m venv .venv
source .venv/bin/activate # (for mac/linux users)
# source .venv/Scripts/Activate (for windows users)

(.venv) pip install --upgrade pip
(.venv) pip install wheel
(.venv) pip install -r requirements.txt

Using ChatGPT with the OpenAI API

The OpenAI API makes it possible to send a request to OpenAI language models and obtain a response back. In order to use the API an OpenAI account is necessary. An API key is also needed in order to use the API (more info here: https://beta.openai.com/docs/api-reference/authentication). Once you have an OpenAI account and have obtained an API key associated to the account create a .env file in the root folder of the repository

OPENAI_API_KEY=<your_api_key_here>

At CHM an Azure OpenAI instance of the gpt-3.5-turbo model is available with an API key.

Running the analysis script

In order to run the analysis script ai_participant.py specify the economic game, the action you want to perform and the type of language model you want to use. Economic game can be one of the following options:

  • ultimatum_sender (sender perspective)
  • ultimatum_receiver (receiver perspective)
  • dictator_sequential
  • dictator_sender (sender perspective)
  • dictator_binary

The action to perform can be one of the following options:

  • test_assumptions (this action sends prompts that test the language model's knowledge of the selected game and checks for the assumptions that the language model has on the game dynamics)
  • prepare_prompts (this action prepares all prompts in a csv file or dataframe)
  • convert_prompts_to_json (this action prepares all prompts from csv file to jsonl file, relevant for using EC2)
  • send_prompts_baseline (this action sends prompts for a specific game, in baseline condition)
  • send_prompts_experimental (this action sends prompts for a specific game, in experimental condition)
  • convert_jsonl_results_to_csv (this action prepares all results from jsonl file to csv file, relevant for using EC2)

The language model type can be one of the following options:

  • text-davinci-003 (refers to the text-davinci class of language models, in particular this option stands for text-davinci-003) NOTE: DISCARDED
  • gpt-35-turbo (refers to the gpt-3.5-turbo language model)

E.g. run the script locally on the trust game from the sender perspective, send all prompts to the language model and save responses

(.venv) python ai_participant.py --game trust_sender --mode send_prompts --model gpt-35-turbo

Sending prompts on AWS EC2

Calling the API from the script can be a slow process; for this study we try to make the process faster by running the code on an AWS EC2 instance located in the US (where the Azure OpenAI model "resides"). We also adapt a parallel processor script provided by OpenAI (see https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py) that allows parallel api requests using async.

Note that using this method requires data to be organized in a jsonl file, where each line contains the prompt to send and metadata such as temperature or frequency penalty (the jsonl file can be prepared beforehand using the ai_participant.py script).

To run the script api_helper.py on EC2 navigate to your project directory and execute the following command on the EC2 terminal (the language model used by default is gpt-35-turbo, and we explicitly say whether we want to collect the baseline or experimental prompts)

(.venv) python3 api_helper.py --game dictator_sender --exptype baseline

Factor Levels for prompts

For each game a parameter file in the params folder of the repository is created. The parameter file contains all prompts used in the study, organized by different languages and by prompt type (baseline or experimental). Additionally, the parameter file includes all factors and language model parameters that are used in the study.

Data files

Each economic game will be associated to a set of data files:

  • {game}_{exp_type}_prompts : this contains the prompts that have been sent to the language model, in csv and jsonl format
  • {game}_{exp_type}_results : this contains the results for each prompt, in csv and jsonl format (results include also the original metadata like temperature or age/gender levels for each prompt)
  • {game}_{exp_type}_completion_ids_to_check these txt files give a list of the completion ids (response ids) that Python didn't manage to parse in json format. Despite asking the language model to return responses in json format, sometimes the response returned donesn't follow json syntax, and is not recognized by Python. For these ids manual check in the corresponding jsonl result file is needed to see if a response can be still extracted or not.