Skip to content

Latest commit

 

History

History
75 lines (52 loc) · 2.75 KB

README.md

File metadata and controls

75 lines (52 loc) · 2.75 KB
Raw

DAACS-NLP

project set up

1 download

git clone wherever you want to run this project from

2 build your python virtualenv

python3 -m venv .venv     # build your project virtual env
source .venv/bin/activate # activate your virual env
.venv/bin/python -m pip install --upgrade pip
.venv/bin/python -m pip install -r ./requirements.txt

Now you can activate and deactivate that specific python env and path like this:

user@host: source ./.venv/bin/activagte
user@host: deactivate

Clean you pip packages

Do this every now and then! This uninstall/reinstalls all pip packages back to exactly what is in requirements.txt.

pip freeze | xargs pip uninstall -y

Edit your .env file.

Make sure you have python path and OPENAPI_KEY in there, like this.

cat ./.env

PYTHONPATH=/Users/afraser/Documents/src/daacs-nlp/src:/Users/afraser/Documents/src/daacs-nlp/.venv/lib/python3.11/site-packages
OPENAI_API_KEY=sk-2r7jkiZZNSdJEBrZKnkuT3Blb*******STUFFGETYOUROWN*********

Edit your .venv/bin/activate

Should look about like this:

# ... stuff 
VIRTUAL_ENV="/Users/afraser/Documents/src/daacs-nlp/.venv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
# ... enter it here, where other envs are set.
export PYTHONPATH=/Users/afraser/Documents/src/daacs-nlp/src:/Users/afraser/Documents/src/daacs-nlp/.venv/lib/python3.11/site-packages
##

Logging into hugging face, to use the llama2 vectorizer

  1. Create an HF account, and an organization for yourself
  2. Go to your hf account, click create token, make sure it's a write token, not just a read token
  3. In your shell, activate your virtual env
  4. In HF, go find the model, then fill out the form to gain access. Currently is this. : https://huggingface.co/meta-llama/Meta-Llama-Guard-2-8B You'll have to be approved, so you might have to wait a few hours.
  5. run huggingface-cli login and paste in your write token.
  6. ALSO set your token as a shell variable, like this. (.venv) hurricane:daacs-nlp afraser$ export HUGGINGFACE_TOKEN=hf_fdtykEqnsbhEF-GET-YOUR-OWNClfjMgDlHPHkLGnr. Agin note the shell should be
  7. Before you run (.venv) hurricane:daacs-nlp afraser$ python ./src/daacs/runnable/build_vector_cache.py you can test if you're set up correctly by running (.venv) hurricane:daacs-nlp afraser$ huggingface-cli 'download' "meta-llama/Meta-Llama-Guard-2-8B