Deep text clustering with language models.

This repo contains the code and data for my term paper "Fine-tuning language models on text clustering."

Project structure

transformers_clustering

This folder contains the clustering module. It should be installed using pip.

experiments

Contains the code for all experiments presented in the term paper (and artefacts of old experiments).

datasets

Stores the datasets used for all experiments.

results

Will contain the results of each experiments.

misc

Contains some notebooks used to create plots and tables and some relicts of the development process.

Dependencies:

Install all dependencies, either using pip or conda.

# Very likely it won't work:
pip install -r requirements.txt

or

# This will create a new conda environment.
# Either:
conda create --name <env> --file requirements_conda.txt

# Or (better):
conda env create -f conda_environment.yml
# The latter will create a conda env named deep_text_clustering, containing nearly all required packages.

If you are using conda, it is necessary to install these packages manually:

• plotly-orca
• torchtext
• umap-learn
• transformers
• cudatoolkit
• sacred

Installation

Install the clustering module using pip:

# From project root.
pip install - e .

Create datasets

Each datasets come with a script that downloads and creates the dataset as csv files.

AG_News

To create the AG_News datasets first run the make_ag_news_datasets.py from the datasets/ag_news folder. To create each subset and save the train and validation splits run the ag_news_create_splits.py from its location inside the datasets folder.`

Trec 6 and 20 Newsgroups

To create these datasets just run their creation script from inside their folders.

Run the experiments

The experiments have to be started from inside the ``experiments` folder. Sacred is used to keep track of all experiments. If you want to enable sacred to save all results using MongoDB, you can specify the following environments variables:

• MONGO_SACRED_ENABLED = true
  • (If it is set to false the experiments will only be tracked using local files)
• mongo_user = MongoDB username
• mongo_pass = Password for the user
• mongo_host = Host of the MongoDB instance
• mongo_port = Port of the MongoDB instance

To run certain experiments without MongoDB tracking, you start it with the flag:

MONGO_SACRED_ENABLED=false python <experiment>.py

No matter if MongoDB is enabled each experiment will be tracked locally and the results are written to results/sacred_runs Parameters of each run can be changed using the sacred syntax:

python <experiment>.py with lr=1.0 n_epochs=1

The results of each run will be written to a folder following the pattern: <project_root>/results/<experiment_name>/<timestamp>