Mozi: A Scientific Large-scale Language Model

Team of Beijing Institute of Technology: Tian Lan^*, Tianyi Che*, Zewen Chi, Xuhao Hu and Xian-ling Mao

---

Catalogue:

• 1. Introduction
• 2. Running Mozi Demo
  • 2.1. Environment Installation
  • 2.2. Prepare Mozi Checkpoint
  • 2.5. Deploying Demo
• 3. Train Your Own SciDPR model
• 3. Train Your Own Mozi models
  • 3.1. Data Preparation
  • 3.2. Training Configurations
  • 3.3. Training Mozi
    • 3.3. Scientific Pre-Training Mozi Models
    • 3.3. Supervised Fine-tuning Mozi Models
• Citation
• Technical Report
• Acknowledgments

---

1. Introduction: [Back to Top]

Mozi is the first large-scale language model for the scientific paper domain, such as question answering and emotional support. With the help of the large-scale language and evidence retrieval models, SciDPR, Mozi generates concise and accurate responses to users' questions about specific papers and provides emotional support for academic researchers.

We will explore more real-world application scenarios for Mozi in the future, making it a foundation model for solving various scientific tasks.

---

2. Running Mozi Models: [Back to Top]

2.1. Environment Installation:

To install the required environment, please run

pip install -r requirements.txt

Then install the Pytorch package with the correct cuda version, for example

pip install torch==1.13.1+cu117 -f https://download.pytorch.org/whl/torch/

2.2. Prepare Mozi Checkpoint:

The Mozi model weights (pre-trained on scientific corpus) consists of the pre-trained large-scale language and the LoRA weights.

1. First of all, please download LLaMA-7B checkpoint and Baichuan-7B checkpoint.

2. Then, please download the LoRA weights for these two models from:

   LoRA checkpoints	Huggingface Delta Weights Address
   Baichuan-7B delta weight	mozi_baichuan_7b
   LLaMA-7B delta weight	mozi_llama_7b

3. We also release the delta LoRA model weights for scientific emotional dialogue, which can be found in here. The emotional dialogue delta weights are built on Baichuan-7B model. In the future, we will directly optimize this scientific emotional dialogue instruction tuning dataset with other instruction dataset, such as paper-ground question answering and scientific information retrieval.

Now, the model parameters are all prepared.

2.3. Deploying Demo:

Upon completion of previous steps, you can run the demo locally as

./scripts/deploy.sh

# #!/bin/bash
# CUDA_VISIBLE_DEVICES=0 python deploy.py \
#     --model scillm-sft\
#     --model_path baichuan-inc/baichuan-7B\
#     --delta_model_path ../ckpt/scillm-emotional-sft/18\
#     --port 23333

This script runs the Mozi model emotional model on 23333 port and the input POST request should be like:

{
    "decoding_method": "greedy",
    "top_p": 0.7,
    "top_k": 10,
    "penalty_alpha": 0.5,
    "max_new_tokens": 128,
    "history": [
        "Human: 最近科研压力真的好大啊"
    ]
}

If you want to test the paper-ground dialog model, please replace the --delta_model_path with the corresponding model checkpoints weights that you download. For the paper-ground dialog, the input POST request should be like:

{
    "decoding_method": "greedy",
    "top_p": 0.7,
    "top_k": 10,
    "penalty_alpha": 0.5,
    "max_new_tokens": 128,
    "evidences": [
        "During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive ...",
        "One way of circumventing this problem is to anonymise the data by removing, ...",
        "Given that this paper is concerned with text documents (e.g. medical records), the involved techniques are related to Natural Language Processing (NLP) ..."
    ],
    "question": "Which dataset do the author use in this paper?"
}

---

3. Train SciDPR model for Mozi Model: [Back to Top]

SciDPR servers the evidence retrieval component for Mozi paper-ground question answering models, which retrieves related evidences of user's queries. For model details about SciDPR model, please refer to this README.md file.

---

4. Train Your Own Mozi Model: [Back to Top]

Prerequisites: Before training the model, making sure the environment is properly installed and the checkpoints of LLaMA-7B and Baichuan-7B are downloaded.

4.1. Data Preparation: [Back to Top]

Declaimer: To ensure the reproducibility of our results, we have released our training dataset. The dataset must be used for research purpose only. The use of the dataset must comply with the licenses from original sources, i.e. QASPER and SciMRC. These datasets may be taken down when requested by the original authors.

Training Task	Dataset Address
Scientific Pre-training	Redpajama Dataset
Paper-ground Dataset QASPER	QASPER-v0.3 dataset
Paper-ground Dataset SciMRC	SciMRC dataset
Emotional Dataset	scientific-emotional-dialog

Due to the limited computation resources, we only collect 4B tokens from Redpajama arXiv corpus for the first version of scientific pre-training, and the downloading scripts could be found in this scripts.

After downloading, put the downloaded file and unzip them under the ./data/ directory, and the directory should look like:

.
├── pretrain
│   ├── download_from_hf.py
│   ├── test
│   │   └── collect.py
│   └── train
│       ├── combine_chinese_corpus.sh
│       └── split.sh
└── sft
    ├── alpaca
    │   └── alpaca_data.json
    ├── combine.py
    ├── dolly
    │   ├── download_from_hf.py
    │   └── train.json
    ├── emotional
    │   └── train.json
    ├── processed_qasper_test_set.json
    ├── processed_scimrc_test_set.json
    ├── qasper
    │   ├── README-test.md
    │   ├── README.md
    │   ├── collect.py
    │   ├── qasper-dev-v0.3.json
    │   ├── qasper-test-and-evaluator-v0.3.tgz
    │   ├── qasper-test-v0.3.json
    │   ├── qasper-train-dev-v0.3.tgz
    │   ├── qasper-train-v0.3.json
    │   ├── qasper_dev_sft.json
    │   ├── qasper_evaluator.py
    │   ├── qasper_sft.json
    │   ├── qasper_test_sft.json
    │   ├── qasper_train_sft.json
    │   ├── qasper_yes_no_test_sft.json
    │   ├── qasper_yes_no_train_sft.json
    │   └── scillm_test.json
    ├── scimrc
    │   ├── collect.py
    │   ├── scimrc_dev_sft.json
    │   ├── scimrc_test_sft.json
    │   ├── scimrc_train_sft.json
    │   ├── scimrc_yes_no_test_sft.json
    │   ├── scimrc_yes_no_train_sft.json
    │   ├── smrc_dev.jsonl
    │   ├── smrc_test.jsonl
    │   └── smrc_train.jsonl
    └── train.json

After downloading these datasets and saving them at the proper path, please refer to Dataset Prepareing Tutorial for preprocessing these for corpus for following training.

4.2 Training Configurations: [Back to Top]

To train the model properly, we use the QLoRA and deepspeed toolkit. Before running, please make sure these essential toolkit are downloaded.

The configurations about training are shown as follows:

Models	Model Name	Training Configurations
Scientific Pretraining	scillm	scillm-train; scillm-deepspeed
Paper-ground SFT	scillm-sft	scillm-sft-train; scillm-deepspeed
Paper-ground SFT	scillm-sft	scillm-sft-train; scillm-deepspeed

Please refer to these configuration file for more training details. Note that these models are pre-trained on 8 x 3090 (24G) GPUs for over 9 days (over 4B tokens). As for paper-ground question answering SFT, the training process cost less than 3 hours (with 2000 steps).

4.3. Training Mozi Models: [Back to Top]

4.3.1. Scientific Pre-training Mozi Models: [Back to Top]

To pre-train Mozi on scientific pre-training corpus with 4B tokens, please run the following commands:

./scripts/train_pretrain.sh

The key arguments of the training script are as follows:

• --model: The model name listed in config/base.json.
• --model_path: The checkpoint for large-scale langauge models, baichuan-inc/baichuan-7B for baichuan-7B model and decapoda-research/llama-7b-hf for LLaMA-7B.
• --train_data_path: The path saves the pretraining corpus.
• --log_path: The directory that saves the pre-trained log in tensorboard format. This directory will be automatically created.
• --save_path: The directory which saves the trained QLoRA delta weights. This directory will be automatically created.

Note that the total training steps can be set in the total_step argument at ./config/base.yaml file. Set this argument carefully to make sure all the tokens will be used during training.

4.3.2. Supversied Fine-tuning Mozi Models: [Back to Top]

Furthermore, to supervised fine-tune Mozi models on paper-ground question answering corpus, first make sure the dataset is set as QASPERDataset, and then please run the following commands:

./scripts/train_sft.sh

The key arguments of the training script are as follows:

• --model: The model name listed in config/base.json.
• --model_path: The checkpoint for large-scale langauge models, baichuan-inc/baichuan-7B for baichuan-7B model and decapoda-research/llama-7b-hf for LLaMA-7B.
• --delta_model_path: The LoRA checkpoint weighted pre-traing in Section 3.3.1. The SFT process will continue optimize these LoRA weights for paper-ground question answering task.
• --train_data_path: The path saves the pretraining corpus.
• --log_path: The directory that saves the pre-trained log in tensorboard format. This directory will be automatically created.
• --save_path: The directory which saves the trained QLoRA delta weights. This directory will be automatically created.

Note that the total training steps can be set in the total_step argument at ./config/base.yaml file. Set this argument carefully to make sure all the tokens will be used during training (2000 steps is enough in our hardware settings).

If you want to train the emotional dialog task, just simply replace the dataset with the path of emotional dataset path,and make sure the dataset_name in config/bash.yaml should also be set as EmotionalDataset.

---

Citation:

If you found Mozi models useful in your research or applications, please kindly cite using the following BibTeX:

...

Technical Report:

You can refer to our technical report for more details, which is saved in this path.

---

Acknowledgments:

This repo benefits from OpenAlpaca, PandaGPT. Thanks for their wonderful works!