Efficient LLM for Taiwan with open weights/datasets/checkpoints and affordable sizes (270M/1.1B)

🤗 Model Collection • 📖 Traditional Chinese Corpus

Taiwan ELM

Taiwan ELM is a family of Efficient LLMs for Taiwan base on apple/OpenELM. The project aims to provide an efficient model for researchers without access to large-scale computing resources.

The model is trained using LLaMA-Factory on 2B Traditional Chinese tokens and 500K instruction samples. We will extend the model to train on larger data sets and different base models if there is sufficient demand.

What is being released?

We release both pre-trained base models and instruction tuned variants with 270M and 1.1B parameters. Along with the model, datasets used to train the models are also released.

In an effort to improve transparency, training checkpoints (including rng/optimizer state) and logs are also released in the model page.

List of released models:

• Taiwan-ELM-270M
• Taiwan-ELM-1_1B
• Taiwan-ELM-270M-Instruct
• Taiwan-ELM-1_1B-Instruct

List of released datasets:

• liswei/Taiwan-Text-Excellence-2B
• liswei/PromptPair-TW
• liswei/wikinews-zhtw-dedup
• liswei/wikipedia-zhtw-dedup
• liswei/coct-en-zhtw-dedup

Some of the datasets are not used for training Taiwan ELM but also released:

• liswei/common-crawl-zhtw
• liswei/c4-zhtw
• liswei/rm-static-zhTW

Codebase:

• Dataset Cleaning
• Tokenizer Training
• Training: LLaMA-Factory - 5177f3b

Usage Examples

For instruction-tuned modesl, we adapt the LLaMA2 template:

<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>

{{ user_message }} [/INST]

The model could be load via AutoModelForCausalLM or text-generation-inference with trust_remote_code=True:

taiwan_elm_270m = AutoModelForCausalLM.from_pretrained("liswei/Taiwan-ELM-270M", trust_remote_code=True)

We also support additional generation methods and speculative generation, please find reference at OpenELM#usage.