Repositories list

• warc2text-runner

  Public
  Scripts for parallelized extraction of plain texts from WARC archieves. Aiming at common and reproducible extraction approach.

  HTML

  •0•3•5•0•Updated Nov 9, 2024Nov 9, 2024

• OpusPocus

  Public
  Marian machine translation training pipeline for thousands of models

  Python

  •0•2•19•0•Updated Nov 8, 2024Nov 8, 2024

• data-analytics-tool

  Public
  Data Analytics Tool

  JavaScript

  •1•9•0•0•Updated Nov 7, 2024Nov 7, 2024

• bitextor-slurm

  Public
  Scripts for running bitextor jobs

  Shell

  •1•0•1•0•Updated Nov 6, 2024Nov 6, 2024

• monotextor-slurm

  Public
  Set of scripts to run monotextor-like pipeline under slurm HPCs

  Rust

  •

  GNU General Public License v3.0

  •0•2•0•0•Updated Nov 4, 2024Nov 4, 2024

• monolingual-multilingual-instruction-tuning

  Public
  Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca

  large-language-modelsinstruction-tuninginstruction-data

  Python

  •0•8•0•0•Updated Nov 2, 2024Nov 2, 2024

• release2_inspection

  Public
  Jupyter Notebook

  •5•1•0•1•Updated Oct 29, 2024Oct 29, 2024

• bitextor-mt-models

  Public
  Shell

  •0•1•3•0•Updated Oct 17, 2024Oct 17, 2024

• cc-download

  Public
  Shell

  •0•0•0•0•Updated Oct 15, 2024Oct 15, 2024

• OpusTrainer

  Public
  Curriculum training

  machine-translationtraining-pipeline

  Python

  •

  MIT License

  •5•16•19•0•Updated Sep 14, 2024Sep 14, 2024

• OpusCleaner

  Public
  OpusCleaner is a web interface that helps you select, clean and schedule your data for training machine translation models.

  machine-translationdata-cleaning

  Python

  •13•48•56•1•Updated Sep 7, 2024Sep 7, 2024

• ia-download

  Public
  Internet archive downloader

  Jupyter Notebook

  •0•2•1•0•Updated Aug 7, 2024Aug 7, 2024

• HPLT-WP4

  Public
  Information and pipelines on WP4: language models training

  Python

  •

  Creative Commons Zero v1.0 Universal

  •2•1•0•0•Updated Jul 11, 2024Jul 11, 2024

• sacremoses

  Public
  Python port of Moses tokenizer, truecaser and normalizer

  nlptokenizermachine-translation

  Python

  •

  MIT License

  •57•487•27•4•Updated May 26, 2024May 26, 2024

• document-aligner

  Public
  tf/idf-based document aligner from Bitextor

  C++

  •

  Apache License 2.0

  •0•0•0•1•Updated Mar 19, 2024Mar 19, 2024

• OPUS-MT-dashboard

  Public
  PHP

  •

  MIT License

  •1•0•0•0•Updated Mar 9, 2024Mar 9, 2024

• HPLT-MT-Models

  Public
  This contains the configuration and scripts for HPLT MT model releases.

  Python

  •0•4•1•0•Updated Mar 6, 2024Mar 6, 2024

• OpusFilter

  Public
  OpusFilter - Parallel corpus processing toolkit

  Python

  •

  MIT License

  •18•0•0•0•Updated Jan 3, 2024Jan 3, 2024

• clianer

  Public
  A lightweight command-line frontend to OpusCleaner

  Python

  •

  MIT License

  •1•0•0•0•Updated Nov 27, 2023Nov 27, 2023

• paracrawl-dashboard

  Public
  Make-shift interface for managing Paracrawl processing and exploring its outputs

  HTML

  •1•0•0•0•Updated Oct 10, 2023Oct 10, 2023

• MT-winterschool-2023

  Public
  0•1•0•0•Updated Feb 7, 2023Feb 7, 2023