Skip to content

Latest commit

 

History

History
64 lines (39 loc) · 1.64 KB

README.md

File metadata and controls

64 lines (39 loc) · 1.64 KB
Raw

Yarn Cleaner

This module can be used to monitor and clean the temporary directories that are populated during Yarn jobs.

Features

  • Checks disk usage of Yarn and kills the application that is using up more than a given threshold
  • Connects to a remote node via SSH and runs commands

Prerequisites

  • python3
  • paramiko
  • argparse

Installation

Clone this repository:

git clone https://github.com/stvoutsin/yarncleaner.git

cd into the repository:

cd yarncleaner

Install dependencies:

  pip install -r requirements.txt

Usage

Using the Command Line This module can be run from the command line with the following command:

sh

python yarncleaner.py [--workers [WORKERS [WORKERS ...]]] [--ssh-username SSH_USERNAME] [--ssh-key-file SSH_KEY_FILE] [--usercache-dir USERCACHE_DIR] [--threshold-percent THRESHOLD_PERCENT]

Arguments

Argument Description
--workers A list of workers
--ssh-username The username to use for SSH
--ssh-key-file The path to the private key file to use for SSH
--usercache-dir The directory where the usercache is located
--threshold-percent The percentage of disk usage to trigger a clean

Using the API

This module can also be used as an API:

from yarncleaner import YarnCleaner
workers = ["worker01", "worker02", "worker03"]
ssh_username = "username"
ssh_key_file = "/path/to/key/file.pem"
cleaner = YarnCleaner(workers=workers, ssh_username=ssh_username, ssh_key_file=ssh_key_file)
cleaner.clean()

License

This module is licensed under the GNU GENERAL PUBLIC LICENSE.