diff --git a/README.md b/README.md index c8606e8..288116d 100644 --- a/README.md +++ b/README.md @@ -1,190 +1,156 @@ -# LLM Alignment Assistant +# ๐ LLM Alignment Assistant - Your Template for Aligning Language Models -## ๐ Overview +## ๐ Introduction -**LLM Alignment Assistant** is an advanced tool designed to assist in aligning large language models (LLMs) with desired human values and objectives. This project offers a full-stack approach to training, fine-tuning, deploying, and monitoring LLMs using **Reinforcement Learning from Human Feedback (RLHF)**. The system also incorporates evaluation metrics to ensure ethical and effective use of language models. The assistant provides a user-friendly interface for exploring the alignment, visualization of training metrics, and deploying the system at scale using cloud-native technologies. +**LLM Alignment Assistant** is not just a comprehensive tool for aligning large language models (LLMs), but also serves as a **powerful template** for building your own LLM alignment application. This repository is designed to provide a full stack of functionality, acting as a starting point to customize and extend for your own LLM alignment needs. Whether you are a researcher, developer, or data scientist, this template provides a solid foundation for efficiently creating and deploying LLMs tailored to align with human values and objectives. -![โจ Architecture Diagram](assets/architecture_diagram.png) -## โจ Key Features +## โจ Features -- **๐ฅ๏ธ User-Friendly Web Interface**: A sleek, intuitive UI for interacting with the LLM and viewing alignment results. -- **๐ Interactive Training**: Train models using RLHF, with dynamic metrics displayed in real-time. -- **๐ ๏ธ Data Augmentation & Preprocessing**: Advanced preprocessing scripts, including tokenization, cleaning, and data augmentation using NLP techniques. -- **โ๏ธ Scalable Deployment**: Easy deployment via Docker and Kubernetes, with horizontal scaling capabilities. -- **๐ Explainability & Monitoring**: Incorporates SHAP or LIME-based explainability features along with live monitoring dashboards. +- **๐ Interactive Web Interface**: A user-friendly interface for interacting with the LLM, training models, and viewing alignment metrics. +- **๐ง Training with RLHF**: Reinforcement Learning from Human Feedback to ensure model alignment with human preferences. +- **๐ ๏ธ Data Augmentation & Preprocessing**: Advanced preprocessing, tokenization, and data augmentation with back-translation and paraphrasing. +- **๐ Transfer Learning**: Utilize pre-trained models like BERT for improved performance on specific tasks. +- **๐ฆ Scalable Deployment**: Docker and Kubernetes-based deployment with Horizontal Pod Autoscaling (HPA). +- **๐ Model Explainability**: SHAP-based dashboards for understanding model decisions. +- **๐ User Feedback Loop**: Collection of user ratings for fine-tuning models continuously. -## ๐๏ธ Project Structure +## ๐ Project Structure -- **๐ app/**: Contains the UI and the backend logic of the web interface. - - `ui.py`: Manages routes and interactions with the UI. - - `static/`: Contains styles and JavaScript for an appealing UI. - - `templates/`: HTML templates for rendering the web pages. -- **๐ data/**: Scripts and datasets for generating, downloading, and processing data. -- **๐ deployment/**: Docker, Kubernetes configurations, and Helm charts to manage deployments. -- **๐ src/**: Core functionality, including training, evaluation, and preprocessing scripts. -- **๐ tests/**: Unit and integration tests to ensure quality across the different components. +- **`app/`**: Contains API and UI code. + - `auth.py`, `feedback.py`, `ui.py`: API endpoints for user interaction, feedback collection, and general interface management. + - **Static Files**: JavaScript (`app.js`, `chart.js`), CSS (`styles.css`), and Swagger API documentation (`swagger.json`). + - **Templates**: HTML templates (`chat.html`, `feedback.html`, `index.html`) for UI rendering. -## ๐ ๏ธ Setup +- **`src/`**: Core logic and utilities for preprocessing and training. + - **Preprocessing** (`preprocessing/`): + - `preprocess_data.py`: Combines original and augmented datasets and applies text cleaning. + - `tokenization.py`: Handles tokenization. + - **Training** (`training/`): + - `fine_tuning.py`, `transfer_learning.py`, `retrain_model.py`: Scripts for training and retraining models. + - `rlhf.py`, `reward_model.py`: Scripts for reward model training using RLHF. + - **Utilities** (`utils/`): Common utilities (`config.py`, `logging.py`, `validation.py`). -### ๐ Prerequisites +- **`dashboards/`**: Performance and explainability dashboards for monitoring and model insights. + - `performance_dashboard.py`: Displays training metrics, validation loss, and accuracy. + - `explainability_dashboard.py`: Visualizes SHAP values to provide insight into model decisions. -- Python 3.8+ -- Docker & Docker Compose -- Kubernetes (Minikube or any cloud provider) -- Node.js (for front-end enhancements) +- **`tests/`**: Unit, integration, and end-to-end tests. + - `test_api.py`, `test_preprocessing.py`, `test_training.py`: Various unit and integration tests. + - **End-to-End Tests** (`e2e/`): Cypress-based UI tests (`ui_tests.spec.js`). + - **Load Testing** (`load_testing/`): Uses Locust (`locustfile.py`) for load testing. -### ๐ง Installation +- **`deployment/`**: Configuration files for deployment and monitoring. + - **Kubernetes Configurations** (`kubernetes/`): Deployment and Ingress configurations for scaling and canary releases. + - **Monitoring** (`monitoring/`): Prometheus (`prometheus.yml`) and Grafana (`grafana_dashboard.json`) for performance and system health monitoring. -1. **๐ฅ Clone the Repository**: - ```bash - git clone https://github.com/yourusername/LLM-Alignment-Assistant.git - cd LLM-Alignment-Assistant - ``` +## โ๏ธ Setup -2. **๐ฆ Set Up the Virtual Environment**: - ```bash - python3 -m venv venv - source venv/bin/activate - pip install -r requirements.txt - ``` +### Prerequisites + +- ๐ Python 3.8+ +- ๐ณ Docker & Docker Compose +- โธ๏ธ Kubernetes (Minikube or a cloud provider) +- ๐ข Node.js (for front-end dependencies) -3. **๐ฆ Install Node.js Dependencies** (optional for enhanced UI): +### ๐ฆ Installation + +1. **Clone the Repository**: ```bash - cd app/static - npm install + git clone https://github.com/yourusername/LLM-Alignment-Assistant.git + cd LLM-Alignment-Assistant ``` -### ๐ Running Locally +2. **Install Dependencies**: + - Python dependencies: + ```bash + pip install -r requirements.txt + ``` + - Node.js dependencies (optional for UI improvements): + ```bash + cd app/static + npm install + ``` -To run the application locally: +### ๐ Running Locally -1. **๐ณ Build Docker Image**: +1. **Build Docker Images**: ```bash docker-compose up --build ``` -2. **๐ Access the UI**: - Visit `http://localhost:5000` in your web browser. +2. **Access the Application**: + - Open a browser and visit `http://localhost:5000`. -## ๐ฆ Deployment +## ๐ข Deployment -### ๐ข Docker and Kubernetes +### โธ๏ธ Kubernetes Deployment -- **๐ณ Docker**: A Dockerfile is provided for containerization. -- **โธ๏ธ Kubernetes**: Use the provided `deployment/kubernetes/deployment.yml` and `service.yml` files to deploy the app to a Kubernetes cluster. -- **๐ Helm Charts**: Helm charts are available in the `deployment/helm/` directory for easier reusability and scalability. +- **Deploy to Kubernetes**: + - Apply the deployment and service configurations: + ```bash + kubectl apply -f deployment/kubernetes/deployment.yml + kubectl apply -f deployment/kubernetes/service.yml + ``` + - **Horizontal Pod Autoscaler**: + ```bash + kubectl apply -f deployment/kubernetes/hpa.yml + ``` -### ๐ CI/CD Pipeline +### ๐ Canary Deployment -A GitHub Actions workflow is included to automate building, testing, and deployment: +- Canary deployments are configured using `deployment/kubernetes/canary_deployment.yml` to roll out new versions safely. -- **โ Lint & Test**: Linting and unit tests are run at every pull request. -- **๐ Docker Build & Push**: Docker images are built and pushed to Docker Hub. -- **โธ๏ธ Kubernetes Deployment**: Automatically deploy to the Kubernetes cluster upon merging. +### ๐ Monitoring and Logging -## ๐ค Training and Fine-Tuning +- **Prometheus and Grafana**: + - Apply Prometheus and Grafana configurations in `deployment/monitoring/` to enable monitoring dashboards. +- **๐ Centralized Logging**: The **ELK Stack** is configured with Docker using `docker-compose.logging.yml` for centralized logs. -### ๐ก Reinforcement Learning from Human Feedback (RLHF) +## ๐ง Training and Evaluation -The training module includes: -- **๐ Fine-Tuning**: Using the `training/fine_tuning.py` script, models can be fine-tuned on specific datasets. -- **๐ Reward Models**: Implemented in `training/reward_model.py` for evaluating the appropriateness of responses. -- **๐ Distributed Training**: Support for distributed training using `training/rlhf.py`. +### ๐ Transfer Learning -### ๐๏ธ Hyperparameter Tuning +The training module (`src/training/transfer_learning.py`) uses pre-trained models like **BERT** to adapt to custom tasks, providing a significant performance boost. -For hyperparameter tuning, **Optuna** has been integrated to provide automated exploration of the training parameters, ensuring optimal model performance. +### ๐ Data Augmentation -## ๐ Data Pipeline +The `data_augmentation.py` script (`src/data/`) applies augmentation techniques like back-translation and paraphrasing to improve data quality. -- **๐ ๏ธ Data Augmentation**: Using advanced NLP techniques, including back-translation and embedding-based masking, available in `preprocessing/augmentation.py`. -- **โ Validation**: Thorough validation scripts (`preprocess_data.py` and `validate_data.py`) to maintain data quality. -- **โ๏ธ Automation with Apache Airflow**: Data pipeline orchestration using Airflow, ensuring proper data flow between stages. +### ๐ง Reinforcement Learning from Human Feedback (RLHF) -## ๐ Evaluation and Monitoring +- **Reward Model Training**: Uses the `rlhf.py` and `reward_model.py` scripts to fine-tune models based on human feedback. +- **Feedback Collection**: Users rate responses via the feedback form (`feedback.html`), and the model retrains with `retrain_model.py`. -- **๐ Metrics**: The `evaluation/metrics.py` script provides a detailed analysis of model performance, including bias detection and fairness metrics. -- **๐ก๏ธ Safety Testing**: Ethical AI assessments using `evaluation/safety_tests.py`. -- **๐ Dashboard**: Real-time monitoring with **Streamlit**, displaying key metrics, including training loss, accuracy, and reward trends. +### ๐ Explainability Dashboard -## ๐ Web Interface Improvements - -- **๐จ Improved UI with TailwindCSS**: We've enhanced the CSS for modern and engaging aesthetics. -- **๐ Interactive Visualizations**: Added **Chart.js** visualizations to present alignment metrics in a clear, graphical format. -- **๐ฌ Chatbot Integration**: A conversational UI element to interact directly with the trained LLM. +The `explainability_dashboard.py` script uses **SHAP** values to help users understand why a model made specific predictions. ## ๐งช Testing -- **โ Unit Tests**: Located in `tests/`, covering training, preprocessing, and evaluation. -- **๐ Integration Tests**: End-to-end tests that simulate full pipeline execution. -- **๐งช Mock Testing**: Use of `pytest-mock` to simulate API calls and external integrations. - -## ๐ Monitoring and Logging - -- **๐ Monitoring**: Kubernetes monitoring using **Prometheus** and **Grafana**, with Horizontal Pod Autoscaling (HPA) for scalability. -- **๐ Explainability**: SHAP and LIME explainability metrics are added to the evaluation process, providing insights into model behavior. -- **๐ Logging**: Centralized logging using **ELK Stack** (Elasticsearch, Logstash, Kibana). - -## ๐ Cloud Deployment Instructions (AWS) - -To deploy the LLM Alignment Assistant on **AWS**, you can utilize **Elastic Kubernetes Service (EKS)** or **AWS Sagemaker** for model training: - -1. **AWS Elastic Kubernetes Service (EKS)**: - - Create an EKS cluster using AWS CLI or the console. - - Apply the Kubernetes deployment files: - ```bash - kubectl apply -f deployment/kubernetes/deployment.yml - kubectl apply -f deployment/kubernetes/service.yml - ``` - - Configure the **Horizontal Pod Autoscaler (HPA)** to ensure scalability: - ```bash - kubectl apply -f deployment/kubernetes/hpa.yml - ``` - -2. **AWS Sagemaker for Model Training**: - - Modify the `training/fine_tuning.py` to integrate with AWS Sagemaker. - - Use the Sagemaker Python SDK to launch a training job: - ```python - import sagemaker - from sagemaker.pytorch import PyTorch - - role = "arn:aws:iam::your-account-id:role/service-role/AmazonSageMaker-ExecutionRole-2023" - - pytorch_estimator = PyTorch( - entry_point='training/fine_tuning.py', - role=role, - instance_count=1, - instance_type='ml.p2.xlarge', - framework_version='1.8.0', - py_version='py3' - ) - - pytorch_estimator.fit({'training': 's3://your-bucket-name/training-data'}) - ``` - - Ensure IAM roles and permissions are properly set for accessing **S3** and **Sagemaker**. - +- **โ Unit Tests**: Located in `tests/`, covering API, preprocessing, and training functionalities. +- **๐ฅ๏ธ End-to-End Tests**: Uses **Cypress** to test UI interactions. +- **๐ Load Testing**: Implemented with **Locust** (`tests/load_testing/locustfile.py`) to ensure stability under load. -## ๐ Future Work +## ๐ฎ Future Work -- **๐ Multi-Language Support**: Expand the LLM's training to support multiple languages. -- **โ๏ธ Ethical AI Enhancements**: Further enhance bias detection and mitigation techniques. -- **โ๏ธ Cloud-Native Deployment**: Implement cloud services like **AWS SageMaker** for training at scale. +- **๐ User Roles and Permissions**: Adding a role-based access control system. +- **๐ Advanced Monitoring**: Further enhance Prometheus alerts for anomaly detection. +- **๐ Public Demo Deployment**: Deploy a public version on Heroku or AWS for showcasing. -## ๐ค Getting Involved +## ๐ค Contributing -Contributions are welcome! Feel free to submit issues, pull requests, or suggestions for new features. +Contributions are welcome! Please submit pull requests or issues for improvements or new features. ## ๐ License -This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details. +This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information. ## ๐ฌ Contact -- **โ๏ธ Email**: [amirsina.torfi@gmail.com](mailto:amirsina.torfi@gmail.com) -- **๐ Website**: [Portfolio](https://astorfi.github.io) +- **๐ง Email**: [amirsina.torfi@gmail.com](mailto:amirsina.torfi@gmail.com) +- **๐ Website**: [Your Portfolio](https://astorfi.github.io) --- -
Made with โค๏ธ by Amirsina Torfi
- +Developed with โค๏ธ by Your Name