-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
101 additions
and
135 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,190 +1,156 @@ | ||
| # LLM Alignment Assistant | ||
| # 🌌 LLM Alignment Assistant - Your Template for Aligning Language Models | ||
|
|
||
| ## 🌟 Overview | ||
| ## 📌 Introduction | ||
|
|
||
| **LLM Alignment Assistant** is an advanced tool designed to assist in aligning large language models (LLMs) with desired human values and objectives. This project offers a full-stack approach to training, fine-tuning, deploying, and monitoring LLMs using **Reinforcement Learning from Human Feedback (RLHF)**. The system also incorporates evaluation metrics to ensure ethical and effective use of language models. The assistant provides a user-friendly interface for exploring the alignment, visualization of training metrics, and deploying the system at scale using cloud-native technologies. | ||
| **LLM Alignment Assistant** is not just a comprehensive tool for aligning large language models (LLMs), but also serves as a **powerful template** for building your own LLM alignment application. This repository is designed to provide a full stack of functionality, acting as a starting point to customize and extend for your own LLM alignment needs. Whether you are a researcher, developer, or data scientist, this template provides a solid foundation for efficiently creating and deploying LLMs tailored to align with human values and objectives. | ||
|
|
||
|  | ||
|
|
||
| ## ✨ Key Features | ||
| ## ✨ Features | ||
|
|
||
| - **🖥️ User-Friendly Web Interface**: A sleek, intuitive UI for interacting with the LLM and viewing alignment results. | ||
| - **📊 Interactive Training**: Train models using RLHF, with dynamic metrics displayed in real-time. | ||
| - **🛠️ Data Augmentation & Preprocessing**: Advanced preprocessing scripts, including tokenization, cleaning, and data augmentation using NLP techniques. | ||
| - **⚙️ Scalable Deployment**: Easy deployment via Docker and Kubernetes, with horizontal scaling capabilities. | ||
| - **🔍 Explainability & Monitoring**: Incorporates SHAP or LIME-based explainability features along with live monitoring dashboards. | ||
| - **🌐 Interactive Web Interface**: A user-friendly interface for interacting with the LLM, training models, and viewing alignment metrics. | ||
| - **🧠 Training with RLHF**: Reinforcement Learning from Human Feedback to ensure model alignment with human preferences. | ||
| - **🛠️ Data Augmentation & Preprocessing**: Advanced preprocessing, tokenization, and data augmentation with back-translation and paraphrasing. | ||
| - **🔄 Transfer Learning**: Utilize pre-trained models like BERT for improved performance on specific tasks. | ||
| - **📦 Scalable Deployment**: Docker and Kubernetes-based deployment with Horizontal Pod Autoscaling (HPA). | ||
| - **🔍 Model Explainability**: SHAP-based dashboards for understanding model decisions. | ||
| - **📊 User Feedback Loop**: Collection of user ratings for fine-tuning models continuously. | ||
|
|
||
| ## 🗂️ Project Structure | ||
| ## 📂 Project Structure | ||
|
|
||
| - **📁 app/**: Contains the UI and the backend logic of the web interface. | ||
| - `ui.py`: Manages routes and interactions with the UI. | ||
| - `static/`: Contains styles and JavaScript for an appealing UI. | ||
| - `templates/`: HTML templates for rendering the web pages. | ||
| - **📁 data/**: Scripts and datasets for generating, downloading, and processing data. | ||
| - **📁 deployment/**: Docker, Kubernetes configurations, and Helm charts to manage deployments. | ||
| - **📁 src/**: Core functionality, including training, evaluation, and preprocessing scripts. | ||
| - **📁 tests/**: Unit and integration tests to ensure quality across the different components. | ||
| - **`app/`**: Contains API and UI code. | ||
| - `auth.py`, `feedback.py`, `ui.py`: API endpoints for user interaction, feedback collection, and general interface management. | ||
| - **Static Files**: JavaScript (`app.js`, `chart.js`), CSS (`styles.css`), and Swagger API documentation (`swagger.json`). | ||
| - **Templates**: HTML templates (`chat.html`, `feedback.html`, `index.html`) for UI rendering. | ||
|
|
||
| ## 🛠️ Setup | ||
| - **`src/`**: Core logic and utilities for preprocessing and training. | ||
| - **Preprocessing** (`preprocessing/`): | ||
| - `preprocess_data.py`: Combines original and augmented datasets and applies text cleaning. | ||
| - `tokenization.py`: Handles tokenization. | ||
| - **Training** (`training/`): | ||
| - `fine_tuning.py`, `transfer_learning.py`, `retrain_model.py`: Scripts for training and retraining models. | ||
| - `rlhf.py`, `reward_model.py`: Scripts for reward model training using RLHF. | ||
| - **Utilities** (`utils/`): Common utilities (`config.py`, `logging.py`, `validation.py`). | ||
|
|
||
| ### 📋 Prerequisites | ||
| - **`dashboards/`**: Performance and explainability dashboards for monitoring and model insights. | ||
| - `performance_dashboard.py`: Displays training metrics, validation loss, and accuracy. | ||
| - `explainability_dashboard.py`: Visualizes SHAP values to provide insight into model decisions. | ||
|
|
||
| - Python 3.8+ | ||
| - Docker & Docker Compose | ||
| - Kubernetes (Minikube or any cloud provider) | ||
| - Node.js (for front-end enhancements) | ||
| - **`tests/`**: Unit, integration, and end-to-end tests. | ||
| - `test_api.py`, `test_preprocessing.py`, `test_training.py`: Various unit and integration tests. | ||
| - **End-to-End Tests** (`e2e/`): Cypress-based UI tests (`ui_tests.spec.js`). | ||
| - **Load Testing** (`load_testing/`): Uses Locust (`locustfile.py`) for load testing. | ||
|
|
||
| ### 🔧 Installation | ||
| - **`deployment/`**: Configuration files for deployment and monitoring. | ||
| - **Kubernetes Configurations** (`kubernetes/`): Deployment and Ingress configurations for scaling and canary releases. | ||
| - **Monitoring** (`monitoring/`): Prometheus (`prometheus.yml`) and Grafana (`grafana_dashboard.json`) for performance and system health monitoring. | ||
|
|
||
| 1. **📥 Clone the Repository**: | ||
| ```bash | ||
| git clone https://github.com/yourusername/LLM-Alignment-Assistant.git | ||
| cd LLM-Alignment-Assistant | ||
| ``` | ||
| ## ⚙️ Setup | ||
|
|
||
| 2. **📦 Set Up the Virtual Environment**: | ||
| ```bash | ||
| python3 -m venv venv | ||
| source venv/bin/activate | ||
| pip install -r requirements.txt | ||
| ``` | ||
| ### Prerequisites | ||
|
|
||
| - 🐍 Python 3.8+ | ||
| - 🐳 Docker & Docker Compose | ||
| - ☸️ Kubernetes (Minikube or a cloud provider) | ||
| - 🟢 Node.js (for front-end dependencies) | ||
|
|
||
| 3. **📦 Install Node.js Dependencies** (optional for enhanced UI): | ||
| ### 📦 Installation | ||
|
|
||
| 1. **Clone the Repository**: | ||
| ```bash | ||
| cd app/static | ||
| npm install | ||
| git clone https://github.com/yourusername/LLM-Alignment-Assistant.git | ||
| cd LLM-Alignment-Assistant | ||
| ``` | ||
|
|
||
| ### 🚀 Running Locally | ||
| 2. **Install Dependencies**: | ||
| - Python dependencies: | ||
| ```bash | ||
| pip install -r requirements.txt | ||
| ``` | ||
| - Node.js dependencies (optional for UI improvements): | ||
| ```bash | ||
| cd app/static | ||
| npm install | ||
| ``` | ||
|
|
||
| To run the application locally: | ||
| ### 🏃 Running Locally | ||
|
|
||
| 1. **🐳 Build Docker Image**: | ||
| 1. **Build Docker Images**: | ||
| ```bash | ||
| docker-compose up --build | ||
| ``` | ||
|
|
||
| 2. **🌐 Access the UI**: | ||
| Visit `http://localhost:5000` in your web browser. | ||
| 2. **Access the Application**: | ||
| - Open a browser and visit `http://localhost:5000`. | ||
|
|
||
| ## 📦 Deployment | ||
| ## 🚢 Deployment | ||
|
|
||
| ### 🚢 Docker and Kubernetes | ||
| ### ☸️ Kubernetes Deployment | ||
|
|
||
| - **🐳 Docker**: A Dockerfile is provided for containerization. | ||
| - **☸️ Kubernetes**: Use the provided `deployment/kubernetes/deployment.yml` and `service.yml` files to deploy the app to a Kubernetes cluster. | ||
| - **📜 Helm Charts**: Helm charts are available in the `deployment/helm/` directory for easier reusability and scalability. | ||
| - **Deploy to Kubernetes**: | ||
| - Apply the deployment and service configurations: | ||
| ```bash | ||
| kubectl apply -f deployment/kubernetes/deployment.yml | ||
| kubectl apply -f deployment/kubernetes/service.yml | ||
| ``` | ||
| - **Horizontal Pod Autoscaler**: | ||
| ```bash | ||
| kubectl apply -f deployment/kubernetes/hpa.yml | ||
| ``` | ||
|
|
||
| ### 🔄 CI/CD Pipeline | ||
| ### 🌟 Canary Deployment | ||
|
|
||
| A GitHub Actions workflow is included to automate building, testing, and deployment: | ||
| - Canary deployments are configured using `deployment/kubernetes/canary_deployment.yml` to roll out new versions safely. | ||
|
|
||
| - **✅ Lint & Test**: Linting and unit tests are run at every pull request. | ||
| - **🐋 Docker Build & Push**: Docker images are built and pushed to Docker Hub. | ||
| - **☸️ Kubernetes Deployment**: Automatically deploy to the Kubernetes cluster upon merging. | ||
| ### 📈 Monitoring and Logging | ||
|
|
||
| ## 🤖 Training and Fine-Tuning | ||
| - **Prometheus and Grafana**: | ||
| - Apply Prometheus and Grafana configurations in `deployment/monitoring/` to enable monitoring dashboards. | ||
| - **📋 Centralized Logging**: The **ELK Stack** is configured with Docker using `docker-compose.logging.yml` for centralized logs. | ||
|
|
||
| ### 💡 Reinforcement Learning from Human Feedback (RLHF) | ||
| ## 🧠 Training and Evaluation | ||
|
|
||
| The training module includes: | ||
| - **📊 Fine-Tuning**: Using the `training/fine_tuning.py` script, models can be fine-tuned on specific datasets. | ||
| - **🏆 Reward Models**: Implemented in `training/reward_model.py` for evaluating the appropriateness of responses. | ||
| - **🌐 Distributed Training**: Support for distributed training using `training/rlhf.py`. | ||
| ### 🔄 Transfer Learning | ||
|
|
||
| ### 🎛️ Hyperparameter Tuning | ||
| The training module (`src/training/transfer_learning.py`) uses pre-trained models like **BERT** to adapt to custom tasks, providing a significant performance boost. | ||
|
|
||
| For hyperparameter tuning, **Optuna** has been integrated to provide automated exploration of the training parameters, ensuring optimal model performance. | ||
| ### 📊 Data Augmentation | ||
|
|
||
| ## 🔄 Data Pipeline | ||
| The `data_augmentation.py` script (`src/data/`) applies augmentation techniques like back-translation and paraphrasing to improve data quality. | ||
|
|
||
| - **🛠️ Data Augmentation**: Using advanced NLP techniques, including back-translation and embedding-based masking, available in `preprocessing/augmentation.py`. | ||
| - **✅ Validation**: Thorough validation scripts (`preprocess_data.py` and `validate_data.py`) to maintain data quality. | ||
| - **⚙️ Automation with Apache Airflow**: Data pipeline orchestration using Airflow, ensuring proper data flow between stages. | ||
| ### 🧠 Reinforcement Learning from Human Feedback (RLHF) | ||
|
|
||
| ## 📈 Evaluation and Monitoring | ||
| - **Reward Model Training**: Uses the `rlhf.py` and `reward_model.py` scripts to fine-tune models based on human feedback. | ||
| - **Feedback Collection**: Users rate responses via the feedback form (`feedback.html`), and the model retrains with `retrain_model.py`. | ||
|
|
||
| - **📊 Metrics**: The `evaluation/metrics.py` script provides a detailed analysis of model performance, including bias detection and fairness metrics. | ||
| - **🛡️ Safety Testing**: Ethical AI assessments using `evaluation/safety_tests.py`. | ||
| - **📊 Dashboard**: Real-time monitoring with **Streamlit**, displaying key metrics, including training loss, accuracy, and reward trends. | ||
| ### 🔍 Explainability Dashboard | ||
|
|
||
| ## 🌐 Web Interface Improvements | ||
|
|
||
| - **🎨 Improved UI with TailwindCSS**: We've enhanced the CSS for modern and engaging aesthetics. | ||
| - **📈 Interactive Visualizations**: Added **Chart.js** visualizations to present alignment metrics in a clear, graphical format. | ||
| - **💬 Chatbot Integration**: A conversational UI element to interact directly with the trained LLM. | ||
| The `explainability_dashboard.py` script uses **SHAP** values to help users understand why a model made specific predictions. | ||
|
|
||
| ## 🧪 Testing | ||
|
|
||
| - **✅ Unit Tests**: Located in `tests/`, covering training, preprocessing, and evaluation. | ||
| - **🔄 Integration Tests**: End-to-end tests that simulate full pipeline execution. | ||
| - **🧪 Mock Testing**: Use of `pytest-mock` to simulate API calls and external integrations. | ||
|
|
||
| ## 📊 Monitoring and Logging | ||
|
|
||
| - **📈 Monitoring**: Kubernetes monitoring using **Prometheus** and **Grafana**, with Horizontal Pod Autoscaling (HPA) for scalability. | ||
| - **🔍 Explainability**: SHAP and LIME explainability metrics are added to the evaluation process, providing insights into model behavior. | ||
| - **📜 Logging**: Centralized logging using **ELK Stack** (Elasticsearch, Logstash, Kibana). | ||
|
|
||
| ## 🚀 Cloud Deployment Instructions (AWS) | ||
|
|
||
| To deploy the LLM Alignment Assistant on **AWS**, you can utilize **Elastic Kubernetes Service (EKS)** or **AWS Sagemaker** for model training: | ||
|
|
||
| 1. **AWS Elastic Kubernetes Service (EKS)**: | ||
| - Create an EKS cluster using AWS CLI or the console. | ||
| - Apply the Kubernetes deployment files: | ||
| ```bash | ||
| kubectl apply -f deployment/kubernetes/deployment.yml | ||
| kubectl apply -f deployment/kubernetes/service.yml | ||
| ``` | ||
| - Configure the **Horizontal Pod Autoscaler (HPA)** to ensure scalability: | ||
| ```bash | ||
| kubectl apply -f deployment/kubernetes/hpa.yml | ||
| ``` | ||
|
|
||
| 2. **AWS Sagemaker for Model Training**: | ||
| - Modify the `training/fine_tuning.py` to integrate with AWS Sagemaker. | ||
| - Use the Sagemaker Python SDK to launch a training job: | ||
| ```python | ||
| import sagemaker | ||
| from sagemaker.pytorch import PyTorch | ||
| role = "arn:aws:iam::your-account-id:role/service-role/AmazonSageMaker-ExecutionRole-2023" | ||
| pytorch_estimator = PyTorch( | ||
| entry_point='training/fine_tuning.py', | ||
| role=role, | ||
| instance_count=1, | ||
| instance_type='ml.p2.xlarge', | ||
| framework_version='1.8.0', | ||
| py_version='py3' | ||
| ) | ||
| pytorch_estimator.fit({'training': 's3://your-bucket-name/training-data'}) | ||
| ``` | ||
| - Ensure IAM roles and permissions are properly set for accessing **S3** and **Sagemaker**. | ||
|
|
||
| - **✅ Unit Tests**: Located in `tests/`, covering API, preprocessing, and training functionalities. | ||
| - **🖥️ End-to-End Tests**: Uses **Cypress** to test UI interactions. | ||
| - **📊 Load Testing**: Implemented with **Locust** (`tests/load_testing/locustfile.py`) to ensure stability under load. | ||
|
|
||
| ## 🚀 Future Work | ||
| ## 🔮 Future Work | ||
|
|
||
| - **🌍 Multi-Language Support**: Expand the LLM's training to support multiple languages. | ||
| - **⚖️ Ethical AI Enhancements**: Further enhance bias detection and mitigation techniques. | ||
| - **☁️ Cloud-Native Deployment**: Implement cloud services like **AWS SageMaker** for training at scale. | ||
| - **🔑 User Roles and Permissions**: Adding a role-based access control system. | ||
| - **📉 Advanced Monitoring**: Further enhance Prometheus alerts for anomaly detection. | ||
| - **🚀 Public Demo Deployment**: Deploy a public version on Heroku or AWS for showcasing. | ||
|
|
||
| ## 🤝 Getting Involved | ||
| ## 🤝 Contributing | ||
|
|
||
| Contributions are welcome! Feel free to submit issues, pull requests, or suggestions for new features. | ||
| Contributions are welcome! Please submit pull requests or issues for improvements or new features. | ||
|
|
||
| ## 📜 License | ||
|
|
||
| This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more details. | ||
| This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for more information. | ||
|
|
||
| ## 📬 Contact | ||
|
|
||
| - **✉️ Email**: [amirsina.torfi@gmail.com](mailto:amirsina.torfi@gmail.com) | ||
| - **🌐 Website**: [Portfolio](https://astorfi.github.io) | ||
| - **📧 Email**: [amirsina.torfi@gmail.com](mailto:amirsina.torfi@gmail.com) | ||
| - **🌐 Website**: [Your Portfolio](https://astorfi.github.io) | ||
|
|
||
| --- | ||
|
|
||
| <p align="center">Made with ❤️ by Amirsina Torfi</p> | ||
| <p align="center">Developed with ❤️ by Your Name</p> |