Skip to content

Latest commit

 

History

History
63 lines (44 loc) · 1.96 KB

README.md

File metadata and controls

63 lines (44 loc) · 1.96 KB
Raw

Multi-Agent Misconception Analysis System

A multi-agent system designed to enhance learning environments by predicting student misconceptions based on their answers to questions. This project uses dspy and OpenAI to process questions, generate potential misconceptions, and evaluate semantic similarities between these misconceptions, facilitating a deeper understanding of student reasoning.

Features

  • Multi-Agent Communication: Implements the Exchange-of-Thought (EoT) framework inspired by large language model communication paradigms like Memory, Report, Relay, and Debate.
  • Misconception Prediction: Identifies the reasoning behind incorrect answers provided by students.
  • Semantic Evaluation: Analyzes and ranks misconceptions based on their similarity to student reasoning.

Getting Started

Prerequisites

  • Python 3.8+
  • dspy library
  • OpenAI API key

Installation

  1. Clone the repository:

    git clone https://github.com/CS194Group/multi-agent-misconceptions.git
cd multi-agent-misconceptions

  2. Install required dependencies:

    pip install -r requirements.txt

Usage

Training and Evaluation

Run the main script to process questions and evaluate misconceptions: bash python main.py

Data

Place your training, testing, and validation datasets in the data/ directory.

Project Structure

  • agents/: Contains multi-agent logic for generating and evaluating misconceptions.
  • data/: Includes training, testing, and validation datasets.
  • dataloader.py: Preprocessing and data loading utilities.
  • evaluation.py: Performance evaluation and reporting tools.
  • main.py: Entry point for training and inference.
  • predictmode.py: Prediction logic for generating misconceptions.

Structure of .env File

LAMBDA_API_KEY=""
LAMBDA_API_BASE="https://api.lambdalabs.com/v1/"
LAMBDA_API_MODEL="llama3.1-70b-instruct-berkeley"
OPENAI_API_KEY=""
WANDB_API_KEY=""