Preprocess updates

[RileyLePrell]

Staff Facing App
The app is built in Streamlit and designed with a user-friendly interface comprising three main pages:

1. Home Page

   • Serves as an introduction to the website.
   • Provides a high-level overview of the app's purpose and functionality.

2. Upload Files

   • Allows staff to upload various types of files, including:
     • Audio recordings (e.g., meeting recordings).
     • Minutes (PDF format).
     • Agendas (PDF format).
   • Metadata is required before uploading files for better organization.

3. View Documents

   • Displays all files currently stored in Azure Blob Storage.
   • Allows staff to view files at various stages of the preprocessing pipeline.

Preprocessing Pipeline
The pipeline processes files uploaded through the app, enabling seamless integration with Weaviate Cloud for advanced querying. Below is a breakdown of the pipeline steps:

1. Metadata Input

   • Users input essential metadata (e.g., meeting date, type, and file type) before uploading files.

2. File Conversion

   • Audio files: Transcribed using AssemblyAI and stored as a "dirty" text file.
   • PDF files: (Minutes/Agendas) Converted directly into "dirty" text files.

3. Text Cleaning

   • Employs OpenAI's GPT with tailored prompt engineering to clean and standardize the "dirty" text.

4. Tokenization and Vector Embedding

   • Tokenizer: Integrated Tiktoken, configured with chunk sizes of 250 tokens for efficient processing.
   • Embedder: Utilizes OpenAI’s "text-embedding-ada-002" model to embed text.

5. Storage

   • All processed data is stored in Weaviate Cloud, enabling efficient storage and search functionalities.

Cloud Functionality

1. Azure Blob Storage:

   • Serves as the file repository.
   • Streamlit app interacts with the Azure Blob for storing files at various stages of the pipeline (e.g., raw, dirty, clean).

2. Weaviate Cloud:

   • Hosts the tokenized and embedded vectors.
   • Facilitates future querying and advanced analytics.

Current Status

• Pipeline: Fully operational.
• Staff Facing App:
  • Pages implemented and functional.
  • Metadata handling and file upload integrated with preprocessing pipeline.
• View Documents Page: Displays uploaded files from Azure Blob Storage.
• Tokenizer and Embedder:
  • Tokenization via Tiktoken.
  • Embedding with text-embedding-ada-002.