ElevenLabs hackathon project exploring what an interactive podcast listening experience would where you can pause and ask questions to the host and get answers in real-time.
- Assembly AI is used for real-time speech-to-text, so users can ask the podcast host questions using their voice.
- Dexa AI is used to generate realistic answers to text queries in the style of Lex Fridman, with a custom private RAG API trained on the transcripts across all ~400 episodes of the Lex Fridman Podcast.
- Dexa uses Pinecone hybrid search to retrieve the most relevant podcast chunks, applies cross-validation and re-ranking to ensure we only take into account transcript chunks that are relevant to the question, uses RAG + GPT-4 to generate an answer in the style of Lex Fridman, and then applies a post-processing step to minimize hallucination.
- ElevenLabs is used for text-to-speech to convert the resulting answer to audio using a custom voice trained on ~7 minutes of Lex Fridman's voice.
Note, we may keep this running for a while but we will shut it down if it becomes expensive for us. Sowwy. But you should be able to run it on your own with the instructions specified under "Build Locally" section.
- Install
pnpm - Run
pnpm ito install dependencies cp .env.example .envand fill in all of the environment variables:
ELEVEN_LABS_API_KEY=''
ASSEMBLY_AI_API_KEY=''
DEXA_API_BASE_URL=''
AWS_ACCESS_KEY_ID=''
AWS_SECRET_ACCESS_KEY=''
AWS_REGION=''
AWS_S3_BUCKET=''
GOOGLE_API_KEY=''
- Run
pnpm dev
PRs are more than welcome.
- Currently we only support the "Lex Fridman Podcast" since we have eleven labs' voice trained only on him. You can help us support more users. [hint, more female speakers!!]
- Currently only YouTube URLs are supported. Help us support more Podcast/audio type URLs.
- Add some way to store Q/A, timestamp, clip info in a database of some sort
- A way for users to view other people's Q/As on the audio, we are thinking that it should kinda look like SoundCloud's Audio Waveform. See this
MIT © Sasha Sheng and Travis Fischer