knowledge-storm v1.0.0 & EMNLP 2024 Paper Accepted!

We’re thrilled to announce the release of Co-STORM, a major update to the STORM project that brings collaborative human-AI knowledge curation into the spotlight! This new version empowers users to engage with language models in a more interactive and aligned way, transforming how we explore and curate knowledge together.

The knowledge-storm package is now at v1.0.0—make sure to upgrade by running:

pip install knowledge-storm --upgrade

News 🔥

• [2024/09] Co-STORM is now live and fully integrated into the knowledge-storm Python package. Try it out by upgrading the package today!
• [2024/09] Our paper on Co-STORM has been accepted to EMNLP 2024! You can read it here.

New Features 🎉

🚀 Co-STORM Engine

• Co-STORM is now integrated into the knowledge-storm package. Check out the API documentation for more details.
• We introduce the Agent interface in Co-STORM, providing a unified framework for defining different LM agent policies for information seeking and knowledge curation. You can explore this interface here.
  • Co-STORM LLM Experts: These agents ground their responses in external knowledge sources and ask follow-up questions based on the discourse.
  • Moderator Agent: Guides the conversation by generating insightful questions, drawing attention to discovered but underexplored areas.
  • Human Engagement: Users can inject their own utterances to steer the conversation, enabling an interactive and collaborative experience.

🧠 Dynamic Mind Map

Co-STORM introduces the KnowledgeBase data class that structures retrieved information into a concept-oriented hierarchy, forming a shared conceptual space between the user and the system. This is presented as a mind map in the graphical UI, allowing users to easily navigate and explore deep knowledge curation processes.

💻 Graphical UI Update

An interactive graphical UI of Co-STORM will be available soon on our live research preview website. Stay tuned for updates!

🔧 Modularity and Flexibility

• Just like STORM, Co-STORM is built using the dspy library, ensuring full modularity. You can easily customize both language models (LMs) and retrieval modules (RMs) for advanced use cases. Check out the customization guide.
• Co-STORM supports the same LMs and RMs as STORM. See the full list of supported models here.

Contributors 🙌

This release wouldn’t be possible without the hard work of:

• @Yucheng-Jiang
• @shaoyijia
• @dekunma

v0.2.4

Bump up knowledge-storm package to v0.2.4 to include recent change. Minor fix on demo light (#118).

New Model Support

• GoogleModel (#105)
• DeepSeekModel (#84)

New Retriever Support

• SerperRM (#102)
• BraveRM (#134)
• Improve VectorRM by decoupling the vector store construction from the retriever class (#126).

v0.2.3 Python Package Release

Python Package Available at PyPI

• You can now install storm via pip install knowledge-storm. (#80)
• We update the README.md to explain our current API.
• If you prefer using the source code, note that pulling the new release will lead to breaking change as we rename src/ to knowledge_storm/.

Ollama Support

#81, credit to: @zhjain

Roadmap & Contribution

Our team is actively working on:

1. Human-in-the-Loop Functionalities: Supporting user participation in the knowledge curation process.
2. Information Abstraction: Developing abstractions for curated information to support presentation formats beyond the Wikipedia-style report.

PRs for integrating more language models into knowledge_storm/lm.py and search engines/retrievers into knowledge_storm/rm.py are highly appreciated!

v0.2.0 New UI release; support customized documents retrieval

Support retrieving information from customized documents

• Add VectorRM to support grounding on user-provided documents, complementing existing support of search engines (YouRM, BingSearch). (#58)
• We provide an example script here and detailed instructions here.

UI updates

• We release a new UI for users to directly interact with STORM, available at https://storm.genie.stanford.edu/. This update brings improved stability and a new feature of discovering popular topics. (Thanks dekunma from Yale for the collaboration)
• We release demo light for developers, a minimal user interface built with streamlit framework in Python, handy for local development and demo hosting (#54)

Miscs

• In-person talk in San Francisco: Our team member will give an invited talk at Wikipedians Meetup @ San Francisco on July 11. Come to chat with us in person!
• Upcoming major update: A major update supporting human-AI collaboration mode will be released over the summer. Stay tuned!

v0.1.0 Refactored codebase

Pipeline Refactoring and Module Interfaces

We release the refactored code for STORM pipeline to make it easier to run/customize/develop STORM.
Interfaces for pipeline modules and data classes have been defined, allowing for better modularity and extensibility.

Language Model Customization Support

STORM now supports the customization of language models across various families and clients, including:

• GPT family
• Claude family
• VLLM client
• TGI client
• Together Client

Examples demonstrating language model customization are provided in the examples/ directory. For more details, please refer to the README.

Retriever Module Customization Support

STORM introduces the ability to customize the Retriever module. As a knowledge curation engine, STORM consumes information provided by the Retriever module.
More information about retriever module customization can be found in the README.

---

This release focuses on enhancing the modularity and flexibility of the STORM pipeline, enabling users to seamlessly integrate and customize language models and retriever modules according to their specific requirements. Contributions to integrating more language model support, search engine/retrieval model support are highly appreciated!