LangSuit⋅E is a systematic and simulation-free testbed for evaluating embodied capabilities of large language models (LLMs) across different tasks in embodied textual worlds. The highlighted features include:
- Embodied Textual Environments: The testbed provides a general simulation-free textual world that supports most embodied tasks, including navigation, manipulation, and communications. The environment is based on Gymnasium and inherits the design patterns.
- Embodied Observations and Actions: All agents' observations are designed to be embodied with customizable
max_view_distance,max_manipulate_distance,focal_length, etc. - Customizable Embodied Agents: The agents in LangSuit⋅E are fully-customizable w.r.t their action spaces and communicative capabilities, i.e., one can easily adapt the communication and acting strategy from one task to another.
- Multi-agent Cooperation: The testbed supports planning, acting and communication among multiple agents, where each agent can be customized to have different configurations.
- Human-agent Communication: Besides communication between agents, the testbed supports communication and cooperation between humans and agents.
- Full support to LangChain library: The LangSuitE testbed supports full usage of API language models, Open-source language models, tool usages, Chain-of-Thought (CoT) strategies, etc..
- Expert Trajectory Generation: We provide expert trajectory generation algorithms for most tasks.
- 🔍 Overview
⚠️ Upgrade Warning !!!- 📦 Benchmark and Dataset
- 🛠 Getting Started
- 📝 Citation
- 📄 Acknowledgements
We are currently refactoring and upgrading LangSuit·E. As of now, the following features are temporarily offline. If you have an urgent need, you can find our old version in the dev branch. If you discover any bugs in the new version or wish for us to prioritize certain features, feel free to tell us in the issues.
Below is our priority list:
- Support interactive WebUI (currently WebUI can render the picture of scene, but do not support interactive actions)
- Support Gymnasium API (will provide better support for BabyAI and more tasks)
- Refactored Expert Agent for IQA, Rearrenage and BabyAI.
- More deeply integration with LangChain
- Support multi-room and multi-agent (TEACh and CWAH need it)
- Better support for AI2-THOR and ProcTHOR based tasks
- Support VirtualHome format (CWAH need it)
We form a benchmark by adapting from existing annotations of simulated embodied engines, a by-product benefit of pursuing a general textual embodied world. Below showcase 6 representative embodied tasks, with variants of the number of rooms, the number of agents, and the action spaces of agents (whether they can communicate with each other or ask humans).
| Task | Simulator | # of Scenes | # of Tasks | # of Actions | Multi-Room | Multi-Agent | Communicative |
|---|---|---|---|---|---|---|---|
| BabyAI | Mini Grid | 105 | 500 | 6 | ✓ | ✗ | ✗ |
| Rearrange | AI2Thor | 120 | 500 | 8 | ✗ | ✗ | ✗ |
| IQA | AI2Thor | 30 | 3,000 | 5 | ✗ | ✗ | ✓ |
| ALFred | AI2Thor | 120 | 506 | 12 | ✗ | ✗ | ✗ |
| TEACh | AI2Thor | 120 | 200 | 13 | ✗ | ✓ | ✓ |
| CWAH | Virtual Home | 2 | 50 | 6 | ✓ | ✓ | ✓ |
- Clone this repository
git clone https://github.com/langsuite/langsuite.git
cd langsuite- Create a conda environment with
Python3.8+and install python requirements
conda create -n langsuite python=3.8
conda activate langsuite
pip install -e .- Export your
OPENAI_API_KEYby
export OPENAI_API_KEY="your_api_key_here"or you can customize your APIs by
cp api.config.yml.example api.config.ymland add or update your API configurations. For a full API agent list, please refer to LangChain Chat Models.
- Download the task dataset by
bash ./data/download.sh <data name>Currently supported datasets include: alfred, babyai, cwah, iqa, rearrange.
langsuite task <config-file.yml>
- Start langsuite server
langsuite serve <config-file.yml>- Start webui
langsuite webuiThe user inferface will run on http://localhost:8501/
task: AlfredTask_V0
template: ./templates/alfred/alfred_react.json
world:
agents:
- type: ChatAgent
inventory_capacity: 5
focal_length: 10
max_manipulate_distance: 2
max_view_distance: 2
step_size: 0.25
{
"intro": {
"default": [
"As an autonomous intelligent agent, you are now navigating a virtual home, and your task is to perform household tasks using specific actions. You will have access to the following information: ..."
]
},
"example": {
"default": [
"Task: put a clean lettuce in diningtable.\nObs: In front of you, You see a stoveburner_2. On your left, you see a stoveburner_1; a sinkbasin_1. On your right, you see a countertop_1; a tomato_0; a toaster_0.\n> Act: turn_left ..."
]
},
"InvalidAction": {
"failure.invalidObjectName": [
"Feedback: Action failed. There is no the object \"{object}\" in your view space. Please operate the object in sight.\nObs: {observation}"
],
...
},
...
}
If you find our work useful, please cite
@inproceedings{langsuite2023,
author = {Jia, Zixia and Wang, Mengmeng and Tong, Baichen and Zheng, Zilong},
title = {LangSuit⋅E: Controlling, Planning, and Interacting with Large Language Models in Embodied Text Environments},
year = {2024},
booktitle = {Findings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024)},
url = {https://github.com/bigai-nlco/langsuite}
}For any questions and issues, please contact nlp@bigai.ai.
Some of the tasks of LangSuit⋅E are based on the datasets and source-code proposed by previous researchers, including BabyAI, AI2Thor, ALFred, TEAch, CWAH.