Merge develop into master (#29)

* lint-frontend  Files.tsx (All-Hands-AI#1089)

for lint-frontend exhaustive deps

Co-authored-by: Robert Brennan <accounts@rbren.io>

* Add build-frontend to build (All-Hands-AI#1137)

* build: fix workspace variable name in dev setup (All-Hands-AI#1138)

* Revamp docker build process (All-Hands-AI#1121)

* refactor docker building

* change to buildx

* disable branch filter

* disable tags

* matrix for building

* fix branch filter

* rename workflow

* sanitize ref name

* fix sanitization

* fix source command

* fix source command

* add push arg

* enable for all branches

* logs

* empty commit

* try freeing disk space

* try disk clean again

* try alpine

* Update ghcr.yml

* Update ghcr.yml

* move checkout

* ignore .git

* add disk space debug

* add df h to build script

* remove pull

* try another failure bypass

* remove maximize build space step

* remove df -h debug

* add no-root

* multi-stage python build

* add ssh

* update readme

* remove references to config.toml

* fix(): build out opendevin modal component (All-Hands-AI#1141)

* corrected port (All-Hands-AI#1159)

* fix All-Hands-AI#1028, /select-file api call on deleted file in Code Editor caused Error (All-Hands-AI#1158)

* fix: /select-file on deleted file exception, detail: All-Hands-AI#1028

* fix: lint.

* fix: lint.

---------

Co-authored-by: aaren.xzh <aaren.xzh@antfin.com>

* Remove custom signal handlers (All-Hands-AI#1153)

* (fix) OpenDevin works on OpenDevin issues (All-Hands-AI#1149)

* Use SANDBOX_TYPE=exec, use docker image

* Use updated image

* fix: fix multiple frontend warnings (All-Hands-AI#1143)

* test(frontend): add unit tests (All-Hands-AI#1076)

* test(frontend): add unit tests for getCachedConfig

* test(frontend): add unit tests for getCachedConfig

* add unit test for the useTypingEffect hook

* add unit test for the useInputComposition hook

* create unit test for auth service

* remove outdated and failing component test

* create unit test for session service

* break down saveSettings into smaller functions for testability and create unit test for new mergeAndUpdateSettings

---------

Co-authored-by: Robert Brennan <accounts@rbren.io>

* Logging additions and fixes (All-Hands-AI#1139)

* Refactor print_to_color into a color formatter

misc fixes

catch ValueErrors and others from Router initialization

add default methods

* Tweak console log formatting, clean up after rebasing exceptions out

* Fix prompts/responses

* clean up

* keep regular colors when no msg_type

* fix filename

* handle file log first

* happy mypy

* ok, mypy

---------

Co-authored-by: Robert Brennan <accounts@rbren.io>

* fix: make typing chat bubble adhere to max width (All-Hands-AI#1142)

* fix: fix delete messages endpoint url (All-Hands-AI#1140)

Co-authored-by: Robert Brennan <accounts@rbren.io>

* Run docker publish on tags (All-Hands-AI#1162)

* run ghcr on tags

* Update .github/workflows/ghcr.yml

Co-authored-by: RaGe <foragerr@users.noreply.github.com>

---------

Co-authored-by: RaGe <foragerr@users.noreply.github.com>

* Document LLM prompt and response logging. (All-Hands-AI#1166)

* Refactor actions a bit (All-Hands-AI#1165)

* refactor to action_manager

* make all actions awaitable

* move task logic into tasks

* Update opendevin/action/tasks.py

* feat: Improve docker compose services integration

* feat: Add ollama, support, memGPT services

* feat: Configure OpenDevin UI container

* feat: Docker services

* hotfix: Add OS locales

* Merge develop into clean-install

* Add ollama, support, memGPT services

* feat: Docker services

* hotfix: Backend app service dependencies fix under Conda

hotfix: Frontend integration

hotfix: Restore useTranslation()

* feat: Add API startup script

hotfix: Integration fix

* feat: Add FastAPI server and Vite dev server logging for debug and live modes

* hotfix: Frontend and API integration. Build improvements.

* Add ollama, support, memGPT services

* feat: Docker services

hotfix: Add OS locales

hotfix: Fix start configuration

hotfix: Temp fix of build errors

feat: Configure OpenDevin UI container

hotfix: Run OpenDevin application container

chore: Update README

feat: Docker Configuration for backend services

feat: Ubuntu 20.04 images with CUDA and Miniconda3

refactor: Update docker environment configuration

feat: Decouple UI and Python app into services.

Co-authored-by: Jim Su <jimsu@protonmail.com>

* hotfix: Frontend integration

* hotfix: Update Miniconda source package and SHA256 hash

* hotfix: Make ESLint 'no-duplicate-imports' to report warnings instead of errors

* chore: Arrange '.env' and '.env.dist' files

* chore: Rename 'nginx.conf' to 'nginx.conf.dist'

* chore: Update Docker services configuration

* feat: Add developer.Dockerfile to Devin web UI Docker config. Move Dockerfile to end-uder.Dockerfile

* feat: Update NGINX configuration with HTTP/HTTPS server config

* feat: Update Devin API host and port

* chore: Remove unnecessary docker-compose.yml file version tag

* chore: Revert 'packageManager' to pnpm

* feat: Add frontend port EXPOSE

* feat: Add API ports EXPOSE directives

* hotfix: Path to 'env_debug' script

* feat: Add developer.Dockerfile

* chore: Synchronize with OpenDevin main branch

* replace Jest with Vitest as the frontend test runner (All-Hands-AI#1163)

* Fix python linter inconsistent behaviour with quotes (All-Hands-AI#1112)

* This has been a headache for a long time, and we had All-Hands-AI#1071 and All-Hands-AI#1100 with the hope to fix the inconsistent behaviour across linters and environments. However, we recently found out that double-quote-string-fixer plugin in pre-commit-hook has inconsistent behaviour on python 3.11 and 3.12. See discussion here. This is sad because while this plugin enforces single quote behaviour with 3.11, it doesn't always enforce so with 3.12. Specifically, with fstr syntax, this plugin allows both single quotes and double quotes with python 3.12.

The problem is, some developers have black linter installed/integrated with their IDE, which is probably the most popular linter in python world (ranked by GitHub stars). This linter insists on always using double quotes. Now we have black and double-quote-string-fixer fight each other (iff the developer uses python 3.12) for some quotes (fstr syntax).

After a lot of research, I couldn't find a way to enforce single quote behaviour without introducing a new dependency, flake8, together with a plugin for it to enforce quotes' behavior. I believe it's better off introducing the more popular black if we have to introduce a new linter. Since black and autopep8 sometimes fight each other, and they mostly overlap, I further remove autopep8.

The unfortunate consequence of this PR is that I had to revert all single quotes back to double quotes. This might cause some inconvenience to existing PRs as they have to resolve conflicts, but I believe the headache will be gone soon. That being said, I am open to abandon this PR if anyone has a better idea to solve the headache.

* Remove black

* Prevent black from changing quotes

* Use flake8 to enforce single quotes

* Fix quotes in config.py

* Add back autopep8

* Add make lint to run linters

* hotfix: PR fix

* Hotfix/clean install (#21)

* hotfix: Added missing variables. Removed .env from repo

---------

Co-authored-by: 808vita <97225946+808vita@users.noreply.github.com>
Co-authored-by: Robert Brennan <accounts@rbren.io>
Co-authored-by: RaGe <foragerr@users.noreply.github.com>
Co-authored-by: Alex Bäuerle <alex@a13x.io>
Co-authored-by: Akki <akhilvc10@gmail.com>
Co-authored-by: மனோஜ்குமார் பழனிச்சாமி <smartmanoj42857@gmail.com>
Co-authored-by: Xia Zhenhua <zhhuaxia@gmail.com>
Co-authored-by: aaren.xzh <aaren.xzh@antfin.com>
Co-authored-by: Boxuan Li <liboxuan@connect.hku.hk>
Co-authored-by: sp.wack <83104063+amanape@users.noreply.github.com>
Co-authored-by: Engel Nyst <enyst@users.noreply.github.com>
Co-authored-by: Jim Su <jimsu@protonmail.com>

.env.dist

@@ -57,3 +57,6 @@ NPM_VERSION=10.5.2
 MITMPROXY_VERSION=10.2.4
 # mitmproxy directory INSIDE THE CONTAINER
 MITMPROXY_DIR=/home/mitmproxy/.mitmproxy
+
+REDIS_IMAGE=redis/redis-stack
+UI_BUILD_DOCKERFILE=developer.Dockerfile

.github/workflows/dogfood.yml

@@ -13,25 +13,18 @@ jobs:
   open-devin:
     if: github.event.label.name == 'dogfood-this'
     runs-on: ubuntu-latest
-    environment: OpenAI
+    container:
+      image: ghcr.io/opendevin/opendevin
+      volumes:
+        - /var/run/docker.sock:/var/run/docker.sock
 
     steps:
+    - name: install git, github cli
+      run: apt-get install -y git gh
+
     - name: Checkout Repository
       uses: actions/checkout@v4
 
-    - name: Install poetry 
-      run: pipx install poetry 
-
-    - name: Set up Python
-      uses: actions/setup-python@v5
-      with:
-        python-version: '3.12'
-        cache: 'poetry'
-
-    - name: Install Dependencies
-      run: |
-        make build
-
     - name: Write Task File
       env:
         ISSUE_TITLE: ${{ github.event.issue.title }}
@@ -48,12 +41,15 @@ jobs:
         ISSUE_TITLE: ${{ github.event.issue.title }}
         ISSUE_BODY: ${{ github.event.issue.body }}
         LLM_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        SANDBOX_TYPE: exec
       run: |
-        poetry run python ./opendevin/main.py -d "./" -i 50 -f task.txt
+        python ./opendevin/main.py -d "./" -i 50 -f task.txt -d $GITHUB_WORKSPACE
+        rm task.txt
 
     - name: Setup Git, Create Branch, and Commit Changes
       run: |
         # Setup Git configuration
+        git config --global --add safe.directory $PWD
         git config --global user.name 'OpenDevin'
         git config --global user.email 'OpenDevin@users.noreply.github.com'
 
@@ -76,10 +72,6 @@ jobs:
         # Push changes
         git push --set-upstream origin $BRANCH_NAME
 
-    - name: Install GitHub CLI
-      run: |
-        sudo apt-get install gh
-
     - name: Fetch Default Branch
       env:
         GH_TOKEN: ${{ github.token }}
@@ -101,10 +93,10 @@ jobs:
           --head "${{ github.head_ref }}" \
           --base "${{ env.DEFAULT_BRANCH }}" \
           | grep -o 'https://github.com/[^ ]*')
-        
+
         # Extract PR number from URL
         PR_NUMBER=$(echo "$PR_URL" | grep -o '[0-9]\+$')
-        
+
         # Set environment vars
         echo "PR_URL=$PR_URL" >> $GITHUB_ENV
         echo "PR_NUMBER=$PR_NUMBER" >> $GITHUB_ENV

.github/workflows/ghcr.yml

@@ -1,8 +1,7 @@
-name: Build and publish multi-arch container images
+name: Publish Docker Image
 
 on:
   push:
-    branches: [ main ]
   workflow_dispatch:
     inputs:
       reason:
@@ -14,6 +13,9 @@ jobs:
   ghcr_build_and_push:
     runs-on: ubuntu-latest
     if: github.event_name == 'push' || github.event.inputs.reason != ''
+    strategy:
+      matrix:
+        image: ["app", "evaluation", "sandbox"]
 
     steps:
       - name: checkout
@@ -29,31 +31,5 @@ jobs:
       - name: Log-in to ghcr.io
         run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
 
-      - name: Build and push multi-arch container images
-        run: |
-          # set env for fork repo
-          DOCKER_BUILD_ORG=$(echo "${{ github.repository }}" | tr '[A-Z]' '[a-z]' | cut -d '/' -f 1)
-          # Find directories containing Dockerfile but not containing .dockerfileignore
-          while IFS= read -r dockerfile_dir; do
-          
-            # Check if .dockerfileignore exists in the directory
-            if [ -f "$dockerfile_dir/.dockerfileignore" ]; then
-                echo "$dockerfile_dir/.dockerfileignore exists, skipping build and push"
-                continue
-            fi
-          
-            # Check if image was already exist in ghcr.io
-            pushd "$dockerfile_dir" > /dev/null
-            FULL_IMAGE=$(make get-full-image DOCKER_BUILD_ORG=$DOCKER_BUILD_ORG)
-            popd > /dev/null
-            EXISTS=$(docker manifest inspect "$FULL_IMAGE" > /dev/null 2>&1 && echo "true" || echo "false")
-            if [ "$EXISTS" == "true" ]; then
-              echo "Image $FULL_IMAGE already exists in ghcr.io, skipping build and push"
-              continue
-            fi
-          
-            # Build and push the image to ghcr.io
-            pushd "$dockerfile_dir" > /dev/null
-            make all DOCKER_BUILD_ORG=$DOCKER_BUILD_ORG
-            popd > /dev/null
-          done < <(find . -type f -name Dockerfile -exec dirname {} \; | sort -u)
+      - name: Build and push ${{ matrix.image }}
+        run: ./containers/build.sh ${{ matrix.image }} --push

Makefile

@@ -1,8 +1,14 @@
 # Makefile for OpenDevin project
 
 # Variables
+DOCKER_IMAGE = ghcr.io/opendevin/sandbox
+BACKEND_PORT = 3000
+BACKEND_HOST = "127.0.0.1:$(BACKEND_PORT)"
+FRONTEND_PORT = 3001
+DEFAULT_WORKSPACE_DIR = "./workspace"
 DEFAULT_MODEL = "gpt-3.5-turbo-1106"
 CONFIG_FILE = config.toml
+PRECOMMIT_CONFIG_PATH = "./dev_config/python/.pre-commit-config.yaml"
 
 # ANSI color codes
 GREEN=\033[0;32m
@@ -15,21 +21,45 @@ RESET=\033[0m
 build:
 	@echo "$(GREEN)Building project...$(RESET)"
 	@$(MAKE) -s check-dependencies
-	@$(MAKE) -s docker-build
+	@$(MAKE) -s pull-docker-image
+	@$(MAKE) -s install-python-dependencies
+	@$(MAKE) -s install-frontend-dependencies
+	@$(MAKE) -s install-precommit-hooks
+	@$(MAKE) -s build-frontend
 	@echo "$(GREEN)Build completed successfully.$(RESET)"
 
 check-dependencies:
 	@echo "$(YELLOW)Checking dependencies...$(RESET)"
-	@$(MAKE) -s check-docker-compose
+	@$(MAKE) -s check-python
+	@$(MAKE) -s check-npm
+	@$(MAKE) -s check-docker
 	@$(MAKE) -s check-poetry
 	@echo "$(GREEN)Dependencies checked successfully.$(RESET)"
 
-check-docker-compose:
-	@echo "$(YELLOW)Checking Docker Compose installation...$(RESET)"
+check-python:
+	@echo "$(YELLOW)Checking Python installation...$(RESET)"
+	@if command -v python3 > /dev/null; then \
+		echo "$(BLUE)$(shell python3 --version) is already installed.$(RESET)"; \
+	else \
+		echo "$(RED)Python 3 is not installed. Please install Python 3 to continue.$(RESET)"; \
+		exit 1; \
+	fi
+
+check-npm:
+	@echo "$(YELLOW)Checking npm installation...$(RESET)"
+	@if command -v npm > /dev/null; then \
+		echo "$(BLUE)npm $(shell npm --version) is already installed.$(RESET)"; \
+	else \
+		echo "$(RED)npm is not installed. Please install Node.js to continue.$(RESET)"; \
+		exit 1; \
+	fi
+
+check-docker:
+	@echo "$(YELLOW)Checking Docker installation...$(RESET)"
 	@if command -v docker > /dev/null; then \
 		echo "$(BLUE)$(shell docker --version) is already installed.$(RESET)"; \
 	else \
-		echo "$(RED)Docker is not installed.\nPlease install Docker Desktop to continue.$(RESET)"; \
+		echo "$(RED)Docker is not installed. Please install Docker to continue.$(RESET)"; \
 		exit 1; \
 	fi
 
@@ -44,35 +74,55 @@ check-poetry:
 		exit 1; \
 	fi
 
-docker-build:
-	@read -p "Run 'docker compose down'? [Y/n]: " run_down;
-	@if [ ! -z "$$run_down" ] ; then @docker compose down \
-  	else exit 0; fi
-	@echo "$(YELLOW)Building Docker images...$(RESET)"
-	@docker compose -f docker-compose.yml build --pull > /dev/null
-	@echo "$(GREEN)Docker images generated successfully.$(RESET)"
-
-docker-rebuild:
-	@echo "$(YELLOW)Force rebuilding Docker images...$(RESET)"
-	@read -p "Run 'docker compose down'? [Y/n]: " run_down; \
-    	 if [ ! -z = "$$run_down" ]; then docker compose down; else exit 0; fi
-	@docker compose -f docker-compose.yml build --pull --no-cache
-	@echo "$(GREEN)Docker images updated successfully.$(RESET)"
-
-docker-start:
-	@echo "$(YELLOW)Starting Docker services...$(RESET)"
-	@docker compose up --build
-	@echo "$(GREEN)All Docker services started$(RESET)"
+pull-docker-image:
+	@echo "$(YELLOW)Pulling Docker image...$(RESET)"
+	@docker pull $(DOCKER_IMAGE)
+	@echo "$(GREEN)Docker image pulled successfully.$(RESET)"
+
+install-python-dependencies:
+	@echo "$(GREEN)Installing Python dependencies...$(RESET)"
+	@if [ "$(shell uname)" = "Darwin" ]; then \
+		echo "$(BLUE)Installing `chroma-hnswlib`...$(RESET)"; \
+		export HNSWLIB_NO_NATIVE=1; \
+		poetry run pip install chroma-hnswlib; \
+	fi
+	@poetry install --without evaluation
+	@echo "$(GREEN)Python dependencies installed successfully.$(RESET)"
+
+install-frontend-dependencies:
+	@echo "$(YELLOW)Setting up frontend environment...$(RESET)"
+	@echo "$(YELLOW)Detect Node.js version...$(RESET)"
+	@cd frontend && node ./scripts/detect-node-version.js
+	@cd frontend && \
+		echo "$(BLUE)Installing frontend dependencies with npm...$(RESET)" && \
+		npm install && \
+		echo "$(BLUE)Running make-i18n with npm...$(RESET)" && \
+		npm run make-i18n
+	@echo "$(GREEN)Frontend dependencies installed successfully.$(RESET)"
+
+install-precommit-hooks:
+	@echo "$(YELLOW)Installing pre-commit hooks...$(RESET)"
+	@git config --unset-all core.hooksPath || true
+	@poetry run pre-commit install --config $(PRECOMMIT_CONFIG_PATH)
+	@echo "$(GREEN)Pre-commit hooks installed successfully.$(RESET)"
+
+lint:
+	@echo "$(YELLOW)Running linters...$(RESET)"
+	@poetry run pre-commit run --files opendevin/**/* agenthub/**/* --show-diff-on-failure --config $(PRECOMMIT_CONFIG_PATH)
+
+build-frontend:
+	@echo "$(YELLOW)Building frontend...$(RESET)"
+	@cd frontend && npm run build
 
 # Start backend
 start-backend:
 	@echo "$(YELLOW)Starting backend...$(RESET)"
-	@docker compose up devin
+	@poetry run uvicorn opendevin.server.listen:app --port $(BACKEND_PORT)
 
 # Start frontend
 start-frontend:
 	@echo "$(YELLOW)Starting frontend...$(RESET)"
-	@docker compose up web_ui
+	@cd frontend && BACKEND_HOST=$(BACKEND_HOST) FRONTEND_PORT=$(FRONTEND_PORT) npm run start
 
 # Run the app
 run:
@@ -83,7 +133,7 @@ run:
 	fi
 	@mkdir -p logs
 	@echo "$(YELLOW)Starting backend server...$(RESET)"
-	@$(MAKE) docker-start
+	@poetry run uvicorn opendevin.server.listen:app --port $(BACKEND_PORT) &
 	@echo "$(YELLOW)Waiting for the backend to start...$(RESET)"
 	@until nc -z localhost $(BACKEND_PORT); do sleep 0.1; done
 	@echo "$(GREEN)Backend started successfully.$(RESET)"
@@ -125,13 +175,14 @@ setup-config-prompts:
 
 	@read -p "Enter your workspace directory [default: $(DEFAULT_WORKSPACE_DIR)]: " workspace_dir; \
 	 workspace_dir=$${workspace_dir:-$(DEFAULT_WORKSPACE_DIR)}; \
-	 echo "WORKSPACE_DIR=\"$$workspace_dir\"" >> $(CONFIG_FILE).tmp
+	 echo "WORKSPACE_BASE=\"$$workspace_dir\"" >> $(CONFIG_FILE).tmp
 
 # Help
 help:
 	@echo "$(BLUE)Usage: make [target]$(RESET)"
 	@echo "Targets:"
 	@echo "  $(GREEN)build$(RESET)               - Build project, including environment setup and dependencies."
+	@echo "  $(GREEN)lint$(RESET)                - Run linters on the project."
 	@echo "  $(GREEN)setup-config$(RESET)        - Setup the configuration for OpenDevin by providing LLM API key,"
 	@echo "                        LLM Model name, and workspace directory."
 	@echo "  $(GREEN)start-backend$(RESET)       - Start the backend server for the OpenDevin project."
@@ -141,4 +192,4 @@ help:
 	@echo "  $(GREEN)help$(RESET)                - Display this help message, providing information on available targets."
 
 # Phony targets
-.PHONY: build check-dependencies check-docker check-poetry setup-config setup-config-prompts docker-build docker-start help
+.PHONY: build check-dependencies check-python check-npm check-docker check-poetry pull-docker-image install-python-dependencies install-frontend-dependencies install-precommit-hooks lint start-backend start-frontend run setup-config setup-config-prompts help

agenthub/monologue_agent/utils/memory.py

@@ -5,6 +5,7 @@
 from llama_index.vector_stores.chroma import ChromaVectorStore
 
 from opendevin import config
+from opendevin.logger import opendevin_logger as logger
 from . import json
 
 embedding_strategy = config.get('LLM_EMBEDDING_MODEL')
@@ -83,6 +84,7 @@ def add_event(self, event: dict):
             },
         )
         self.thought_idx += 1
+        logger.debug("Adding %s event to memory: %d", t, self.thought_idx)
         self.index.insert(doc)
 
     def search(self, query: str, k: int = 10):