Make the docker setup consistent with local #1199

mikekelly · 2023-04-13T22:16:45Z

Background

The project structure in Docker was different from local setup for no apparent reason, this brings it in line and cleans things up.

Changes

Dockerfile now copies the full directory in to the container
The docker-compose volume configuration was simplified.
Only calling apt-get update and apt-get install once each
Separating apt-get update and apt-get install into separate steps so the former can hit docker cache when building
Install deps globally using /tmp dir to adjust the requirements.txt file to remove non Docker deps
Use conventional /app dir
Revert to using root as per base container, resolving permissions issues when creating logs in mounted volume.

Testing

Image built successfully and container ran successfully.

PR Quality Checklist

My pull request is atomic and focuses on a single change.
I have thoroughly tested my changes with multiple different prompts.
I have considered potential risks and mitigations for my changes.
I have documented my changes clearly and comprehensively.
I have not snuck in any "extra" small tweaks changes

tech3br · 2023-04-13T22:21:25Z

Thanks man! It's useful.

nponeccop · 2023-04-14T06:43:08Z

@mikekelly There are conflicts now

scripts/logger.py

nponeccop · 2023-04-15T14:31:17Z

@mikekelly There are conflicts now

github-actions · 2023-04-17T15:58:02Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions · 2023-04-17T15:58:03Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions · 2023-04-17T15:58:03Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

github-actions · 2023-04-17T15:58:04Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

mikekelly · 2023-04-19T08:16:09Z

I tried using this but got some permissions issues:
Permission denied: '/home/appuser/app/logs/activity.log

I'm not experiencing this. What are the permissions on the files in that dir? (ls -la)

mikekelly · 2023-04-19T08:18:42Z

You may want to ensure that your working dir is completely clean before using docker-compose run auto-gpt. You can clean the dir of any additionally created files by using git clean -fdx but just be aware this will delete your .env too so make sure you keep a backup of it

montanaflynn · 2023-04-19T08:37:02Z

@mikekelly the permissions are:

drwxr-xr-x  2 root root  4096 Apr 19 10:33 logs

Also I tried using a fresh git clone, and just making the changes in this pull request and got same error.

I'm running debian 11 and Docker Compose version v2.17.2

docker compose run auto-gpt
[+] Running 1/0
 ✔ Container auto-gpt-redis-1  Running                                                                                                                                                                            0.0s 
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/app/autogpt/__main__.py", line 6, in <module>
    from autogpt.agent.agent import Agent
  File "/app/autogpt/agent/__init__.py", line 1, in <module>
    from autogpt.agent.agent import Agent
  File "/app/autogpt/agent/agent.py", line 3, in <module>
    from autogpt.app import execute_command, get_command
  File "/app/autogpt/app.py", line 5, in <module>
    from autogpt.agent.agent_manager import AgentManager
  File "/app/autogpt/agent/agent_manager.py", line 7, in <module>
    from autogpt.llm_utils import create_chat_completion
  File "/app/autogpt/llm_utils.py", line 11, in <module>
    from autogpt.logs import logger
  File "/app/autogpt/logs.py", line 202, in <module>
    logger = Logger()
             ^^^^^^^^
  File "/app/autogpt/config/singleton.py", line 15, in __call__
    cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/autogpt/logs.py", line 49, in __init__
    self.file_handler = logging.FileHandler(
                        ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1181, in __init__
    StreamHandler.__init__(self, self._open())
                                 ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/logging/__init__.py", line 1213, in _open
    return open_func(self.baseFilename, self.mode,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
PermissionError: [Errno 13] Permission denied: '/app/logs/activity.log'

mikekelly · 2023-04-19T11:08:45Z

@montanaflynn please can you try this branch (https://github.com/mikekelly/Auto-GPT/tree/docker-permissions-on-linux) in another completely fresh clone and see if it resolves your issue

montanaflynn · 2023-04-19T12:38:11Z

@mikekelly no permissions issues with that branch, although I did get this warning which I hadn't seen before:

Warning: The file 'AutoGpt.json' does not exist. Local memory would not be saved to a file.

mikekelly · 2023-04-19T13:34:15Z

@montanaflynn I don't think that warning relates to this change

Pwuts

Looks pretty good. I'd like to request a few tweaks, nothing big though.

Pwuts · 2023-04-19T21:53:09Z

Dockerfile


-# Install Xvfb and other dependencies for headless browser testing


This comment was actually useful imo, can you add it back in some form?

Pwuts · 2023-04-19T21:55:34Z

docker-compose.yml

-      - ".env:/app/.env"
-    profiles: ["exclude-from-up"]
+      - "./:/app"
+    profiles: ["exclude-from-up"] # Run and attach to container instead with: `docker-compose run auto-gpt`


Suggested change

profiles: ["exclude-from-up"] # Run and attach to container instead with: `docker-compose run auto-gpt`

profiles: ["exclude-from-up"] # Run and attach to container instead with: `docker-compose run --rm auto-gpt`

--rm to prevent 1001 dead containers

Pwuts · 2023-04-19T21:58:39Z

docker-compose.yml

    env_file:
      - .env


Since redis is started by default, it seems logical to also connect redis by default. This will be overridden by any explicit settings in .env.

Suggested change

env_file:

- .env

env_file:

- .env

environment:

MEMORY_BACKEND: ${MEMORY_BACKEND:-redis}

REDIS_HOST: ${REDIS_HOST:-redis}

Pwuts · 2023-04-19T22:00:45Z

Dockerfile


-# Copy the application files
-COPY --chown=appuser:appuser autogpt/ ./autogpt
+COPY . .


We don't need all files in the container build. docker-compose.yml contains a non-selective volume mount, that's fine because it is intended for local use. Please be more selective here though, since we will be shipping images using this config.

Pwuts · 2023-04-20T13:50:25Z

Dockerfile

 RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \
    && apt-get update \
    && apt-get install -y chromium firefox-esr


Chromium is already installed by apt-get install chromium-driver. This might entirely be replaced by apt-get install -y firefox-esr.

Suggested change

RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \

&& echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list \

&& apt-get update \

&& apt-get install -y chromium firefox-esr

RUN apt-get install -y firefox-esr

Pwuts · 2023-04-20T13:52:21Z

Dockerfile

-RUN apt-get -y update
-RUN apt-get -y install git chromium-driver
+RUN apt-get update
+RUN apt-get install -y git chromium-driver wget gnupg2 libgtk-3-0 libdbus-glib-1-2 dbus-x11 xvfb ca-certificates


I'd prefer keeping X11 and the Xvfb etc out of the docker image, as docker containers are headless anyways, and clearly state in the docs that only headless browsing is supported in the docker container. Headless browsing should work fine since #1473.

github-actions · 2023-04-21T18:08:49Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

Pwuts · 2023-04-24T11:16:32Z

Closing in favor of #1843. Sorry, but we're trying to move quickly and there wasn't any activity on this PR.

mikekelly force-pushed the docker-consistency branch from dc40016 to 5185086 Compare April 14, 2023 10:57

nponeccop previously approved these changes Apr 14, 2023

View reviewed changes

mikekelly dismissed nponeccop’s stale review via cf9e6fa April 14, 2023 11:37

mikekelly force-pushed the docker-consistency branch 3 times, most recently from 5cc4ba3 to c940762 Compare April 14, 2023 11:48

mikekelly mentioned this pull request Apr 14, 2023

Improve Dockerfile with best practices and optimizations #1231

Merged

5 tasks

mikekelly force-pushed the docker-consistency branch from c940762 to 62dadc2 Compare April 14, 2023 11:57

This was referenced Apr 14, 2023

Fix/dockerfile permissions #1271

Closed

correctly set the path for Dockerfile and docker-compose in order to … #1277

Closed

nponeccop reviewed Apr 14, 2023

View reviewed changes

scripts/logger.py Outdated Show resolved Hide resolved

nponeccop previously approved these changes Apr 14, 2023

View reviewed changes

mikekelly dismissed nponeccop’s stale review via 22f6cf5 April 14, 2023 12:18

mikekelly force-pushed the docker-consistency branch from 62dadc2 to 22f6cf5 Compare April 14, 2023 12:18

nponeccop previously approved these changes Apr 14, 2023

View reviewed changes

nponeccop mentioned this pull request Apr 14, 2023

PR Batch 5 (WIP) #1162

Closed

1 task

Korri mentioned this pull request Apr 15, 2023

Update docker-compose.yml #1342

Closed

5 tasks

stuszynski mentioned this pull request Apr 15, 2023

PermissionError: [Errno 13] Permission denied: '../logs' #1328

Closed

Pwuts mentioned this pull request Apr 16, 2023

Improve docker setup & config #1843

Merged

5 tasks

nponeccop mentioned this pull request Apr 16, 2023

PR Batch 6 #1699

Closed

1 task

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Apr 17, 2023

mikekelly dismissed nponeccop’s stale review via e938c59 April 17, 2023 16:02

mikekelly force-pushed the docker-consistency branch from 22f6cf5 to e938c59 Compare April 17, 2023 16:02

mikekelly dismissed nponeccop’s stale review via d5f2d7a April 19, 2023 08:20

mikekelly force-pushed the docker-consistency branch 2 times, most recently from d5f2d7a to 4549b97 Compare April 19, 2023 08:21

mikekelly force-pushed the docker-consistency branch from 4549b97 to a1749d5 Compare April 19, 2023 13:33

mikekelly force-pushed the docker-consistency branch from a1749d5 to 6171920 Compare April 19, 2023 16:53

mikekelly changed the title ~~make the docker setup consistent with local~~ Make the docker setup consistent with local Apr 19, 2023

make the docker setup consistent with local

cc48b09

mikekelly force-pushed the docker-consistency branch from 6171920 to cc48b09 Compare April 19, 2023 17:07

nponeccop added docker Obsolete? and removed B8 labels Apr 19, 2023

Pwuts requested changes Apr 19, 2023

View reviewed changes

Pwuts removed the bug Something isn't working label Apr 20, 2023

Pwuts self-assigned this Apr 20, 2023

Pwuts mentioned this pull request Apr 20, 2023

Optimize Dockerfile apt task #2671

Closed

1 task

Pwuts reviewed Apr 20, 2023

View reviewed changes

Pwuts mentioned this pull request Apr 21, 2023

Allows the Docker Container to actually use redis to write to the local machine via a network #2331

Closed

5 tasks

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Apr 21, 2023

Pwuts closed this Apr 24, 2023

lutzseverino mentioned this pull request May 9, 2023

Permission denied to '/app/logs/activity.log' #4048

Closed

1 task

SquareandCompass pushed a commit to SquareandCompass/Auto-GPT that referenced this pull request Oct 21, 2023

document response fields (Significant-Gravitas#1199)

3a3e115

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the docker setup consistent with local #1199

Make the docker setup consistent with local #1199

mikekelly commented Apr 13, 2023 •

edited

Loading

tech3br commented Apr 13, 2023

nponeccop commented Apr 14, 2023

nponeccop commented Apr 15, 2023

github-actions bot commented Apr 17, 2023

github-actions bot commented Apr 17, 2023

github-actions bot commented Apr 17, 2023

github-actions bot commented Apr 17, 2023

mikekelly commented Apr 19, 2023

mikekelly commented Apr 19, 2023

montanaflynn commented Apr 19, 2023

mikekelly commented Apr 19, 2023

montanaflynn commented Apr 19, 2023

mikekelly commented Apr 19, 2023

Pwuts left a comment

Pwuts Apr 19, 2023

Pwuts Apr 19, 2023

Pwuts Apr 19, 2023

Pwuts Apr 19, 2023

Pwuts Apr 20, 2023

Pwuts Apr 20, 2023 •

edited

Loading

github-actions bot commented Apr 21, 2023

Pwuts commented Apr 24, 2023


		# Install Xvfb and other dependencies for headless browser testing

	profiles: ["exclude-from-up"] # Run and attach to container instead with: `docker-compose run auto-gpt`
	profiles: ["exclude-from-up"] # Run and attach to container instead with: `docker-compose run --rm auto-gpt`

-    env_file:
-      - .env
+    env_file:
+      - .env
+    environment:
+      MEMORY_BACKEND: ${MEMORY_BACKEND:-redis}
+      REDIS_HOST: ${REDIS_HOST:-redis}

Make the docker setup consistent with local #1199

Make the docker setup consistent with local #1199

Conversation

mikekelly commented Apr 13, 2023 • edited Loading

Background

Changes

Testing

PR Quality Checklist

tech3br commented Apr 13, 2023

nponeccop commented Apr 14, 2023

nponeccop commented Apr 15, 2023

github-actions bot commented Apr 17, 2023

github-actions bot commented Apr 17, 2023

github-actions bot commented Apr 17, 2023

github-actions bot commented Apr 17, 2023

mikekelly commented Apr 19, 2023

mikekelly commented Apr 19, 2023

montanaflynn commented Apr 19, 2023

mikekelly commented Apr 19, 2023

montanaflynn commented Apr 19, 2023

mikekelly commented Apr 19, 2023

Pwuts left a comment

Choose a reason for hiding this comment

Pwuts Apr 19, 2023

Choose a reason for hiding this comment

Pwuts Apr 19, 2023

Choose a reason for hiding this comment

Pwuts Apr 19, 2023

Choose a reason for hiding this comment

Pwuts Apr 19, 2023

Choose a reason for hiding this comment

Pwuts Apr 20, 2023

Choose a reason for hiding this comment

Pwuts Apr 20, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Apr 21, 2023

Pwuts commented Apr 24, 2023

mikekelly commented Apr 13, 2023 •

edited

Loading

Pwuts Apr 20, 2023 •

edited

Loading