Refactor docker, fix unhandled error (#64)

* Fix unhandled error handling

* Rework docker, remove unnecessary source map support

* Fix confusing debug log, improve warning

* Add --rm flag, self review, fix tests

Dockerfile

@@ -0,0 +1,55 @@
+# The common multistage Dockerfile for all packages in the monorepo. The idea is to create a "build" stage that is a
+# base for building all packages. The advantage is that Docker reuses the "build" resulting in faster builds. Some pnpm
+# features also make Docker caching more efficient. That said, pnpm documentation regarding Docker is not very good and
+# their official example for monorepo (https://pnpm.io/docker) doesn't work at the time of writing. See also:
+# - https://github.com/pnpm/pnpm/issues/3114#issuecomment-1177144513
+# - https://github.com/pnpm/pnpm/issues/3114#issuecomment-1162821282
+#
+# Note, that Dockerfile assumes the context (path) is the root of the monorepo in order to generate the common "build"
+# stage.
+#
+# Debugging tips (assuming CWD = ./packages/pusher):
+#   1. Build: docker build --target pusher --tag pusher:latest ../../
+#   2. Inspect: docker run -it --init -v $(pwd)/config:/app/config --env-file .env --entrypoint /bin/sh pusher:latest
+# The above commands will allow you to inspect the output of the build stage. You can change the target to debug other
+# stages and verify that the image is correct.
+
+# We use the alpine image because of its small size. The alternative considered was the "slim" image, but it is larger
+# and we already use alpine (without issues) in other projects, so the size reduction seems worth it.
+FROM node:18-alpine AS build
+WORKDIR /app
+RUN npm install -g pnpm
+# Copy just the "pnpm-lock.yaml" file and use "pnpm fetch" to download all dependencies just from the lockfile. This
+# command is specifically designed to improve building a docker image because it only installs the dependencies if the
+# lockfile has changed (otherwise uses the cached value).
+COPY pnpm-lock.yaml /app
+RUN pnpm fetch
+# Copies all of the contents (without files listed in .dockerignore) of the monorepo into the image.
+COPY . /app
+# Ideally, we would use "--offline" option, but it seems pnpm has a bug. Fortunately, the instalation times are similar.
+# See: https://github.com/pnpm/pnpm/issues/6058 for details.
+RUN pnpm install --recursive --prefer-offline
+# Build all packages in the monorepo.
+RUN pnpm run --recursive build
+
+# Create a separate stage for pusher package. We create a temporary stage for deployment and then copy the result into
+# the final stage. Only the production dependencies and package implementation is part of this last stage.
+FROM build AS deployed-pusher
+RUN pnpm --filter=pusher --prod deploy deployed-pusher
+FROM node:18-alpine as pusher
+WORKDIR /app
+ENV NODE_ENV=production
+COPY --from=deployed-pusher /app/deployed-pusher .
+# Enabling source maps adds small runtime overhead, but improves debugging experience and the performance is worth it.
+ENTRYPOINT ["node", "--enable-source-maps", "dist/index.js"]
+
+# Create a separate stage for api package. We create a temporary stage for deployment and then copy the result into
+# the final stage. Only the production dependencies and package implementation is part of this last stage.
+FROM build AS deployed-api
+RUN pnpm --filter=api --prod deploy deployed-api
+FROM node:18-alpine as api
+WORKDIR /app
+ENV NODE_ENV=production
+COPY --from=deployed-api /app/deployed-api .
+# Enabling source maps adds small runtime overhead, but improves debugging experience and the performance is worth it.
+ENTRYPOINT ["node", "--enable-source-maps", "dist/index.js"]

package.json

@@ -8,6 +8,7 @@
   "scripts": {
     "build": "pnpm recursive run build",
     "clean": "pnpm recursive run clean",
+    "docker:build": "pnpm recursive run docker:build",
     "eslint:check": "pnpm recursive run eslint:check",
     "eslint:fix": "pnpm recursive run eslint:fix",
     "prettier:check": "pnpm recursive run prettier:check",

packages/api/README.md

@@ -50,8 +50,6 @@ AWS_S3_BUCKET_NAME=my-config-bucket
 AWS_S3_BUCKET_PATH=configs/my-app/signed-api.json
 ```
 
-<!-- TODO: Document how to pass ENVs through docker -->
-
 <!-- NOTE: Keep the logger configuration in-sync with logger and pusher. -->
 
 #### `LOGGER_ENABLED`
@@ -158,27 +156,43 @@ To deploy on premise you can use the Docker instructions below.
 
 ## Docker
 
-The API is also dockerized. Docker needs to publish the port of the server (running inside the docker) to the port on
-the host machine. By default, it expects the server is running on port `8090` and publishes it to the `8090` on the
-host. To change this, you need to modify both `signed-api.json` configuration and use `PORT` environment variable with
-the same value.
+The API is also dockerized. To run the dockerized APi, you need to:
+
+1. Publish the port of the API to the host machine using the `--publish` flag.
+2. Mount config folder to `/app/config`. The folder should contain the `signed-api.json` file.
+3. Pass the `-it --init` flags to the docker run command. This is needed to ensure the docker is stopped gracefully. See
+   [this](https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals) for details.
+4. Specify the `--env-file` with the path to the `.env` file containing the [ENV configuration](#environment-variables).
+5. Optionally, pass the `--rm` flag to remove the container after it is stopped.
 
-In order to run the API from a docker, run:
+For example:
 
-```bash
-# Starts the API on port 8090
-pnpm run docker:start
-# Or in a detached mode
-pnpm run docker:detach:start
-# Optionally specify port (also make sure the same port is specified inside `signed-api.json`)
-PORT=5123 pnpm run docker:start
+```sh
+# Assuming the current folder contains the "config" folder and ".env" file and the API port is 8090.
+docker run --publish 8090:8090 -it --init --volume $(pwd)/config:/app/config --env-file .env --rm api:latest
+```
+
+As of now, the docker image is not published anywhere. You need to build it locally. To build the image run:
+
+```sh
+docker build --target api --tag api:latest ../../
+```
+
+### Development only docker instructions
+
+You can use shorthands from package.json. To understand how the docker image is built, read the
+[Dockerfile](../../Dockerfile).
+
+```sh
+pnpm run docker:build
+pnpm run docker:run
 ```
 
 ### Examples
 
 Here are some examples of how to use the API with `curl`. Note, the port may differ based on the configuration.
 
-```bash
+```sh
 # Upsert batch of signed data (HTTP POST)
 curl --location 'http://localhost:8090' \
 --header 'Content-Type: application/json' \

packages/api/package.json

@@ -10,23 +10,20 @@
     "build": "tsc --project tsconfig.build.json",
     "clean": "rm -rf coverage dist",
     "dev": "nodemon --ext ts,js,json,env  --exec \"pnpm ts-node src/index.ts\"",
-    "docker:build": "docker compose --file docker/docker-compose.yml build",
-    "docker:start:detach": "docker compose --file docker/docker-compose.yml up --detach",
-    "docker:start": "docker compose --file docker/docker-compose.yml up",
-    "docker:stop": "docker compose --file docker/docker-compose.yml down",
+    "docker:build": "docker build --target api --tag api:latest ../../",
+    "docker:start": "docker run --publish 8090:8090 -it --init --volume $(pwd)/config:/app/config --env-file .env --rm api:latest",
     "eslint:check": "eslint . --ext .js,.ts --max-warnings 0",
     "eslint:fix": "eslint . --ext .js,.ts --fix",
     "prettier:check": "prettier --check \"./**/*.{js,ts,md,json}\"",
     "prettier:fix": "prettier --write \"./**/*.{js,ts,md,json}\"",
-    "start-prod": "node dist/src/index.js",
+    "start-prod": "node --enable-source-maps dist/index.js",
     "test": "jest",
     "tsc": "tsc --project ."
   },
   "license": "MIT",
   "devDependencies": {
     "@types/express": "^4.17.18",
     "@types/lodash": "^4.14.199",
-    "@types/source-map-support": "^0.5.8",
     "nodemon": "^3.0.1"
   },
   "dependencies": {
@@ -37,7 +34,6 @@
     "express": "^4.18.2",
     "lodash": "^4.17.21",
     "signed-api/common": "workspace:common@*",
-    "source-map-support": "^0.5.21",
     "zod": "^3.22.2"
   }
 }

packages/api/src/index.ts

@@ -1,9 +1,3 @@
-// Imports the source-map-support module and registers it to enable source map support in Node.js. This allows stack
-// traces to show information from the original code, rather than the compiled JavaScript.
-//
-// You can check how this works by following the demo from https://github.com/evanw/node-source-map-support#demos. Just
-// create a test script with/without the source map support, build the project and run the built script using node.
-import 'source-map-support/register';
 import { startServer } from './server';
 import { logger } from './logger';
 import { fetchAndCacheConfig } from './config';

packages/pusher/README.md

@@ -21,7 +21,13 @@ The pusher needs a configuration in order to run. The `config` folder contains e
 To start the the pusher in dev mode run the following:
 
 1. `cp pusher.example.json pusher.json` - To copy the pusher configuration from the example. Note, the `pusher.json`
-   file is ignored by git.
+   file is ignored by git. If you are using Docker Desktop, you need to change the URL from localhost to
+   `host.docker.internal`. For example:
+
+   ```jsonc
+   "url": "http://host.docker.internal:8090"
+   ```
+
 2. `cp secrets.example.env secrets.env` - To copy the secrets.env needed for the configuration. This file is also
    ignored by git.
 3. Set the `NODARY_API_KEY` inside the secrets file. Ask someone from development team for the key.
@@ -69,8 +75,6 @@ LOG_FORMAT=json
 LOG_LEVEL=info
 ```
 
-<!-- TODO: Document how to pass ENVs through docker -->
-
 <!-- NOTE: Keep the logger configuration in-sync with logger and API. -->
 
 #### `LOGGER_ENABLED`
@@ -341,31 +345,33 @@ To deploy on premise you can use the Docker instructions below.
 
 ## Docker
 
-Pusher is also dockerized. The dockerized pusher needs expects environment variable `CONFIG_PATH` to be defined,
-pointing to a directory with `pusher.json` and `secrets.env` files.
+Pusher is also dockerized. To run the dockerized pusher you need to:
 
-In order to run the pusher from a docker, run:
+1. Mount config folder to `/app/config`. The folder should contain the `pusher.json` and `secrets.env` files.
+2. Pass the `-it --init` flags to the docker run command. This is needed to ensure the docker is stopped gracefully. See
+   [this](https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals) for details.
+3. Specify the `--env-file` with the path to the `.env` file containing the [ENV configuration](#environment-variables).
+4. Optionally, pass the `--rm` flag to remove the container after it is stopped.
+
+For example:
 
-```bash
-CONFIG_PATH=$(pwd)/config pnpm run docker:start
-# or in a detached mode
-CONFIG_PATH=$(pwd)/config pnpm run docker:detach:start
+```sh
+# Assuming the current folder contains the "config" folder and ".env" file.
+docker run -it --init --volume $(pwd)/config:/app/config --env-file .env --rm pusher:latest
 ```
 
-To stop a running pusher in a detached mode, run:
+As of now, the docker image is not published anywhere. You need to build it locally. To build the image run:
 
-```bash
-pnpm run docker:stop
+```sh
+docker build --target pusher --tag pusher:latest ../../
 ```
 
 ### Development only docker instructions
 
-By default the `CONFIG_PATH` is points to the `pusher/config` directory. This means it's possible to run:
+You can use shorthands from package.json. To understand how the docker image is built, read the
+[Dockerfile](../../Dockerfile).
 
-```bash
+```sh
 pnpm run docker:build
-# or
-pnpm run docker:start
+pnpm run docker:run
 ```
-
-without the need to set `CONFIG_PATH` explicitly.

packages/pusher/package.json

@@ -9,15 +9,13 @@
     "build": "tsc --project tsconfig.build.json",
     "clean": "rm -rf coverage dist",
     "dev": "nodemon --ext ts,js,json,env  --exec \"pnpm ts-node src/index.ts\"",
-    "docker:build": "docker compose --file docker/docker-compose.yml build",
-    "docker:start:detach": "docker compose --file docker/docker-compose.yml up --detach",
-    "docker:start": "docker compose --file docker/docker-compose.yml up",
-    "docker:stop": "docker compose --file docker/docker-compose.yml down",
+    "docker:build": "docker build --target pusher --tag pusher:latest ../../",
+    "docker:start": "docker run -it --init --volume $(pwd)/config:/app/config --env-file .env --rm pusher:latest",
     "eslint:check": "eslint . --ext .js,.ts --max-warnings 0",
     "eslint:fix": "eslint . --ext .js,.ts --fix",
     "prettier:check": "prettier --check \"./**/*.{js,ts,md,yml,json}\"",
     "prettier:fix": "prettier --write \"./**/*.{js,ts,md,yml,json}\"",
-    "start-prod": "node dist/src/index.js",
+    "start-prod": "node --enable-source-maps dist/index.js",
     "test": "jest",
     "tsc": "tsc --project ."
   },
@@ -36,13 +34,11 @@
     "express": "^4.18.2",
     "lodash": "^4.17.21",
     "signed-api/common": "workspace:common@*",
-    "source-map-support": "^0.5.21",
     "zod": "^3.22.2"
   },
   "devDependencies": {
     "@types/express": "^4.17.18",
     "@types/lodash": "^4.14.199",
-    "@types/source-map-support": "^0.5.8",
     "nodemon": "^3.0.1"
   }
 }