Infrastructure

Naming

Server	External host	Branch	Description
sd1.52k	spacedock.info	master	Current production server
sd1a.52k	sd1a.52k.de	-	Production container A
sd1b.52k	sd1b.52k.de	-	Production container B
sd-prod-live.52k	sd-prod-live.52k.de	master	Alias for sd1.52k, always pointing to the active container
sd2.52k	beta.spacedock.info	beta	Stable-ish changes to be merged into production
sd6.52k	alpha.spacedock.info	alpha	Latest raw changes merged from fix/feature branches

The two production containers (sd1a & sd1b) get swapped during upgrades, in a blue-green-deployment like fashion.

Web server setup

The backend is a Flask application run with gunicorn. An Apache Web Server in the same container handles requests to static files (mod zips, background images, JS/CSS/fonts), serving them directly from disk, and proxies dynamic requests to gunicorn.

In front of all web servers of all containers is an Apache Traffic Server, which handles all the caching.

                                 ┌────────────────────────────────────────────┐
                                 │                                            │
                                 │ Production container                       │
                                 │                                            │
                                 ├────────┐   ┌─────────────────────────────┐ │
                                 │        │   │                             │ │
                                 │        ├──►│ gunicorn backend            │ │
              ┌─────────┐        │        │   │ (SpaceDock's Python code)   │ │
─────────────►│         │        │        │   │                             │ │
              │ Apache  │        │ Apache │   └─────────────────────────────┘ │
─────────────►│ Traffic ├───┬───►│ Web    │                                   │
              │ Server  │   │    │ Server │   ┌─────────────────────────────┐ │
─────────────►│         │   │    │        │   │                             │ │
              └─────────┘   │    │        ├──►│ Storage (via X-Sendfile or  │ │
                            │    │        │   │          ProxyPass / Alias) │ │
                            │    │        │   │                             │ │
                            │    ├────────┘   └─────────────────────────────┘ │
                            │    │                                            │
                            │    └────────────────────────────────────────────┘
                            │
                            │    ┌────────────────────────────┐
                            │    │                            │
                            ├───►│ Alpha (same as Production) │
                            │    │                            │
                            │    └────────────────────────────┘
                            │
                            │    ┌────────────────────────────┐
                            │    │                            │
                            ├───►│ Beta  (same as Production) │
                            │    │                            │
                            │    └────────────────────────────┘
                            │
                            │    ┌────────────────────────────┐
                            │    │                            │
                            └───►│ Other stuff VITAS hosts    │
                                 │                            │
                                 └────────────────────────────┘

Analyzing request path

One can figure out through which servers a request went by looking at the via: header. Whenever one of the web servers passes it along to the next layer, it adds an entry for itself. E.g.

• via: http/1.1 localhost (52K) means ATS (identifying itself as localhost (52k)) got the request and passed it to AW, which then served it from the disk without contacting gunicorn.
• via: 1.1 spacedock.info, http/1.1 localhost (52K) means ATS received the request and passed it to AW (identifying as spacedock.info), which received it and passed it to gunicorn.

Note that this header is "cached" alongside the actual content, i.e. if ATS serves the file from memory cache, it still attaches the via part as if it forwarded it to AW.

Storage

The storage for the mods is a CIFS/SMB share mounted from a Hetzner storage box. The concrete mount situation is unclear, whether it's mounted on the host and bind-mounted into the container, or directly mounted in the container using Proxmox' UI, or mounted via a custom script.

The mount options:

//xxxx.your-storagebox.de/backup on /storage/sdmods type cifs (rw,relatime,vers=3.0,sec=ntlmssp,cache=strict,username=xxxxx,uid=33,forceuid,gid=33,forcegid,addr=XXX.XXX.XXX.XXX,file_mode=0777,dir_mode=0777,seal,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1)

Some unknown output:

mp0: /mnt/storage3/storage/sdmods,mp=/storage/sdmods

• mounted twice?
• try out cache=loose and compare performance
• try out fsc and compare performance