+ B.Sc. Engineering Science |{' '}
+
+ Currently pursuing his M.Sc. Informatics
+ {' '}
+ | Architecture of Pyra 4, Development of Pyra 4 Core, CLI, and GUI
+ and the Project's DevOps and Documentation |{' '}
+
+ https://github.com/dostuffthatmatters
+ {' '}
+ | moritz.makowski[at]tum.de
+
+
+
+
+## Bug Reports & Feature Requests
+
+You can track things we are actively working on GitHub: https://github.com/tum-esm/pyra/issues. We gladly accept bug reports and feedback on Pyra. Feel free to ask us any questions about possible features or discuss similar research endeavors.
+
+Planned updates are listed as [milestones on GitHub](https://github.com/tum-esm/pyra/milestones).
diff --git a/packages/docs/docs/developer-guide/_category_.json b/packages/docs/docs/developer-guide/_category_.json
new file mode 100644
index 00000000..21214fb9
--- /dev/null
+++ b/packages/docs/docs/developer-guide/_category_.json
@@ -0,0 +1,7 @@
+{
+ "label": "Developer Guide",
+ "position": 4,
+ "link": {
+ "type": "generated-index"
+ }
+}
diff --git a/packages/docs/docs/developer-guide/architecture.mdx b/packages/docs/docs/developer-guide/architecture.mdx
new file mode 100644
index 00000000..fec1afec
--- /dev/null
+++ b/packages/docs/docs/developer-guide/architecture.mdx
@@ -0,0 +1,52 @@
+---
+sidebar_position: 1
+---
+
+# Architecture
+
+**The hierarchy between the three parts of Pyra:**
+
+
+
+**Inside Pyra Core's main loop:**
+
+- The Core will infinitely loop over the modules `MeasurementConditions`, `EnclosureControl`, `SunTracking`, `OpusMeasurements`, and `SystemChecks`
+- The Core will start the Upload- and Helios-Thread when they should be running but are not
+- The threads will stop themselves based on the config
+
+**The communication via the `state.json` file:**
+
+
+
+
+
+
+## Pyre Core Directory structure
+
+### Responsibilities
+
+- `types` contains all types used in the codebase. The whole codebase contains static type hints. A static-type analysis can be done using MyPy (see `scripts/`).
+- `utils` contains all supporting functionality used in one or more places.
+- `interfaces` includes the "low-level" code to interact with the PLC, the operating system, and the config- and state files.
+- `modules` contains the different steps that Pyra Core runs sequentially on the main thread.
+- `threads` contains the logic that Pyra Core runs in parallel to the main thread.
+
+### Import hierarchy
+
+```mermaid
+ graph LR;
+ A["types"] -- imported by --> B;
+ B["utils"] -- imported by --> C;
+ C["interfaces"] -- imported by --> D;
+ C -- imported by --> E;
+ D["modules"] -- imported by --> F;
+ E["threads"] -- imported by --> F["main.py"];
+```
+
+_\* the graph is transient_
diff --git a/packages/docs/docs/developer-guide/external-interfaces.mdx b/packages/docs/docs/developer-guide/external-interfaces.mdx
new file mode 100644
index 00000000..4a55f9dd
--- /dev/null
+++ b/packages/docs/docs/developer-guide/external-interfaces.mdx
@@ -0,0 +1,79 @@
+---
+sidebar_position: 3
+sidebar_label: External Interfaces
+---
+
+# Interfaces with external programs
+
+## OPUS
+
+We are using the `pywin32` Python library, which provides an interface to the [Windows DDE APIs](https://en.wikipedia.org/wiki/Dynamic_Data_Exchange). Pyra Core uses DDE to communicate with OPUS: Loading experiment files, starting and stopping macros, and stopping OPUS itself.
+
+```python
+import dde
+
+# initialize DDE client
+server = dde.CreateServer()
+server.Create("Client")
+conversation = dde.CreateConversation(server)
+
+# connect to OPUS
+conversation.ConnectTo("OPUS", "OPUS/System")
+assert self.conversation.Connected()
+
+# send a command to OPUS
+macro_path = "..."
+answer = conversation.Request(f"RUN_MACRO {macro_path}")
+logger.info(f"Started OPUS macro: {macro_path}")
+```
+
+:::note
+
+We are working on improving the reliability of that connection: https://github.com/tum-esm/pyra/issues/124
+
+:::
+
+## CamTracker
+
+The `CamTrackerConfig.txt` file contains coordinates that Pyra Core uses in its Astronomy utility class. The coordinates can be found by looking for the `$1` mark inside the file:
+
+```
+...
+
+$1
+48.15
+11.57
+0.54
+
+...
+```
+
+The `LEARN_Az_Elev.dat` file contains CamTracker's logs about mirror position and currently estimated sun position. Example:
+
+```dat
+Julian Date, Tracker Elevation, Tracker Azimuth, Elev Offset from Astro, Az Offset from Astro, Ellipse distance/px
+ 2458332.922778,107.490800,149.545000,0.197305,0.188938,705.047211
+ 2458332.922836,107.494400,149.761400,0.192179,0.365420,736.914133
+ 2458332.922905,107.498400,150.208200,0.188914,0.778934,736.914133
+ 2458332.922975,107.499200,149.811600,0.179557,0.335728,736.914133
+ 2458332.923032,107.508400,149.647800,0.182958,0.145281,736.914133
+ ...
+```
+
+The `SunIntensity.dat` file contains CamTracker's own evaluation of the sun conditions. This is currently not used in Pyra Core, but might be in the future. Example:
+
+```dat
+Julian Date, Date UTC[yyyy/MM/dd HH:mm:ss], Intensity[px*ms], big and small obj good
+ 2457057.199028, 2015/02/03 16:46:36, 0.000000, good
+ 2457057.200347, 2015/02/03 16:48:30, 0.000000, good
+ 2458332.906019, 2018/08/02 09:44:40, 0.000000, good
+ 2458332.906088, 2018/08/02 09:44:46, nan, bad
+ 2458332.906169, 2018/08/02 09:44:53, nan, bad
+ ...
+```
+
+When stopping CamTracker, we have to add a `stop.txt` inside CamTracker's code directory and the application will shut down gracefully.
+
+## TUM PLC
+
+We are using the Snap7 Python library to communicate with the TUM PLC. Here is a manual for the underlying Snap7 API: http://snap7.sourceforge.net/sharp7.html
diff --git a/packages/docs/docs/developer-guide/internal-interfaces.mdx b/packages/docs/docs/developer-guide/internal-interfaces.mdx
new file mode 100644
index 00000000..cdaef3c7
--- /dev/null
+++ b/packages/docs/docs/developer-guide/internal-interfaces.mdx
@@ -0,0 +1,236 @@
+---
+sidebar_position: 2
+sidebar_label: Internal Interfaces
+---
+
+# Interfaces used internally
+
+## `config.json`
+
+The config file under `config/config.json` contains all parameters to tweak Pyra's operation. Schema:
+
+```typescript
+export type config = {
+ general: {
+ version: '4.0.5';
+ seconds_per_core_interval: number;
+ test_mode: boolean;
+ station_id: string;
+ min_sun_elevation: number;
+ };
+ opus: {
+ em27_ip: string;
+ executable_path: string;
+ experiment_path: string;
+ macro_path: string;
+ username: string;
+ password: string;
+ };
+ camtracker: {
+ config_path: string;
+ executable_path: string;
+ learn_az_elev_path: string;
+ sun_intensity_path: string;
+ motor_offset_threshold: number;
+ };
+ error_email: {
+ sender_address: string;
+ sender_password: string;
+ notify_recipients: boolean;
+ recipients: string;
+ };
+ measurement_decision: {
+ mode: 'automatic' | 'manual' | 'cli';
+ manual_decision_result: boolean;
+ cli_decision_result: boolean;
+ };
+ measurement_triggers: {
+ consider_time: boolean;
+ consider_sun_elevation: boolean;
+ consider_helios: boolean;
+ start_time: { hour: number; minute: number; second: number };
+ stop_time: { hour: number; minute: number; second: number };
+ min_sun_elevation: number;
+ };
+ tum_plc: null | {
+ ip: string;
+ version: number;
+ controlled_by_user: boolean;
+ };
+ helios: null | {
+ camera_id: number;
+ evaluation_size: number;
+ seconds_per_interval: number;
+ measurement_threshold: number;
+ edge_detection_threshold: number;
+ save_images: boolean;
+ };
+ upload: null | {
+ host: string;
+ user: string;
+ password: string;
+ upload_ifgs: boolean;
+ src_directory_ifgs: string;
+ dst_directory_ifgs: string;
+ remove_src_ifgs_after_upload: boolean;
+ upload_helios: boolean;
+ dst_directory_helios: string;
+ remove_src_helios_after_upload: boolean;
+ };
+};
+```
+
+In addition to this schema, there are value rules that are validated when parsing the config object:
+
+```python
+assertions: list[Callable[[], None]] = [
+ lambda: assert_min_max("general.seconds_per_core_interval", 5, 600),
+ lambda: assert_min_max("general.min_sun_elevation", 0, 90),
+ lambda: assert_ip_address("opus.em27_ip"),
+ lambda: assert_file_path("opus.executable_path"),
+ lambda: assert_file_path("opus.experiment_path"),
+ lambda: assert_file_path("opus.macro_path"),
+ lambda: assert_file_path("camtracker.config_path"),
+ lambda: assert_file_path("camtracker.executable_path"),
+ lambda: assert_file_path("camtracker.learn_az_elev_path"),
+ lambda: assert_file_path("camtracker.sun_intensity_path"),
+ lambda: assert_min_max("camtracker.motor_offset_threshold", -360, 360),
+ lambda: assert_min_max("measurement_triggers.min_sun_elevation", 0, 90),
+ lambda: assert_min_max("measurement_triggers.start_time.hour", 0, 23),
+ lambda: assert_min_max("measurement_triggers.stop_time.hour", 0, 23),
+ lambda: assert_min_max("measurement_triggers.start_time.minute", 0, 59),
+ lambda: assert_min_max("measurement_triggers.stop_time.minute", 0, 59),
+ lambda: assert_min_max("measurement_triggers.start_time.second", 0, 59),
+ lambda: assert_min_max("measurement_triggers.stop_time.second", 0, 59),
+ lambda: assert_ip_address("tum_plc.ip"),
+ lambda: assert_min_max("helios.camera_id", 0, 999999),
+ lambda: assert_min_max("helios.evaluation_size", 1, 100),
+ lambda: assert_min_max("helios.seconds_per_interval", 5, 600),
+ lambda: assert_min_max("helios.measurement_threshold", 0.1, 1),
+ lambda: assert_ip_address("upload.host"),
+]
+```
+
+## `state.json` and `persistent-state.json`
+
+The state file is generated under `runtime-data/state.json`. Pyra Core writes its internal values to this file. The state file is used to communicate between modules as well as with the "outside" world (UI, CLI). Its schema:
+
+```typescript
+type coreState = {
+ helios_indicates_good_conditions: boolean | null;
+ measurements_should_be_running: boolean;
+ enclosure_plc_readings: {
+ last_read_time: string | null;
+ actors: {
+ fan_speed: number | null;
+ current_angle: number | null;
+ };
+ control: {
+ auto_temp_mode: boolean | null;
+ manual_control: boolean | null;
+ manual_temp_mode: boolean | null;
+ sync_to_tracker: boolean | null;
+ };
+ sensors: {
+ humidity: number | null;
+ temperature: number | null;
+ };
+ state: {
+ cover_closed: boolean | null;
+ motor_failed: boolean | null;
+ rain: boolean | null;
+ reset_needed: boolean | null;
+ ups_alert: boolean | null;
+ };
+ power: {
+ camera: boolean | null;
+ computer: boolean | null;
+ heater: boolean | null;
+ router: boolean | null;
+ spectrometer: boolean | null;
+ };
+ connections: {
+ camera: boolean | null;
+ computer: boolean | null;
+ heater: boolean | null;
+ router: boolean | null;
+ spectrometer: boolean | null;
+ };
+ };
+ os_state: {
+ cpu_usage: number | null;
+ memory_usage: number | null;
+ last_boot_time: string | null;
+ filled_disk_space_fraction: number | null;
+ };
+};
+```
+
+The persistent-state file is generated under `logs/persistent-state.json`. It is currently only used by the Core and will not be reset with a restart of the Core. Its schema:
+
+```typescript
+type persistentCoreState = {
+ active_opus_macro_id: null | number;
+ current_exceptions: string[];
+};
+```
+
+## Validation strategy
+
+[MyPy](https://github.com/python/mypy) will make full use of the schemas included above (see [testing](/docs/developer-guide/testing-and-ci)). Whenever loading the config- or state files, the respective schema validation will run. Hence, Pyra will detect when a JSON file does not have the expected schema and raise a precise Exception. All internal code interfaces (function calls, etc.) are covered by the strict MyPy validation.
+
+:::info
+
+With `pyra-cli config get` only the schema will be validated, not the value rules. This is because the GUI can deal with invalid values, but not with an invalid schema.
+
+:::
+
+## How the UI reads logs and state
+
+The UI reads the log files and the state file periodically using [Tauri's file system API](https://tauri.app/v1/api/js/modules/fs). We tested using sockets or file watchers, but both did not work well on Windows, and reading it periodically is the most basic implementation.
+
+## Logging
+
+All scripts that output messages at runtime should use the `Logger` class:
+
+```python
+from packages.core import utils
+
+logger = utils.Logger()
+
+logger.debug("...")
+logger.info("...")
+logger.warning("...")
+logger.critical("...")
+logger.error("...")
+
+
+# By default, it will log from a "pyra.core" origin
+logger = utils.Logger()
+
+# Here, it will log from a "camtracker" origin
+logger = utils.Logger(origin="camtracker")
+```
+
+Messages from all log levels can be found in `logs/debug.log`, and messages from levels INFO/WARNING/CRITICAL/ERROR can be found in `logs/info.log`.
+
+## Activity Log
+
+_Pyra Core_ will write important events to "activity logs" stored in `logs/activity/activity-YYYY-MM-DD.json`. This is duplicate information as in the regular log files, but significantly easier to parse.
+
+```typescript
+export type activityHistory = {
+ localTime: string;
+ event:
+ | 'start-core'
+ | 'stop-core'
+ | 'start-measurements'
+ | 'stop-measurements'
+ | 'error-occured'
+ | 'errors-resolved';
+}[];
+```
+
+## Pyra CLI commands from UI
+
+All write operations from the UI (update config, etc.) are done by running Pyra CLI commands. This is the reason why we have to use the global Python interpreter instead of a virtual environment: We did not make it work that the [shell interface from Tauri](https://tauri.app/v1/api/js/modules/shell) can make use of a virtual environment.
diff --git a/packages/docs/docs/developer-guide/repository-organization.mdx b/packages/docs/docs/developer-guide/repository-organization.mdx
new file mode 100644
index 00000000..71cebe2a
--- /dev/null
+++ b/packages/docs/docs/developer-guide/repository-organization.mdx
@@ -0,0 +1,97 @@
+---
+sidebar_position: 6
+---
+
+# Repository Organization
+
+## Directory Structure
+
+```
+📁 .github (contains github actions workflows)
+
+📁 config (configuration parameters used by pyra core)
+ 📄 config.json (not in the repository, used by pyra core)
+ 📄 config.default.json (template)
+ 📄 helios.config.default.json (template)
+ 📄 tum_plc.config.default.json (template)
+ 📄 upload.config.default.json (template)
+
+📁 logs (logs and other output that pyra core produces)
+ 📄 info.log (log lines from the last 60 minutes)
+ 📄 debug.log (log lines from the last 60 minutes)
+ 📁 archive (daily log files for all older log lines)
+ 📁 activity (activity logs, jsons, only major events)
+ 📁 helios (images generated by helios)
+ 📁 helios-autoexposure (images from helios' autoexposure routine)
+
+📁 packages (contains the code for pyra core, cli, and ui)
+ 📁 core (entry point is "main.py")
+ 📁 cli (entry point is "main.py")
+ 📁 ui (project structure based on Tauri and Vite)
+ 📁 docs (documentation generated with docusaurus)
+
+📁 runtime-data (where pyra core will put temporary data)
+
+📁 scripts (scripts for manual things)
+ 📄 get_upload_dir_checksum.py (calculate an upload dir checksum)
+ 📄 run_type_analysis.py (run mypy on core and cli)
+ 📄 sync_version_numbers.py (sync version number in all locations)
+
+📁 tests (tests which can be run using pytest)
+ 📁 cli (can be run in a ci environment)
+ 📁 integration (can only be run on working em27 systems)
+ 📁 repository (can be run in a ci environment)
+```
+
+## Branch Structure
+
+**Branches:** `development-...`, `integration-x.y.z`, `main`, `release`, `prerelease`
+
+**Hierarchy:**
+
+- `development-...` contains stuff in active development and will be merged into `integration-x.y.z`.
+- `integration-x.y.z` is used during active integration on the stations and will be merged into `main`.
+- `main` contains the latest running version that passed the integration and will be merged into `release` once enough changes have accumulated. Any branch can be released into `prerelease` to run the CI-Pipeline on demand.
+- `prerelease` will not be merged into anything else and is just used for development purposes.
+
+```mermaid
+graph TD;
+ A["development-person2"]-- "on finishing a set of features" -->C;
+ B["development-person1"]-- "on finishing a set of features" -->C;
+ C["integration-x.y.z"]-- "after successful integration" -->D;
+ D["main"]-- "when creating a release" -->E["release"];
+ F["any-branch"]-- "on-demand to bundle the UI" -->G["prerelease"];
+```
+
+## Issues and Releases
+
+Things we work on are managed via issues - which are bundled into milestones (each milestone represents a release). The issues should be closed once they are on the `main` branch via commit messages or pull requests ("closes #87", "fixes #70", etc. see [this list of keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)).
+
+**Issue Labels:**
+
+-
+ implemented
+
+ : Issues that have been finished but are not on the `main` branch yet.
+-
+ backlog
+
+ : Issues that are not urgent (feature ideas, improvements, etc.)
+-
+ update: ...
+
+ : Used, once we have a set of things for a good release package
+
+Planned releases are described on [GitHub](https://github.com/tum-esm/pyra/milestones).
+
+:::info
+
+We integrate new releases on two different machines. When the new software is running smoothly for some time, we make it an official release.
+
+:::
+
+## Version numbers
+
+Versions up to 4.0.4 are alpha and beta versions that should not be used regularly. Pyra can be generally used starting from version 4.0.5.
+
+Inside the codebase, the version number is included 3 times: `pyproject.toml`, `packages/ui/package.json`, `packages/ui/src-tauri/tauri.conf.json`. The script `scripts/sync_version_numbers.py` takes the version number from the `.toml` file and pastes it into the other locations. This script can be run in a [git-hook](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks).
diff --git a/packages/docs/docs/developer-guide/running-pyra-manually.mdx b/packages/docs/docs/developer-guide/running-pyra-manually.mdx
new file mode 100644
index 00000000..946f78f1
--- /dev/null
+++ b/packages/docs/docs/developer-guide/running-pyra-manually.mdx
@@ -0,0 +1,46 @@
+---
+sidebar_position: 4
+---
+
+# Running Pyra manually
+
+## Install dependencies
+
+Dependency management is done using [Poetry](https://python-poetry.org). `poetry install` installs all dependencies in the currently activated interpreter environment.
+
+## Pyra Core
+
+Normally, Pyra should be started via the CLI:
+
+```bash
+pyra-cli core start
+pyra-cli core is-running
+pyra-cli core stop
+```
+
+The start command will execute `python run-pyra-core.py` in a background process.
+
+## Upload/Helios threads
+
+In Pyra Core's regular operation, it will start and stop Upload and Helios in dedicated threads. These threads will write their output into the Core's log files. You can run these two threads in headless mode (without Pyra Core) and they will print all of their logs to the console:
+
+```bash
+python run-headless-helios-thread.py
+python run-headless-upload-thread.py
+```
+
+## Building the frontend
+
+Inside the `packages/ui` directory:
+
+- `yarn` will install all dependencies
+- `yarn tauri dev` will start a development server (with hot reload)
+- `yarn tauri build` will bundle the Tauri application and put the installer for your operating system into the `packages/ui/src-tauri/target/release/bundle` directory
+
+## Building the documentation
+
+Inside the `packages/docs` directory:
+
+- `yarn` will install all dependencies
+- `yarn develop` will start a development server (with hot reload)
+- `yarn build` will render the static HTML and bundle the SPA into the `packages/docs/build` directory
diff --git a/packages/docs/docs/developer-guide/testing-and-ci.mdx b/packages/docs/docs/developer-guide/testing-and-ci.mdx
new file mode 100644
index 00000000..5ebe343d
--- /dev/null
+++ b/packages/docs/docs/developer-guide/testing-and-ci.mdx
@@ -0,0 +1,73 @@
+---
+sidebar_position: 5
+sidebar_label: Testing & CI
+---
+
+# Testing and Continuous Integration
+
+## Testing Strategy
+
+The full software only runs when OPUS and CamTracker are installed and an EM27 is connected. However, in test mode, the software can run on its own without interacting with any modules that require a special setup. Most functionality is tested by running the software and conducting measurements. Since Pyra only starts/stops the measurement process but does not affect output files, these output files from OPUS do not have to be tested.
+
+## Testing with Pytest
+
+Testing is done using [PyTest](https://github.com/pytest-dev/pytest/). All tests can be found in the `tests/` directory. There are two types of test functions:
+
+```python
+# can be run without config.json/hardware setup
+@pytest.mark.ci
+def test_something():
+ pass
+
+# require config.json and hardware setup
+@pytest.mark.integration
+def test_something_else():
+ pass
+```
+
+Run the two test categories with:
+
+```bash
+python -m pytest -m "ci" tests
+python -m pytest -m "integration" tests
+```
+
+The following can be used to test whether **emailing and uploading** is configured correctly:
+
+```bash
+python -m pytest tests/integration/test_emailing.py
+python -m pytest tests/integration/test_upload.py
+```
+
+## Static Typing for Pyra Core and CLI
+
+All our python code contains static type annotations. We are using [MyPy](https://github.com/python/mypy) to test the code's integrity. Run this analysis using:
+
+```bash
+bash scripts/run_type_analysis.sh
+```
+
+We are using the strict MyPy settings with a few exceptions that can be found in `pyproject.toml`.
+
+## Static Typing for Pyra UI
+
+The whole frontend is written in [Typescript](https://github.com/microsoft/TypeScript). Testing the code's integrity can be done by building the UI:
+
+```bash
+cd packages/ui
+yarn build
+```
+
+## Continuous Integration
+
+**Test on Main:** Will run all CI tests, MyPy, and build the UI _on every commit on `main` or every PR on `main`_.
+
+**Build on Prerelease:** Will build the UI and generates a release named "vX.Y.Z-prerelease (generated by CI)" that has the `.msi` file attached to it _on every commit on `prerelease`_.
+
+**Build on Release:** Will build the UI and generates a release named "vX.Y.Z (generated by CI)" that has the `.msi` file attached to it _on every commit on `release`_.
+
+:::info
+
+The `release` branch should only contain code from the latest official release. Use the `prerelease` branch to bundle the installable UI on demand.
+
+:::
diff --git a/packages/docs/docs/developer-guide/upload-in-detail.mdx b/packages/docs/docs/developer-guide/upload-in-detail.mdx
new file mode 100644
index 00000000..f9bf08e5
--- /dev/null
+++ b/packages/docs/docs/developer-guide/upload-in-detail.mdx
@@ -0,0 +1,27 @@
+---
+sidebar_position: 4
+sidebar_label: Upload in Detail
+---
+
+# The Upload Process in Detail
+
+## Upload Steps
+
+1. Create an empty `upload-meta.json` for new upload processes and upload it to the server
+2. Download the `upload-meta.json` from the server if the upload has been started before but interrupted
+3. When files are found in the `fileList` that do not exist locally: raise an exception
+4. When the remote meta says `complete=True` but there are local files not in `fileList`: raise an exception
+5. For every file that is found locally but not in the server side meta, upload it with `scp`
+6. Every 25 files - for performance reasons - update the server-side meta-file
+7. Upload the file `/scripts/get_upload_dir_checksum.py` to the server and calculate the checksum of the `YYYYMMDD` directory that is on the server (based on `fileList`).
+8. Calculate the local checksum using the same script
+9. When both checksums match, set `complete=True` in the remote meta and possibly remove the local `YYYYMMDD` directory
+
+## Causes for an `InvalidUploadState` exception
+
+Whenever the upload encounters an invalid state, it raises an `InvalidUploadState` exception, which appears in the logs, and continues with the next `YYYYMMDD` directory. Possible causes are:
+
+- The upload thread crashed during the removal of the local `YYYYMMDD` directory
+- Files to the local `YYYYMMDD` directory were added after that day
+- Files from the local `YYYYMMDD` directory were removed manually
+- The remote meta file was manipulated by hand
diff --git a/packages/docs/docs/index.mdx b/packages/docs/docs/index.mdx
new file mode 100644
index 00000000..38b97944
--- /dev/null
+++ b/packages/docs/docs/index.mdx
@@ -0,0 +1,49 @@
+---
+sidebar_position: 1
+sidebar_label: Index
+title: Index
+---
+
+# Pyra 4 Documentation
+
+## What is Pyra?
+
+Pyra (name based on [Python]() and [Ra](https://en.wikipedia.org/wiki/Ra)) is a software that automates the operation of [EM27/SUN](https://www.bruker.com/en/products-and-solutions/infrared-and-raman/remote-sensing/em27-sun-solar-absorption-spectrometer.html) measurement setups. Operating EM27/SUN devices requires a lot of human interaction. Pyra makes it possible to autonomously operate these devices 24/7.
+
+Pyra has enabled the [Technical University of Munich](https://www.tum.de/en/) to collect continuous data from 5 stations around the city of Munich since 2019 using [MUCCnet](https://atmosphere.ei.tum.de/). Versions 1 to 3 of Pyra have been experimental tools, improved internally since 2016.
+
+
+
+The goal of Pyra version 4 is to make it even more stable, easy to understand and extend, and usable by the broad EM27/SUN community.
+
+The software is licensed under GPLv3 and is [open-sourced on GitHub](https://github.com/tum-esm/pyra). Whenever using Pyra, please make sure to cite the codebase.
+
+## About This Documentation
+
+This documentation page serves as a guide for both users and developers. By user, we mean people operating Pyra on their EM27/SUN stations. By developers, we target people working on the Pyra 4 codebase itself.
+
+**For users:**
+
+- Overview of [what Pyra 4 does](/docs/user-guide/functionality)
+- How to [set it up](/docs/user-guide/setup)
+- How to [use it and work with the command line interface](/docs/user-guide/usage)
+- How the [upload](/docs/user-guide/upload) works
+- What the [TUM PLC and Helios](/docs/user-guide/tum-plc-and-helios)
+- Answers to [frequently asked euestions](/docs/user-guide/faq)
+
+**For developers:**
+
+- High level view of the [architecture](/docs/developer-guide/architecture)
+- [Interfaces between program parts](/docs/developer-guide/internal-interfaces)
+- [Interfaces to external programs](/docs/developer-guide/external-interfaces)
+- How to [manually run the parts of Pyra](/docs/developer-guide/running-pyra-manually)
+- How [the repository is organized](/docs/developer-guide/repository-organization)
+- How [testing and continuous integration](/docs/developer-guide/testing-and-ci) is set up
+
+The [contact section](/docs/contact) contains author and contact information, as well as ways to report bugs and suggest new features.
+
+:::info
+
+This documentation is **not an API reference**. Pyra 4's code is statically typed, and all functions contain doc strings - hence, the API reference is in the codebase itself. The developer guide serves as a high-level intro to the codebase structure.
+
+:::
diff --git a/packages/docs/docs/user-guide/_category_.json b/packages/docs/docs/user-guide/_category_.json
new file mode 100644
index 00000000..dccbf015
--- /dev/null
+++ b/packages/docs/docs/user-guide/_category_.json
@@ -0,0 +1,7 @@
+{
+ "label": "User Guide",
+ "position": 3,
+ "link": {
+ "type": "generated-index"
+ }
+}
diff --git a/packages/docs/docs/user-guide/faq.mdx b/packages/docs/docs/user-guide/faq.mdx
new file mode 100644
index 00000000..8e238f56
--- /dev/null
+++ b/packages/docs/docs/user-guide/faq.mdx
@@ -0,0 +1,22 @@
+---
+sidebar_position: 6
+sidebar_label: FAQ
+---
+
+# FAQ
+
+## Pyra Shows a `filelock._error.Timeout` Exception
+
+In theory it is possible that Pyra encounters a deadlock when reading state or log files - although we have never encountered it yet. The error message will say something like this:
+
+```
+filelock._error.Timeout: The file lock 'C:\Users\ga56fem\Downloads\pyra\pyra-4.0.6\config\.state.lock' could not be acquired.
+```
+
+You can use the CLI command `pyra-cli remove-filelocks` to resolve that.
+
+```note
+
+We are using the python library `filelock` to enforce exclusive use of state- and log-files: There should not be two processes interacting with one of these files at the same time. This is necessary because we have at least 4 processes that work with these files: the CLI and the main-, upload- and helios-thread.
+
+```
diff --git a/packages/docs/docs/user-guide/functionality.mdx b/packages/docs/docs/user-guide/functionality.mdx
new file mode 100644
index 00000000..2c78ab80
--- /dev/null
+++ b/packages/docs/docs/user-guide/functionality.mdx
@@ -0,0 +1,13 @@
+---
+sidebar_position: 1
+---
+
+# Functionality
+
+Measurements with an EM27/SUN require you to run CamTracker - the software that tracks the sun position and operates the mirrors accordingly - and OPUS - the software that sends commands to the EM27/SUN and collects the output data into interferogram files.
+
+
+
+Based on certain conditions - time, sun angle, or sun condition (see [Helios](/docs/user-guide/tum-plc-and-helios)) - Pyra decides whether to perform measurements or not.
+
+Pyra can handle both interfaces to start and stop measurement activity. This can either be done manually (clicking a button in the GUI), automatically or via a CLI command. In addition, Pyra monitors the measurement process and operating system stability (disk space, CPU usage, etc.), permanently logs its activity, and sends out emails when errors occur/have been resolved.
diff --git a/packages/docs/docs/user-guide/setup.mdx b/packages/docs/docs/user-guide/setup.mdx
new file mode 100644
index 00000000..5cb6f887
--- /dev/null
+++ b/packages/docs/docs/user-guide/setup.mdx
@@ -0,0 +1,194 @@
+---
+sidebar_position: 2
+sidebar_label: Setup
+---
+
+# Hot to set it up
+
+## Software Requirements
+
+:::note
+
+The following manual steps are only required at the initial system setup. All future updates and new installations of Pyra can be done with the [Pyra Setup Tool](#pyra-setup-tool).
+
+:::
+
+:::tip
+
+In some cases, you might get an error message saying that "executing `.ps1` scripts is not allowed due to an ExecutionPolicy". This can be solved with the following command (credits to https://stackoverflow.com/a/49112322/8255842):
+
+```
+Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Unrestricted
+```
+
+:::
+
+### OPUS and CamTracker
+
+Integration tests are done using **OPUS 7.8.44** and **CamTracker 3.9.2015.02.03**.
+
+### Python 3.10
+
+https://www.python.org/
+
+Your system's python3.10 interpreter should be available as shell environment variable `python`. Therefore, you need to add system environment variables to point the `python` command (and `pip`) to the Python3.10 interpreter by checking the box marked in the image below.
+
+
+
+
+
+ Incomplete installation:
+
+
+
+```
+$ python --version
+Python was not found; run without arguments to install
+from the Microsoft Store, or disable this shortcut from
+Settings > Manage App Execution Aliases.
+```
+
+
+
+ Complete installation:
+
+
+
+```
+$ python --version
+Python 3.10.x
+```
+
+If your `python` command doesn't resolve to the installed version, please look at the section ["Add Environment Paths"](#add-environment-paths) and add the path ".../Python310" and "Python310/Scripts" where you just installed Python.
+
+### Install Poetry
+
+https://python-poetry.org/
+
+### Git & GitHub CLI
+
+https://git-scm.com/ and https://cli.github.com/
+
+:::info
+
+Git and the GitHub CLI are used by the Pyra Setup Tool.
+
+:::
+
+## Pyra Setup Tool
+
+The [**Pyra Setup Tool**](https://github.com/tum-esm/pyra-setup-tool) can be used to install and migrate between Pyra versions. The tool looks for the "Documents" directory on your system and generates a directory structure like this:
+
+```
+📁
+ 📁 pyra
+ 📁 pyra-4.0.5
+ 📁 pyra-4.0.6
+ 📁 ...
+```
+
+On the first time, clone the setup tool repo:
+
+```bash
+cd %userprofile%/Documents
+gh repo clone tum-esm/pyra-setup-tool
+```
+
+The setup tool uses the `colorama` library:
+
+```bash
+pip install colorama
+```
+
+**Run the tool with (from this point on it will auto-update itself):**
+
+```bash
+cd %userprofile%/Documents/pyra-setup-tool
+python run.py
+```
+
+The setup tool automates the following steps:
+
+1. Download each release's codebase
+2. Install all python dependencies into the global python interpreter environment
+3. Download and runs the GUI installer
+4. Create desktop shortcuts to the `pyra-x.y.z` directory
+5. (Optional) Migrate the `config.json` file to keep the previous settings
+
+## Pyra CLI Command
+
+You need to actively tell Windows about the `pyra-cli` command. When using the installer, it will tell you which path to [add to the environment variables](#add-environment-paths) on every install process.
+
+
+
+:::note
+
+The edit of the path variable is only required once per Pyra computer - when setting it up initially.
+
+:::
+
+## Add Environment Paths
+
+1. Look for the "Edit the system environment variables" utility
+
+
+
+2. Click on "Environment Variables"
+
+
+
+3. Select "PATH" in the system section and click on "Edit"
+
+
+
+4. Add the required paths to the list
+
+## Releases
+
+All Pyra releases can be found here: https://github.com/tum-esm/pyra/releases. Please only use the releases starting with `4.0.5`. Older versions are experimental.
+
+## Error Email Setup
+
+For Pyra to send you error emails (see [Usage section](/docs/user-guide/usage#error-emails)), you need an email account that allows this kind of connection.
+
+We are using Gmail for this. You have to use an "App Password" for Pyra, which requires 2FA for your Google Account. Please refer to this article on how to set up an "App Password": https://support.google.com/accounts/answer/185833. Inside the config tab, you have to add this "App Password" to receive emails:
+
+
+
+:::note
+
+There is a test script to test whether email sending works. See [/docs/developer-guide/testing-and-ci](/docs/developer-guide/testing-and-ci).
+
+We plan to add a button "test emailing" to the config tab: https://github.com/tum-esm/pyra/issues/109
+
+:::
+
+:::caution
+
+The "Gmail password" and the "app password" are not the same.
+
+Whenever changing/resetting the "Gmail password", all "app passwords" will be invalidated.
+
+:::
+
+## Upload
+
+You need a server running Linux that Pyra can connect to via SSH - using a password. The server has to have `python3.10` installed.
+
+In the config.upload section, you can set connection details, and source and destination directories for your interferograms. How the upload works is described [here](/docs/user-guide/upload) in detail.
diff --git a/packages/docs/docs/user-guide/tum-plc-and-helios.mdx b/packages/docs/docs/user-guide/tum-plc-and-helios.mdx
new file mode 100644
index 00000000..7dddff59
--- /dev/null
+++ b/packages/docs/docs/user-guide/tum-plc-and-helios.mdx
@@ -0,0 +1,52 @@
+---
+sidebar_position: 5
+---
+
+# TUM PLC & Helios
+
+The hardware for the TUM weather protection enclosure and Helios setup is not available to buy off the shelf yet. Please [contact us](/docs/contact) if you are interested in acquiring our hardware.
+
+
+
+The TUM hardware used by [MUCCnet](https://atmosphere.ei.tum.de/) has been described in [Dietrich et al. 2021 (https://doi.org/10.5194/amt-14-1111-2021)](https://doi.org/10.5194/amt-14-1111-2021).
+
+## What does Helios do?
+
+Helios evaluates the current sun state - whether direct or diffuse sunlight is present. EM27/SUN applications require direct sunlight for measurement data to be useful. Additionally, CamTracker will lose track of the sun with very diffuse light conditions and requires a restart.
+
+The Helios hardware is made up of a transparent globe with black stripes glued to it and a translucent milky glass with a camera pointing upwards attached below.
+
+
+
+
+
+Helios will periodically take images with that camera, process them, and evaluate shadow conditions. The processing will detect the lens circle and cut off the outer 10% of the radius - a lot of unwanted shadows can occur on the edges due to dirt from the outside. Finally, it will use a canny edge filter and count the pixels where "fast transitions between light and dark" can be found. If the pixel count is above a certain threshold (configurable), the sunlight conditions are categorized as "good".
+
+Example images in **bad** conditions:
+
+