diff --git a/README.md b/README.md index 5e44be3d9..90fb175b7 100644 --- a/README.md +++ b/README.md @@ -22,18 +22,16 @@ inside a server and exposes endpoints to send commands/receive data. Useful for installation at labs where multiple people may control equipment, possibly from remote locations. -![concept][] - The main premise of blueapi is to minimize the boilerplate required to get plans and devices up and running by generating an API for your lab out of type-annotated plans. For example, take the following plan: ```python - import bluesky.plans as bp - from blueapi.core import MsgGenerator +import bluesky.plans as bp +from blueapi.core import MsgGenerator - def my_plan(foo: str, bar: int) -> MsgGenerator: - yield from bp.scan(...) +def my_plan(foo: str, bar: int) -> MsgGenerator: + yield from bp.scan(...) ``` Blueapi's job is to detect this plan and automatically add it to the lab's API diff --git a/docs/explanations/architecture.md b/docs/explanations/architecture.md index f3c8cf848..a60c6b784 100644 --- a/docs/explanations/architecture.md +++ b/docs/explanations/architecture.md @@ -1,6 +1,5 @@ # Architecture - Blueapi performs a number of tasks: * Managing the Bluesky [RunEngine](https://nsls-ii.github.io/bluesky/run_engine_api.html), giving it instructions and handling its errors. Traditionally this job has been done by a human with an [IPython](https://ipython.org/) terminal, so it requires automating. @@ -9,26 +8,7 @@ Blueapi performs a number of tasks: These responsibilities are kept separate in the codebase to ensure a clean, maintainable architecture. -## Key Components - - -![blueapi architecture main components](../images/blueapi-architecture.png) - - -### The `BlueskyContext` Object - -Holds registries of plans and devices as well as a number of helper methods for -registering en-masse from a normal Python module. - -### The Worker Object - -Wraps the Bluesky `RunEngine` and accepts requests to run plans. The requests include the name -of the plan and a dictionary of parameters to pass. The worker validates the parameters against -the known expectations of the plan, passes it to the `RunEngine` and handles any errors. - +![blueapi main components](../images/blueapi.png) -### The Service Object -Handles communications and the API layer. This object holds a reference to the worker -can interrogate it/give it instructions in response to messages it receives from the message -bus. It can also forward the various events generated by the worker to topics on the bus. +Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and optional connection to the downstream message broker. diff --git a/docs/explanations/events.md b/docs/explanations/events.md index dbd4ac98d..ff0388fe2 100644 --- a/docs/explanations/events.md +++ b/docs/explanations/events.md @@ -9,7 +9,7 @@ Since the `RunEngine` is traditionally used by a human in front of an IPython te sometimes assumes intuitive behaviour. The worker replaces the human and so must fill in the gaps. -The base engine programmatically emits data events conforming to the `bluesky event model`_. These +The base engine programmatically emits data events conforming to the [bluesky event model](https://blueskyproject.io/event-model). These are meant to be handled by other subscribing code (e.g. databroker) and are decoupled from concerns such as whether a plan has started, finished, paused, errored etc. See the example below: diff --git a/docs/explanations/extension-code.md b/docs/explanations/extension-code.md new file mode 100644 index 000000000..0e39cbe28 --- /dev/null +++ b/docs/explanations/extension-code.md @@ -0,0 +1,20 @@ +# Home of Plans and Devices + +## Dodal + +[Dodal](https://github.com/DiamondLightSource/dodal) is a repository for DLS device configuration, providing classes and factory functions for devices used at DLS. +For specific advice on creating new device types and adding them to new or existing beamlines, see [Create a Beamline](https://diamondlightsource.github.io/dodal/main/how-to/create-beamline.html) and [Device Standards](https://diamondlightsource.github.io/dodal/main/reference/device-standards.html) in the dodal documentation. + +## Other Repositories + +Plans and devices can be in any pip-installable package, such as: + +* A package on pypi +* A Github repository +* A local directory via the [scratch area](../how-to/edit-live.md). + +The easiest place to put the code is a repository created with the [`python-copier-template`](https://diamondlightsource.github.io/python-copier-template/main/index.html). Which can then become any of the above. [Example for the I22 beamline](https://github.com/DiamondLightSource/i22-bluesky). + +:::{seealso} +Guide to setting up a new Python project with an environment and a standard set of tools: [`Create a new repo from the template`](https://diamondlightsource.github.io/python-copier-template/main/tutorials/create-new.html) +::: diff --git a/docs/explanations/lifecycle.md b/docs/explanations/lifecycle.md index 862f3a754..d11d607f7 100644 --- a/docs/explanations/lifecycle.md +++ b/docs/explanations/lifecycle.md @@ -2,43 +2,44 @@ The following demonstrates exactly what the code does with a plan through its lifecycle of being written, loaded and run. Take the following plan. -``` - from typing import Any, List, Mapping, Optional, Union - - import bluesky.plans as bp - from blueapi.core import MsgGenerator - from dls_bluesky_core.core import inject - from bluesky.protocols import Readable - - - def count( - detectors: List[Readable] = [inject("det")], # default valid for Blueapi only - num: int = 1, - delay: Optional[Union[float, List[float]]] = None, - metadata: Optional[Mapping[str, Any]] = None, - ) -> MsgGenerator: - """ - Take `n` readings from a collection of detectors - - Args: - detectors (List[Readable]): Readable devices to read: when being run in Blueapi - defaults to fetching a device named "det" from its - context, else will require to be overridden. - num (int, optional): Number of readings to take. Defaults to 1. - delay (Optional[Union[float, List[float]]], optional): Delay between readings. - Defaults to None. - metadata (Optional[Mapping[str, Any]], optional): Key-value metadata to include - in exported data. + +```python +from typing import Any, List, Mapping, Optional, Union + +import bluesky.plans as bp +from blueapi.core import MsgGenerator +from dls_bluesky_core.core import inject +from bluesky.protocols import Readable + + +def count( + detectors: List[Readable] = [inject("det")], # default valid for Blueapi only + num: int = 1, + delay: Optional[Union[float, List[float]]] = None, + metadata: Optional[Mapping[str, Any]] = None, +) -> MsgGenerator: + """ + Take `n` readings from a collection of detectors + + Args: + detectors (List[Readable]): Readable devices to read: when being run in Blueapi + defaults to fetching a device named "det" from its + context, else will require to be overridden. + num (int, optional): Number of readings to take. Defaults to 1. + delay (Optional[Union[float, List[float]]], optional): Delay between readings. Defaults to None. - - Returns: - MsgGenerator: _description_ - - Yields: - Iterator[MsgGenerator]: _description_ - """ - - yield from bp.count(detectors, num, delay=delay, md=metadata) + metadata (Optional[Mapping[str, Any]], optional): Key-value metadata to include + in exported data. + Defaults to None. + + Returns: + MsgGenerator: _description_ + + Yields: + Iterator[MsgGenerator]: _description_ + """ + + yield from bp.count(detectors, num, delay=delay, md=metadata) ``` @@ -53,50 +54,46 @@ will build a [pydantic](https://docs.pydantic.dev/) model of the parameters to v like this: -``` - from pydantic import BaseModel - - class CountParameters(BaseModel): - detectors: List[Readable] = ["det"] - num: int = 1 - delay: Optional[Union[float, List[float]]] = None - metadata: Optional[Mapping[str, Any]] = None - - class Config: - arbitrary_types_allowed = True - validate_all = True - - This is for illustrative purposes only, this code is not actually generated, but an object - resembling this class is constructed in memory. - The default arguments will be validated by the context to inject the "det" device when the - plan is run. The existence of the "det" default device is not checked until this time. -``` +```python +from pydantic import BaseModel + +class CountParameters(BaseModel): + detectors: List[Readable] = [inject("det")] + num: int = 1 + delay: Optional[Union[float, List[float]]] = None + metadata: Optional[Mapping[str, Any]] = None -The model is also stored in the context. + class Config: + arbitrary_types_allowed = True + validate_all = True +``` +This is for illustrative purposes only, this code is not actually generated, but an object +resembling this class is constructed in memory. +The default arguments will be validated by the context to inject the "det" device when the +plan is run. The existence of the "det" default device is not checked until this time. The model is also stored in the context. ## Startup On startup, the context is passed to the worker, which is passed to the service. The worker also holds a reference to the `RunEngine` that can run the plan. - ## Request A user can send a request to run the plan to the service, which includes values for the parameters. It takes the form of JSON and may look something like this: -``` - { - "name": "count", - "params": { - "detectors": [ +```json +{ + "name": "count", + "params": { + "detectors": [ "andor", "pilatus" - ], - "num": 3, - "delay": 0.1 - } + ], + "num": 3, + "delay": 0.1 } +} ``` The `Service` receives the request and passes it to the worker, which holds it in an internal queue @@ -114,13 +111,13 @@ See also [type validators](./type_validators.md) ## Execution -The validated parameter values are then passed to the plan function, which is passed to the RunEngine. +The validated parameter values are then passed to the plan function, which is passed to the `RunEngine`. The plan is executed. While it is running, the `Worker` will publish * Changes to the state of the `RunEngine` * Changes to any device statuses running within a plan (e.g. when a motor changes position) * Event model documents emitted by the `RunEngine` -* When a plan starts, finishes or fails. +* When the plan starts, finishes or fails. If an error occurs during any of the stages from "Request" onwards it is sent back to the user over the message bus. diff --git a/docs/explanations/plans.md b/docs/explanations/plans.md new file mode 100644 index 000000000..95fdef4de --- /dev/null +++ b/docs/explanations/plans.md @@ -0,0 +1,44 @@ +# Plans + +While the bluesky project uses `plan` in a general sense to refer to any `Iterable` of `Msg`'s which may be run by the `RunEngine`, blueapi distinguishes between a `plan` and a `stub`. This distinction is made to allow for a subset of `stub`'s to be exposed and run, as `stub`'s may not make sense to run alone. + +Generally, a `plan` includes at least one `open_run` and `close_run` and is a complete description of an experiment. If it does not, it is a `stub`. This distinction is made in the bluesky core library between the `plan`'s and `plan_stub`'s modules. + + +## Allowed Argument Types + +When added to the blueapi context, `PlanGenerator`'s are formalised into their schema - [a Pydantic BaseModel](https://docs.pydantic.dev/1.10/usage/models) with the expected argument types and their defaults. + +Therefore, `PlanGenerator`'s must only take as arguments [those types which are valid Pydantic fields](https://docs.pydantic.dev/dev/concepts/types) or Device types which implement `BLUESKY_PROTOCOLS` defined in dodal, which are fetched from the context at runtime. + +Allowed argument types for Pydantic BaseModels include the primitives, types that extend `BaseModel` and `dict`'s, `list`'s and other `sequence`'s of supported types. Blueapi will deserialise these types from JSON, so `dict`'s should use `str` keys. + + +## Stubs + +Some functionality in your plans may make sense to factor out to allow re-use. These pieces of functionality may or may not make sense outside of the context of a plan. Some will, such as nudging a motor, but others may not, such as waiting to consume data from the previous position, or opening a run without an equivalent closure. + +To enable blueapi to expose the stubs that it makes sense to, but not the others, blueapi will only expose a subset of `MsgGenerator`'s under the following conditions: + +- `__init__.py` in directory has `__exports__`: List[str]: only those named in `__exports__` +- `__init__.py` in directory has `__all__`: List[str] but no `__exports__`: only those named in `__all__` + +This allows other python packages (such as `plans`) to access every function in `__all__`, while only allowing a subset to be called from blueapi as standalone. + +```python +# Rehomes all of the beamline's devices. May require to be run standalone +from .package import rehome_devices +# Awaits a standard callback from analysis. Should not be run standalone +from .package import await_callback + +# Exported from the module for use by other modules +__all__ = [ + "rehome_devices", + "await_callback", +] + +# Imported by instances of blueapi and allowed to be run +__exports__ = [ + "rehome_devices", +] +``` diff --git a/docs/explanations/type_validators.md b/docs/explanations/type_validators.md index 3f38a37e9..c086237e9 100644 --- a/docs/explanations/type_validators.md +++ b/docs/explanations/type_validators.md @@ -3,15 +3,15 @@ ## Requirement Blueapi takes the parameters of a plan and internally creates a [pydantic](https://docs.pydantic.dev/) model for future validation e.g. -``` - def my_plan(a: int, b: str = "b") -> Plan - ... +```python +def my_plan(a: int, b: str = "b") -> Plan + ... - # Internally becomes something like +# Internally becomes something like - class MyPlanModel(BaseModel): - a: int - b: str = "b" +class MyPlanModel(BaseModel): + a: int + b: str = "b" ``` @@ -19,9 +19,9 @@ That way, when the plan parameters are sent in JSON form, they can be parsed and However, it must also cover the case where a plan doesn't take a simple dictionary, list or primitive but instead a device, such as a detector. -``` - def my_plan(a: int, b: Readable) -> Plan: - ... +```python +def my_plan(a: int, b: Readable) -> Plan: + ... ``` An Ophyd object cannot be passed over the network as JSON because it has state. @@ -29,18 +29,18 @@ Instead, a string is passed, representing an ID of the object known to the `Blue At the time a plan's parameters are validated, blueapi must take all the strings that are supposed to be devices and look them up against the context. For example with the request: -``` - { - "name": "count", - "params": { - "detectors": [ - "andor", - "pilatus" - ], - "num": 3, - "delay": 0.1 - } +```json +{ + "name": "count", + "params": { + "detectors": [ + "andor", + "pilatus" + ], + "num": 3, + "delay": 0.1 } +} ``` `andor` and `pilatus` should be looked up and replaced with Ophyd objects. @@ -59,30 +59,32 @@ object returned from validator method is not checked by pydantic so it can be the actual instance and the plan never sees the runtime generated reference type, only the type it was expecting. -> **_NOTE:_** This uses the fact that the new types generated at runtime have access to - the context that required them via their closure. This circumvents the usual - problem of pydantic validation not being able to access external state when - validating or deserialising. +:::{note} +This uses the fact that the new types generated at runtime have access to +the context that required them via their closure. This circumvents the usual +problem of pydantic validation not being able to access external state when +validating or deserializing. +::: -``` - def my_plan(a: int, b: Readable) -> Plan: - ... +```python +def my_plan(a: int, b: Readable) -> Plan: + ... - # Becomes +# Becomes - class MyPlanModel(BaseModel): - a: int - b: Reference[Readable] +class MyPlanModel(BaseModel): + a: int + b: Reference[Readable] ``` This also allows `Readable` to be placed at various type levels. For example: -``` - def my_weird_plan( - a: Readable, - b: List[Readable], - c: Dict[str, Readable], - d: List[List[Readable]], - e: List[Dict[str, Set[Readable]]]) -> Plan: - ... +```python +def my_weird_plan( + a: Readable, + b: List[Readable], + c: Dict[str, Readable], + d: List[List[Readable]], + e: List[Dict[str, Set[Readable]]]) -> Plan: + ... ``` diff --git a/docs/how-to/add-plans-and-devices.md b/docs/how-to/add-plans-and-devices.md index a2c12e0b2..2da795535 100644 --- a/docs/how-to/add-plans-and-devices.md +++ b/docs/how-to/add-plans-and-devices.md @@ -1,103 +1,34 @@ # Add Plans and Devices to your Blueapi Environment -Custom plans and devices, tailored to individual experimental environments, can be added via configuration. In both cases the relevant code must be in your Python path, for example, part of a library installed in your Python environment. For editing and tweaking you can use an editable install, see below. +:::{seealso} +[The bluesky documentation](https://blueskyproject.io/bluesky/main/index.html) for an introduction to the nature of plans and devices and why you would want to customize them for your experimental needs. +::: -## Home of Code - -The code can be in any pip-installable package, such as: - -* A package on pypi -* A Github repository -* A local directory with a `pyproject.toml` file or similar. - -The easiest place to put the code is a repository created with the [`python-copier-template`](https://diamondlightsource.github.io/python-copier-template/main/index.html). Which can then become any of the above. - -See also: Guide to setting up a new Python project with an environment and a standard set of tools: [`Create a new repo from the template`](https://diamondlightsource.github.io/python-copier-template/main/tutorials/create-new.html) - -For development purposes this code should be installed into your environment with - -``` - pip install -e path/to/package -``` - -## Format - -Plans in Python files look like this: - -> **_NOTE:_** The type annotations (e.g. `: str`, `: int`, `-> MsgGenerator`) are required as blueapi uses them to generate an API! You can define as many plans as you like in a single Python file or spread them over multiple files. -``` - from bluesky.protocols import Readable, Movable - from blueapi.core import MsgGenerator - from typing import Mapping, Any - - def my_plan( - detector: Readable, - motor: Movable, - steps: int, - sample_name: str, - extra_metadata: Mapping[str, Any]) -> MsgGenerator: - - # logic goes here - ... -``` - -Devices are made using the [dodal](https://github.com/DiamondLightSource/dodal) style available through factory functions like this: - -> **_NOTE:_** The return type annotation `-> MyTypeOfDetector` is required as blueapi uses it to determine that this function creates a device. Meaning you can have a Python file where only some functions create devices and they will be automatically picked up. - -Similarly, these functions can be organized per-preference into files. -``` - from my_facility_devices import MyTypeOfDetector - - def my_detector(name: str) -> MyTypeOfDetector: - return MyTypeOfDetector(name, {"other_config": "foo"}) -``` - - -See also: dodal for many more examples - -An extra function to create the device is used to preserve side-effect-free imports. Each device must have its own factory function. +Blueapi can be configured to load custom code at startup that defines plans and devices. The code must be in your Python environment (via `pip install `) or your [scratch area](./edit-live.md). ## Configuration - -See also: [configure app](./configure-app.md) +:::{seealso} +[Configure the Application](./configure-app.md) +::: First determine the import path of your code. If you were going to import it in a Python file, what would you put? For example: -``` - import my_plan_library.tomography.plans +```python +import my_plan_library.tomography.plans ``` You would add the following into your configuration file: -``` - env: - sources: - - kind: dodal - # note, this code does not have to be inside dodal just because it uses - # the dodal kind. The module referenced contains a dodal-style function - # for initializing a particular device e.g. MyTypeOfDetector in my_lab. - module: dodal.my_beamline - - kind: planFunctions - module: my_plan_library.tomography.plans +```yaml +env: + sources: + - kind: planFunctions + module: my_plan_library.tomography.plans ``` - You can have as many sources for plans and devices as are needed. - -## Scratch Area on Kubernetes - -Sometimes in-the-loop development of plans and devices may be required. If running blueapi out of a virtual environment local packages can be installed with `pip install -e path/to/package`, but there is also a way to support editable packages on Kubernetes with a shared filesystem. - -Blueapi can be configured to install editable Python packages from a chosen directory, the helm chart can mount this directory from the -host machine, include the following in your `values.yaml`: -``` - scratch: - hostPath: path/to/scratch/area # e.g. /dls_sw//software/blueapi/scratch - -``` - -You can then clone projects into the scratch directory and blueapi will automatically incorporate them on startup. You must still include configuration to load the plans and devices from specific modules within those packages, see above. - +:::{seealso} +[Home of Plans and Devices](../explanations/extension-code.md) for an introduction to the nature of plans and devices and why you would want to customize them for your experimental needs. +::: diff --git a/docs/how-to/configure-app.md b/docs/how-to/configure-app.md index a42fd8740..7bee9cc52 100644 --- a/docs/how-to/configure-app.md +++ b/docs/how-to/configure-app.md @@ -1,18 +1,10 @@ -# Configure the application +# Configure the Application -By default, configuration options are ingested from pydantic BaseModels, -however the option exists to override these by defining a yaml file which -can be passed to the `blueapi` command. +Blueapi's default configuration can be overridden +by defining a yaml file which can be passed to the `blueapi` command. -An example of this yaml file is found in `config/defaults.yaml`, which follows -the schema defined in `src/blueapi/config.py` in the `ApplicationConfig` -object. +To set your own application configuration create a file and pass it to the CLI: -To set your own application configuration, edit this file (or write your own) -and simply pass it to the CLI by typing:: - -``` blueapi --config path/to/file.yaml ``` - -where `path/to/file.yaml` is the relative path to the configuration file you -wish to use. Then, any subsequent calls to child commands of blueapi will -use this file. +``` +blueapi --config path/to/file.yaml +``` diff --git a/docs/how-to/edit-live.md b/docs/how-to/edit-live.md new file mode 100644 index 000000000..8f1eb8c7c --- /dev/null +++ b/docs/how-to/edit-live.md @@ -0,0 +1,58 @@ +# Edit Plans and Device Live + +You may want to tweak/edit your palns and devices live, i.e. without having to make a new release of a Python module, `pip install` it and restart blueapi. Blueapi can be configured to use a special directory called the "scratch area" where source code can be checked out and installed in [development mode](https://setuptools.pypa.io/en/latest/userguide/development_mode.html). + +## Configuration + +Blueapi can be configured to install editable Python packages from a chosen directory: + +```yaml +# my-scratch.yaml + +scratch: + root: /path/to/my/scratch/directory + repositories: + # Repository for DLS devices + - name: dodal + remote_url: https://github.com/DiamondLightSource/dodal.git + + # Example repository full of custom plans for a particular science technique + - name: mx-bluesky + remote_url: https://github.com/DiamondLightSource/mx-bluesky.git +``` + +## Synchronization + +Blueapi will synchronize reality with the configuration if you run + +``` +blueapi -c my-scratch.yaml setup-scratch +``` + +The goal of synchronization is to make the scratch directory resemble the YAML specification without accidentally overwriting uncommited/unpushed changes that may already be there. For each specified repository, blueapi will clone it if it does not exist and otherwise ignore it. If it exists in a broken state, this can cause problems, and you may have to manually delete it from your scratch area. + +## Reloading + +:::{warning} +This will abort any running plan and delete and re-initialize all ophyd devices +::: + +If you add or remove packages from the scratch area, you will need to restart blueapi. However if you edit code that is already checked out you can tell the server to perform a hot reload via + +``` +blueapi controller env -r +``` + +## Kubernetes + +The helm chart can be configured to mount a scratch area from the +host machine, include the following in your `values.yaml`: + +```yaml + scratch: + hostPath: path/to/scratch/area # e.g. /dls_sw//software/blueapi/scratch +``` + +:::{note} +If you do this then the value of `scratch.root` in your blueapi configuration is no longer particularly important, it only specifies where to mount the scratch area _inside_ the container. +::: diff --git a/docs/how-to/run-cli.md b/docs/how-to/run-cli.md index 58d8f2960..5bba4878a 100644 --- a/docs/how-to/run-cli.md +++ b/docs/how-to/run-cli.md @@ -1,39 +1,32 @@ # Control the Worker via the CLI -The worker comes with a minimal CLI client for basic control. It should be noted that this is -a test/development/debugging tool and not meant for production! - - `./run-container` and `../tutorials/installation` - - - +Blueapi comes with a minimal CLI client for basic control/debugging. ## Basic Introspection The worker can tell you which plans and devices are available via: ``` - blueapi controller plans - blueapi controller devices +blueapi controller plans +blueapi controller devices +``` By default, the CLI will talk to the worker via a message broker on `tcp://localhost:61613`, -but you can customize this. +but you can customize this via a [configuration file](./configure-app.md). -``` - blueapi controller -h my.host -p 61614 plans +```yaml +# custom-address.yaml + +api: + host: example.com + port: 8082 ``` -## Running Plans +Then run -You can run a plan and pass arbitrary JSON parameters. -``` - # Run the sleep plan - blueapi controller run sleep '{"time": 5.0}' - - # Run the count plan - blueapi controller run count '{"detectors": ["current_det", "image_det"]}' +``` +blueapi -c custom-address.yaml controller plans ``` -The command will block until the plan is finished and will forward error/status messages -from the server. +The CLI has a number of features including [running plans](../tutorials/run-plan.md) and See also [Full CLI reference](../reference/cli.md) diff --git a/docs/how-to/run-container.md b/docs/how-to/run-container.md index dd86a779a..fb9d73bc6 100644 --- a/docs/how-to/run-container.md +++ b/docs/how-to/run-container.md @@ -1,4 +1,4 @@ -# Run in a container +# Run in a Container Pre-built containers with blueapi and its dependencies already installed are available on `Github Container Registry @@ -8,10 +8,14 @@ installed are available on `Github Container Registry To pull the container from github container registry and run:: - `docker run ghcr.io/diamondlightsource/blueapi:main --version` +``` +docker run ghcr.io/diamondlightsource/blueapi:latest +``` with `podman`: - `podman run ghcr.io/diamondlightsource/blueapi:main --version` +``` +podman run ghcr.io/diamondlightsource/blueapi:latest +``` -To get a released version, use a numbered release instead of `main`. +To get a released version, use a numbered release instead of `latest`. diff --git a/docs/how-to/write-devices.md b/docs/how-to/write-devices.md new file mode 100644 index 000000000..6abaa58f2 --- /dev/null +++ b/docs/how-to/write-devices.md @@ -0,0 +1,70 @@ +# Write Devices for Blueapi + +:::{seealso} +[Home of Plans and Devices](../explanations/extension-code.md) for information about where device code usually lives. +::: + + +## Format + +:::{seealso} +[Dodal](https://github.com/DiamondLightSource/dodal) for many more examples +::: + +Devices are made using the [dodal](https://github.com/DiamondLightSource/dodal) style available through factory functions like this: + +```python +from my_facility_devices import MyTypeOfDetector + +def my_detector(name: str) -> MyTypeOfDetector: + return MyTypeOfDetector(name, {"other_config": "foo"}) +``` + +The return type annotation `-> MyTypeOfDetector` is required as blueapi uses it to determine that this function creates a device. Meaning you can have a Python file where only some functions create devices and they will be automatically picked up. Similarly, these functions can be organized per-preference into files. + +The device is created via a function rather than a global to preserve side-effect-free imports. Each device must have its own factory function. + +# How to Configure Detectors to Write Files + +:::{note} +**This is an absolute requirement to write data onto the Diamond Filesystem**. This decorator must be used every time a new data collection is intended to begin. For an example, see below. +::: + +Dodal defines a decorator, `@attach_metadata`, for configuring `ophyd-async` detectors to write data to a common location. + +```python + @attach_metadata + def ophyd_async_snapshot( + detectors: List[Readable], + metadata: Optional[Mapping[str, Any]] = None, + ) -> MsgGenerator: + Configures a number of devices, which may be Ophyd-Async detectors and require + knowledge of where to write their files, then takes a snapshot with them. + Args: + detectors (List[Readable]): Devices, maybe including Ophyd-Async detectors. + Returns: + MsgGenerator: Plan + Yields: + Iterator[MsgGenerator]: Bluesky messages + yield from count(detectors, 1, metadata or {}) + + def repeated_snapshot( + detectors: List[Readable], + metadata: Optional[Mapping[str, Any]] = None, + ) -> MsgGenerator: + Configures a number of devices, which may be Ophyd-Async detectors and require + knowledge of where to write their files, then takes multiple snapshot with them. + Args: + detectors (List[Readable]): Devices, maybe including Ophyd-Async detectors. + Returns: + MsgGenerator: Plan + Yields: + Iterator[MsgGenerator]: Bluesky messages + @attach_metadata + def inner_function(): + yield from count(detectors, 1, metadata or {}) + + + for _ in range(5): + yield from inner_function() +``` diff --git a/docs/how-to/write-plans.md b/docs/how-to/write-plans.md index eb6d6f503..90c66a27e 100644 --- a/docs/how-to/write-plans.md +++ b/docs/how-to/write-plans.md @@ -1,173 +1,93 @@ -# Writing Bluesky plans for Blueapi - - -For an introduction to bluesky plans and general forms/advice, `see the bluesky documentation `__. Blueapi has some additional requirements, which are explained below. - -## Plans - -While the bluesky project uses `plan` in a general sense to refer to any `Iterable` of `Msg`\ s which may be run by the `RunEngine`, blueapi distinguishes between a `plan` and a `stub`. This distinction is made to allow for a subset of `stub`\ s to be exposed and run, as `stub`\ s may not make sense to run alone. - -Generally, a `plan` includes at least one `open_run` and `close_run` and is a complete description of an experiment. If it does not, it is a `stub`. This distinction is made in the bluesky core library between the `plan`\ s and `plan_stub`\ s modules. - -## Type Annotations - -To be imported into the blueapi context, `plan`\ s and `stub`\ s must be the in the form of a `PlanGenerator`: any function that return a `MsgGenerator` (a python `Generator` that yields `Msg`\ s). `PlanGenerator` and `MsgGenerator` types are available to import from `dodal`. - -``` - def foo() -> MsgGenerator: - # The minimum PlanGenerator acceptable to blueapi - yield from {} +# Write Bluesky Plans for Blueapi + +:::{seealso} +[The bluesky documentation](https://blueskyproject.io/bluesky/main/index.html) for an introduction to bluesky plans and general forms/advice. Blueapi has some additional requirements, which are explained below. +::: + +## Format + +:::{seealso} +[Explanation of why blueapi treats plans in a special way](../explanations/plans.md) +::: + +Plans in Python files look like this: + +```python +from bluesky.protocols import Readable, Movable +from bluesky.utils import MsgGenerator +from typing import Mapping, Any + +def my_plan( + detector: Readable, + motor: Movable, + steps: int, + sample_name: str, + extra_metadata: Mapping[str, Any]) -> MsgGenerator: + + # logic goes here + ... ``` -> **_NOTE:_** `PlanGenerator` arguments must be annotated to enable blueapi to generate their schema - -**Input annotations should be as broad as possible**, the least specific implementation that is sufficient to accomplish the requirements of the plan. +The type annotations (e.g. `: str`, `: int`, `-> MsgGenerator`) are required as blueapi uses them to detect that this function is intended to be a plan and generate its runtime API. -For example, if a plan is written to drive a specific implementation of Movable, but never calls any methods on the device and only yields bluesky `'set'` Msgs, it can be generalised to instead use the base protocol `Movable`. - -``` - def move_to_each_position(axis: Movable) -> MsgGenerator: - # Originally written for SpecificImplementationMovable - for _ in range(i): - yield from abs_set(axis, location) -``` - -## Allowed Argument Types - -When added to the blueapi context, `PlanGenerator`\ s are formalised into their schema- `a Pydantic BaseModel `__ with the expected argument types and their defaults. - -Therefore, `PlanGenerator`\ s must only take as arguments `those types which are valid Pydantic fields `__ or Device types which implement `BLUESKY_PROTOCOLS` defined in dodal, which are fetched from the context at runtime. - - Allowed argument types for Pydantic BaseModels include the primitives, types that extend `BaseModel` and `dict`\ s, `list`\ s and other `sequence`\ s of supported types. Blueapi will deserialise these types from JSON, so `dict`\ s should use `str` keys. +**Input annotations should be as broad as possible**, the least specific implementation that is sufficient to accomplish the requirements of the plan. For example, if a plan is written to drive a specific motor (`MyMotor`), but only uses the general methods on the [`Movable` protocol](https://blueskyproject.io/bluesky/main/hardware.html#bluesky.protocols.Movable), it should take `Movable` as a parameter annotation rather than `MyMotor`. ## Injecting defaults -Often when writing a plan, it is known which device the plan will mostly or always be run with, but at the time of writing the plan the device object has not been instantiated: dodal defines device factory functions, but these cannot be injected as default arguments to plans. +Some plans are created for specific sets of devices, or will almost always be used with the same devices, it is useful to be able to specify defaults. Dodal defines device factory functions, but these cannot be injected as default arguments to plans. + +Dodal defines an `inject` function which allows defaulting devices, so long as there is a device of that name in the context which conforms to the type annotation. -Dodal defines an `inject` function which bypasses the type checking of the constructed schemas, defering to the blueapi contexting fetching of the device when the plan is imported. This allows defaulting devices, so long as there is a device of that name in the context which conforms to the type annotation. +```python +from dodal.common import inject -``` - def touch_synchrotron(sync: Synchrotron = inject("synchrotron")) -> MsgGenerator: - # There is only one Synchrotron device, so we know which one it will always be. - # If there is no device named "synchrotron" in the blueapi context, it will except. - sync.specific_function() - yield from {} +def touch_synchrotron(sync: Synchrotron = inject("synchrotron")) -> MsgGenerator: + # There is only one Synchrotron device, so we know which one it will always be. + # If there is no device named "synchrotron" in the blueapi context, it will except. + sync.specific_function() + yield from {} ``` -### Metadata +## Injecting Metadata The bluesky event model allows for rich structured metadata to be attached to a scan. To enable this to be used consistently, blueapi encourages a standard form. -> **_NOTE:_** Plans **should** include `metadata` as their final argument, if they do it **must** have the type Optional[Mapping[str, Any]], `and a default of None `__\, with the plan defaulting to an empty dict if passed `None`. If the plan calls to a stub/plan which takes metadata, the plan **must** pass down its metadata, which may be a differently named argument. +Plans ([as opposed to stubs](../explanations/plans.md)) **should** include `metadata` as their final parameter, if they do it **must** have the type `Mapping[str, Any] | None`, [and a default of None](https://stackoverflow.com/questions/26320899/why-is-the-empty-dictionary-a-dangerous-default-value-in-python). If the plan calls to a stub/plan which takes metadata, the plan **must** pass down its metadata, which may be a differently named parameter. -``` - def pass_metadata(x: Movable, metadata: Optional[Mapping[str, Any]] = None) -> MsgGenerator: - yield from bp.count{[x], md=metadata or {}} +```python +def pass_metadata(x: Movable, metadata: Mapping[str, Any] | None = None) -> MsgGenerator: + yield from bp.count{[x], md=metadata or {}} ``` ## Docstrings -Blueapi plan schemas include includes the docstrings of imported Plans. **These should therefore explain as much about the scan as cannot be ascertained from its arguments and name**. This may include units of arguments (e.g. seconds or microseconds), its purpose in the function, the purpose of the plan etc. - -``` - def temp_pressure_snapshot( - detectors: List[Readable], - temperature: Movable = inject("sample_temperature"), - pressure: Movable = inject("sample_pressure"), - target_temperature: float = 273.0, - target_pressure: float = 10**5, - metadata: Optional[Mapping[str, Any]] = None, - ) -> MsgGenerator: - """ - Moves devices for pressure and temperature (defaults fetched from the context) - and captures a single frame from a collection of devices - Args: - detectors (List[Readable]): A list of devices to read while the sample is at STP - temperature (Optional[Movable]): A device controlling temperature of the sample, - defaults to fetching a device name "sample_temperature" from the context - pressure (Optional[Movable]): A device controlling pressure on the sample, - defaults to fetching a device name "sample_pressure" from the context - target_pressure (Optional[float]): target temperature in Kelvin. Default 273 - target_pressure (Optional[float]): target pressure in Pa. Default 10**5 - Returns: - MsgGenerator: Plan - Yields: - Iterator[MsgGenerator]: Bluesky messages - """ - yield from move({temperature: target_temperature, pressure: target_pressure}) - yield from count(detectors, 1, metadata or {}) -``` - -### Decorators - -Dodal defines a decorator for configuring any `ophyd-async` devices- which will be the majority of devices at Diamond- to write to a common location. - -> **_NOTE:_** **This is an absolute requirement to write data onto the Diamond Filesystem**. This decorator must be used every time a new data collection is intended to begin. For an example, see below. - -``` - @attach_metadata - def ophyd_async_snapshot( - detectors: List[Readable], - metadata: Optional[Mapping[str, Any]] = None, -### ) -> MsgGenerator: - Configures a number of devices, which may be Ophyd-Async detectors and require - knowledge of where to write their files, then takes a snapshot with them. - Args: - detectors (List[Readable]): Devices, maybe including Ophyd-Async detectors. - Returns: - MsgGenerator: Plan - Yields: -### Iterator[MsgGenerator]: Bluesky messages - yield from count(detectors, 1, metadata or {}) - - def repeated_snapshot( - detectors: List[Readable], - metadata: Optional[Mapping[str, Any]] = None, -### ) -> MsgGenerator: - Configures a number of devices, which may be Ophyd-Async detectors and require - knowledge of where to write their files, then takes multiple snapshot with them. - Args: - detectors (List[Readable]): Devices, maybe including Ophyd-Async detectors. - Returns: - MsgGenerator: Plan - Yields: -### Iterator[MsgGenerator]: Bluesky messages - @attach_metadata - def inner_function(): - yield from count(detectors, 1, metadata or {}) - - - for _ in range(5): - yield from inner_function() -``` - -### Stubs - -Some functionality in your plans may make sense to factor out to allow re-use. These pieces of functionality may or may not make sense outside of the context of a plan. Some will, such as nudging a motor, but others may not, such as waiting to consume data from the previous position, or opening a run without an equivalent closure. - -To enable blueapi to expose the stubs that it makes sense to, but not the others, blueapi will only expose a subset of `MsgGenerator`\ s under the following conditions: - -| `__init__.py` in directory has `__exports__`: List[str]: only - those named in `__exports__` -| `__init__.py` in directory has `__all__`: List[str] but no - `__exports__`: only those named in `__all__` - -This allows other python packages (such as `plans`) to access every function in `__all__`, while only allowing a subset to be called from blueapi as standalone. - -``` - # Rehomes all of the beamline's devices. May require to be run standalone - from .package import rehome_devices - # Awaits a standard callback from analysis. Should not be run standalone - from .package import await_callback - - # Exported from the module for use by other modules - __all__ = [ - "rehome_devices", - "await_callback", - ] - - # Imported by instances of blueapi and allowed to be run - __exports__ = [ - "rehome_devices", - ] +Blueapi exposes the docstrings of plans to clients, along with the parameter types. It is therefore worthwhile to make these detailed and descriptive. This may include units of arguments (e.g. seconds or microseconds), its purpose in the function, the purpose of the plan etc. + +```python +def temp_pressure_snapshot( + detectors: List[Readable], + temperature: Movable = inject("sample_temperature"), + pressure: Movable = inject("sample_pressure"), + target_temperature: float = 273.0, + target_pressure: float = 10**5, + metadata: Optional[Mapping[str, Any]] = None, +) -> MsgGenerator: + """ + Moves devices for pressure and temperature (defaults fetched from the context) + and captures a single frame from a collection of devices + Args: + detectors (List[Readable]): A list of devices to read while the sample is at STP + temperature (Optional[Movable]): A device controlling temperature of the sample, + defaults to fetching a device name "sample_temperature" from the context + pressure (Optional[Movable]): A device controlling pressure on the sample, + defaults to fetching a device name "sample_pressure" from the context + target_pressure (Optional[float]): target temperature in Kelvin. Default 273 + target_pressure (Optional[float]): target pressure in Pa. Default 10**5 + Returns: + MsgGenerator: Plan + Yields: + Iterator[MsgGenerator]: Bluesky messages + """ + yield from move({temperature: target_temperature, pressure: target_pressure}) + yield from count(detectors, 1, metadata or {}) ``` diff --git a/docs/images/blueapi.png b/docs/images/blueapi.png index 412035b38..41539cc02 100644 Binary files a/docs/images/blueapi.png and b/docs/images/blueapi.png differ diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 59980ffaf..ed6181545 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -2,4 +2,8 @@ Full reference for the CLI: +```{eval-rst} +.. click:: blueapi.cli:main + :prog: blueapi :show-nested: +``` \ No newline at end of file diff --git a/docs/reference/messaging-spec.md b/docs/reference/messaging-spec.md index 363577c80..a59b77106 100644 --- a/docs/reference/messaging-spec.md +++ b/docs/reference/messaging-spec.md @@ -5,3 +5,7 @@ bus, allowing subscribers to keep track of the status of plans, as well as other status changes. This page documents the channels to which clients can subscribe to receive these messages and their structure. +```{eval-rst} +.. literalinclude:: ./asyncapi.yaml + :language: yaml +``` \ No newline at end of file diff --git a/docs/reference/rest-spec.md b/docs/reference/rest-spec.md index 8db73b848..346867023 100644 --- a/docs/reference/rest-spec.md +++ b/docs/reference/rest-spec.md @@ -1,6 +1,8 @@ # REST Specification Blueapi runs a FastAPI server through which the blueapi worker can be -interacted with. Here the [openapi docs page](./openapi.yaml) documents all possible endpoints of this server. - +interacted with. Here the [openapi schema](./openapi.yaml) documents all possible endpoints of this server. +```{eval-rst} +.. openapi:: ./openapi.yaml +``` diff --git a/docs/tutorials/installation.md b/docs/tutorials/installation.md index effcec222..01dd6404d 100644 --- a/docs/tutorials/installation.md +++ b/docs/tutorials/installation.md @@ -2,7 +2,7 @@ ## Check your version of python -You will need python 3.8 or later. You can check your version of python by +You will need python 3.10-3.11. You can check your version of python by typing into a terminal: ``` python3 --version diff --git a/docs/tutorials/quickstart.md b/docs/tutorials/quickstart.md index 1eb1c42b8..5866f45f5 100644 --- a/docs/tutorials/quickstart.md +++ b/docs/tutorials/quickstart.md @@ -1,34 +1,21 @@ -# Quickstart Guide +# Quickstart guide Blueapi acts as a worker that can run bluesky plans against devices for a specific laboratory setup. It can control devices to collect data and export events to tell downstream services about the data it has collected. -## Start ActiveMQ - -The worker requires a running instance of ActiveMQ, the simplest -way to start it is to run it via a container: - -``` - docker run -it --rm --net host rmohr/activemq:5.15.9-alpine -``` - -``` - podman run -it --rm --net host rmohr/activemq:5.15.9-alpine -``` - ## Start Worker To start the worker: ``` - blueapi serve +blueapi serve ``` The worker can also be started using a custom config file: ``` - blueapi --config path/to/file serve +blueapi --config path/to/file serve ``` ## Test that the Worker is Running @@ -36,7 +23,7 @@ The worker can also be started using a custom config file: Blueapi comes with a CLI so that you can query and control the worker from the terminal. ``` - blueapi controller plans +blueapi controller plans ``` The above command should display all plans the worker is capable of running. diff --git a/docs/tutorials/run-bus.md b/docs/tutorials/run-bus.md new file mode 100644 index 000000000..9b9b22f29 --- /dev/null +++ b/docs/tutorials/run-bus.md @@ -0,0 +1,43 @@ +# Run with local message bus + +Blueapi can publish updates to a message bus asynchronously, the CLI can then view these updates display them to the user. + +## Start Message Bus + +The worker can publish updates to a running instance of a message bus such as ActiveMQ, the simplest +way to start it is to run it via a container: + +``` +docker run -it --rm --net host rmohr/activemq:5.15.9-alpine +``` + +``` +podman run -it --rm --net host rmohr/activemq:5.15.9-alpine +``` + +## Config File + +Create a YAML file for configuring blueapi: + +```yaml +# stomp.yaml + +# Edit this if your message bus of choice is running on a different host, +# if it has different credentials, +# or if its STOMP plugin is running on a different port +stomp: + host: localhost + port: 61613 + auth: + username: guest + # This is for local development only, production systems should use good passwords + password: guest +``` + +## Run the Server + +``` +blueapi --config stomp.yaml serve +``` + +It should print a message about being connected to the console, otherwise it will print an error. \ No newline at end of file diff --git a/docs/tutorials/run-plan.md b/docs/tutorials/run-plan.md new file mode 100644 index 000000000..49d56b6e8 --- /dev/null +++ b/docs/tutorials/run-plan.md @@ -0,0 +1,54 @@ +# Run a Plan + +:::{note} +You will need [a running server connected to a message bus](./run-bus.md) to complete this tutorial. +::: + +With a [running worker](./quickstart.md), you can then run a plan. In a new terminal: + +``` +blueapi controller run sleep '{"time": 5}' +``` + +## Example Plans + +Move a Motor + +``` +blueapi -c stomp.yaml controller run move \ +'{ + "moves": {"x": 5} +}' +``` + +Take a Snapshot on a Detector + +``` +blueapi -c stomp.yaml controller run count \ +'{ + "detectors": ["image_det"] +}' +``` + +Run a Scan + +``` +blueapi -c stomp.yaml controller run scan \ +'{ + "detectors": ["image_det"], + "spec": { + "type": "Line", + "axis": "x", + "start": 0, + "stop": 10, + "num": 10 + }, + "axes_to_move": {"x": "x"} +}' +``` + +The names of the devices used (`"image_det"` and `"x"`) can be found via: + +``` +blueapi controller devices +``` diff --git a/docs/tutorials/dev-run.md b/docs/tutorials/run-with-dev.md similarity index 59% rename from docs/tutorials/dev-run.md rename to docs/tutorials/run-with-dev.md index 308253a96..139db54e1 100644 --- a/docs/tutorials/dev-run.md +++ b/docs/tutorials/run-with-dev.md @@ -7,7 +7,7 @@ Assuming you have setup a developer environment, you can run a development versi Ensure you are inside your virtual environment: ``` - source venv/bin/activate +source venv/bin/activate ``` You will need to follow the instructions for setting up ActiveMQ as in [run cli instructions](../how-to/run-cli.md). @@ -21,8 +21,6 @@ debugging capabilities. [debug in vscode](../images/debug-vscode.png) -## Develop devices - -When you select the 'scratch directory' option - where you have devices (dodal) and plans (BLxx-beamline) in a place like `/dls_sw/BLXX/software/blueapi/scratch`, then the list of devices available will refresh without interfacing with the K8S cluster. Just run the command `blueapi env -r` or `blueapi env --reload`. - -With this setup you get a developer loop: "write devices - write plans - test them with blueapi". +:::{seealso} +[Scratch Area](../how-to/edit-live.md) for in-the-loop development of plans and devices +::: \ No newline at end of file