Overhaul documentation

README.md

@@ -22,18 +22,16 @@ inside a server and exposes endpoints to send commands/receive data. Useful for
 installation at labs where multiple people may control equipment, possibly from
 remote locations.
 
-![concept][]
-
 The main premise of blueapi is to minimize the boilerplate required to get plans
 and devices up and running by generating an API for your lab out of
 type-annotated plans. For example, take the following plan:
 
 ```python
-    import bluesky.plans as bp
-    from blueapi.core import MsgGenerator
+import bluesky.plans as bp
+from blueapi.core import MsgGenerator
 
-    def my_plan(foo: str, bar: int) -> MsgGenerator:
-        yield from bp.scan(...)
+def my_plan(foo: str, bar: int) -> MsgGenerator:
+    yield from bp.scan(...)
 ```
 
 Blueapi's job is to detect this plan and automatically add it to the lab's API

docs/explanations/architecture.md

@@ -1,6 +1,5 @@
 # Architecture
 
-
 Blueapi performs a number of tasks:
 
 * Managing the Bluesky [RunEngine](https://nsls-ii.github.io/bluesky/run_engine_api.html), giving it instructions and handling its errors. Traditionally this job has been done by a human with an [IPython](https://ipython.org/) terminal, so it requires automating.
@@ -9,26 +8,7 @@ Blueapi performs a number of tasks:
 
 These responsibilities are kept separate in the codebase to ensure a clean, maintainable architecture.
 
-## Key Components
-
-
-![blueapi architecture main components](../images/blueapi-architecture.png)
-
-
-### The `BlueskyContext` Object
-
-Holds registries of plans and devices as well as a number of helper methods for 
-registering en-masse from a normal Python module. 
-
-### The Worker Object
-
-Wraps the Bluesky `RunEngine` and accepts requests to run plans. The requests include the name 
-of the plan and a dictionary of parameters to pass. The worker validates the parameters against
-the known expectations of the plan, passes it to the `RunEngine` and handles any errors.
-
+![blueapi main components](../images/blueapi.png)
 
-### The Service Object
 
-Handles communications and the API layer. This object holds a reference to the worker 
-can interrogate it/give it instructions in response to messages it receives from the message
-bus. It can also forward the various events generated by the worker to topics on the bus.
+Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and optional connection to the downstream message broker.

docs/explanations/events.md

@@ -9,7 +9,7 @@ Since the `RunEngine` is traditionally used by a human in front of an IPython te
 sometimes assumes intuitive behaviour. The worker replaces the human and so must fill in the
 gaps.
 
-The base engine programmatically emits data events conforming to the `bluesky event model`_. These
+The base engine programmatically emits data events conforming to the [bluesky event model](https://blueskyproject.io/event-model). These
 are meant to be handled by other subscribing code (e.g. databroker) and are decoupled from concerns such as whether
 a plan has started, finished, paused, errored etc. See the example below:
 

docs/explanations/extension-code.md

@@ -0,0 +1,17 @@
+# Home of Plans and Devices
+
+Plans and devices can be in any pip-installable package, such as:
+
+* A package on pypi
+* A Github repository
+* A local directory via the [scratch area](../how-to/edit-live.md).
+
+The easiest place to put the code is a repository created with the [`python-copier-template`](https://diamondlightsource.github.io/python-copier-template/main/index.html). Which can then become any of the above.
+
+:::{seealso}
+Guide to setting up a new Python project with an environment and a standard set of tools: [`Create a new repo from the template`](https://diamondlightsource.github.io/python-copier-template/main/tutorials/create-new.html)
+:::
+
+## Dodal
+
+[Dodal](https://github.com/DiamondLightSource/dodal) is a repository for DLS device configuration, providing factory functions for devices used at DLS. If you work at DLS, new devices should be added there.

docs/explanations/lifecycle.md

@@ -2,43 +2,44 @@
 
 The following demonstrates exactly what the code does with a plan through its lifecycle 
 of being written, loaded and run. Take the following plan.
-```
-    from typing import Any, List, Mapping, Optional, Union
-
-    import bluesky.plans as bp
-    from blueapi.core import MsgGenerator
-    from dls_bluesky_core.core import inject
-    from bluesky.protocols import Readable
-
-
-    def count(
-        detectors: List[Readable] = [inject("det")],  # default valid for Blueapi only
-        num: int = 1,
-        delay: Optional[Union[float, List[float]]] = None,
-        metadata: Optional[Mapping[str, Any]] = None,
-    ) -> MsgGenerator:
-        """
-        Take `n` readings from a collection of detectors
-
-        Args:
-            detectors (List[Readable]): Readable devices to read: when being run in Blueapi
-                                        defaults to fetching a device named "det" from its
-                                        context, else will require to be overridden.
-            num (int, optional): Number of readings to take. Defaults to 1.
-            delay (Optional[Union[float, List[float]]], optional): Delay between readings.
-                                                                Defaults to None.
-            metadata (Optional[Mapping[str, Any]], optional): Key-value metadata to include
-                                                            in exported data.
+
+```python
+from typing import Any, List, Mapping, Optional, Union
+
+import bluesky.plans as bp
+from blueapi.core import MsgGenerator
+from dls_bluesky_core.core import inject
+from bluesky.protocols import Readable
+
+
+def count(
+    detectors: List[Readable] = [inject("det")],  # default valid for Blueapi only
+    num: int = 1,
+    delay: Optional[Union[float, List[float]]] = None,
+    metadata: Optional[Mapping[str, Any]] = None,
+) -> MsgGenerator:
+    """
+    Take `n` readings from a collection of detectors
+
+    Args:
+        detectors (List[Readable]): Readable devices to read: when being run in Blueapi
+                                    defaults to fetching a device named "det" from its
+                                    context, else will require to be overridden.
+        num (int, optional): Number of readings to take. Defaults to 1.
+        delay (Optional[Union[float, List[float]]], optional): Delay between readings.
                                                             Defaults to None.
-
-        Returns:
-            MsgGenerator: _description_
-
-        Yields:
-            Iterator[MsgGenerator]: _description_
-        """
-
-        yield from bp.count(detectors, num, delay=delay, md=metadata)
+        metadata (Optional[Mapping[str, Any]], optional): Key-value metadata to include
+                                                        in exported data.
+                                                        Defaults to None.
+
+    Returns:
+        MsgGenerator: _description_
+
+    Yields:
+        Iterator[MsgGenerator]: _description_
+    """
+
+    yield from bp.count(detectors, num, delay=delay, md=metadata)
 ```
 
 
@@ -53,50 +54,46 @@ will build a [pydantic](https://docs.pydantic.dev/) model of the parameters to v
 like this:
 
 
-```
-    from pydantic import BaseModel
-
-    class CountParameters(BaseModel):
-        detectors: List[Readable] = ["det"]
-        num: int = 1
-        delay: Optional[Union[float, List[float]]] = None
-        metadata: Optional[Mapping[str, Any]] = None
-
-        class Config:
-            arbitrary_types_allowed = True
-            validate_all = True
-
-    This is for illustrative purposes only, this code is not actually generated, but an object
-    resembling this class is constructed in memory.
-    The default arguments will be validated by the context to inject the "det" device when the
-    plan is run. The existence of the "det" default device is not checked until this time.
-```
+```python
+from pydantic import BaseModel
+
+class CountParameters(BaseModel):
+    detectors: List[Readable] = [inject("det")]
+    num: int = 1
+    delay: Optional[Union[float, List[float]]] = None
+    metadata: Optional[Mapping[str, Any]] = None
 
-The model is also stored in the context.
+    class Config:
+        arbitrary_types_allowed = True
+        validate_all = True
+```
 
+This is for illustrative purposes only, this code is not actually generated, but an object
+resembling this class is constructed in memory.
+The default arguments will be validated by the context to inject the "det" device when the
+plan is run. The existence of the "det" default device is not checked until this time. The model is also stored in the context.
 
 ## Startup
 
 On startup, the context is passed to the worker, which is passed to the service.
 The worker also holds a reference to the `RunEngine` that can run the plan.
 
-
 ## Request
 
 A user can send a request to run the plan to the service, which includes values for the parameters.
 It takes the form of JSON and may look something like this:
-```
-    {
-        "name": "count",
-        "params": {
-            "detectors": [
+```json
+{
+    "name": "count",
+    "params": {
+        "detectors": [
             "andor",
             "pilatus"
-            ],
-            "num": 3,
-            "delay": 0.1
-        }
+        ],
+        "num": 3,
+        "delay": 0.1
     }
+}
 ```
 
 The `Service` receives the request and passes it to the worker, which holds it in an internal queue
@@ -114,13 +111,13 @@ See also [type validators](./type_validators.md)
 
 ## Execution
 
-The validated parameter values are then passed to the plan function, which is passed to the RunEngine.
+The validated parameter values are then passed to the plan function, which is passed to the `RunEngine`.
 The plan is executed. While it is running, the `Worker` will publish
 
 * Changes to the state of the `RunEngine`
 * Changes to any device statuses running within a plan (e.g. when a motor changes position)
 * Event model documents emitted by the `RunEngine`
-* When a plan starts, finishes or fails.
+* When the plan starts, finishes or fails.
 
 If an error occurs during any of the stages from "Request" onwards it is sent back to the user
 over the message bus.

docs/explanations/plans.md

@@ -0,0 +1,44 @@
+# Plans
+
+While the bluesky project uses `plan` in a general sense to refer to any `Iterable` of `Msg`'s which may be run by the `RunEngine`, blueapi distinguishes between a `plan` and a `stub`. This distinction is made to allow for a subset of `stub`'s to be exposed and run, as `stub`'s may not make sense to run alone.
+
+Generally, a `plan` includes at least one `open_run` and `close_run` and is a complete description of an experiment. If it does not, it is a `stub`. This distinction is made in the bluesky core library between the `plan`'s and `plan_stub`'s modules.
+
+
+## Allowed Argument Types
+
+When added to the blueapi context, `PlanGenerator`'s are formalised into their schema - [a Pydantic BaseModel](https://docs.pydantic.dev/1.10/usage/models) with the expected argument types and their defaults. 
+
+Therefore, `PlanGenerator`'s must only take as arguments [those types which are valid Pydantic fields](https://docs.pydantic.dev/dev/concepts/types) or Device types which implement `BLUESKY_PROTOCOLS` defined in dodal, which are fetched from the context at runtime.
+
+Allowed argument types for Pydantic BaseModels include the primitives, types that extend `BaseModel` and `dict`'s, `list`'s  and other `sequence`'s of supported types. Blueapi will deserialise these types from JSON, so `dict`'s should use `str` keys.
+
+
+## Stubs
+
+Some functionality in your plans may make sense to factor out to allow re-use. These pieces of functionality may or may not make sense outside of the context of a plan. Some will, such as nudging a motor, but others may not, such as waiting to consume data from the previous position, or opening a run without an equivalent closure.
+
+To enable blueapi to expose the stubs that it makes sense to, but not the others, blueapi will only expose a subset of `MsgGenerator`'s under the following conditions:
+
+- `__init__.py` in directory has `__exports__`: List[str]: only those named in `__exports__`
+- `__init__.py` in directory has `__all__`: List[str] but no `__exports__`: only those named in `__all__`
+
+This allows other python packages (such as `plans`) to access every function in `__all__`, while only allowing a subset to be called from blueapi as standalone.
+
+```python
+# Rehomes all of the beamline's devices. May require to be run standalone
+from .package import rehome_devices
+# Awaits a standard callback from analysis. Should not be run standalone
+from .package import await_callback
+
+# Exported from the module for use by other modules
+__all__ = [
+    "rehome_devices",
+    "await_callback",
+]
+
+# Imported by instances of blueapi and allowed to be run
+__exports__ = [
+    "rehome_devices",
+]
+```