Overhaul documentation #674

callumforrester · 2024-10-17T12:04:42Z

Fixes #447
Fixes #361
Fixes #671

…xplanations

codecov · 2024-10-17T12:07:56Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.56%. Comparing base (cd91f0f) to head (b691f06).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #674      +/-   ##
==========================================
- Coverage   92.62%   92.56%   -0.07%     
==========================================
  Files          35       35              
  Lines        1654     1654              
==========================================
- Hits         1532     1531       -1     
- Misses        122      123       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DiamondJoseph

How much is this just re-formatting the docs to pass current linting/build and how much is it intending to update the docs? There's several things that we're actively moving away from that aren't being changed, but also more changes than just updating the formatting.

DiamondJoseph · 2024-10-17T12:10:34Z

docs/explanations/architecture.md

-Handles communications and the API layer. This object holds a reference to the worker 
-can interrogate it/give it instructions in response to messages it receives from the message
-bus. It can also forward the various events generated by the worker to topics on the bus.
+Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and optional connection to the downstream message broker.


Suggested change

Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and optional connection to the downstream message broker.

Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and external connections.

Do we want to be general or specific?

I made this a bit more general because I didn't think a list of major classes and their purposes was that useful and was prone to being out of date. It may be more useful when blueapi is more stable.

DiamondJoseph · 2024-10-17T12:12:32Z

docs/explanations/extension-code.md

@@ -0,0 +1,17 @@
+# Home of Plans and Devices
+
+Plans and devices can be in any pip-installable package, such as:


I'd move the device/dodal part first, so that it's clear that devices belong in dodal, rather than someone adding devices to their repo then realising it's meant to be in dodal.

DiamondJoseph · 2024-10-17T12:14:30Z

docs/explanations/extension-code.md

+
+## Dodal
+
+[Dodal](https://github.com/DiamondLightSource/dodal) is a repository for DLS device configuration, providing factory functions for devices used at DLS. If you work at DLS, new devices should be added there.


new devices = new device classes?

This is mostly true, get devices into dodal, then plan to move to ophyd-async. Let's treat mostly true as true for the purposes of introductory docs.

Should be expanded to explain that beamlines should also be added there, with a link to dodal how to make a new beamline.

This is a problem in general, some of these docs belong in dodal with links from blueapi. I think we can and should do that migration gradually though. I'm going to advocate for removing this sentence and making an issue in dodal for adding a page/pages explaining how to:

Make a new device

Add a device to a beamline

Make a new beamline

The reason I asked is there is already docs in dodal for at least part of that
https://diamondlightsource.github.io/dodal/main/how-to/create-beamline.html
https://diamondlightsource.github.io/dodal/main/reference/device-standards.html (?)

Ah cool! I'll reference those

DiamondJoseph · 2024-10-17T12:14:50Z

docs/explanations/extension-code.md

+* A Github repository
+* A local directory via the [scratch area](../how-to/edit-live.md).
+
+The easiest place to put the code is a repository created with the [`python-copier-template`](https://diamondlightsource.github.io/python-copier-template/main/index.html). Which can then become any of the above.


Could: an example, like i22?

DiamondJoseph · 2024-10-17T12:15:15Z

docs/explanations/lifecycle.md

+
+
+def count(
+    detectors: List[Readable] = [inject("det")],  # default valid for Blueapi only


Is this change going in prior to deprecating inject or after?

Update to >=3.10 typing (List->list, Union[x, y] -> x | y, Optional[x] -> x | None)

I am going to make a separate issue for moving all of the code snippets to Python files and checking that they still work and lint properly at docs build time, similar to ophyd-async.

DiamondJoseph · 2024-10-17T12:17:45Z

docs/explanations/type_validators.md

+```python 
+def my_plan(a: int, b: str = "b") -> Plan
+    ...

-    # Internally becomes something like
+# Internally becomes something like

-    class MyPlanModel(BaseModel):
-        a: int
-        b: str = "b"
+class MyPlanModel(BaseModel):
+    a: int
+    b: str = "b"


Is this repeating what is explained in plans.md?

Probably does need a bit of consolidation, my read is that plans.md explains which parts of blueapi plans are special and why, and type_validators.md explains the very technical implementation of plan parameter parsing.

DiamondJoseph · 2024-10-17T12:18:33Z

docs/explanations/type_validators.md

+    b: List[Readable],
+    c: Dict[str, Readable],
+    d: List[List[Readable]],
+    e: List[Dict[str, Set[Readable]]]) -> Plan:
+    ...


Python 3.10ify

#677 unless you strongly want it here

DiamondJoseph · 2024-10-17T12:35:06Z

docs/how-to/configure-app.md

-however the option exists to override these by defining a yaml file which
-can be passed to the `blueapi` command.
+Blueapi's default configuration can be overridden 
+by defining a yaml file which can be passed to the `blueapi` command.


memo: this is about to get changed by changes to overriding config values from env vars etc, make sure to update it then

DiamondJoseph · 2024-10-17T12:36:56Z

docs/how-to/write-devices.md

+Devices are made using the [dodal](https://github.com/DiamondLightSource/dodal) style available through factory functions like this:
+
+```python 
+from my_facility_devices import MyTypeOfDetector
+
+def my_detector(name: str) -> MyTypeOfDetector:
+    return MyTypeOfDetector(name, {"other_config": "foo"})
+```
+
+The return type annotation `-> MyTypeOfDetector` is required as blueapi uses it to determine that this function creates a device. Meaning you can have a Python file where only some functions create devices and they will be automatically picked up. Similarly, these functions can be organized per-preference into files. 
+
+The device is created via a function rather than a global to preserve side-effect-free imports. Each device must have its own factory function.


Should: defer to dodal's docs on writing device classes and beamlines, and ensure they are up to date?

Agreed, see other discussions, I think we should migrate some of this to dodal as a separate PR

DiamondJoseph · 2024-10-17T12:38:45Z

docs/how-to/write-plans.md

+Blueapi exposes the docstrings of plans to clients, along with the parameter types. It is therefore worthwhile to make these detailed and descriptive. This may include units of arguments (e.g. seconds or microseconds), its purpose in the function, the purpose of the plan etc.
+
+```python
+def temp_pressure_snapshot(
+    detectors: List[Readable],
+    temperature: Movable = inject("sample_temperature"),
+    pressure: Movable = inject("sample_pressure"),
+    target_temperature: float = 273.0,
+    target_pressure: float = 10**5,
+    metadata: Optional[Mapping[str, Any]] = None,
+) -> MsgGenerator:
+    """
+    Moves devices for pressure and temperature (defaults fetched from the context)
+    and captures a single frame from a collection of devices
+    Args:
+        detectors (List[Readable]): A list of devices to read while the sample is at STP
+        temperature (Optional[Movable]): A device controlling temperature of the sample,
+            defaults to fetching a device name "sample_temperature" from the context
+        pressure (Optional[Movable]): A device controlling pressure on the sample,
+            defaults to fetching a device name "sample_pressure" from the context
+        target_pressure (Optional[float]): target temperature in Kelvin. Default 273
+        target_pressure (Optional[float]): target pressure in Pa. Default 10**5
+    Returns:
+        MsgGenerator: Plan
+    Yields:
+        Iterator[MsgGenerator]: Bluesky messages
+    """
+    yield from move({temperature: target_temperature, pressure: target_pressure})
+    yield from count(detectors, 1, metadata or {})


Should: replacing this form of docstring with using Annotated/Field arguments so that the values propagate into json schema?

Oh, I did not realise they didn't propagate, that's quite bad. I was hoping to make the plans as vanilla-bluesky as we could, so vanilla python function docstrings are definitely desirable...

Lemme get a couple of example plans into blueapi and compare the json schemas

@attach_data_session_metadata_decorator() @validate_call(config={"arbitrary_types_allowed": True}) def linkam_plan_1( trajectory: LinkamTrajectory, linkam: Linkam3 = LINKAM, shutter_time: float = 0.04, ) -> MsgGenerator: """ Follow a trajectory in temperature, collecting a number of frames either at equally spaced positions or while continually scanning. e.g. for 2 segments, the first stepped and the 2nd flown:\n trajectory start v v final segment stop\n \\ /\n stepped segment__\\__ /\n \\ / flown segment\n 1st segment stop \\__ /\n exposures: xx xx xx 1/N seconds Args: trajectory: Trajectory for the scan to follow. linkam: Temperature controller shutter_time: Time allowed (s) for opening shutter before triggering detectors. Returns: MsgGenerator: Plan Yields: Iterator[MsgGenerator]: Bluesky messages """ ... @attach_data_session_metadata_decorator() @validate_call(config={"arbitrary_types_allowed": True}) def linkam_plan_2( trajectory: Annotated[ LinkamTrajectory, Field(description="Trajectory for the scan to follow.") ], linkam: Annotated[Linkam3, Field(LINKAM, description="Temperature controller.")], shutter_time: Annotated[ float, Field( 0.04, description="Time allowed for opening shutter before triggering detectors.", json_schema_extra={"units": "s"}, ), ], ) -> MsgGenerator: """ Follow a trajectory in temperature, collecting a number of frames either at equally spaced positions or while continually scanning. e.g. for 2 segments, the first stepped and the 2nd flown:\n trajectory start v v final segment stop\n \\ /\n stepped segment__\\__ /\n \\ / flown segment\n 1st segment stop \\__ /\n exposures: xx xx xx 1/N seconds """ ... @attach_data_session_metadata_decorator() @validate_call(config={"arbitrary_types_allowed": True}) def linkam_plan_3( trajectory: LinkamTrajectory = Field( description="Trajectory for the scan to follow." ), linkam: Linkam3 = Field(LINKAM, description="Temperature controller."), shutter_time: float = Field( 0.04, description="Time allowed for opening shutter before triggering detectors.", json_schema_extra={"units": "s"}, ), ) -> MsgGenerator: """ Follow a trajectory in temperature, collecting a number of frames either at equally spaced positions or while continually scanning. e.g. for 2 segments, the first stepped and the 2nd flown:\n trajectory start v v final segment stop\n \\ /\n stepped segment__\\__ /\n \\ / flown segment\n 1st segment stop \\__ /\n exposures: xx xx xx 1/N seconds """ ...

linkam_plan_1.json
linkam_plan_2.json
linkam_plan_3.json

I've just gone through all that just to prove they're basically identical and the Pydantic model doesn't actually get the information about the Fields 🙃

callumforrester · 2024-10-18T10:07:11Z

@DiamondJoseph regarding the fact that some of this is going out of date "soon": I have been bitten too many times by writing code and docs in anticipation of changes that are due "in a week or so" and end up taking 6 months. I'm strongly against putting anything in here that is not true in main right now.

Ideally docs should be updated in the same PR as the change they reference, second-best option is after the PR is merged, worst-best is before the PR is merged.

callumforrester added 12 commits October 15, 2024 10:29

Update supported versions

48d2244

Complete basic set of tutorials

806fef7

Split out writing plans and devices section, move lots of detail to e…

975c6c3

…xplanations

Add info about writing devices

242552f

Make tutorial actually work

ba2d5fd

Tidy up how tos

22d2d9b

Tidy up explanations

8948887

Fix references

9b76249

Consistent title capitalisation

d804cf5

Remove lie

d212ab4

Remove broken image from readme

652e357

Fix indent

265c327

callumforrester requested a review from DiamondJoseph October 17, 2024 12:05

DiamondJoseph requested changes Oct 17, 2024

View reviewed changes

callumforrester mentioned this pull request Oct 18, 2024

Lint and execute code snippets in docs #677

Open

Action suggestions for home of plans and devices page

b691f06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul documentation #674

Overhaul documentation #674

callumforrester commented Oct 17, 2024

codecov bot commented Oct 17, 2024 •

edited

Loading

DiamondJoseph left a comment

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 17, 2024

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 18, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 17, 2024

DiamondJoseph Oct 17, 2024

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024 •

edited

Loading

DiamondJoseph Oct 17, 2024

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 17, 2024

callumforrester Oct 18, 2024

DiamondJoseph Oct 18, 2024

DiamondJoseph Oct 18, 2024

callumforrester commented Oct 18, 2024

	Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and optional connection to the downstream message broker.
	Above are the main components of blueapi. The main process houses the REST API and manages the subprocess, which wraps the `RunEngine`, devices and external connections.

		@@ -0,0 +1,17 @@
		# Home of Plans and Devices

		Plans and devices can be in any pip-installable package, such as:


		## Dodal

		[Dodal](https://github.com/DiamondLightSource/dodal) is a repository for DLS device configuration, providing factory functions for devices used at DLS. If you work at DLS, new devices should be added there.



		def count(
		detectors: List[Readable] = [inject("det")], # default valid for Blueapi only

Overhaul documentation #674

Are you sure you want to change the base?

Overhaul documentation #674

Conversation

callumforrester commented Oct 17, 2024

codecov bot commented Oct 17, 2024 • edited Loading

Codecov Report

DiamondJoseph left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

callumforrester Oct 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

callumforrester commented Oct 18, 2024

codecov bot commented Oct 17, 2024 •

edited

Loading

callumforrester Oct 18, 2024 •

edited

Loading