Strategies for dynamic selection of transitions/…

[hartytp]

I'm trying to get my head around the idiomatic ndscan way of doing things.

One thing that has come up is how to handle Fragments that take build-time parameters whose value is determined by user input (e.g. a generic diagnostic experiment that can run on on one out of a few different transitions).

In regular artiq python the following feels idiomatic and works as one would expect:

import artiq.experiment as aq


class MyEnv(aq.HasEnvironment):
    def build(self, arg):
        self.arg= arg

    def do(self):
        print(self.arg)


class MyExp(aq.EnvExperiment):
    def build(self):
        arg= self.get_argument(
            'argument',
            aq.EnumerationValue(['foo', 'bar'], default='foo')
            )
        self.my_env = MyEnv(self, arg)

    def run(self):
        self.my_env.do()

The obvious ndscan equivalent is something like

import artiq.experiment as aq
import ndscan.experiment as nd


class MyFrag(nd.Fragment):
    def build_fragment(self, arg):
        self.arg = arg
        self.setattr_param("param", nd.FloatParam, "parameter", 0)

    def do(self):
        print(f"argument {self.arg}")
        print(f"parameter {self.param.get()}")


class MyExp(nd.ExpFragment):
    def build_fragment(self):
        arg = self.get_argument(
            'argument',
            aq.EnumerationValue(['foo', 'bar'], default='foo')
            )
        self.setattr_fragment("my_frag", MyFrag, arg)

    def run_once(self):
        self.my_frag.do()

MyScan = nd.make_fragment_scan_exp(MyExp)

If we now want to override param things break. The fqn is MyFrag_none.param which is incorrect (it should be MyFrag_bar.param) and results in a runtime error Parameter schema not found....

Is there an obvious way around this? Is this an unreasonable thing to want to do? How should I think about this use-case?

[hartytp]

NB since kwargs are not included in the ndscan fqn, the following does achieve the immediately desired effect. I'm pretty sure this is a terrible idea and breaks things in some way that isn't currently immediately obvious to me, but I haven't got my head around the design enough to see how!

import artiq.experiment as aq
import ndscan.experiment as nd


class MyFrag(nd.Fragment):
    def build_fragment(self, *, arg):
        self.arg = arg
        self.setattr_param("param", nd.FloatParam, "parameter", 0)

    def do(self):
        print(f"argument {self.arg}")
        print(f"parameter {self.param.get()}")


class MyExp(nd.ExpFragment):
    def build_fragment(self):
        arg = self.get_argument(
            'argument',
            aq.EnumerationValue(['foo', 'bar'], default='foo')
            )

        self.setattr_fragment("my_frag", MyFrag, arg=arg)

    def run_once(self):
        self.my_frag.do()

MyScan = nd.make_fragment_scan_exp(MyExp)

Edit: one thing that is affected by the above is ndscan_show since that relies on extracting the schema from the magic PARAMS_ARG_KEY argument. See

ndscan/ndscan/results/arguments.py

Line 10 in a1bb524

def extract_param_schema(arguments: Dict[str, Any]) -> Dict[str, Any]:

ndscan/ndscan/experiment/entry_point.py

Line 171 in 9d46a98

self._params = self.get_argument(PARAMS_ARG_KEY, PYONValue(default=desc))

What's less clear to me is whether there is a simple way of achieving the desired affect without this (presumably) undesired side effect. e.g. could store the scan args in a dataset (which is what FragmentScanExperiment states that we do).

[hartytp]

@dnadlinger do you have any thoughts on how this should be handled? The kind of use-cases I find are e.g. doing some thermometry code that should work on a few different transitions. This feels like a reasonable (and somewhat common) use case which horribly breaks things with ndscan at present since the parameter tree (dataset paths) and some build_fragment parameters are not known the first time build is called.

[dnadlinger]

@hartytp One of the core ndscan design elements is the fact that the fragment and parameter tree structure is known in advance and can be queried from the outside. This allows things like the automatic generation of the parameter override/scan UI and scan loops, the long-lived-kernel thing I've been meaning to implement for ages, etc., and also provides a simple mental model: An ndscan ExpFragment is a function in a (loose) mathematical sense from its params to its results; the parameters and result channels are part of the function "type", and do not change dynamically.

Trying to find ways to get away with breaking this assumption strikes me as the wrong approach. One could design a system where parameters can be dynamically defined depending on the value of some other parameters (which would then really be more of a free-form key-value store than a parameter list). However, it is not clear how a sensible UI for that would look like, especially in the ARTIQ model with the strict lifetime separation between experiment submission and run time. There is also a big potential for user mistakes and confusion – one of my design considerations was to avoid an annoying problem with regular ARTIQ paramters, which I'm sure you've also run into at some point, where you changed the name/… of a parameter in the source code but still have an old version open in the dashboard, leading to user inputs being silently ignored.

My approach would simply be to step back and avoid getting into a round peg/square hole situation in the first place. It is not clear to me where a non-trivial amount of "dynamism" would even be needed in the examples you mention. Note that the "works as expected" regular ARTIQ example in your initial post doesn't have anything of that sort either – MyEnv doesn't call anything like self.get_argument(f"{arg} frequency", …) either, because that runs into many of the issues (e.g. how would you supply a value in the user interface?).

Now, how to do that really depends on your specific situation and code structure. Here is a loose collection of ideas, in terms of the thermometry example:

• Instantiate the fragments for all the transition types, and just select between them at runtime. If you have a large number of possible choices, or the setup/cleanup is expensive (in terms of time, hardware profile count, …), you could add a Fragment method (freeze/suspend/mute/ignore/…?) to effectively ignore parts of the fragment tree at runtime in device_setup_subfragments, etc.
• Instead of baking the transition name into the fragment tree, have all the fragment parameters in relative units (e.g. detuning) and read the base values (e.g. centre frequency) at runtime according to the selected transition.
• If you prefer absolute units for simple experiments that just operate on one transition, you could combine the last few points and make a wrapper that exposes parameters in relative units and forwards everything to whichever transition child fragment was selected at runtime.
• Make a separate experiment per transition, i.e. select the transition type in the experiment explorer. You could put them into their own folder (package).

On a completely separate note, the mangling of arguments into the FQN, which allows easy programmatic instantiation of "fragment templates" with different schemata, is a bit of an ad-hoc design choice that could use some cleanup (e.g. fixing the kwargs discrepancy you mentioned). However, this is orthogonal to the issue discussed here; no matter how this is implemented in detail, the fragment tree remains static.

[dnadlinger]

I've added a new "documentation" label, as a use case study/"best practices"/… discussion of this would certainly be useful for others too.

[hartytp]

Thanks @dnadlinger. What does a "good solution" look like here? IMHO

1. For all parameters with an associated dataset, the values in the GUI should default to the dataset value (e.g. not to 0). Otherwise, to get a scan to run with its default parameters the user has to figure out which dataset is linked to the parameter and copy-paste the value. This is time consuming and error prone.
2. The user should never have to do manual dataset arithmetic to update a dataset after a scan. For example, suppose I want to scan a transition frequency, I want to be able to update the associated dataset directly from the GUI. A bad solution here would be scanning an offset from the nominal value, reading the optimal offset from the GUI, manually finding the correct dataset and doing arithmetic to calculate the new value and updating that manually
3. We should be able to browse the entire parameter tree in the GUI. This requires the parameter tree to be known the first time build is called
4. The parameter tree should only have parameters relevant to the experiment that we're running. e.g. we don't want a large number of parameters associated with fragments which are not used in the experiment. Doing so just confuses things and makes it too easy to scan the wrong thing. Not a huge deal, just a potentially annoying source of confusion.
5. We ideally want to avoid compiling a lot of code that is never used (the compiler is slow enough already)
6. We don't want to have a large number of experiments which all do the same thing but for different argument values (e.g. to scan across different transitions). This becomes particularly painful if one wants to support permutations of arguments.
7. We don't want to make invasive modifications to ARTIQ

AFAICT (but correct me if I'm wrong) there is no solution that satisfies all of these. So we're in the business of compromise. The more I think about this I'm coming around to the view that (6) is the best one to give up, which is one of the solutions you suggest.

Make a separate experiment per transition, i.e. select the transition type in the experiment explorer. You could put them into their own folder (package).

To keep the number of separate experiments tractable I think one has to work in the mindset of "create the permutations one actually needs" rather than "support everything that is physically possible" (which is what runtime fragments allow). This solution still feels a bit clunky, but perhaps it's the best we can do.

In terms of the other proposed solutions

Instantiate the fragments for all the transition types, and just select between them at runtime. If you have a large number of possible choices, or the setup/cleanup is expensive (in terms of time, hardware profile count, …), you could add a Fragment method (freeze/suspend/mute/ignore/…?) to effectively ignore parts of the fragment tree at runtime in device_setup_subfragments, etc.

This fails to satisfy (1) (2) (4) and (5). IMHO (1) and (2) are probably the worst of these to fail on; eliminating manual dataset arithmetic is one of the killer features of ndscan and I'm loathe to give it up.

Instead of baking the transition name into the fragment tree, have all the fragment parameters in relative units (e.g. detuning) and read the base values (e.g. centre frequency) at runtime according to the selected transition.

As above except we now satisfy (1).

If you prefer absolute units for simple experiments that just operate on one transition, you could combine the last few points and make a wrapper that exposes parameters in relative units and forwards everything to whichever transition child fragment was selected at runtime.

This improves on the previous one because it automates the forwarding from parameters to the relevant fragments (i.e. handling the overrides etc) but it still fails on the important points above.

[hartytp]

Note that the "works as expected" regular ARTIQ example in your initial post doesn't have anything of that sort either – MyEnv doesn't call anything like self.get_argument(f"{arg} frequency", …) either, because that runs into many of the issues (e.g. how would you supply a value in the user interface?).

On a completely separate note, the mangling of arguments into the FQN, which allows easy programmatic instantiation of "fragment templates" with different schemata, is a bit of an ad-hoc design choice that could use some cleanup (e.g. fixing the kwargs discrepancy you mentioned). However, this is orthogonal to the issue discussed here; no matter how this is implemented in detail, the fragment tree remains static.

One point to note here is that there is a difference between runtime arguments which affect the parameter tree (including associated datasets) and those which don't. As a contrived example, suppose changing the selected transition changes the parameter tree (because different transitions have different associated pi-time datasets) but changing between a red/blue sideband does not (because we use the same datasets and just change how frequencies are computed from them at run time). In the latter case the dynamism should be easily supportable.

Going back to my example at the top of this thread, if the argument in question is the sideband we still hit problems because ndscan encodes the sideband into the fqn -- despite the fact that changes to its value do not alter the parameter tree in any way. Perhaps the answer here is to view arguments of build_fragment as always being intended to have consequences for the parameter tree and to handle arguments which do not alter the parameter tree differently (e.g. expose a setter method).