Federate Filter creation and integration

[trevorhardy]

@kdheepak: I'm trying to write a filter federate for the User Guide Python example suite. as best I can tell (with help from @nightlark from the ns-3 example), I need to use two APIs:

1. Register the filter - ns-3 example - Is helicsFederateRegisterFilter the equivalent API?
2. Set a callback - ns-3 example - What is the pyhelics API for this? The C++ API has two parameters: the endpoint and the function for the callback. The only thing close to this I can find in pyhelics is helicsSetCustomCallback but the signature doesn't match.

Any guidance?

[trevorhardy]

So @nightlark pointed out that the callback is specific to the ns-3 implementation and not a HELICS thing.

So maybe this is a question for @phlptp: once I register my federate as a filter, is there anything else to do? helicsFederateRegisterFilter returns a filter object so maybe I just helicsAddSourceFilterToEndpoint with the registered filters name? And then any time a message comes in on an endpoint with this filter federate it will get granted a time?

[kdheepak]

I think you want helicsFilterSetCustomCallback. See an example implementation in the tests:

https://github.com/GMLC-TDC/pyhelics/blob/3a799617db4b455cce60a5e2ecd2d47a915f1651/tests/test_filters.py#L764-L767

https://github.com/GMLC-TDC/pyhelics/blob/3a799617db4b455cce60a5e2ecd2d47a915f1651/tests/test_filters.py#L799-L802

[phlptp]

in the C API when you create a filter you specify a type with it. Unless you specify a custom type the callback function won't do much and might error. The other types might have some configuration but otherwise don't require a callback.

[trevorhardy]

So if I'm implementing a filter federate (where I'm going to do arbitrary stuff to the messages), I'm assuming that's a custom filter, right?

[trevorhardy]

So, things I need clarity on:

1. A filter federate is a message federate that registers a federate filter on another federates endpoint, right? If so, I'm assuming that's a custom filter.
2. Does a filter federate need to have an endpoint for HELICS to send the messages to be filtered? I think so.
3. Can the filter federate just act like a normal message federate and process these messages as it sees fit or is the custom filter callback the preferred or required method?
4. helicsFederateRegisterFilter only specifies which federate the filter is created on; I assume there needs to be an assignment of that filter to an endpoint but I'm not seeing an appropriate API. There is a helicsBrokerAddSourceFilterToEndpoint; is that the best/only way to do this?

[phlptp]

1). A filter federate is a message federate that registers a federate filter on another federates endpoint, right?
It doesn't have to be a message Federate, a value Federate can do it too.

2). Does a filter federate need to have an endpoint for HELICS to send the messages to be filtered? I think so.
No

3). Can the filter federate just act like a normal message federate and process these messages as it sees fit or is the custom filter callback the preferred or required method?

these are two very distinct modes of operation.
If you just want to receive and process the message and forward them on then that is a rerouting filter and some endpoints in your federate, just operate on the message objects and send on to the original destination when done. It is pretty simple, you don't have to mess with callbacks, and you are pretty flexible in what you can do. The disadvantage is it takes a time grant on the federate that does the operation. So the final destination will not receive the message until the next time grant by the filtering federate.

The other mode is a custom filter with a callback. This operates inline meaning there is no simulation time advance required by the filtering process and it all happens behind the scenes so to speak. The callback will execute in a different thread if that matters though.

[trevorhardy]

Thanks; this helps a lot.

So when I register the filter on the federate, I still have to associate it with particular endpoints on said federate with something like helicsBrokerAddSourceFilterToEndpoint right?

[phlptp]

yes when you register it it is just a filter that doesn't have any connections. You have to make the source or destination connections after it is registered.

[phlptp]

filters can be registered on a core as well if you don't want to use a federate

[trevorhardy]

OK, I've written an initial version and have begun testing/debugging. Right now the federation gets stuck at its initial time grant (all federates waiting for something). As I've been trying to debug it I decided to check to see if the rerouting filters the filter federate was installing on the other federates' endpoints.

Here is a sample of the filter federate config:

"name": "EV_filter_config",
  "loglevel": 1,
  "coreType": "zmq",
  "period": 0.001,
  "uninterruptible": false,
  "terminate_on_error": true,
  "endpoints": [
    {
      "name": "filter/main",
      "global": true
    }
  ],
  "filters": [
    {
      "name": "fromEV1.soc",
      "sourcetargets": "EV_federate/EV1.soc",
      "operation": "reroute",
      "properties": [
        {
          "name": "newdestination",
          "value": "filter/main"
        }
      ],
      "info": "EV_federate"
    },
    {
      "name": "fromEV2.soc",
      "sourcetargets": "EV_federate/EV2.soc",
      ...

To confirm these filters are getting installed, I put a ten second pause on EV_federate prior to entering execution mode and then print out filter_count = h.helicsFederateGetFilterCount(fed) after entering execution mode. This ensures that the filter federate (which does not have the pause) will finish configuration, including installing the rerouting filters, and enter execution before EV_federate.

When watching these log files in real-time as the federation runs, the pause is very apparent (giving me confidence it is working as expected) but the filter count is zero. It seems like the installation of the filters by the filter federate onto the EV federate isn't happening. I don't know if this explains why the federation is hanging, though.

Any thoughts?

[phlptp]

I don't think the filter count on the federate you are connecting to will change. those are counts of interfaces.
you might try the data_flow_graph query and see if it shows up in there.

[trevorhardy]

I'm comparing what I've done to what I can see in example_filters.json and I can't see any reason why this shouldn't be working.

{
      "name": "filtername2", //filters can have names (optional)
      "sourcetargets": ["filterFed/ept2"], //this is a key field specifying the source targets can be an array
      //"dest":["dest targets"],  // field specifying destination targets
      "operation": "reroute", //currently valid operations are "delay","clone","cloning","randomdelay","randomdrop","reroute","custom"
      //additional properties for filters are specified in a property array or object if there is just a single one
      "properties": {
        "name": "newdestination", //A reroute filter takes a new destination
        "value": "ept1" //the value here is the endpoint that should be the new destination
      }

[trevorhardy]

Previous attempts were made under HELICS v2.6.0. I tried running under v2.4.2 and am having the same problem.

[trevorhardy]

Based on the comments from @phlptp, let's consider my filter-count efforts a red herring.

Getting back to the base issue: right now the federation hangs when the filter federate is included and doesn't when its excluded.

Right now I have four federates making the following time requests and none are being granted a time:
Battery: 180
Charger: 120
Filter: HELICS_MAX_TIME
Controller: HELICS_MAX_TIME

(The filter is setup to operate on the messages sent between the charger and the controller.)

This doesn't seems like it should be happening even if the reroute filter that were put in place aren't working properly. The co-simulation should be able to advance in time but the results will be wrong as the charging control coordination between the filter and controller would be broken. Somebody should be granted a time, though.

Any thoughts or suggestions on further debugging? Right now, I'm stuck 😖

[trevorhardy]

I removed the rerouting filter definitions from the filter federate config and re-ran: everything runs fine. The three other federates run as normal and the filter federate (which is requesting HELICS_MAX_TIME) is granted that time when the rest of the federation is done executing. Without any rerouted messages to process, it has nothing to do.

[trevorhardy]

As an experiment, I changed the filter operation field to an invalid type:

Original:

{
  ...
  "operation": "reroute",
  ...
}

Experiment:

{
  ...
  "operation": "reroute3",
  ...
}

Same behavior and no error messages; this makes me think there could be something subtly wrong in my definition and I'm not seeing the error message from it.

[trevorhardy]

I added a similar error to the general portion of the config JSON and it did throw an error (as expected):

Original:

{
  ...
  "period": 1,
  ...
}

Experiment:

{
  ...
  "period": "bobo",
  ...
}

So maybe the code responsible for parsing the filters isn't included or working?

[trevorhardy]

Just checked this on the v3.0.0-alpha.2 release and the co-sim runs to the point when the first message should be sent (and the filter intercepts it) and all federates are stuck waiting for a time grant. The Controller and the Filter are requesting infinite time and the Battery and Controller are requesting 180s. The message should be sent on this timestep from Charger to Controller.

[trevorhardy]

@eberleim has been working on this for a week or two and hopefully will have figured out what's going on soon.

[trevorhardy]

There are still on-going challenges (even in v3) with rerouting filters and filter federates. This issue has been captures in #1995 so I'm going to close the discussion here.