Dummy GNMI Target

[agardnerIT]

For anyone building exporters, we need a quick way to get GNMI metrics. We should not need to rely on having a physical device available.

My Usecase

I need a working GNMI target so that the gateway can ingest metrics. Once done, I will build an exporter to export GNMI metrics into my solution.

[0xmc]

Hi Adam,

If you have a Linux system, I've had a lot of luck with containerlab recently for setting up virtual infrastructure for this type of development work. And if you're looking for an image to run, you can register for account with Arista and get access to their containerized EOS (cEOS) lab images.

If the containerlab/cEOS route works for you, give it a try and let me know if you have any trouble setting it up.

If not, I'll give you a few more suggestions, but I haven't found a good, easy way of running a local, "fake" gnmi target.

mc

[agardnerIT]

OK, I've got an Ubuntu 20 VM with containerlab installed. I've registered with Arista. Trying to run through their quickstart but I can't find a license.key file anywhere.

As this is related to the creation of my exporter, perhaps we can work on this together?

Fundamentally, my exporter is going to POST metrics to an endpoint like so:

curl -L -X POST 'https://example.com/api/v2/metrics/ingest' \
-H 'Authorization: Api-Token abcdefjhij1234567890' \
-H 'Content-Type: text/plain' \
--data-raw 'cpu.temperature 55'

The raw data should have one metric per line (limited to 1000 metrics per POST request). The line is in the following format where dimensions are Key=Value pairs to split the metric. Dimensions are optional.

metric.key,dimensions payload

Using dimensions would look like this:

gnmi.interface3.myMetric,portNumber=32 12.34
gnmi.interface3.myMetric,portNumber=65 43.12

[agardnerIT]

I think I'm closer now. I'm using https://github.com/openconfig/gnmi to generate a fake server.

Have a look at ~/gnmi/testing/fake/gnmi/cmd/gen_fake_config/config.pb.txt for the pb definition.

Here's the complete working script, I'm creating all files with cat <<EOF>> directives to make it clearer what I'm building

https://gist.github.com/agardnerIT/676e08b2d8ef6926fd25d115c186e42d

This works to check things are working:

/home/$USER/gnmi/cmd/gnmi_cli/gnmi_cli -a localhost:8080 -q '*' -logtostderr -insecure -qt s

When I run that, I get:

{
  "interfaces": 4
}
{
  "interfaces": 4
}
{
  "interfaces": 4
}

Perhaps we could arrange a quick Zoom session. Conceptually, I know what I need to do but I think I'm going to struggle to get this 100% right without your assistance.

[0xmc]

Hi Adam,

I'm not sure I'll be able to find some time to talk in the near future, especially since, according to your profile, you are in Brisbane. But I might be able to give a few pointers. First, good on you for finding fake_server in the gnmi repo. You may also want to take a peek at https://github.com/google/gnxi/tree/master/gnmi_target though, to be honest, I've not had much luck with it.

As for the Arista container, EOS does not require a license file. With an EOS download in hand, this will get you an EOS image (update the image name to whatever image you're using)

docker import --change 'VOLUME /mnt/flash/' cEOS-lab-4.25.1F.tar ceos:4.25.1F

And this is what your topology file will look like

name: foo

mgmt:
  network: custom_mgmt_foo
  ipv4_subnet: 172.16.0.0/24

topology:
  nodes:
    ceos:
      kind: ceos
      image: ceos:4.25.1F
      mgmt_ipv4: 172.16.0.2

One last thing. In two weeks I'll be helping with the NANOG 82 Hackathon. We'll providing virtual infrastructure, do you won't have to muck about with containerlab too much. And you might be able to find some people willing to help you develop your exporter.

[agardnerIT]

Thanks for responding. I've registered for that Hackathon. I'm happy to fit timings around you. Living here, I'm very used to calls at all hours of the day and night as the rest of the world works!

For others (or future me), the Arista downloads are here: https://www.arista.com/en/support/software-download

wget https://LINK_TO_TAR_XZ_FILE -O /tmp/cEOS-lab-4.26.0.1F.tar.xz
docker import --change 'VOLUME /mnt/flash/' /tmp/cEOS-lab-4.26.0.1F.tar.xz ceos:4.26.0.1F

So now I have the containerlab running with the topology above:

sudo containerlab deploy --topo ~/clab-quickstart/mytopo.yml

$ sudo containerlab deploy --topo ~/clab-quickstart/mytopo.yml
INFO[0000] Parsing & checking topology file: mytopo.yml
INFO[0000] Creating lab directory: /home/.../gnmi-gateway/clab-foo
INFO[0000] Creating docker network: Name='custom_mgmt_foo', IPv4Subnet='172.16.0.0/24', IPv6Subnet='', MTU='1500'
INFO[0000] Creating container: ceos
INFO[0000] Restarting 'ceos' node
INFO[0000] Writing /etc/hosts file
+---+---------------+--------------+----------------+------+-------+---------+---------------+--------------+
| # |     Name      | Container ID |     Image      | Kind | Group |  State  | IPv4 Address  | IPv6 Address |
+---+---------------+--------------+----------------+------+-------+---------+---------------+--------------+
| 1 | clab-foo-ceos | 0ce8dfe2e7e6 | ceos:4.26.0.1F | ceos |       | running | 172.16.0.2/24 | NA           |
+---+---------------+--------------+----------------+------+-------+---------+---------------+--------------+

Now I need to figure out my targets.json. What I don't know:

• Contents of the subscription should be
• IP and port I need to specify (is that the GNMI server: localhost:9339 or point at the target 172.16.0.2. If so, which port?

My targets.json:

{
  "request": {
    "default": {
      "subscribe": {
        "prefix": {
        },
        "subscription": [{
            "path": {
              "elem": [{
                  "name": "interfaces"
                }]
            }
          }]
      }
    }
  },
  "target": {
    "myrouter.example.com": {
      "addresses": [
        "172.16.0.2:9339"
      ],
      "credentials": {
        "username": "admin"
      },
      "request": "default",
      "meta": {
        "NoTLSVerify": "yes"
      }
    }
  }
}

GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info ./gnmi-gateway -EnableGNMIServer -ServerTLSCert=server.crt -ServerTLSKey=server.key -TargetLoaders=json -TargetJSONFile=targets.json -Exporters=debug

[agardnerIT]

If it helps:

$ docker exec -it clab-foo-ceos Cli
ceos>en
ceos#show version
 cEOSLab
Hardware version:
Serial number:
Hardware MAC address: 001c.730d.c827
System MAC address: 001c.730d.c827

Software image version: 4.26.0.1F-21994874.42601F (engineering build)
Architecture: i686
Internal build version: 4.26.0.1F-21994874.42601F
Internal build ID: e41b7ab2-f5ed-45cb-ba9c-f320cb81332f

cEOS tools version: 1.1
Kernel version: 5.4.0-1045-aws

Uptime: 0 weeks, 0 days, 0 hours and 13 minutes
Total memory: 3969504 kB
Free memory: 2825124 kB

ceos#show interfaces status
Port       Name   Status       Vlan     Duplex Speed  Type         Flags Encapsulation
Ma0               connected    routed   a-full a-1G   10/100/1000

[colinmcintosh]

Hey Adam, just going through this. That gist you wrote was super helpful for getting setup the same as you. You're definitely on the right track with a fake gNMI server or a virtual router to generate test data. I ran your example and I initially didn't see data coming through but upon closer inspection it looks like the fake gNMI server is generating data but the fields that are being generated are deprecated fields that gnmi-gateway does not process correctly. Here's the log output from my setup which is similair to yours, note the line I marked that just contains the value 4.000000:

[cmcintosh@Apogee .../github.com/openconfig/gnmi-gateway:release] :) make build && ./gnmi-gateway -EnableGNMIServer -ServerTLSCert=server.crt -ServerTLSKey=server.key -TargetLoaders=json -TargetJSONFile=targets2.json -Exporters=dynatrace
rm -f gnmi-gateway
rm -f cover.out
go build -o gnmi-gateway -ldflags "-X github.com/openconfig/gnmi-gateway/gateway.Version="v0.9.0-7-g62091ad-62091ad" -X github.com/openconfig/gnmi-gateway/gateway.Buildtime=2021-06-01T16:21:03Z" .
./gnmi-gateway -version
gnmi-gateway version v0.9.0-7-g62091ad-62091ad (Built 2021-06-01T16:21:03Z)
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Starting GNMI Gateway."}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Clustering is NOT enabled. No locking or cluster coordination will happen."}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Starting connection manager."}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Starting gNMI server on 0.0.0.0:9339."}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Starting dynatrace exporter..."}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Connection manager received a target control message: 1 inserts 0 removes"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Initializing target localhost ([localhost:8888]) map[NoTLSVerify:yes]."}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Target localhost: Connecting"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Target localhost: Subscribing"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Target localhost: Connected"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Target localhost: Synced"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"[dynatrace.go] Got: meta_connected, 1.000000"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"[dynatrace.go] Got: , 4.000000"}      <-- SEE THIS LINE HERE --
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"[dynatrace.go] Got: meta_sync, 1.000000"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Value is: 0.000000"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"[dynatrace.go] Skipping value because its not a number: 0.000000"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Value is: 0.000000"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"[dynatrace.go] Skipping value because its not a number: 0.000000"}
{"level":"info","time":"2021-06-01T09:21:07-07:00","message":"Target localhost: Disconnected"}
E0601 09:21:07.947796    1281 reconnect.go:114] client.Subscribe (target "localhost") failed: <nil>; reconnecting in 1.104660288s

This marked line is the data being generated by the the fake server but the deprecated path field is not being displayed. It's also only displaying a single update because gnmi-gateway (and the underlying gNMI cache) deduplicates unchanged values.

I'm not too familiar with the deprecated fields or the fake server so I can chat with some other OpenConfig folks about what the best way to proceed is: if we should see that the fake server is updated, if we should honor deprecated fields in gnmi-gateway, or something else. Another thing I'll note is that the fake server is very barebones and doesn't generate practical traffic; I think @0xmc had a good suggestion for using containerlab or another virtual router to generate gNMI and that will probably give you a much more realistic (and reliable) test scenario.

[0xmc]

Hi @agardnerIT ,

How goes it? If you're still stuck, let's find some time to chat next week.

[0xmc]

Hi @agardnerIT. Checking back in to see how things are going. Let us know if you'd still like to find some time to chat.

[agardnerIT]

Hi Colin. Sorry for the radio silence. Since we've last spoken things have changed somewhat (for the better) in Dynatrace.

It's now possible to define your own entity types alongside hosts, processes, process groups, services and applications. This means we can define entity types like switch, router, load balancer or anything else we need to model.

Once we have that entity type, we create instances of it: switch 1, switch 2 etc.

Once an individual instance of an entity exists, we can attach metrics to it.

With that in mind, step 1 would seem to be modelling entity types from the data feed.

• Is there a standard field for "device type" that we can leverage?
• Is there a hook we can subscribe to which notifies when a new "type" of device becomes known?

One thing I'd like to investigate is linking this into Keptn because Keptn can act as that workflow engine middleware layer, so gnmi-gateway fires a CloudEvent into Keptn (eg. sh.keptn.event.dev.newdevicetypefound.triggered) Then I can write a microservice which listens for that event and does whatever I need it to do (eg. POST into Dynatrace).

The above architecture would be extensible for other tooling because if someone else (not Dynatrace) wanted to react on that newdevicetypefound.triggered event, they could do so with no changes to the gnmi-gateway code.