How to set target_key to a variable value

[nishaprabhakar]

Hello @jshlbrd!

I'm trying to reformat a log in this format:

{
    "key": "A", 
    "value": {
         "stringValue": "B"
     }
}

to:

{
   "A": "B"
}

I'm struggling to understand how to have the target_key input be a variable, or alternatively how to grab the value of "key" and store it as a variable.
Here's an example of what I'm trying to do:

sub.transform.object.copy(settings={object: {source_key: 'value.stringValue', target_key:[input["key"]]}})

Any ideas on how to do this? Thanks :)

[jshlbrd]

Hey @nishaprabhakar! 👋 Feel free to tag @brexhq/substation next time if you want to get the team's attention.

I don't think this is possible to do, and at the moment I'm not sure how to best support that functionality (maybe another team member will have other ideas). Usually I would recommend formatting JSON objects into key-value pairs where the key is consistent -- for example, without knowing your use case, I would convert the original event to this:

{
  "type": "A",
  "value": "B"
}

The issue you're likely to run into with a format like {"A":"B"} are dynamic values for the key ("A"). You might end up with JSON objects that look like this:

{
  "foo": "fooer"
}
{
  "bar": "barre"
}
{
  "baz": "bazar"
}

These objects will be difficult to access in most systems that rely on a JSON schema. But, if that doesn't work for you, then the only solution I can think of that fits the design of the project is a new meta transform that would access the value that will become the JSON key and run an embedded transform. It might look something like this...

sub.tf.meta.object.invert({
  object: {target_key: 'key'},
  transform: sub.obj.copy({object: {source_key: 'value.stringValue'}),
}),

... where the key placed into the JSON object is the value of key and the value is value.stringValue. Something like that would turn your input into this (and you could delete the fields you don't need):

{
    "key": "A", 
    "value": {
         "stringValue": "B"
     },
    "A": "B"
}

[nishaprabhakar]

Thank you! To provide more context, the larger log looks like:

 {
      "timeUnixNano": "1720026285502545978",
      "observedTimeUnixNano": "1720026285532804533",
      "body": {
        "stringValue": "RAW STRING"
      },
      "attributes": [
        {
          "key": "time",
          "value": {
            "stringValue": "2024-07-03T17:04:45.502545978Z"
          }
        },
        {
          "key": "log.iostream",
          "value": {
            "stringValue": "stdout"
          }
        },
        {
          "key": "log.file.path",
          "value": {
            "stringValue": "FILE PATH"
          }
        },
        {
          "key": "logtag",
          "value": {
            "stringValue": "F"
          }
        },
        {
          "key": "log",
          "value": {
            "STRING LOG"
          }
        }
      ],
      "traceId": "",
      "spanId": ""
}

and I'd like to convert it to:

{
      "timeUnixNano": "1720026285502545978",
      "observedTimeUnixNano": "1720026285532804533",
      "log.file.path": "FILE PATH",
      "time": "2024-07-03T17:04:45.502545978Z",
      "log.iostream": "stdout",
      "logtag": "F",
      "log": "STRING LOG",
      "traceId": "",
      "spanId": ""
}

This schema is consistent across the logs. So just to make sure I understand, I will need to create a new meta transform to make this work?

Thanks!

[jshlbrd]

Ah I see. I think there's a way to do this without adding a new transform by building a Jsonnet function. I'll get back to you on this tomorrow.

[jshlbrd]

@nishaprabhakar Since your JSON schema is consistent, this will work for what you want to do:

local sub = import '../../../../../build/config/substation.libsonnet';

// invert_kv copies a nested value and moves it to the top 
// level as a key.
local invert_kv(keys) = [
  // The key is interpolated into the nested path.
  local key_string_val = std.format('attributes.#(key==%s).value.stringValue', key);

  // The source is the location of the nested value.
  // The target is the key input with dots escaped.
  sub.tf.object.copy({ object: { source_key: key_string_val, target_key: std.strReplace(key, '.', '\\.') } }),

  // This for-loop must be at the bottom of the function.
  for key in keys
];

local keys = [
  'time',
  'log.iostream',
  'log.file.path',
  'logtag',
  'log',
];

{
  transforms: invert_kv(keys) + [
    // Delete attributes since it is no longer needed.
    sub.tf.obj.delete({object: {source_key: 'attributes'}}),
    // Delete body since this was requested by nishaprabhakar.
    sub.tf.obj.delete({object: {source_key: 'body'}}),
    // Send to stdout.
    sub.tf.send.stdout(),
  ]
}

When the input is this (I made it single line to ease testing) ...

{"timeUnixNano":"1720026285502545978","observedTimeUnixNano":"1720026285532804533","body":{"stringValue":"RAW STRING"},"attributes":[{"key":"time","value":{"stringValue":"2024-07-03T17:04:45.502545978Z"}},{"key":"log.iostream","value":{"stringValue":"stdout"}},{"key":"log.file.path","value":{"stringValue":"FILE PATH"}},{"key":"logtag","value":{"stringValue":"F"}},{"key":"log","value":{"stringValue":"STRING LOG"}}],"traceId":"","spanId":""}

... the output is this (pretty printed):

{
  "timeUnixNano": "1720026285502545978",
  "observedTimeUnixNano": "1720026285532804533",
  "traceId": "",
  "spanId": "",
  "time": "2024-07-03T17:04:45.502545978Z",
  "log.iostream": "stdout",
  "log.file.path": "FILE PATH",
  "logtag": "F",
  "log": "STRING LOG"
}

So all you need to do is maintain the list of JSON keys and the function will move values to the top-level correctly. If the nested key location is dynamic, then this works too (values are only copied if the key exists in the object):

local invert_kv(keys) = [
    // The key is interpolated into the nested path.
    local key_string_val = std.format('attributes.#(key==%s).value.stringValue', key);

    // The source is the location of the nested value.
    // The target is the key input with dots escaped.
    sub.tf.object.copy({ object: { source_key: key_string_val, target_key: std.strReplace(key, '.', '\\.') } }),

    // This for-loop must be at the bottom of the function.
    for key in keys
  ] + [
    local key_multi_val = std.format('attributes.#(key==%s).value.multiValue', key);
    sub.tf.object.copy({ object: { source_key: key_multi_val, target_key: std.strReplace(key, '.', '\\.') } }),

    for key in keys
];

Hope that helps!

[nishaprabhakar]

Thank you, this was super helpful.