Jolt

Jolt provides JSON to JSON transformation

This is a map-type SmartModule that transforms JSON records leveraging Fluvio Jolt library, which has its own DSL (Domain Specific Language) to remove the need for coding simple transformations.

The transformations in Jolt are a set of operations that are sequentially performed over incoming records.

-> Jolt only works on JSON text records.

There are three main types of operations:

• Shift - move the field from one location to another
• Default - specify a default value for the field, if not present
• Remove - delete the field from the object

There can be a mix and match of transformations applied at the same time. Let's see below:

Specification example

[
    {
      "operation": "remove", // remove field $.id from incoming JSON object
      "spec": {
        "id": ""
      }
    },
    {
      "operation": "shift", // move everything inside $.data
      "spec": {
        "*": "data.&0",
      }
    },
    {
      "operation": "default", // if $.data.source does not exist, add it with value "http-connector"
      "spec": {
        "data": {
            "source": "http-connector"
        }
      }
    }
]

The Jolt SmartModule applies the operations in sequence: remove followed by shift followed by default.

Usage example

To demonstrate how Jolt SmartModule works we will use SMDK tool.

info

All smartmodule versions are marked with x.y.z. To find the latest version, run:

• fluvio hub smartmodule list

First, we need to download it to our cluster:

$ fluvio hub smartmodule download infinyon/jolt@x.y.z

Second, we create a file transform.yaml with transformation specification defined above:

# transform.yaml
transforms:
  - uses: infinyon/jolt@x.y.z
      spec:
        - operation: remove
          spec:
            id: ""
        - operation: shift
          spec:
            "*": "data.&0"
        - operation: default
          spec:
            data:
              source: "http-connector"

Let's use smdk test to see it in action:

$ smdk test --text '{}' --transforms-file ./transform.yaml
1 records outputted
{"data":{"source":"http-connector"}}

$ smdk test --text '{"id":1, "name": "John Smith", "account": "1111" }' --transforms-file ./transform.yaml
{"data":{"account":"1111","name":"John Smith","source":"http-connector"}}

$ smdk test --text '{"id":1, "name": "John Smith", "account": "1111", "type": "custom" }' --transforms-file ./transform.yaml
{"data":{"account":"1111","name":"John Smith","source":"http-connector","type":"custom"}}

$ smdk test --text '{"id":1, "name": "John Smith", "source":"mqtt-connector" }' --transforms-file ./transform.yaml
{"data":{"name":"John Smith","source":"mqtt-connector"}}

For additional examples checkout the tutorials:

• [Build HTTP to SQL Pipeline]
• [Build MQTT to SQL Pipeline]