Use Webhooks

This tutorial assumes that fluvio is installed, and logged-in to InfinyOn Cloud. Follow the Quick Start to get set up.

Webhooks are special connectors with an associated external url. Users can send data to their topics via a HTTP POST request.

Webhook Configuration

The webhook configuration file has the following structure:

meta:
  name: my-webhook        # required
  topic: my-webhook       # required
  logLevel: debug         # default: info, options: [trace, debug, info, warn, error]
  secrets:
    - name: my-key
  producer:
    linger: 100ms
    batch-size: 1mb
    compression: snappy    # default: none, options: [none, gzip, snappy, lz4, zstd]
webhook:
  outputParts: full        # default: body, options: [full, body]
  outputType: json         # default: text, options: [text, json]
  outputUri: full          # default: none, options: [full, none, path, query]
transforms:
  - uses: infinyon/jolt@x.y.z
    with:
      key: ${{ secrets.my_key }}

The minimum configuration has needs the top 2 required fields: name and topic. The name is the name of the webhook, and the topic is the topic that the webhook will publish to.

Create a simple Webhook

Let's create an example configuration file

# example-webhook.yaml
meta:
  name: my-webhook
  topic: my-webhook-topic

Add a webhook to InfinyOn Cloud

$ fluvio cloud webhook create --config example-webhook.yaml

Your output should look similar to this. We'll cover sending data to this url.

Webhook "my-webhook" created with url: https://infinyon.cloud/webhooks/v1/[random string]

If you need this url again, you can run this command to list your webhooks, and their urls.

$ fluvio cloud webhook list

Example output

 NAME        TOPIC             URL
 my-webhook  my-webhook-topic  https://infinyon.cloud/webhooks/v1/[random string]

Send data to webhook

We'll be sending json data {"key": "value"} to our webhook using curl. Replace the url so [random string] matches your unique url. Keep this command close because we'll refer to this example curl command later.

$ curl -v -X POST -d 'Hello World!' https://infinyon.cloud/webhooks/v1/[uri-key]

In another terminal, star a consumer that reads form the beginning:

$ fluvio consume my-webhook-topic -B

We should see the following output:

Hello World!

Create a JSON Webhook and embed HTTP parts

Next we'll send json records, but before we do that we'll modify outputParts, outputType, and outputUri in the example-webhook.yaml configuration file:

# example-webhook.yaml
meta:
  name: my-webhook
  topic: my-webhook-topic

webhook:
  outputParts: full
  outputType: json
  outputUri: full

Run this command to update your webhook.

$ fluvio cloud webhook update --config example-webhook.yaml
Webhook "my-webhook" updated

Let's restart the consumer to a JSON formatter:

$ fluvio consume my-webhook-topic -O json

Run another curl with a json payload:

$ curl -v -X POST -H "Content-Type: application/json" -d '{"key": "value"}' https://infinyon.cloud/webhooks/v1/[uri-key]

The consumer should now show the following output:

{
  "body": {
    "key": "value"
  },
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip, br",
    "content-length": "16",
    "content-type": "application/json",
    "host": "infinyon.cloud",
    "user-agent": "curl/x.y.z",
    "x-forwarded-for": "..."
  },
  "path": "",
  "query": ""
}

Webhook Path and Query info

A curl with path and query info to https://infinyon.cloud/webhooks/v1/[uri-key]/path_a/path_b?param_a=1&param_b=2 can be sent as follows: (note the single quotes around the URI)

$ curl -v -X POST -H "Content-Type: application/json" -d '{"key": "value"}' \
  'https://infinyon.cloud/webhooks/v1/[uri-key]/path_a/path_b?param_a=1&param_b=2'

The consumer should show return path and query information as follows:

{
  "body": {
    "key": "value"
  },
  "headers": {
    "accept": "*/*",
    "accept-encoding": "gzip, br",
    "content-length": "16",
    "content-type": "application/json",
    "host": "infinyon.cloud",
    "user-agent": "curl/x.y.z",
    "x-forwarded-for": "..."
  },
  "path": "path_a/path_b",
  "query": "param_a=1&param_b=2"
}

note

Json output is the recommended way to receive uri path and query information so that Smartmodule transformation can be directly applied. See Webhook Config for more details.

Conclusion

You now know how to create and configure the output of Webhooks. Check out the Webhook Config reference to see how to configure other transformations.