Skip to main content
Version: latest

Connector Configuration File

This is a template YAML connector file that needs to be populated to be functional. In the next section, we will explore the different sections of the connector configuration file and how to properly configure each one.

apiVersion: 0.1.0

meta:
 name:
 version:
 type:
 topic:

 # optional (inbound connectors)
 producer:
   # optional
   linger:
   # optional
   batch-size:
   # optional
   compression:

 # optional (outbound connectors)
 consumer:
   # optional as default, but required if offset strategy is provided
   id:
   # optional
   offset:
    strategy:
    start:
    flush:
   # optional
   partition:
   # optional
   max_bytes:

 # optional
 secrets:
   - name: secret_1

# optional
transforms:
 - uses: smartmodule_name
   with:
     param_name: param_value

Connector apiVersion configuration​

The apiVersion is the version of the connector API that the connector uses to parse the configuration file. The current accepted version is 0.1.0.

Connector meta configuration​

The meta section contains the metadata for the connector:

  • The name is the name of the connector. e.g. my-connector.
  • The type is the type of the connector. e.g. http-source, http-sink, mqtt-source.
    • See the connectors section for the full list of connectors supported.
  • The version is the version of the connector. e.g. 0.1.0.
  • The topic is the topic that the connector will connect to. e.g. my-topic. The topic will be created automatically if it does not exist.
  • The secrets(optional) is a list of secrets that the connector will use. This accepts a list of objects with the key name.
    • See the [secrets] section for more information.
  • The producer(optional) is the producer configuration for the connector. Currently, this is only used for source/inbound connectors. The current supported configurations are linger, compression and batch_size. All configurations are optional. See examples to a list of valid values for each configuration.
  • The consumer(optional) is the consumer configuration for the connector. Currently, this is only used for sink/outbound connectors. The current supported configurations are id, partition, max_bytes and offset. All configurations are optional. See examples to a list of valid values for each configuration.

At minimum connector configuration would look like:

apiVersion: 0.1.0
meta:
 name: my-connector
 type: my-connector-type
 version: x.y.z
 topic: my-topic

All other fields are optional.

info

The x.y.z version should be updated with the version of the latest connector in the Hub.

Connector transforms configuration​

Connectors support transforms. Records can be modified before they are sent to the topic. The transforms section is a list of transform objects. Each transform object has an uses and a with section.

  • uses is the reference to the SmartModule used in the transform.
  • with is the configuration for the transform

Consumer with Offset​

Consumers can be configured with different start offsets to determine where to begin consuming records from a topic.

The following options are available for the start offset:

  • absolute: x: Starts consuming from the absolute index x of the topic.
  • beginning: Starts consuming from the very beginning of the topic.
  • from-beginning: x: Starts consuming from index x from the beginning of the topic.
  • end: Starts consuming from the end of the topic.
  • from-end: x: Starts consuming from index x from the end of the topic.

Offset Management​

Offset management is a Fluvio feature that stores the offset of the last consumed record. This is especially useful when a connector is stopped and restarted, as it allows the connector to resume consumption from the last saved offset rather than starting over.

The behavior of offset management depends on the strategy field. When strategy is set to auto or manual, offset management is enabled, and the start rule is based on the saved offset rather than the entire topic.

The strategy field supports the following values:

  • auto: Automatically commits the offset after each record is consumed.
  • manual: Requires the user to commit the offset manually.
  • none: Disables offset management entirely.

Example Configuration​

A consumer connector with offset management might look like this:

apiVersion: 0.1.0
meta:
  name: my-connector
  type: my-connector-type
  version: x.y.z
  topic: my-topic
consumer:
  id: my-consumer
  offset:
    strategy: auto
    start: beginning 
    flush: 10s

References​

The following references will provide additional information for: