Skip to main content
Version: latest

Transform Dependencies

In this example, we will show how import rust crates otherwise not included. In this example, we will import the regex library and verify that the source topic reads proper https website links.

Prerequisites

This guide uses local Fluvio cluster. If you need to install it, please follow the instructions at here.

Transformation

Below is the syntax for importing multiple crates.

transforms:
- operator: (transfrom operator)
dependencies:
- name: (library a)
version: (version number)
- name: (library b)
version: (version number)
(... more crates ...)
run: |
(...)

In our example, the following is our transfrom operator.

transforms:
- operator: filter
dependencies:
- name: regex
version: "1"
run: |
fn verify(input: String) -> Result<bool> {
let re = regex::Regex::new(r"^(https?:\/\/)?([a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(\/[^\s]*)?$")
.unwrap();
Ok(re.is_match(&input))
}

Running the Example

Copy and paste following config and save it as dataflow.yaml.

# dataflow.yaml
apiVersion: 0.5.0
meta:
name: regex-example
version: 0.1.0
namespace: examples

config:
converter: raw

topics:
sentences:
schema:
value:
type: string
validlink:
schema:
value:
type: string

services:
verify-service:
sources:
- type: topic
id: sentences

transforms:
- operator: filter
dependencies:
- name: regex
version: "1"
run: |
fn verify(input: String) -> Result<bool> {
let re = regex::Regex::new(r"^(https?:\/\/)?([a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(\/[^\s]*)?$")
.unwrap();
Ok(re.is_match(&input))
}

sinks:
- type: topic
id: validlink

To run example:

$ sdf run

Produce sentences to in sentence topic:

$ echo "hello world" | fluvio produce sentences
$ echo "https://www.fluvio.io" | fluvio produce sentences

Consume topic validlink to retrieve the result in another terminal:

$ fluvio consume validlink -Bd
https://www.fluvio.io

Cleanup

Exit sdf terminal and clean-up. The --force flag removes the topics:

$ sdf clean --force

Conclusion

We just covered another basic operator in SDF, the Flat-Map Operator. The Flat-Map is a powerful operator. As a matter of fact, the Flat-Map operator can be used inplace of other operators.