Transform Dependencies

In this example, we will show how import rust crates otherwise not included. In this example, we will import the regex library and verify that the source topic reads proper https website links.

Prerequisites

This guide uses local Fluvio cluster. If you need to install it, please follow the instructions at here.

Transformation

Below is the syntax for importing multiple crates.

transforms:
  - operator: (transfrom operator)
    dependencies:
      - name: (library a)
        version: (version number)
      - name: (library b)
        version: (version number)
      (... more crates ...)
    run: |
      (...)

In our example, the following is our transfrom operator.

transforms:
  - operator: filter
    dependencies:
      - name: regex
        version: "1"
    run: |
      fn verify(input: String) -> Result<bool> {
        let re = regex::Regex::new(r"^(https?:\/\/)?([a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(\/[^\s]*)?$")
                    .unwrap();
        Ok(re.is_match(&input))
      }

Running the Example

Copy and paste following config and save it as dataflow.yaml.

# dataflow.yaml
apiVersion: 0.5.0
meta:
  name: regex-example
  version: 0.1.0
  namespace: examples

config:
  converter: raw

topics:
  sentences:
    schema:
      value:
        type: string
  validlink:
    schema:
      value:
        type: string

services:
  verify-service:
    sources:
      - type: topic
        id: sentences

    transforms:
      - operator: filter
        dependencies:
          - name: regex
            version: "1"
        run: |
          fn verify(input: String) -> Result<bool> {
            let re = regex::Regex::new(r"^(https?:\/\/)?([a-zA-Z0-9-]+\.)+[a-zA-Z]{2,}(\/[^\s]*)?$")
                        .unwrap();
            Ok(re.is_match(&input))
          }

    sinks:
      - type: topic
        id: validlink

To run example:

$ sdf run

Produce sentences to in sentence topic:

$ echo "hello world" | fluvio produce sentences
$ echo "https://www.fluvio.io" | fluvio produce sentences

Consume topic validlink to retrieve the result in another terminal:

$ fluvio consume validlink -Bd
https://www.fluvio.io

Cleanup

Exit sdf terminal and clean-up. The --force flag removes the topics:

$ sdf clean --force

Conclusion

We just covered another basic operator in SDF, the Flat-Map Operator. The Flat-Map is a powerful operator. As a matter of fact, the Flat-Map operator can be used inplace of other operators.