Skip to main content
Version: 0.12.0 (stable)

Array Map

SmartModule ArrayMaps are used to break apart Records into smaller pieces. This can be very useful for working with your data at a fine granularity. Often, each record in a Topic may actually represent many data points, but we'd like to be able to analyze and manipulate those data points independently. ArrayMap allows us to dig in and break apart these composite records into the smaller units of data that we want to work with.

SmartModule ArrayMap

Let's take a look at an example ArrayMap and walk through how it works and what some sample input and output data might look like.

Generic Example: Transform JSON arrays to recordsโ€‹

A common use case is to transform JSON arrays and produce a stream of the records of those arrays. For example, suppose that each element we receive in Fluvio is a JSON array, though we want to interact with the elements of these arrays rather than the arrays themselves. Then, using an ArrayMap, we can transform composite arrays to records that look like this (where this line is a single record):

["a", "b", "c"]

But, we want those elements as a distinct record, like this (where each line is a distinct record):

"a"
"b"
"c"

If you'd like to see a practical example of ArrayMap in action, check out our blog on using ArrayMap to break apart paginated API requests.

Let's dive in and see how to this up in Fluvio.

Create a SmartModule Projectโ€‹

Run smdk generate with the name of the filter and choose the "filter" options:

$ smdk generate array-map
Generating new SmartModule project: array-map
project-group => 'john'
fluvio-smartmodule-cargo-dependency => '"0.3.0"'
๐Ÿ”ง Destination: ~/smdk/array-map ...
๐Ÿ”ง Generating template ...
โœ” ๐Ÿคท Will your SmartModule use init parameters? ยท false
โœ” ๐Ÿคท Which type of SmartModule would you like? ยท array-map
โœ” ๐Ÿคท Will your SmartModule be public? ยท false
Ignoring: /var/folders/5q/jwc86771549058kmbkbqjcdc0000gn/T/.tmp4imt4g/cargo-generate.toml
[1/5] Done: Cargo.toml
[2/5] Done: README.md
[3/5] Done: SmartModule.toml
[4/5] Done: src/lib.rs
[5/5] Done: src
๐Ÿ”ง Moving generated files into: `~/smdk/array-map`...
๐Ÿ’ก Initializing a fresh Git repository
โœจ Done! New project created ~/smdk/array-map

Code Generator for ArrayMapโ€‹

The code in this generated project takes JSON arrays as input records and returns the elements of those arrays as output records. Let's take a look at the full source, then we'll cover it piece by piece. Let's look at src/lib.rs:

$ cd array-map && cat src/lib.rs
use fluvio_smartmodule::{smartmodule, SmartModuleRecord, RecordData, Result};

#[smartmodule(array_map)]
pub fn array_map(record: &SmartModuleRecord) -> Result<Vec<(Option<RecordData>, RecordData)>> {
// Deserialize a JSON array with any kind of values inside
let array: Vec<serde_json::Value> = serde_json::from_slice(record.value.as_ref())?;

// Convert each JSON value from the array back into a JSON string
let strings: Vec<String> = array
.into_iter()
.map(|value| serde_json::to_string(&value))
.collect::<core::result::Result<_, _>>()?;

// Create one record from each JSON string to send
let records: Vec<(Option<RecordData>, RecordData)> = strings
.into_iter()
.map(|s| (None, RecordData::from(s)))
.collect();
Ok(records)
}

This ArrayMap essentially has three steps it takes:

  1. Deserialize a JSON array as input and store it in a Vec<Value>
  2. Converts each Value back into a JSON string
  3. Converts each JSON string into a distinct output Record

Let's take this for a test drive and see it in action.

Build the SmartModuleโ€‹

Let's make sure our code compiles. If eveything works as expected, there is a .wasm file generated in the target directory.

$ smdk build
...
Compiling array-map v0.1.0 (~/smdk/array-map)
Finished release-lto [optimized] target(s) in 11.31s

Your SmartModule WASM binary is now ready for use.

Test with SMDKโ€‹

Let's test our work using the command line test facility.

$ smdk test --text='["a", "b", "c"]'
loading module at: ~/smdk/array-map/target/wasm32-unknown-unknown/release-lto/array_map.wasm
3 records outputed
"a"
"b"
"c"

Great, everything works as expected. Let's test on cluster.

Test on Clusterโ€‹

Let's create a new Fluvio topic to produce the sample records we want to consume with our SmartModule:

$ fluvio topic create array-map
topic "array-map" created

Now we can produce the sample data to our topic.

$ fluvio produce array-map
> ["a", "b", "c"]
Ok!
> ["d", "e", "f"]
Ok!
> ^C

Let's double check it's all there.

$ fluvio consume array-map -dB
Consuming records from the beginning of topic 'array-map'
["a", "b", "c"]
["d", "e", "f"]

Load SmartModule to Fluvioโ€‹

The SmartModule can be loaded to local Fluvio Cluster or [InfinyOn Cloud], as determined by the [current profile]. In this example, the profile points to InfinyOn Cloud.

$ smdk load
Found SmartModule package: array-map
loading module at: ~/smdk/array-map/target/wasm32-unknown-unknown/release-lto/array_map.wasm
Trying connection to fluvio router.infinyon.cloud:9003
Creating SmartModule: array-map

Rust fluvio smartmodule list to ensure your SmartModule has been uploaded:

$ fluvio smartmodule list
SMARTMODULE SIZE
john/array-map@0.1.0 157.8 KB

SmartModules that have been uploaded on the cluster can be used by other areas of the system (consumers, producers, connectors, etc):

$ fluvio consume array-map -dB --smartmodule=john/array-map@0.1.0
Consuming records from the beginning of topic 'array-map'
"a"
"b"
"c"
"d"
"e"
"f"

Congratulations! ๐ŸŽ‰ Eveything worked as expected!

Publish to SmartModule Hubโ€‹

It turns out this SmartModule was requested by other data streaming teams in the organization, so we've decided to [publish] it on [SmartModule Hub].

$ smdk publish
Creating package john/array-map@0.1.0
.. fill out info in hub/package-meta.yaml
Package hub/array-map-0.1.0.ipkg created
Package uploaded!

Let's double check that the SmartModule is available for download:

$ fluvio hub list
SMARTMODULE Visibility
john/array-map@0.1.0 private
...

Congratulations! ๐ŸŽ‰ Your SmartModule is now available for download in the SmartModule Hub.

[publish]: [InfinyOn Cloud]: [current profile]: [SmartModule Hub]: