Skip to main content
Version: latest

How to use the Hub

The InfinyOn Hub serves as a centralized repository for InfinyOn Connectors, SmartModules, and Dataflows. It facilitates the discovery and deployment of extensions to enhance Fluvio's data streaming capabilities. The hub provides developers and data engineers a platform to access pre-built solutions or share their extensions easily.

Connectors​

List the Connectors available for download. The official connectors (prepended with infinyon/) can run on InfinyOn Cloud and in your local environment.

List Connectors​

Use fluvio hub connector list or cdk hub list to list the available connectors.

$ fluvio hub connector list
  CONNECTOR                          Visibility 
  infinyon-labs/graphite-sink@0.2.0  public     
  infinyon/duckdb-sink@0.1.0         public     
  infinyon/http-sink@0.2.11          public     
  infinyon/http-source@0.4.3         public     
  infinyon/ic-webhook-source@0.1.4   public     
  infinyon/inf-webhook-source@0.1.9  public     
  infinyon/kafka-sink@0.2.9          public     
  infinyon/kafka-source@0.2.7        public     
  infinyon/mqtt-source@0.2.9         public     
  infinyon/sql-sink@0.4.3            public     
  qdrant/qdrant-sink@0.1.0           public

Check out Connectors for Developers to learn how to build and publish your own connectors for the Hub.

Use Connectors​

If you use a connector in InfinyOn Cloud, the Cloud infrastructure will automatically download and run the connector package. If you use connectors in your local environment, you must use the cdk command to download and run the connector package.

Download and Run Locally​

Connectors are independent binaries that run outside of Fluvio. Use the cdk command to download and run connectors locally:

$ cdk hub download infinyon/http-sink@0.2.11

The same connector configuration file can run the connector locally or in InfinyOn Cloud.

$ cdk deploy start --ipkg infinyon-http-sink-0.2.11.ipkg --config <CONFIG>

To stop a connector, use the cdk deploy shutdown command.

Deploy on InfinyOn Cloud​

Connectors are automatically downloaded to InfinyOn Cloud. The management of InfinyOn Cloud is performed using the fluvio cloud commands. To run a connector in InfinyOn Cloud, use the command fluvio cloud connector create.

$ fluvio cloud connector create --config <CONFIG>

Note List your profiles (fluvio profile list) to ensure your CLI points to the InfinyOn Cloud cluster.

SmartModules​

SmartModules are custom-defined programs that connectors use to transform data records during processing. Similar to connectors, SmartModules can be published to the Hub and made available for download. However, unlike connectors, SmartModules are downloaded to the cluster and are not executed outside of Fluvio.

List SmartModules​

List the SmartModules available for download. The official SmartModules (prepended with infinyon/).

$ fluvio hub smartmodule list
  SMARTMODULE                              Visibility 
  infinyon-labs/array-map-json@0.1.1       public     
  infinyon-labs/csv-json@0.2.0             public     
  infinyon-labs/dedup-filter@0.0.2         public     
  infinyon-labs/json-formatter@0.1.3       public     
  infinyon-labs/key-gen-json@0.1.1         public     
  infinyon-labs/regex-map-json@0.1.2       public     
  infinyon-labs/regex-map@0.1.1            public     
  infinyon-labs/rss-json@0.1.1             public     
  infinyon-labs/stars-forks-changes@0.1.4  public     
  infinyon/csv-json@0.2.0                  public     
  infinyon/jolt@0.4.1                      public     
  infinyon/json-sql@0.2.1                  public     
  infinyon/regex-filter@0.2.0              public

Download and Run SmartModules​

Use fluvio hub smartmodule download to download specific SmartModules to a local or InfinyOn Cloud cluster. Use fluvio profile list to identify the target cluster.

$ fluvio hub smartmodule download infinyon/jolt@0.4.1

SmartModules downloaded to a cluster can be used by the producer CLI or consumer CLI to transform records. They can also be accessed within the Connectors via the configuration file.

To see the SmartModules currently installed on the cluster, use the following command:

$ fluvio smartmodule list

References​