Troubleshooting

This document will help you troubleshoot common issues with your Fluvio cluster.

Troubleshooting a Local Cluster

To troubleshoot a local Fluvio cluster, you can check the logs of the SC and SPUs processes.

• Run cat ~/.fluvio/log/flv_sc.log for SC logs
• Run cat ~/.fluvio/log/spu_log_XXXX.log for each SPU
  • E.g. when running 1 SPU, there will be spu_log_5001.log

You can also use tail -n 100 ~/.fluvio/log/flv_sc.log to just print the last 100 lines of the log file.

Getting help

You can ask for help from the Fluvio Discord group,

The logging information needed for diagnostics can be packaged up with the command fluvio cluster diagnostics. The compressed archive can be shared in the Discord group or sent via PM to Infinyon admins

Upgrading the Cluster

If you receive an error Check Versions match failed Check Versions match failed: cannot resume a x.y.z cluster with fluvio version a.b.c. This is an error that may be shown when upgrading Fluvio version of a locally deployed cluster. The fluvio cluster may need to be upgraded to the new version: see the fluvio cluster upgrade command for more information.

Log directory

By default, Fluvio logs are stored in the ~/.fluvio/log directory. But if you have the FLUVIO_DIR environment variable set, the logs will be stored in the directory specified by the variable.

Setting log level

You can set various log levels as filtering tracing log.

For example, to start cluster using log level info using cluster start

$ fluvio cluster start --rust-log=info

For individual binaries, you can use RUST_LOG env variable:

$ RUST_LOG=info fluvio run sc --local

The log level can be set to trace, debug, info, warn, error.

SPU Server Port Conflict

By default, the SPU server uses port 9010. If this port is already in use, the server will fail to start and report the following error:

Error binding TcpListener: Address already in use

To verify the issue, check the SPU logs using the command: Run cat ~/.fluvio/log/spu_log_5001.log for spu with id 5001

Solution: Use an Alternative Port

To resolve the port conflict, first stop any currently running SPU if necessary. Next, start the Fluvio cluster using the --spu=0 flag to prevent automatic SPU provisioning. Then, launch the SPU manually with your preferred port numbers. Finally, register the SPU server so it can be recognized by the cluster.

1. fluvio cluster start --spu=0
2. fluvio run spu -i 5001 -p 0.0.0.0:9020 -v 0.0.0.0:9021 --log-base-dir ~/.fluvio/data > ~/.fluvio/log/spu_log_5001.log 2>&1 &
3. fluvio cluster spu register --id 5001 --public-server localhost:9020 --private-server localhost:9021

Producer

Max Record Size Exceeded

if you are having issues with the producer, like the error record size (35707200 bytes), exceeded maximum request size (33554432 bytes) you can increase the maximum record size by setting the max_record_size parameter in the producer command.

fluvio produce hello-node --max-request-size 37748736