Skip to main content

This Week in Fluvio #24

· 3 min read

Welcome to This Week in Fluvio, our weekly newsletter for development updates to Fluvio open source. Fluvio is a distributed, programmable streaming platform written in Rust.


Data Retention

If you are producing a lot of data into Fluvio, you might be interested in keeping that data fresh for your consumers. Older data can be automatically pruned.

The default retention on new topics is 7 days with a default segment size of 1 GB. So any data residing in older 1 GB segments will be pruned 7 days after the last write. The current segment is left alone.

Example data lifecycle

For a given topic with a retention of 7 days using 1 GB segments

  • Day 0: 2.5 GB is written (total topic data: 2.5 GB)
Topic Segment #Segment sizeDays since last write
01 GB0
11 GB0
20.5 GBN/A
  • Day 6: Another 2 GB is written (total topic data: 4.5 GB,)
Topic Segment #Segment sizeDays since last write
01 GB6
11 GB6
21 GB0
31 GB0
40.5 GBN/A
  • Day 7: 2 segments from Day 0 are 7 days old. They are pruned (total topic data: 2.5 GB)
Topic Segment #Segment sizeDays since last write
21 GB1
31 GB1
40.5 GBN/A
  • Day 14: 2 segments from Day 7 are 7 days old. They are pruned (total topic data: 0.5 GB)
Topic Segment #Segment sizeDays since last write
40.5 GBN/A

The newest segment is left alone and only begins to age once a new segment is being written to.

For more detail check out the docs for more about data retention in Fluvio


Get in touch with us on Github Discussions or join our Discord channel and come say hello! Watch videos on our InfinyOn Youtube Channel

For the full list of changes this week, be sure to check out our CHANGELOG. Until next week!