cloud —

S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way

At Red Canary, storing large amounts of small data is a big deal.

S3 vs. ScyllaDB for tiny telemetry: What Red Canary learned the hard way

Their Linux Endpoint Detection and Response (EDR) sensors run on hundreds of thousands of endpoints (VMs, laptops, EC2 instances, etc.) They continuously generate small payloads with telemetry details (file changes, process launches, and network connections), as well as self-reported health checks. Uploading all these small payloads to S3, which charges per PUT operation, became quite costly at scale.

The logical approach – merging data into larger chunks before uploading them to S3 — did cut costs. However, it also introduced file-size limits, delayed data freshness, and complicated debugging of specific endpoint records. After some experimentation, they cut latency and costs with an unexpected approach: storing large amounts of small data in a write-optimized database, ScyllaDB. "After some experimentation, they cut latency and costs with an unexpected approach: storing large amounts of small data in a write-optimized database, ScyllaDB."

Earlier coverage

Reported by thenewstack.io.

cloud

Related coverage

Elsewhere in this edition