javascript —

Cloudflare Scales D1 Control Plane to Millions of Databases

The D1 control plane architecture is built around Postgres for transactional metadata, ClickHouse for analytics, and Workers KV for globally replicated routing data.

Cloudflare Scales D1 Control Plane to Millions of Databases

It keeps query processing at the edge, separate from centralized management operations, using distributed metadata caching, connection pooling, asynchronous workflows, and periodic health checks to improve scalability and recover from failures. Announced in 2022, Cloudflare D1 is a serverless SQL database built on SQLite and designed to run close to applications on Cloudflare's global network. Over the past year, the number of daily active databases has grown 18-fold.

Cloudflare uses 30-second heartbeats, with per-database jitter, to reconcile each D1 Durable Object with Postgres, the source of truth for database existence and account limits. This mechanism makes failed creation or deletion operations self-correcting: orphaned objects are marked as deleted, and Workers KV routing metadata is repaired if best-effort updates fail. David Laban, full stack software engineer at Cloudflare, and Lambros Petrou, principal engineer at Cloudflare, write: Heartbeats can tolerate slightly out-of-date data, meaning they can be served by Hyperdrive's multiple caching layers or by a Postgres read replica. This heartbeat flow enables the Durable Object to self-correct to reflect the source of truth (Postgres) regarding the creation and deletion of databases.

Earlier coverage

Reported by infoq.com.

javascriptcloud

Related coverage

Elsewhere in this edition