Operations & limits

What it takes to run, what it has been measured to handle, and — stated plainly — what it does not do. Better you know now than discover it in production.

Deployment footprint #

Measured capacity #

These are measured on a production-equivalent node, not extrapolated:

• ~150 concurrent chat sessions comfortably sustained.
• 50+ concurrent uploads per node.
• Calls are bounded structurally, not load-measured — fixed media threads, a room pinned to one loop, a configurable per-room participant cap (64 by default). How many concurrent calls one node carries has not been measured yet.
• A room holds at most 5,000 active members, on every plane. The add that would exceed it is refused with 409 and nothing changes; restoring a removed member or changing a role never needs a seat.
• The connection-establishment knee is the database pool, which is configurable.

Availability #

What is in place:

• Every service restarts automatically after failure or host reboot — verified by deliberately rebooting both environments.
• Background jobs survive worker restarts and are recovered if a worker dies mid-task.
• The cache holds only regenerable state, so it can be flushed or lost without data impact.
• Restore from off-site backup is drilled, with a measured RTO of 64 seconds and the smoke suite run against the restored data.

Deploys #

A release restarts production once. The server is stopped with SIGTERM; every open socket, chat and RTC alike, is closed with 1001 server shutting down within milliseconds of the signal, and the service answers again within a few seconds. A request in the gap gets 502 from the edge. Before 2026-10-09 a release cut clients twice, about a minute apart; the second restart is gone.

• Reconnect on 1001 with exponential backoff and jitter from about one second, then catchup. Durable events are never lost; ephemerals in the gap are gone.
• On the RTC socket, also rebuild the PeerConnection: the SFU's state does not survive a restart, so the old connection has ICE credentials and mids the new instance never saw. You receive a fresh peers snapshot on connect.
• There is no reconnect rate limit on either socket. The limits a client can hit are 30 chat sends per 10 s per principal and 20 broadcast frames a second per socket, each answered with an error ack, not a close.

Data lifecycle #

• On-demand erasure — room, principal or whole tenant. Deletes database rows and object-store blobs, blobs first (keys are unrecoverable once rows cascade). A room being erased is immediately inaccessible.
• Scheduled retention — an optional window after which messages and their attachments are hard-deleted, set by the operator for a tenant or for a plan (the tenant's own window wins). The default is keep forever: nothing auto-deletes unless an operator opts in.
• Abandoned uploads and expired stories are swept automatically, including their objects.
• Usage events are kept indefinitely for billing integrity.
• Directory OTP challenges — a phone number plus the code's hash — are deleted 24 hours after they expire, whether or not anyone signed up.
• The webhook delivery log keeps each event sent to your endpoint for 30 days after the delivery settles, so it can be replayed; then it is deleted.
• Server logs, including the edge's access log with client IP addresses, rotate by size — at most 100 MB per service — so the oldest entries are overwritten as new ones arrive.

Observability #

• Structured JSON logs with request-correlation ids; secrets are masked before anything can log them, and push endpoints/tokens never reach a log line.
• Prometheus metrics: per-route counters, live connection gauges, auth failures, fan-out backlog, job outcomes, and push delivery by outcome and transport. Served on an internal-only port.
• Call quality is measured per subscriber (bandwidth estimate, loss fraction) and published as aggregate distributions — never labelled by room or participant. The same measurement drives which simulcast layer each subscriber receives (see Calls).
• Separate liveness and readiness endpoints for orchestration.

Licensing & resale #

Every component is permissively licensed — MIT, Apache-2.0, BSD, ISC, Zlib or Unicode. No copyleft or source-available licence appears anywhere in the dependency tree, and this is enforced mechanically at every release rather than by manual review. Components were chosen to keep it that way: Valkey rather than Redis, RustFS rather than MinIO, client-supplied video posters rather than GPL-family transcoding.

The practical result: the platform can be sold, sub-licensed, white-labelled or embedded in a proprietary product with no source-disclosure or network-use obligations.

Not included #

Things only you can verify #

Two paths cannot be exercised in CI because they need real third-party accounts. Test both against your own credentials before launch:

• SMS delivery for OTP — your webhook or vendor route, on real handsets.
• Native push — FCM and APNs with your credentials and a real device token, including that apns.env matches your build type.

Both are validated for shape when you configure them, and the request construction is unit-tested, but the vendor round-trip is yours to confirm.