Security Boundaries in CouchDB Replication #

Replication across a fleet of edge nodes, mobile clients, and a centralized cluster is, in practice, a continuous trust negotiation: every stream crosses a security boundary that dictates credential scope, document visibility, and which side is allowed to write a winning revision. This page addresses the exact scenario a distributed-systems team hits when tenant A’s telemetry starts landing in tenant B’s database — a boundary was drawn in the wrong place, and a replication worker faithfully carried data across it. The foundational rules for how revisions move between databases are covered in the parent guide, CouchDB Replication Architecture & Revision Fundamentals; here we operationalize them into enforceable boundaries backed by scoped credentials, per-endpoint _security, deterministic filtering, and automated rotation.

A replication boundary is enforced at three intersecting layers — network transport, database-level access control, and revision authority — and CouchDB evaluates each of them independently on every request. There is no single cross-endpoint handshake that reconciles both databases’ _security up front. Instead, each endpoint authenticates the replication worker using the credentials in the replication document’s source/target auth objects and applies its own _security as the replicator reads from the source and writes to the target.

In edge and IoT deployments, devices frequently operate behind NAT gateways or cellular networks with intermittent connectivity, which makes long-lived interactive sessions impractical. CouchDB accommodates this by authenticating each remote endpoint from the replication document’s own auth objects, while the optional user_ctx sets the local authorization context — and only when a server admin writes the document. If the replication user lacks read access to the source database, the source simply does not return its documents; because CouchDB read access is granted at the database level, per-document exclusion is achieved with filter functions or Mango selectors, not with _security. If the target rejects a write, the replicator records the failure, and a persistent permission error drives the job toward a crashing and eventually failed state. Getting this alignment right is what keeps a mobile offline cache from ever synchronizing a dataset it was never authorized to hold.

Configuration Schema & Required Parameters #

The _replicator database is the control plane for every replication job, so it is also where boundaries are declared. A secure, continuous, single-tenant replication document combines scoped auth payloads, a local user_ctx, and exactly one data-partitioning mechanism. Deploy it using the canonical _replicator document schema; the security-relevant fields are highlighted below.

{
  "_id": "rep_edge_to_core_tenant_a",
  "source": {
    "url": "https://edge-node-01.local:6984/iot_tenant_a",
    "auth": { "basic": { "username": "rep_edge_01", "password": "${EDGE_REP_PASSWORD}" } }
  },
  "target": {
    "url": "https://core-cluster.internal:6984/iot_tenant_a",
    "auth": { "basic": { "username": "rep_core_writer", "password": "${CORE_REP_PASSWORD}" } }
  },
  "continuous": true,
  "create_target": false,
  "user_ctx": { "name": "rep_edge_01", "roles": ["iot_tenant_a_sync"] },
  "filter": "tenant_ddoc/tenant_filter"
}

The parameters that define the boundary — and the failure that results from setting them wrong — are summarized here.

Two constraints are non-negotiable. First, filter, selector, and doc_ids are mutually exclusive — CouchDB rejects a document that sets more than one, so choose a single partitioning mechanism per stream. Second, user_ctx has no effect on remote authentication; a common misconfiguration is expecting user_ctx.roles to authorize a write against a remote target, when in fact only the target’s auth credentials and the target database’s _security decide that.

Streaming Detection / Monitoring Setup #

Boundary violations announce themselves as authentication and authorization failures on the scheduler, not in the application log. The _scheduler/docs endpoint reports the live state and last error for every replication document, and it is the correct feed to poll for 401/403 transitions. The minimal working poller below surfaces any job whose boundary is being rejected.

import time
import requests

COUCH = "https://core-cluster.internal:6984"
AUTH = ("monitor_ro", "${MONITOR_PASSWORD}")


def poll_boundary_failures(interval: int = 15):
    """Yield replication docs whose state or error signals a boundary rejection."""
    session = requests.Session()
    session.auth = AUTH
    while True:
        resp = session.get(f"{COUCH}/_scheduler/docs", timeout=10)
        resp.raise_for_status()
        for doc in resp.json().get("docs", []):
            state = doc.get("state")
            info = doc.get("info") or {}
            reason = (info.get("error") or "") if isinstance(info, dict) else ""
            if state in {"crashing", "failed"} or "unauthorized" in reason.lower():
                yield doc["doc_id"], state, reason
        time.sleep(interval)


if __name__ == "__main__":
    for doc_id, state, reason in poll_boundary_failures():
        print(f"[boundary] {doc_id}: state={state} reason={reason!r}")

Because a permanent permission denial and a transient network blip both surface as crashing, treat repeated unauthorized/forbidden reasons on the same doc_id as a boundary problem and everything else as a candidate for retry. Wiring these transitions into an alerting path is covered in Async Monitoring & Webhooks; the backoff behavior for the transient cases belongs to Error Handling & Retry Logic.

Core Implementation #

Edge environments demand automated credential rotation to contain compromised device keys and stale replication tokens. The orchestrator below rotates a replication user’s credentials without interrupting an active stream: it writes the new secret to _users, patches the _replicator document’s auth payload, and confirms the job returns to running before the old key is retired. It uses structured logging and bounded retries so a rotation event during a mass rollout does not itself become a denial-of-service against the CouchDB cluster.

import logging
import secrets
import time
from dataclasses import dataclass

import requests

log = logging.getLogger("boundary.rotation")


@dataclass
class ReplicationTarget:
    couch_url: str        # e.g. https://core-cluster.internal:6984
    replicator_db: str    # usually "_replicator"
    rep_doc_id: str       # the _replicator document to patch
    rep_username: str     # replication user whose password rotates
    roles: list[str]      # tenant roles the user must retain


class CredentialRotator:
    """Rotate a replication user's password with zero-downtime cutover.

    Steps: generate secret -> update _users -> patch _replicator auth ->
    verify the job is healthy -> only then is the caller safe to discard
    the previous secret.
    """

    def __init__(self, target: ReplicationTarget, admin_auth: tuple[str, str]):
        self.t = target
        self.session = requests.Session()
        self.session.auth = admin_auth  # rotation requires an admin context

    def _request(self, method: str, path: str, **kw):
        url = f"{self.t.couch_url}/{path}"
        for attempt in range(1, 4):  # bounded retry, never unbounded
            try:
                resp = self.session.request(method, url, timeout=10, **kw)
                if resp.status_code < 500:
                    return resp
                log.warning("5xx from %s (attempt %d)", path, attempt)
            except requests.RequestException as exc:
                log.warning("transport error on %s: %s", path, exc)
            time.sleep(2 ** attempt)  # exponential backoff with a hard ceiling
        raise RuntimeError(f"give up after retries: {method} {path}")

    def rotate(self) -> str:
        new_password = secrets.token_urlsafe(32)  # cryptographically strong
        user_id = f"org.couchdb.user:{self.t.rep_username}"

        # 1) Update the user document. CouchDB hashes the password on write.
        user = self._request("GET", f"_users/{user_id}").json()
        user["password"] = new_password
        user["roles"] = self.t.roles  # keep tenant roles intact across rotation
        put = self._request("PUT", f"_users/{user_id}", json=user)
        put.raise_for_status()

        # 2) Patch the active _replicator document's auth payload in place.
        doc_path = f"{self.t.replicator_db}/{self.t.rep_doc_id}"
        rep = self._request("GET", doc_path).json()
        rep.setdefault("source", {}).setdefault("auth", {}).setdefault("basic", {})
        rep["source"]["auth"]["basic"]["password"] = new_password
        self._request("PUT", doc_path, json=rep).raise_for_status()

        # 3) Confirm the job re-authenticated and is healthy before cutover.
        if not self._await_running():
            raise RuntimeError("job did not return to running; keep old secret")
        log.info("rotated credential for %s", self.t.rep_username)
        return new_password

    def _await_running(self, timeout: int = 60) -> bool:
        deadline = time.time() + timeout
        path = f"_scheduler/docs/{self.t.replicator_db}/{self.t.rep_doc_id}"
        while time.time() < deadline:
            state = self._request("GET", path).json().get("state")
            if state == "running":
                return True
            if state == "failed":
                return False
            time.sleep(3)
        return False


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    tgt = ReplicationTarget(
        couch_url="https://core-cluster.internal:6984",
        replicator_db="_replicator",
        rep_doc_id="rep_edge_to_core_tenant_a",
        rep_username="rep_edge_01",
        roles=["iot_tenant_a_sync"],
    )
    CredentialRotator(tgt, admin_auth=("admin", "${ADMIN_PASSWORD}")).rotate()

The cutover ordering matters: patching _replicator before the new secret exists in _users would push the job into crashing, and discarding the old secret before _await_running confirms health would strand the stream. This rotation flow follows the credential-update semantics of HTTP authentication described in RFC 7235, which lets an updated credential take effect on the next request without tearing down the replication session.

Strategy Variants & Trade-offs #

There is no single way to draw a boundary; the right choice depends on how strong an isolation guarantee the tenancy model requires and how much operational surface the team can absorb. Four strategies dominate in production, and they compose — filtered replication over a database-per-tenant layout is common.

• Database-per-tenant isolation — each tenant gets its own database with a dedicated _security object and a dedicated replication user. The strongest boundary, because a read grant simply does not exist across tenants, at the cost of managing many databases and replication documents.
• Filtered replication — a single shared database uses a filter function, Mango selector, or doc_ids allow-list to exclude documents. Cheaper to operate, but the boundary lives in application-authored logic and must be validated against the partitioning rules before it ships.
• Proxy-terminated authentication — a reverse proxy or mutual-TLS layer terminates identity before the request reaches CouchDB, presenting a proxied user_ctx downstream. Centralizes credential policy, but adds an out-of-band component whose failure modes are separate from CouchDB’s.
• Write gating via validate_doc_update — a design-document VDU function rejects writes that violate tenant invariants regardless of how they arrived. A defense-in-depth backstop against a mis-scoped write, but it runs on every write and can become a throughput bottleneck.

Note that a JavaScript filter evaluates every change through the query server, whereas a Mango selector can use an index and is generally the faster boundary for high-churn feeds. When two independently authorized streams converge on the same database, the boundary choice also shapes how divergence is later reconciled by revision tree mechanics.

Deployment & Orchestration #

Package the rotation orchestrator and boundary poller as a small, stateless service so credentials and endpoints arrive as environment variables rather than baked-in files. The service should read COUCH_URL, ADMIN_PASSWORD, and per-tenant secrets from the platform’s secret store — never from the image — and mount TLS material read-only.

Run exactly one replication worker per source partition. Scheduling two continuous jobs with the same computed replication id against the same source produces duplicated read pressure and racing checkpoints; a single replica per partition keeps checkpoint authority unambiguous. In Kubernetes, model each tenant stream as a Deployment with replicas: 1 (or a StatefulSet when a stable identity simplifies auditing), and gate rollout on a health-check endpoint that returns non-200 whenever any managed job is in failed or has been crashing past a threshold.

from http.server import BaseHTTPRequestHandler, HTTPServer

UNHEALTHY_STATES = {"failed"}


class HealthHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        # In production, read cached _scheduler/docs state set by the poller.
        unhealthy = getattr(self.server, "unhealthy_jobs", [])
        code = 503 if unhealthy else 200
        self.send_response(code)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        body = "unhealthy: " + ",".join(unhealthy) if unhealthy else "ok"
        self.wfile.write(body.encode())


if __name__ == "__main__":
    server = HTTPServer(("0.0.0.0", 8080), HealthHandler)
    server.unhealthy_jobs = []
    server.serve_forever()

Map each replication stream to a single tenant namespace, enforced by role-based _security objects and explicit user_ctx declarations. When boundaries are aligned with the network partitions described in sync topology models, a rejected write stays contained to one stream instead of triggering retry loops that exhaust cluster resources and mask legitimate failures.

Troubleshooting & Common Errors #

Boundary failures cluster around a handful of HTTP status codes and scheduler states. Match the symptom to the remediation before assuming a network fault.

When a boundary permanently blocks a write, the scheduler records the cause in the job’s error/reason fields. Parse those to separate transient network errors from permanent permission denials, and route the denials to manual review sync queues for policy reconciliation rather than letting them retry indefinitely.

FAQ #

Related #

• Revision Tree Mechanics
• Conflict Generation Models
• Sync Topology Models
• _replicator Document Schema
• Error Handling & Retry Logic

Part of: CouchDB Replication Architecture & Revision Fundamentals