Handling 409 Conflicts in CouchDB Replication Jobs #

You landed here because a _replicator job is logging HTTP 409 Conflict responses and doc_write_failures is climbing in _active_tasks while your checkpoint refuses to advance. This is the incident-response guide for that exact symptom: how to tell a harmless per-document conflict from a systemic write block, how to drain the backlog with a deterministic resolver, and how to confirm the fix stuck. It sits inside the broader error handling & retry logic recovery layer — retry orchestration keeps the job alive, whereas this page keeps its writes landing. On edge and IoT fleets, mobile backends, and Python sync pipelines, a 409 storm almost always traces back to a stale _rev reaching the target, so the whole procedure below is built around one rule: never write blind, always write against the current leaf revision.

Immediate Triage & Prerequisites #

A single document 409 usually does not halt a replication job — the scheduling replicator records the failure and keeps streaming. So before writing any resolution code, decide whether you have transient per-document conflicts or a hard, systemic write rejection. The two demand different responses.

• Confirm the failure is real and sustained. Query GET /_active_tasks (or GET /_scheduler/jobs) and filter for type: "replication". Read doc_write_failures, missing_revisions_found, and checkpointed_source_seq. A doc_write_failures counter that ticks up while checkpointed_source_seq keeps advancing is normal conflict churn; a rising failure count paired with a static checkpoint is a hard write block, not a network blip.
• Read the job document. GET /_replicator/<job_id> — verify continuous: true (which retries indefinitely) and inspect source/target/user_ctx. Deploy jobs from the standard _replicator document schema so these fields are predictable across restarts.
• Grep the CouchDB log for the canonical signature. doc_update_conflict is the 409 fingerprint; each entry names the conflicting _id and the rejected revision. Attachment collisions and _deleted tombstone races surface as the same signature — there are no special log lines for them.
• Confirm you actually have conflicting leaves. Read a suspect document with curl 'http://localhost:5984/<db>/<id>?conflicts=true'. A non-empty _conflicts array means competing leaves are being retained; CouchDB never auto-prunes them, so they persist through compaction until you tombstone them explicitly.
• Environment. Python 3.10+ and one HTTP client (requests or httpx). Run a single resolver replica per replication partition — two workers racing the same partition just doubles 409 pressure.

Why 409 happens at all is worth internalising: CouchDB’s Multi-Version Concurrency Control rejects a write whose incoming _rev does not match the target’s current leaf. That aligns with the HTTP conflict semantics in RFC 7231 Section 6.5.8, and the deeper model is covered in conflict generation models. The remedy is always the same shape — read the current revision, then write against it.

Step-by-Step Implementation #

The read-current-then-write loop below is the whole 409 remedy. Each step includes a check you can run before moving on.

1. Isolate the offending job. GET /_scheduler/jobs and locate the row whose doc_write_failures is non-zero and whose checkpointed_source_seq is frozen. Verify: assert job["info"]["doc_write_failures"] > 0.

2. Fetch the document with its conflicts. GET /<db>/<id>?conflicts=true returns the current read-winner _rev plus a _conflicts array of losing leaves. Verify: assert body.get("_conflicts"), otherwise there is nothing to resolve on this doc.

3. Fetch every losing leaf body. The _conflicts array holds only revision IDs. Retrieve each with GET /<db>/<id>?rev=<rev> so you can apply merge logic to real field values. Verify: the number of fetched branch bodies equals len(_conflicts) + 1.

4. Pick a winner deterministically. Apply a domain rule — timestamp priority, field-level union, or sensor interpolation. When the discard is acceptable, Implementing Last-Write-Wins in CouchDB is the cheapest choice; the full trade-off lives in algorithm selection for merge. Verify: the winner is chosen by a rule that two parallel workers would agree on (break ties on the lexicographically highest _rev).

5. Commit winner plus tombstones in one batch. POST /<db>/_bulk_docs with the merged survivor (stamped against the current winner _rev) and a {"_id": id, "_rev": rev, "_deleted": true} tombstone for every loser. Writing the winner alone leaves the document conflicted — the conflict only clears when the losers are tombstoned. Verify: assert len(batch["docs"]) == len(_conflicts) + 1.

6. Handle the write-time 409. If the batch returns 409, the winner _rev went stale between read and write. Re-read, rebuild against the fresh _rev, and retry with bounded exponential backoff — the same discipline your error handling & retry logic uses everywhere else. Verify: re-read with ?conflicts=true and assert the _conflicts key is now absent.

There is no built-in resolution policy to lean on here: the _replicator document has no conflict_resolution, client_wins, or server_wins field. Every conflicting revision is retained until your application deletes it, so resolution is always application-side.

Complete Working Example #

This self-contained script triages one replication job, then resolves the conflicts it produced. It reads the job’s failure counters, walks conflicted documents, promotes a winner (Last-Write-Wins by updated_at, timezone-safe), and commits winner-plus-tombstones in one atomic batch with bounded backoff on 409. Set COUCH_URL, DB and JOB_ID in the environment before running.

import os
import time
from datetime import datetime, timezone

import requests


def parse_ts(value: str) -> datetime:
    """Parse an application timestamp as timezone-aware UTC.

    A missing offset is assumed UTC so it can never invert the comparison.
    """
    dt = datetime.fromisoformat((value or "").replace("Z", "+00:00"))
    if dt.tzinfo is None:
        dt = dt.replace(tzinfo=timezone.utc)
    return dt.astimezone(timezone.utc)


def job_is_blocked(couch_url: str, job_id: str) -> bool:
    """True when a job is rejecting writes rather than merely churning."""
    jobs = requests.get(f"{couch_url}/_scheduler/jobs").json().get("jobs", [])
    job = next((j for j in jobs if j.get("id") == job_id or j.get("doc_id") == job_id), None)
    if not job:
        return False
    info = job.get("info") or {}
    # Failing writes + a checkpoint that will not advance = systemic block.
    return (info.get("doc_write_failures", 0) or 0) > 0


def resolve_doc(db_url: str, doc_id: str, max_retries: int = 5) -> dict:
    """Promote the newest leaf by updated_at and tombstone the losers atomically."""
    for attempt in range(1, max_retries + 1):
        winner = requests.get(f"{db_url}/{doc_id}", params={"conflicts": "true"}).json()
        losers = winner.get("_conflicts", [])
        if not losers:
            return {"id": doc_id, "ok": True, "note": "no conflict"}

        branches = [winner] + [
            requests.get(f"{db_url}/{doc_id}", params={"rev": rev}).json() for rev in losers
        ]
        chosen = max(branches, key=lambda d: (parse_ts(d.get("updated_at", "")), d["_rev"]))

        survivor = {k: v for k, v in chosen.items() if not k.startswith("_")}
        survivor["_id"] = doc_id
        survivor["_rev"] = winner["_rev"]  # write against the current read-winner
        tombstones = [{"_id": doc_id, "_rev": rev, "_deleted": True} for rev in losers]

        resp = requests.post(f"{db_url}/_bulk_docs", json={"docs": [survivor, *tombstones]})
        if resp.status_code != 409:
            return {"id": doc_id, "ok": True, "attempt": attempt, "result": resp.json()}

        time.sleep(min(2 ** attempt, 30))  # _rev went stale mid-flight; re-read and retry
    raise RuntimeError(f"exhausted retries resolving {doc_id}")


def sweep(couch_url: str, db: str, job_id: str) -> list[dict]:
    """Report job health, then resolve every conflicted document in the database."""
    db_url = f"{couch_url}/{db}"
    if job_is_blocked(couch_url, job_id):
        print(f"[warn] job {job_id} is reporting write failures — resolving conflicts")

    view = requests.get(
        f"{db_url}/_all_docs", params={"conflicts": "true", "include_docs": "true"}
    ).json()
    conflicted = [r["doc"]["_id"] for r in view.get("rows", []) if r.get("doc", {}).get("_conflicts")]
    return [resolve_doc(db_url, doc_id) for doc_id in conflicted]


if __name__ == "__main__":
    couch = os.environ.get("COUCH_URL", "http://localhost:5984")
    database = os.environ.get("DB", "iot_telemetry")
    job = os.environ.get("JOB_ID", "rep_edge_to_cloud_001")
    for outcome in sweep(couch, database, job):
        print(outcome)

For a continuous pipeline, drop resolve_doc into a _changes worker subscribed with conflicts=true, and call it for every document whose _conflicts array is non-empty. When whole-document replacement is too blunt, escalate to the field-level and CRDT resolvers in custom conflict resolver functions in Python.

Gotchas & Edge Cases #

• Blind PUT is the usual culprit. Clients that write without first reading the current _rev generate cascading 409s during bulk sync windows. Fix the writer, not just the symptom: send If-Match: <rev> or a conditional _bulk_docs batch so a stale write fails fast and locally.
• Serialization strips _rev. Mobile SDKs and ORM layers sometimes drop _rev during JSON mapping, so CouchDB treats each payload as a brand-new revision and every write “succeeds” into a fresh conflicting leaf. Log the outgoing _rev on one device before trusting the client.
• Unbounded retries amplify the storm. A continuous: true job with a large http_connections pool hammers a recovering target and exhausts connection pools upstream. Cap per-request retries with retries_per_request and shrink the pool for cellular or satellite backhauls.
• _conflicts survives compaction. Lowering revs_limit trims non-leaf history but never removes conflicting leaves — those must be tombstoned. POST /<db>/_compact only reclaims storage after the losers are deleted.
• Missing timestamps break LWW. A branch with no updated_at sorts as the empty string and loses every comparison. Route documents lacking a resolution key to review rather than silently dropping a real edit.

Verification & Observability #

Confirm the fix at three levels. First, at the document: re-read each resolved _id with ?conflicts=true and assert the _conflicts key is gone — the authoritative per-document check. Second, at the job: GET /_active_tasks should show doc_write_failures flat and the backlog draining, while GET /_scheduler/jobs reports the job running rather than crashing, and checkpointed_source_seq starts advancing again. Track checkpoint progress continuously by monitoring replication checkpoints via API. Third, at the pipeline: emit your own metrics — conflicts-resolved count, resolution latency, and escalation rate — and alert when the conflict rate fails to fall after a sweep, which almost always means winners were written without their tombstones. When a document is genuinely unresolvable (clock drift, missing keys, semantic ambiguity), do not force a lossy write — route it through fallback resolution chains into manual review sync queues with full audit context. For payload structures and revision-tree traversal detail, the official CouchDB Replication documentation is authoritative.

FAQ #

Related #

• Monitoring Replication Checkpoints via API
• Implementing Last-Write-Wins in CouchDB
• Understanding MVCC in CouchDB Replication

Part of: Error Handling & Retry Logic