Observability and SLOs

Binding for implementation. This page defines what the deployment records about itself (logs, metrics, alert state), how service-level objectives are measured, which alerts fire and how they reach a person, the dead-letter queue consumers, /health and pmail doctor, and the runbooks. Nothing here may record mail content or clear-text addresses (Security › Logging rules).

1. Signals

Analytics Engine from Rust. workers-rs 0.8.7 exposes the binding: Env::analytics_engine(name) returns AnalyticsEngineDataset, written with write_data_point(&AnalyticsEngineDataPoint) or AnalyticsEngineDataPointBuilder::new().indexes(..).add_blob(..).add_double(..).write_to(&dataset) (docs.rs, worker 0.8.7, read 2026-10-09). Limits: 20 blobs, 20 doubles and one index per data point, 16 KB of blobs, an index of at most 96 bytes, 250 data points per invocation (Analytics Engine limits, read 2026-10-09). wrangler dev does not write local data to Analytics Engine, so with PM_ENV = "local" the metrics sink writes each data point as a log line instead (event = "metric"), which the integration tests read.

Required Worker configuration. The generated wrangler.toml must contain:

[observability]
enabled = true
head_sampling_rate = 1

[observability.logs]
invocation_logs = false        # invocation logs record request URLs and the email recipient (FR-PRV-6)

[observability.traces]
enabled = false                # production; staging sets true with head_sampling_rate = 0.1

[[analytics_engine_datasets]]
binding = "METRICS"
dataset = "pylota_mail_metrics"

Cloudflare’s invocation log for a fetch is the method and full URL, and for the email handler it is the recipient address (Workers Logs docs, read 2026-10-09). URLs such as /v1/identities/lookup?address=… contain addresses, so invocation logs are off. Automatic traces record URLs and handler attributes the same way, so traces run only on staging, whose traffic is synthetic. Workers Issues needs Wrangler 4.134.0 or later. The pinned 4.139.0 supports it, but v1 does not depend on it.

METRICS is listed in Configuration › Bindings and in the template in Rust workspace.

2. Structured logs

2.1 Schema

worker::log writes one JSON object per line. Only these fields exist; values are typed (section 12 of Security):

{"ts":"2026-10-09T10:12:03.412Z","level":"info","event":"inbound_accepted","env":"production",
 "version":"1.0.0","commit":"abc1234","handler":"email","request_id":"req_01J9Z5…",
 "tenant_id":"ten_01J9…","identity_id":"idn_01J9…","message_id":"msg_01J9…","bytes":48213,"duration_ms":41}

2.2 Event names

3. Metrics

3.1 Data point layout

Every metric is written to METRICS with the same layout, so one SQL shape reads them all:

• Counters are aggregated in memory per invocation by (index1, name, labels) and written once at the end of the invocation with double1 = count. Observations (durations, sizes) are one data point each. A per-invocation cap of 240 points protects the platform limit; the overflow is counted in metrics_dropped_total.
• Labels are low-cardinality codes. Message, thread and identity IDs are never labels; domain_id and tenant_id are, because their counts are bounded by configuration.
• Writes never fail a request: an error from the binding is logged once per isolate and ignored.

Reading (Analytics SQL API dataset events.analyticsEngine."pylota_mail_metrics", account scope; sample weights are applied to COUNT, SUM and AVG automatically, per the SQL API datasets page read 2026-10-09):

SELECT blob4 AS domain_id, SUM(double1) AS value
FROM events.analyticsEngine."pylota_mail_metrics"
WHERE accountTag = '<ACCOUNT_TAG>'
  AND timestamp >= NOW() - INTERVAL '1' HOUR
  AND blob1 = 'bounces_total' AND blob2 = 'production'
GROUP BY domain_id

3.2 Catalogue

4. Service-level objectives

Windows are rolling 30 days unless stated. “Good” and “total” are counted from the metrics above.

Measured outside production: NFR-QUAL-1…3 by the evaluation harness, NFR-SEC-1 by the attack suite, NFR-SEC-2 by cargo xtask build-worker, NFR-OPS-1 by deploy rehearsals (Testing). NFR-COST-1 is checked by reviewing Cloudflare usage after a week of idling on staging.

5. Alerts

5.1 How alerts reach a person

Custom Alerts are created in the Cloudflare dashboard from the queries in deploy/observability/alerts/; no creation API was found in the documentation on 2026-10-09, so pmail doctor cannot check that they exist. Where Custom Alerts are not available on the account, the operator runs pmail doctor --json on a schedule: it lists firing state alerts and exits non-zero when any is firing.

5.2 Burn-rate rules

Availability SLOs use multi-window burn rates. A Custom Alert with SLO detection fires when (1 − bad/total) × 100 is below its target over both the short and the long window, so each rule’s target is 100 × (1 − burn × (1 − SLO)).

Latency SLOs (95% and 99% targets) have budgets too large for those burn rates, so they use two rules each: page when the good share is below 90% (95% targets) or 97% (99% targets) over 1 h and 5 min, and ticket when it is below the target over 6 h and 30 min. Each rule needs at least 100 events in the window (the Custom Alert “minimum event count”).

5.3 Alert list

5.4 The state alert evaluator

The * * * * * cron runs ops::alerts::evaluate:

1. Read the conditions:
   • dlq_items: per queue, COUNT(*) and MIN(first_seen_at) of open rows; also written as the dlq_items metric;
   • webhook_endpoints with enabled = 1 AND consecutive_failures >= 10, and those disabled with failing in the last minute;
   • domains in failing or suspended;
   • erasure_requests with status = 'failed', or status = 'running' and created_at older than 20 h;
   • when PM_DAILY_SEND_QUOTA is set: SELECT SUM(u.value) FROM usage_daily u JOIN tenants t ON t.id = u.tenant_id WHERE u.day = ?today AND u.metric = 'sends' AND t.mode = 'live' against 80% of the quota. The roll-up runs every 15 minutes, so this alert can lag by up to 15 minutes; SES sends are counted too, which only makes it fire earlier;
   • when SES is configured: SELECT COUNT(*) FROM domains WHERE ses_region IS NOT NULL AND state <> 'removed', plus one for the platform identity, against 9,000 (ses_identities_90pct);
   • conditions reported by objects and crons since the last run (mailbox_size, abuse_pause, rpc_owner_mismatch, inbound_lost, ses_object_lost from the inbound consumer, and ses_sending_paused and ses_rule_missing from the 15-minute SES platform check, which reads GetAccount and the receipt rule set): the reporting code writes an alert.fired audit row itself.
2. Read the current state: for each alert key, the latest audit_log row with action IN ('alert.fired', 'alert.resolved') AND target_id = <alert key>.
3. Transition, with pure rules in core::slo: a true condition on a key that is not firing writes alert.fired (details_json = { "severity", "values" }), logs alert_fired and writes one alert_fired point. A firing key re-notifies (another alert_fired point, no audit row) every 6 hours. A firing key whose condition has been false on two consecutive runs writes alert.resolved.
4. Alert rows use tenant_id of the affected tenant, or NULL for deployment-level alerts. They contain IDs and numbers only.

6. Dashboards

The SQL for each panel is kept in deploy/observability/dashboards/ and runs against the Analytics SQL API, so it works from any tool that can call that API.

7. Health checks

7.1 GET /health

• No authentication, no dependency calls, no tenant data, no alert state.
• 200 {"status": "ok", "version": "1.0.0", "commit": "abc1234", "env": "production"} when the isolate’s configuration is valid. env is PM_ENV, which Configuration says is shown here. When SES is configured, the body also has "ses_region": "eu-west-2" (the value of PM_SES_REGION), so anyone can see where AWS processes mail (Domains on any DNS host §11).
• 200 {"status": "degraded", …} when the Worker runs with a feature off because its configuration is incomplete: "ses": "sns_topic_missing" (SES credentials and region set, PM_SES_SNS_TOPIC_ARN missing: the SES transport is off) or "billing": "stripe_secrets_missing" (PM_BILLING=stripe without its secrets: billing is not started) (Rust workspace › Startup rules).
• 503 with the error envelope (unavailable) when the configuration is invalid (a required variable or secret is missing or malformed, or an optional variable is malformed; config_invalid is logged with the variable’s name, and the body’s details.config_invalid names it). Every other handler returns the same.
• It is a liveness check. Dependency health is pmail doctor’s job.

7.2 pmail doctor

doctor runs every check, prints one line per check with pass, warn or fail, and a fix for each failure (FR-OPS-3). Output format and exit codes are defined in CLI and setup. The checks:

8. Dead-letter queues

Every work queue has a dead-letter queue with a consumer (FR-OPS-4, Rust workspace): pm-inbound-dlq, pm-outbound-dlq, pm-delivery-events-dlq, pm-webhooks-dlq, pm-index-dlq, each with max_batch_size = 100.

8.1 Consumer

For each message:

1. Insert into dlq_items (INSERT OR IGNORE on (queue, message_id), so a repeat is absorbed): a new dlq_ ID, the source queue, the Cloudflare message ID, the body as received, its SHA-256, the tenant_id and kind named by the body if any, and first_seen_at = now.
2. Log dlq_item (queue, message ID, tenant, kind from the body; no body text) and count it.
3. ack(). A failed insert leaves the message for the dead-letter queue’s own retry.

The alert evaluator turns open items into dlq_items metrics and the dlq:{queue} alert (J8).

8.2 Table

dlq_items is defined in Data model: a dlq_ ID, the queue, the Cloudflare message ID (unique per queue), the body as received, its SHA-256, the tenant and kind, and the redrive bookkeeping. Rows are deleted after 14 days by the global retention job (Privacy).

8.3 Listing and redriving

Dead-letter items are read and redriven through the platform API, with a platform key holding platform:ops (REST API › Platform operations). The CLI wraps it as pmail dlq list and pmail dlq redrive (CLI and setup).

• GET /v1/platform/dlq lists items (filters queue, status, tenant_id). It returns the queue, kind, tenant and timestamps, never the stored body: pointers can carry envelope addresses.
• POST /v1/platform/dlq/{dlq_id}/redrive first checks that SHA-256 of body_json still equals body_sha256. A mismatch (the row was edited or damaged) is refused with 500 internal_error, logged as dlq_body_mismatch and nothing is published. Otherwise it publishes the stored body back to its source queue through the Worker’s own producer binding (Q_INBOUND, Q_OUTBOUND, Q_DELIVERY, Q_WEBHOOKS or Q_INDEX), then sets redriven_at and increments redrive_count in the same request, and writes an audit_log row (dlq.redrive). No Cloudflare API token is involved. Every consumer is idempotent (Design conventions), so a redrive is safe to repeat.

9. Runbooks

Every runbook ends by recording what was done in the incident log and checking that the alert resolved.

Bounce spike

1. Diagnose. Overview and Outbound dashboards: which domain, which identities. Read GET /v1/identities/{id}/messages?status=bounced for samples; group deliveries.smtp_code and bounce_type. Hard bounces from one recipient domain usually mean stale addresses; soft bounces with 4.7.x mean throttling or reputation.
2. Mitigate. Pause the sending identities (PATCH /v1/identities/{id} {"status": "paused"}) if the integrator is sending to a bad list. Hard bounces already create suppressions (FR-DLV-2). If a recipient provider is throttling, lower identity_daily_send_cap for the affected tenant.
3. Verify. The bounce rate falls below 2% over the next hour; resume identities.

Complaint spike

1. Diagnose. Which identities and message kinds. Check that marketing mail carries consent and unsubscribe headers (FR-OUT-8) and that the AI disclosure policy is applied.
2. Mitigate. Identities above 0.3% complaints over their last 1,000 sends are already paused (FR-DLV-3). Pause the rest of the affected identities; suspend the tenant if the content is abusive (PATCH /v1/tenants/{id} {"status": "suspended"}). Complaint suppressions are permanent.
3. Verify. No new complaints for 24 hours before resuming, then watch the rate for a week.

Quota exhausted

1. Diagnose. provider_quota_errors_total by provider_code: E_DAILY_LIMIT_EXCEEDED (daily quota) or E_RATE_LIMIT_EXCEEDED (rate). Cloudflare applies the daily quota per account and raises it automatically over time; it is not exposed to the Worker as a number (Email Service limits page, read 2026-10-09). With PM_DAILY_SEND_QUOTA set to the figure shown in the dashboard, provider_quota_80 warns at 80%; without it, provider_quota fires on the first quota error. Update the variable when Cloudflare raises the quota.
2. Mitigate. Nothing is lost: definitely-not-sent messages stay queued and back off for up to 24 hours (G3). Request a higher limit from Cloudflare; move urgent domains to SES (Email Sending outage) if they are pre-verified there. For a tenant’s own cap (quota.warning, 429 daily_cap_reached), raise identity_daily_send_cap or tenant_daily_send_cap in the tenant policy.
3. Verify. transport_outcomes_total{outcome=accepted} resumes; no message reaches failed: quota_exhausted.

Email Sending outage

1. Diagnose. Rising uncertain and retry outcomes or E_INTERNAL_SERVER_ERROR across all domains; check Cloudflare’s status page. Uncertain messages are never resent automatically (FR-OUT-2).

2. Mitigate (J5). For each affected domain that has a verified SES identity (domains.ses_identity set and its Easy DKIM records published), switch its transport with a platform key:

   curl -X PATCH https://mail.example.com/v1/domains/dom_01JA… \
     -H "Authorization: Bearer $PYLOTA_MAIL_KEY" -H "Content-Type: application/json" \
     -d '{"transport": "ses"}'
   

   The change is audit-logged and starts a health check, which re-checks alignment for the new transport. Domains without SES fall back per policy only if they are failing; otherwise their mail waits in the queue.

3. Resolve uncertain sends. After the outage, message.reconciled events settle most of them (FR-DLV-4). For the rest, the integrator checks with the recipient or its own records and calls POST …/messages/{id}/resolve.

4. Verify. Switch transports back ({"transport": "cloudflare"}) once Cloudflare reports recovery, and watch delivery_events_total.

Domain failing

1. Diagnose. GET /v1/domains/{domain_id}/health lists issues with the record and fix; GET /v1/domains/{domain_id}/records shows expected versus observed per resolver. A single resolver disagreeing never changes state (H7).
2. Mitigate. Sending already uses the identity’s platform address (sent_via_fallback, FR-DOM-6). Give the operator the exact record from issues[].fix. For suspended (nameservers, ownership TXT or registration changed), issue a new ownership value with POST /v1/domains/{id}/reprove. For inbound_reject_spike, check whether a domain’s routing points elsewhere or a sender is guessing addresses (dictionary attack); both are visible in inbound_received_total by result. For notification_send_failures, check the platform domain first: system mail (sign-in, invitations, notifications) has no fallback, so while it is failing notification sends fail (reason domain_failing_no_fallback) or are held back (notifications_deferred_total{reason=platform_domain}). The Notifier keeps the items and retries hourly for 24 hours (O25); fixing the platform domain within that time loses nothing. If the platform domain is healthy, follow Email Sending outage.
3. Verify. POST /v1/domains/{id}/verify twice, a minute apart; the state returns to healthy and domain.recovered is emitted.

SES account and receiving

1. Diagnose. pmail doctor --check ses shows production access, the sending status, the receipt rule set and the identity count. For ses_object_lost, the ses_ingest row with status lost names the object and recipient: both the SNS push and the SQS backstop failed to get the message ingested before the 14-day lifecycle rule deleted it. Look for pm-inbound dead-letter items and backstop cron errors in that period.
2. Mitigate.
   • ses_sending_paused: every SES domain already sends through its identities’ platform addresses (FR-DOM-6). Follow AWS’s instructions in the SES console to have sending resumed (the steps are AWS’s; verify them at the time).
   • ses_rule_missing: run pmail setup ses again. It is idempotent, adds pm-deliver to the active rule set and never deactivates another set. Until then, mail to SES domains does not reach the Worker.
   • ses_identities_90pct: the limit of 10,000 identities per Region can be raised only through the AWS account manager (quotas, read 2026-10-09). Ask for it now, or point new customers at nameservers or cloudflare_zone. At 10,000, creating a domain that needs an SES identity fails with 422 transport_unavailable (details.reason = "ses_identity_limit").
   • ses_object_lost: the message cannot be recovered (the sender’s server got a success reply). Tell the affected tenant, then fix why ingestion stalled (DLQ growth, a failing backstop).
3. Verify. pmail doctor --check ses passes and the alert resolves; for a lost object, new mail to the same recipient arrives.

DLQ growth

1. Diagnose. pmail dlq list --queue <queue> (GET /v1/platform/dlq?queue=…): the kind and tenant of each item, and the matching error log lines (by request_id or message_id). inbound_tempfail means R2 writes are failing in email() (J1): senders are retrying, nothing is lost.
2. Mitigate. Fix the cause (a bug: deploy the fix; a dependency outage: wait). Then pmail dlq redrive --queue <queue>, which calls POST /v1/platform/dlq/{dlq_id}/redrive for each open item. Items are kept for 14 days.
3. Verify. Redriven items leave the open set; dlq:{queue} resolves; for inbound items, the messages appear in their mailboxes.

Integrator API down

1. Diagnose. GET /v1/webhooks/{webhook_id}/deliveries?status=failed shows error codes (timeout, tls, dns, status_5xx, ssrf_blocked). Deliveries retry for about 72 hours (J4).

2. Mitigate. Nothing to do while the endpoint is down. After 100 consecutive failures over at least 24 hours the endpoint is disabled. When the integrator is back: re-enable (PATCH /v1/webhooks/{id} {"enabled": true}) and replay what died:

   curl -X POST https://mail.example.com/v1/webhooks/whk_01J9…/replay \
     -H "Authorization: Bearer $PYLOTA_MAIL_KEY" -H "Content-Type: application/json" \
     -d '{"since": "2026-10-08T00:00:00Z", "until": "2026-10-09T00:00:00Z", "status": "dead"}'
   

   Replay reaches back 30 days from each event’s occurred_at (or the tenant’s retention.events_days, if shorter); older events cannot be replayed.

3. Verify. Replayed deliveries succeed; consumers deduplicate on webhook-id.

Parser bug

1. Diagnose. Reproduce with the raw message (GET …/messages/{id}/raw, while within raw_days) against crates/conformance; add the case to the corpus with addresses rewritten to RFC 2606 names.
2. Fix. Release with the fix and an incremented parser_version.
3. Re-parse (J3). Start a reparse job for each affected tenant with a platform key holding platform:ops: POST /v1/platform/jobs with {"kind": "reparse", "tenant_id": "ten_…", "after": "<first affected date>"} (REST API › Platform operations). It re-parses affected messages from raw with the new parser_version and re-emits their events with reprocessed: true (Inbound). Follow it with GET /v1/platform/jobs/{job_id}. Messages past raw_days cannot be re-parsed and are counted in the job’s result.
4. Verify. Spot-check re-parsed messages and their message.received events with reprocessed.

Compromised key

1. Contain (J6). Revoke at once: DELETE /v1/keys/{key_id}. Revoke its descendants too: list keys and revoke every key whose created_by_key_id chain leads to the compromised key (revocation does not cascade).
2. Investigate. GET /v1/audit-events?actor_key_id=key_… lists the key’s administrative actions. Sends are not audit rows (each is recorded by its message, events and delivery log): list the outbound messages of the identities the key reaches, and query Workers Logs for key_id = <key> over the last 7 days (route, status, message_id). Check webhooks created by the key (URLs pointing somewhere unexpected) and keys it created. If the key held identities:sign, its signature_minted lines name the identities it signed as; an assertion lives at most 10 minutes and a signed request at most 5, and the identity’s private key was never exposed, so no identity key needs rotating for this alone.
3. Remediate. Rotate integrator secrets that may have been read through the key (webhook secrets with rotate-secret). Cancel queued sends made by the key (POST …/cancel). If the key was a platform key, review every tenant. For rpc_owner_mismatch, treat it as a possible isolation bug: capture the logged IDs and open a private security advisory.
4. Verify. Requests with the old key return 401 key_revoked.

Abusive identity

1. Diagnose. identity.paused with reason: abuse_threshold carries the complaint and bounce metrics. Review recent outbound messages and recipients.
2. Mitigate. Keep the identity paused (inbound continues). Resume only with a tenant or platform key after the cause is fixed (PATCH … {"status": "active"}, audit-logged). For a whole tenant, suspend it. For mailbox_size (above 70% of 10 GB), set retention.message_days for the tenant or split traffic across identities; raw MIME and attachments are already in R2.
3. Verify. Rates stay below the thresholds for a week after resuming.

Erasure failure

1. Diagnose. GET /v1/erasure-requests/{id} shows failed and the partial receipt; erasure.failed names the step and error. Find job_step and job_failed log lines by job_id.
2. Mitigate. Fix the cause (for example a Vectorize or R2 outage), then submit the same erasure again (POST /v1/erasure-requests with the same scope and target). Erasure is idempotent; the new receipt shows what was still left. NFR-PRV-1 counts from the first request, so act within the 24-hour window.
3. Verify. The new request is completed (or completed_with_holds) with zero probe hits.

Restore from PITR

D1 has Time Travel (30 days on Workers Paid) and SQLite-backed Durable Objects have point-in-time recovery (30 days). R2 has no point-in-time recovery, no object versioning and no bucket replication (PutBucketVersioning and PutBucketReplication are listed as not implemented on the R2 S3 API compatibility page, last updated 2026-07-31, read 2026-10-09). The only copy of a deleted blob is the optional backup bucket (Privacy › R2 backup copy).

1. Scope. Decide what to restore: D1, one or more mailboxes, or both. Pick the target time T.

2. Freeze. Suspend affected tenants (PATCH /v1/tenants/{id} {"status": "suspended"}): inbound gets a temporary failure, so senders retry and nothing is lost; sends are refused.

3. Save what a restore would undo. Before restoring D1, export erasure requests and key revocations made after T: pmail erasure list --json and pmail keys list --json, filtered by time.

4. Restore D1.

   npx --yes wrangler@4.139.0 d1 time-travel info pylota-mail --timestamp=2026-10-09T09:00:00Z
   npx --yes wrangler@4.139.0 d1 time-travel restore pylota-mail --bookmark=<bookmark>
   

   The restore is destructive and in place, cancels in-flight queries, and prints a bookmark that undoes it; record that bookmark (D1 Time Travel docs, read 2026-10-09).

5. Restore a mailbox. Inside the object: ctx.storage.getBookmarkForTime(T), then ctx.storage.onNextSessionRestoreBookmark(bookmark) (which returns an undo bookmark), then abort the object so it restarts restored (Durable Objects SQLite storage API, read 2026-10-09). workers-rs 0.8.7 does not wrap these methods (docs.rs, read 2026-10-09), so the restore tooling (P1) adds externs on Storage::as_raw() behind a platform-key-only operator entry point. The PITR API is not available in local development, so drills run on staging.

6. Reconcile. R2 is not rewound:

   • inbound messages received after T in a restored mailbox still have raw.eml; re-queue their pointers (ingest deduplicates on raw_sha256);
   • outbound messages sent after T lost their rows and their idempotency ledger. The restore tooling lists t/{ten}/i/{idn}/out/ objects uploaded after T whose message is missing from the restored mailbox, and for each re-inserts the message from the stored MIME with status uncertain and flag reprocessed, plus an idempotency row from the object’s idem_key_sha256, fingerprint and operation metadata, with response_json built from the re-inserted row. A retry with the same Idempotency-Key then replays instead of sending again, and a person resolves each uncertain message as usual;
   • re-apply the saved key revocations, then re-submit the saved erasure requests with reason reapply_after_restore:{era_id} (Privacy).

7. Resume the tenants and run pmail doctor --mail-test.

RPO and RTO (NFR-OPS-2): D1 and Durable Object recovery is continuous, which meets the 1-minute RPO for indexes. R2 objects are written once, before the row that points to them, and deleted only by retention and erasure; R2’s durability covers infrastructure loss, which meets the 15-minute RPO for blobs. Against a bug that deletes objects, nothing protects blobs by default; with PM_BACKUP_BUCKET set, the nightly copy limits the loss to objects created since the last run (RPO 24 hours). The 4-hour RTO is rehearsed in the staging drill.

10. Tests