You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3e081f6
Browse filesBrowse the repository at this point in the historyBrowse files
fix(sync): count a Property with a recorded failure in the reconciliation-age alert (#1082)
sync.reconciliation-age left out every Property with a recorded sync failure,
so a Property whose full reads all failed could not raise the alert that
watches for a Property with no completed read. The read now judges every
syncable Property alike. A connection that needs reauthorization, including
one revoked by the provider, is not syncable and stays out: its reconnect
notice is its signal. Runbook, alerts table and ADR 0055 are updated.
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Copy file name to clipboardExpand all lines: docs/adr/0055-stable-review-and-inbox-handling-cycles.md
+4Lines changed: 4 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1148,6 +1148,10 @@ control just inside the limit).
1148
1148
1149
1149
A reply observation's `content_expires_at` was written only when the row was inserted, and a read that finds the same reply records no row. So a reply Google kept showing stopped being served 30 days after it was first stored, while its review's clock moved on every fetch. A restating read now moves the head row's `content_expires_at` to that read's expiry and its `updated_at` to the read's time, forward only and only while the row holds a reply (`renewHeadContentClock`). It writes no row, revision or fact, and leaves `observed_at` and `provider_updated_at` alone. The clock counts from the last fetch that returned the reply, as ADR 0031 requires for a review's content.
1150
1150
1151
+
## Note 2026-11-04 (b) — the reconciliation-age alert counts a Property with a recorded failure
1152
+
1153
+
`sync.reconciliation-age` left out any Property with an `error_class`, because `sync.failed-nonzero` already names a coded failure. That alert raises only when a retry date passes before the next look re-dates the row, so a Property that failed every full read for days raised no standing signal at all. The read now judges every syncable Property alike. A Property whose connection needs reauthorization (`reauth_required`, which is also how `provider_revoked` is recorded) is not syncable, so it is still left to the reconnect notice.
1154
+
1151
1155
## Merged from ADR 0003
1152
1156
1153
1157
Introduce a new `review` bounded context for external platform reviews,
Copy file name to clipboardExpand all lines: docs/operations/alerts-and-security-posture.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -153,7 +153,7 @@ Every alert is defined in `src/shared/observability/alert-definitions.ts` (owner
153
153
|`sync.failed-nonzero`| P2 | any coded sync failure whose retry is due | §13 |
154
154
|`sync.push-missed`| P2 | topic set; any review polling found in 24h, dated within a day of that, with no Google push accepted within 60min | §4 |
155
155
|`sync.push-silent`| P2 | topic set; polling found new reviews in 24h and Pub/Sub accepted no push at all in that time | §4 |
156
-
|`sync.reconciliation-age`| P2 | a syncable Property's newest completed reconciliation is over 8h old (parked coded failures excluded; independent of push) | §13 |
156
+
|`sync.reconciliation-age`| P2 | a syncable Property's newest completed reconciliation is over 8h old (counts failed Properties; independent of push)| §13 |
157
157
|`ai.review-analysis-stalled`| P2 | incomplete enabled coverage with no new settlement for > 15min; monitor failure also fires | §22 |
158
158
|`ai.review-analysis-empty-enrollment`| P2 | any queued/running zero-snapshot enrollment while eligible reviews exist | §22 |
159
159
|`retention.failure`| P1 | latest retention run failed for any subject; monitor failure (the `retention_runs` read) also fires | §8 |
**What the system does.** The snapshot run ends as `provider_rejected` (`review_provider_snapshot_runs.failure_code`) and discards its cursors. The worker writes one warning, `review_snapshot_provider_rejected`, with `providerStatus` (Google's HTTP status, 400 or 404; null if the error carried none), `dispatch` (`answered`) and `runId`. Look the status up in the worker log: 404 means Google no longer has the Location through this connection, 400 means it refuses the request for it. The job then parks the Property like a location over the cap: it records `error_class = provider_rejected`, the failure time and the retry date, logs `review_snapshot_failure_deferred`, and ends normally. It throws nothing, so there is no BullMQ retry, no dead letter, no quarantine entry, no Sentry event and no repair latch (`worker.job-runtime-unready` stays clear), and `markSyncSucceeded` is not called. The wait is the ordinary ladder (15 minutes, 1 hour, 4 hours, 6 hours cap), not a day, so a Location fixed on Google is read again within hours and a completed run clears the failure.
302
302
303
-
**What you see.**Only `sync.failed-nonzero`, with `alert_codes``provider_rejected:N` (N Properties), and only as an edge, not a standing signal. The retry date is written into the future when the failure is recorded, and the alert counts a Property only from that date until the next look fails again and dates a new one: about one 15-minute sweep per rung of the ladder, once per 6 hours at the cap. So it raises and clears on every rung while the Location stays unreachable, and a quiet alert does not mean the Location is readable. The durable view of what is rejected right now is the query under "What to do". 401 and 403 are not this: they are `authorization_changed` / `authorization_denied` (reconnect Google), 429 and 5xx are retried on the same run, and a code-less failure is still `provider_failure` with its loud error line.
303
+
**What you see.**`sync.failed-nonzero`, with `alert_codes``provider_rejected:N` (N Properties), only as an edge, not a standing signal; and `sync.reconciliation-age` (§13), which is standing, once the Property has gone 8 hours without a completed reconciliation. The retry date is written into the future when the failure is recorded, and `sync.failed-nonzero`counts a Property only from that date until the next look fails again and dates a new one: about one 15-minute sweep per rung of the ladder, once per 6 hours at the cap. So it raises and clears on every rung while the Location stays unreachable, and a quiet one does not mean the Location is readable. The durable view of what is rejected right now is the query under "What to do". 401 and 403 are not this: they are `authorization_changed` / `authorization_denied` (reconnect Google), 429 and 5xx are retried on the same run, and a code-less failure is still `provider_failure` with its loud error line.
304
304
305
305
**What to do.** Find the Properties: `select property_id, error_class, last_terminal_error_at, error_retry_at from review_sync_state where error_class = 'provider_rejected'` (read-only session). Then check each Location in Google Business Profile: it is still there, verified, not suspended, and still managed by the account the Organization connected (a Location the account no longer manages is the likeliest 404). If the owner removed the Location on purpose, archive or disconnect the Property in RepKey. If it was fixed on Google, nothing else is needed: the next look reads it and clears the failure. To read it sooner, set `next_incremental_at` and `error_retry_at` of that `review_sync_state` row to now and leave `error_class` in place (see "Retry sooner" above). If every location of one connection is rejected, or the status is 400 for locations that look healthy on Google, treat it as a RepKey request defect instead (the status in the warning and the number of Properties tell the two apart) and take it to the engineers; do not wait on Google.
306
306
@@ -324,7 +324,7 @@ It enqueues no sync. For each Property with a review whose content expires withi
324
324
325
325
**Alert:**`sync.reconciliation-age` (P2) — a Property the sweep still polls has no completed reconciliation (a completed provider snapshot run, recorded as a `listing_complete``review_google_reputation_snapshot_facts` row for its current source epoch; a head fact from `REVIEW_HEAD_CHECKS=persist` does not count) inside the last 8 hours.
326
326
327
-
**What it means:** the six-hour reconciliation is the safety net that finds a review whose push never came (ADR 0055 decision 5), and the only path while push is dark. The sweep guarantees one about every six hours; the alert fires two hours later, so a slow sweep does not page. A Property with no completed reconciliation yet is counted from the day it became syncable (`properties.updated_at`, `google_connections.credential_authorized_at`, as for `sync.sweep-lag`), and a Property restored with an old reconciliation is counted from the day it qualified again. A Property parked with a coded failure (`review_sync_state.error_class`, for example `review_cap_exceeded` for a location over the cap) is left out: it has its own alert, `sync.failed-nonzero`. The alert does not depend on push. It is held while the `health.sync` or `health.syncPush` section is degraded.
327
+
**What it means:** the six-hour reconciliation is the safety net that finds a review whose push never came (ADR 0055 decision 5), and the only path while push is dark. The sweep guarantees one about every six hours; the alert fires two hours later, so a slow sweep does not page. A Property with no completed reconciliation yet is counted from the day it became syncable (`properties.updated_at`, `google_connections.credential_authorized_at`, as for `sync.sweep-lag`), and a Property restored with an old reconciliation is counted from the day it qualified again. A Property with a recorded failure (`review_sync_state.error_class`, for example `provider_rejected`, or `review_cap_exceeded` for a location over the cap) is counted like any other, because a run that fails never completes a reconciliation: `sync.failed-nonzero` names the code, this alert says how long it has been. A Property whose Google connection needs reauthorization (`reauth_required`, which is also how `provider_revoked` is recorded) is not syncable and is not counted: the reconnect notice is its signal. The alert does not depend on push. It is held while the `health.sync` or `health.syncPush` section is degraded.
328
328
329
329
**First things to check:**`GET /api/health/metrics` → `syncPush.reconciliation.overdueCount` and `oldestAgeMs` (`sync.reconciliation_overdue`, `sync.oldest_reconciliation_age_ms`). If `sync.oldest_due_age_ms` is also high the sweep itself is behind: follow the checks above. If the sweep is healthy, read the overdue Properties' runs: a run that keeps failing before it completes (see the sync failure codes on `review_sync_state`), or a location that never finishes its listing, never writes a fact.
0 commit comments