Skip to content

feat: replace permanent event-delivery disable with a bounded cooldown - #256

Open
tanderson-ld wants to merge 9 commits into
mainfrom
ta/sdk-3375/evtproc-cooldown
Open

tanderson-ld wants to merge 9 commits into
mainfrom
ta/sdk-3375/evtproc-cooldown

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What

Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278) in the shared internal events code: on an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.

  • Windows run 5, 10, 20, 40, then 60 minutes (capped at the 1 hour ceiling), stepping once per failed probe. They are deterministic: RETRY 1.4.3 permits zero jitter for a component whose attempts are internally scheduled, and the flush schedule itself paces attempts, which is also why the windows are not floored at the flush interval.
  • While a window is open, every flush (scheduled, flushAsync()/flushBlocking(), or the shutdown flush) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be disposed of. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context deduplicator is flushed when the discarded batch contained an index or identify event, so a later payload can introduce those contexts again.
  • Recording is never gated (1.8.8): evaluations keep updating summary counters and intake continues during a window.
  • A successful probe returns delivery to the flush cadence immediately (1.8.3); retry state resets only after 2 consecutive successful deliveries (1.8.4). No cooldown state persists across restarts (1.8.10). The recoverable-failure retry inside DefaultEventSender (1.7.2) is unchanged.

This is a port of the reviewed Go implementation (launchdarkly/go-sdk-events#68), adapted to this module's threading model: delivery outcomes arrive on flush worker threads and the shared executor rather than on the dispatcher thread, so the cooldown tracker is synchronized pure state, and outcomes are correlated to the window era they were dispatched in via an opaque WindowToken (an outcome from a delivery in flight across a window boundary updates nothing).

API (internal module surface)

  • EventSender.Result gains the unexpected classification with static factories success(Date), failure(), and unexpectedFailure(), and the accessor isUnexpectedFailure(), which also accepts the legacy flag so results from senders written against the old API still get cooldown behavior.
  • The three-argument Result constructor and isMustShutDown() are deprecated. unexpectedFailure() keeps writing the legacy flag for consumers that have not migrated (the server SDK's EventSenderWrapper and the Android SDK).
  • DefaultEventSender migrates off the deprecated isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers to classifyHttpFailure (first commit, behavior-identical) and then returns the new classification (second commit). Transport-level IOExceptions keep their always-retry behavior.
  • EventsConfiguration gains a package-private cooldownNanoTimeFn test seam (deliberately not part of any constructor); the tracker documents that the source must have System.nanoTime() semantics, and all window arithmetic uses overflow-safe difference comparisons.

Design notes

  • Diagnostic event delivery is not gated by the window (matching the spec, which does not cover diagnostics). Diagnostic outcomes drive the cooldown only the way they drove the old permanent disable: an unexpected failure on a diagnostic payload opens a window, while a diagnostic success or ordinary failure has no effect on delivery state - in particular, a diagnostic success is not the probe and cannot count toward the reset threshold, since only analytics deliveries prove the service is accepting analytics payloads again. (Without that asymmetry, periodic diagnostic successes would hold the ladder at 5 minutes forever while analytics payloads kept failing.) A diagnostic dispatched inside a window carries no live token, so its outcome cannot move delivery state even if it lands after the window elapses. This coupling preserves this module's existing shape and is a deliberate divergence from the Go implementation, which excludes diagnostic results entirely; it is flagged for spec follow-up.
  • Server-time calibration from the Date header remains unconditional, exactly as on main.
  • The disabled flag remains as a shutdown-only latch; nothing sets it from delivery results anymore.
  • A delivery that dies in an exception in serialization or the sender is recorded as an ordinary failure, breaking the consecutive-success streak without opening a window. A throw from the response handling itself (for example a custom sender returning null) is caught separately so it cannot kill the delivery worker. The busy-worker count is decremented in a finally, so even an Error (not an Exception) escaping the loop body cannot leave close() waiting forever; the Error still ends that worker thread. (This last part fixes a pre-existing hazard on main, surfaced in review.)
  • The cooldown check and the delivery token issue are one atomic tracker call, so a window opening concurrently on a worker thread either diverts the flush to the discard path or makes the dispatched payload's token stale. The once-per-window Warn is also claimed under the tracker's own lock.
  • The tracker's clock seam is a package-private NanoTimeSource interface rather than java.util.function.LongSupplier, which does not exist on Android below API 24 (this module ships in the Android SDK at minSdk 21); the fix is verified with a d8 --min-api 21 dex sweep showing zero java.util.function references.
  • Logging: one Error when a window opens (with its length), a Warn for the first discarded batch of each window (Debug thereafter), and an Info when delivery resumes.
  • Java has no failed-send metrics surface, so the in-window discard fires no metric; the Warn/Debug discard logs carry the count and remaining window.

Testing

  • DeliveryCooldownTrackerTest (24 tests): window ladder with every rung asserted past the shift-guard bound, reset semantics, boundary-round staleness on both probe success and recoverable probe failure, stale-outcome and in-window-outcome guards, null token, default clock path, boundary instant, the atomic gate, and the once-per-window discard notice.
  • DefaultEventProcessorCooldownTest (17 tests): cooldown entry and resume across a fake clock, legacy shutdown-flag results, contradictory success+shutdown results, in-window disposal with context-introduction recovery through both the index and identify arms, a window open forgetting even already-delivered introductions, empty in-window flushes doing nothing, shutdown during a window, warn-once-then-debug and resume logging, diagnostics being fully decoupled from the cooldown (a diagnostic failure opens no window; diagnostic outcomes do not affect an open window), a null sender result neither stopping delivery nor wedging close(), a sender exception breaking the success streak, simultaneous failures across the full worker pool stepping the ladder exactly once, and the batch-size stat zeroing on discards.
  • DeliveryCooldownTrackerTest (24 tests): the window ladder and ceiling clamp, reset semantics, boundary-round staleness on probe success and recoverable probe failure, stale failures breaking the streak without moving a window, null-token inertness, the atomic gate, the once-per-window discard notice, overflow and wrap safety.
  • diagnosticEventsStopAfter401Error is rewritten as diagnosticEventsContinueAfter401Error; the permanent-disable test is renamed to describe the cooldown contract; a transport-failure sender test pins the deliberate never-unexpected classification of connection errors.
  • Full module suite: 360 tests green; javadoc and checkstyle clean; javadoc warning count unchanged from main.
  • Two multi-agent review rounds (five agent reviews: adversarial, general, and spec-parity on the initial implementation; adversarial and general on the fixed state) ran against this change before it was opened, including adversarial reviews with runnable proofs on a real Android API 23 emulator image, a d8/dexdump Android compatibility sweep, mutation testing of the new tests, and behavioral-parity diffing against the spec and the Go reference. A subsequent human review round folded in further fixes (diagnostic decoupling, the window-open deduplication wipe, the stale-failure streak fix).

Out of scope (follow-ups)

  • The 413 carve-out: classifyHttpFailure has no locally-unrecoverable classification, so a 413 opens a window rather than dropping only the oversized payload. That is a pre-existing classifier gap with its own Jira bug (it also requires unsuppressing the corresponding contract test), deliberately not fixed here.
  • The server SDK changes (dependency bump, public EventSender.Result.STOP doc reinterpretation, wrapper migration, event-delivery-cooldown contract-test capability, and the 413 test suppression) ship in a follow-up PR once this module releases.
  • Android SDK adoption: the Android SDK's own DirectEventProcessor reads the dual-written legacy flag and keeps its current permanent-disable behavior byte-identical before and after this release; migrating it to the cooldown is its own coordinated change.
  • Diagnostics RETRY alignment is separate spec work.

Release notes

  • Minor (feat). Behavior change worth calling out: a custom event sender result carrying the legacy shutdown flag now causes a temporary cooldown rather than a permanent stop.

Note

Overview
Replaces permanent analytics shutdown on unexpected delivery failures (e.g. HTTP 401/403) with a bounded cooldown: windows start at 5 minutes and double up to 1 hour per failed probe, then resume after a successful probe.

Adds DeliveryCooldownTracker with WindowToken-correlated outcomes so concurrent deliveries and window boundaries don’t double-step the ladder or miscount success streaks. DefaultEventProcessor gates flushes through the tracker: in-cooldown flushes discard buffered batches (no network), log once-per-window warn then debug, flush context deduplication when index/identify would be lost, and keep diagnostics on a separate response path that does not drive cooldown state.

EventSender.Result gains success / failure / unexpectedFailure and isUnexpectedFailure() (legacy mustShutDown still maps to unexpected). DefaultEventSender uses classifyHttpFailure for unexpected vs recoverable, treats 413 as drop-without-retry (not unexpected), and keeps transport errors as ordinary failures with one retry.

Hardens flush workers: null sender results and sender exceptions count as ordinary failures; response handling errors and busy-worker decrements use finally so close() cannot wedge.

Reviewed by Cursor Bugbot for commit 6a9a1b0. Bugbot is set up for automated code reviews on this repo. Configure here.

…sifier

DefaultEventSender now classifies HTTP failures with
HttpErrors.classifyHttpFailure instead of the deprecated
isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers. Behavior
and log output are unchanged: the set of statuses treated as unexpected
is identical, unexpected failures still skip the retry and still log at
error level with the same wording, and transport-level exceptions keep
their warn-and-retry-once handling.
…n (SDK-3375)

On an unexpected delivery failure (a non-recoverable HTTP status such as
401 or 403), the event processor now enters a bounded, self-recovering
cooldown window instead of permanently disabling delivery for the
lifetime of the SDK instance, per the EVTPROC spec's Section 1.8.

Windows double from 5 minutes to a 1 hour ceiling, one step per failed
probe. In-window flushes of any kind are no-ops that dispose of the
batch exactly as a sent-and-refused payload would be, without the
network or serialization cost. Recording is never gated, retry state
resets after two consecutive successful analytics deliveries, and no
cooldown state persists across restarts. Diagnostic sends are not gated
by the window; an unexpected diagnostic failure opens one, as it
previously disabled delivery, but diagnostic successes do not drive the
delivery schedule.

EventSender.Result gains the unexpected classification with factory
methods and the isUnexpectedFailure accessor; the legacy constructor and
isMustShutDown are deprecated and still dual-written for consumers that
have not migrated.
Diagnostics obtain their delivery token through the same atomic
cooldown-check-plus-issue call that analytics flushes use, so a delivery
dispatched inside a window carries no token at all and every pre-window
token goes stale when the window opens. All three in-cooldown guards are
therefore uniformly defensive; one of them previously claimed to be
reachable from in-window diagnostic failures.
MockEventSender enqueues the captured payload before it reads the result
field, so a test that awaits the startup diagnostic and then immediately
changes the sender's result can hand the new result to the still-running
diagnostic task. The startup diagnostic then fails unexpectedly and opens
the cooldown window itself, the test's own window-opening flush is
discarded in-window, and the test times out waiting for it - the fake
clock never advances, so the accidental window never elapses. Reproduced
deterministically by widening the descheduling gap; seen once in CI.

The three tests that change the result right after awaiting the startup
diagnostic now wait for the diagnostic task to finish first, using the
single-threaded executor barrier the file already uses elsewhere.
@tanderson-ld
tanderson-ld marked this pull request as ready for review November 9, 2026 19:19
@tanderson-ld
tanderson-ld requested a review from a team as a code owner November 9, 2026 19:19
@tanderson-ld tanderson-ld changed the title feat: replace permanent event-delivery disable with a bounded cooldown (SDK-3375) feat: replace permanent event-delivery disable with a bounded cooldown Nov 9, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8358410. Configure here.

@beekld beekld left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review: The description says the Android SDK's own DirectEventProcessor reads the legacy flag and keeps its permanent disable unchanged. Android's main branch has no DirectEventProcessor; it exists only on unmerged branches. Android main builds this module's DefaultEventSender and DefaultEventProcessor directly (ComponentsImpl.java), pinned at internal 1.9.0. So Android's next bump of this dependency picks up the cooldown and the ungated diagnostics without a coordinated change.

Relatedly, nothing in either repo reads the dual-written legacy flag. EventSenderWrapper only writes it, so it isn't a consumer that has not migrated.

@tanderson-ld

Copy link
Copy Markdown
Contributor Author

AI review comment: Mirror work pending from the Go PR's human review (launchdarkly/go-sdk-events#68, threads by beekld). The Go side landed these in ff661e0 and fcf7d6f; this PR needs the equivalents before merge. Parking the checklist here rather than pushing now.

  1. Comment trims. The tracker and processor comments here carry the same long-form design narration that was trimmed on the Go side (review feedback: comments should state what the code cannot show, with rationale living in the PR description and tests). Mirror the trim across DeliveryCooldownTracker's javadoc and the processor's cooldown comments.

  2. Clear the context deduplicator when a window opens. The refused window-opening batch's index and identify events are lost with it, so without a wipe, post-window payloads reference contexts the service never received until the deduplicator's own flush interval heals it. Java adaptation required: the window opens in noteUnexpectedFailure on a delivery worker or the shared executor thread, while the deduplicator is only ever touched from the dispatcher thread and its implementations are not promised thread-safe. Post a FLUSH_USERS message to the inbox instead of calling flush() directly; the minutes-long window makes the ordering delay irrelevant. Test updates mirror the Go side: a refused opener's context is introduced afresh post-window, the wipe deliberately forgets even delivered introductions, and the summary-only-discard test's premise no longer holds (after an open-wipe, every in-window batch with context events carries introductions).

  3. Break the consecutive-success streak on stale failures. Both record-failure paths should reset the streak before the staleness check, so success, failure, success cannot reset the ladder (RETRY 1.8.3); stale outcomes still must not open, step, or end windows. Critical Java nuance: the null-token guard must come first. In-window-dispatched diagnostics deliberately carry a null token so their straddling outcomes stay inert; a null token must not touch the streak. The existing diagnosticStraddlingTheWindowEndCannotStepTheLadder test doubles as the regression test for that guard. Javadoc here still says stale outcomes update nothing, which becomes false for the streak; the method docs need the same one-clause corrections as the Go side. Mirror the new tracker tests (both stale-failure flavors of the success-failure-success scenario, plus unstamped-token inertness).

  4. No Java analog is needed for the Go side's DiscardedInCooldown metadata field: this module has no failed-send metrics surface.

  5. PR description correction, independent of the code: the Testing section says four multi-agent review rounds ran against this change; the accurate count is two rounds comprising five agent reviews (three agents on the initial implementation, two on the fixed state).

…fixes (SDK-3375)

Diagnostics are now fully out of scope for the event-delivery cooldown, matching
the Go implementation. A diagnostic delivery no longer mints a window token or
feeds the tracker; it only calibrates the clock from the Date header as before.
This removes a blackout mode a reviewer surfaced: a proxy that accepts /bulk but
rejects /diagnostic would otherwise open a new window on every periodic
diagnostic and discard analytics the service would accept. Backpressure on the
diagnostic endpoint itself belongs to the DIAG spec (SDK-2777), not here.

Mirrors the fixes made on the Go PR's human review (launchdarkly/go-sdk-events#68):

- When a window opens, forget every context key so post-window payloads
  re-introduce contexts whose index/identify events were lost with the refused
  batch. The window opens on a flush worker thread while the deduplicator is
  only safe to touch from the dispatcher thread, so the clear is routed through
  the inbox as a FLUSH_USERS message.
- A delivery failure now breaks the consecutive-success streak even when its
  token is stale, so success/failure/success no longer resets the ladder across
  a failure (RETRY 1.8.3). Stale outcomes still cannot open, step, or end a
  window; unstamped (null) tokens remain fully inert.
- Comment trims, and the EventSender.failure() javadoc no longer claims it does
  not affect the delivery schedule.

Tests updated for the window-open wipe: the three diagnostic-coupling tests
become decoupling tests (the straddle case is gone with the token), and the
dedup tests assert via index-event presence rather than exact flush counts.
…wedge close() (SDK-3375)

A flush worker caught only Exception around its send and response handling, so an
Error (StackOverflowError, NoClassDefFoundError, a test AssertionError, etc.)
escaping the loop body skipped the busy-worker-count decrement. The count stayed
elevated and close() blocked forever in waitUntilAllFlushWorkersInactive. Moving
the decrement into a finally guarantees it runs even when an Error ends the
worker thread. Pre-existing on main; surfaced in review of this change.

Adds errorFromEventSenderDoesNotWedgeClose, which times out without the fix.
… failure (SDK-3375)

Two review findings from Cursor Bugbot:

- The transport-failure warning in DefaultEventSender logged e.toString(),
  which can include hostnames or URLs. It now logs LogValues.exceptionSummary(e),
  matching HttpErrors' own transport-failure helper and this module's other
  exception logging.
- A custom EventSender that returns null reported no delivery, but the null was
  silently ignored, leaving the consecutive-success streak intact. It is now
  treated as an ordinary failure, the same as an exception from the sender, so it
  breaks the streak without opening a window. This matches the Go implementation,
  whose value-typed result has an ordinary-failure zero value and so never had
  this gap. Adds nullResultBreaksTheSuccessStreak.
… (SDK-3513)

A 413 Payload Too Large is locally unrecoverable: DefaultEventSender now drops
the payload at the sender call site without retrying it and returns an ordinary
failure, so delivery is neither disabled nor put into a cooldown window and a
subsequent smaller payload can still succeed (EVTPROC 1.7.3.1). This matches the
Go server SDK, which drops immediately (the mobile SDKs retry once before
dropping; the server SDKs follow the spec's no-retry reading). The shared
HttpErrors classifier is deliberately untouched, since data sources share it.

Adds a DefaultEventSenderTest case asserting a 413 is not retried, not
unexpected, and not a shutdown.
@cursor cursor Bot added the risk:medium label Nov 13, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Risk: medium.

Automated risk review (dry run): Medium

This PR would not be automatically approved. Human approval required.

Why: This replaces permanent event-delivery shutdown with a concurrent cooldown state machine (and 413 payload drop) in DefaultEventProcessor, DeliveryCooldownTracker, DefaultEventSender, and EventSender, which is a non-trivial shared-SDK behavior change even with extensive tests.

If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.

Open in Web View Automation 

Sent by Cursor Approval Agent: Dry-Run PR Approver

@tanderson-ld
tanderson-ld requested a review from beekld November 13, 2026 14:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants