Repository navigation
feat: replace permanent event-delivery disable with a bounded cooldown - #256
tanderson-ld wants to merge 9 commits into
Conversation
…sifier DefaultEventSender now classifies HTTP failures with HttpErrors.classifyHttpFailure instead of the deprecated isHttpErrorRecoverable/checkIfErrorIsRecoverableAndLog helpers. Behavior and log output are unchanged: the set of statuses treated as unexpected is identical, unexpected failures still skip the retry and still log at error level with the same wording, and transport-level exceptions keep their warn-and-retry-once handling.
…n (SDK-3375) On an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor now enters a bounded, self-recovering cooldown window instead of permanently disabling delivery for the lifetime of the SDK instance, per the EVTPROC spec's Section 1.8. Windows double from 5 minutes to a 1 hour ceiling, one step per failed probe. In-window flushes of any kind are no-ops that dispose of the batch exactly as a sent-and-refused payload would be, without the network or serialization cost. Recording is never gated, retry state resets after two consecutive successful analytics deliveries, and no cooldown state persists across restarts. Diagnostic sends are not gated by the window; an unexpected diagnostic failure opens one, as it previously disabled delivery, but diagnostic successes do not drive the delivery schedule. EventSender.Result gains the unexpected classification with factory methods and the isUnexpectedFailure accessor; the legacy constructor and isMustShutDown are deprecated and still dual-written for consumers that have not migrated.
Diagnostics obtain their delivery token through the same atomic cooldown-check-plus-issue call that analytics flushes use, so a delivery dispatched inside a window carries no token at all and every pre-window token goes stale when the window opens. All three in-cooldown guards are therefore uniformly defensive; one of them previously claimed to be reachable from in-window diagnostic failures.
MockEventSender enqueues the captured payload before it reads the result field, so a test that awaits the startup diagnostic and then immediately changes the sender's result can hand the new result to the still-running diagnostic task. The startup diagnostic then fails unexpectedly and opens the cooldown window itself, the test's own window-opening flush is discarded in-window, and the test times out waiting for it - the fake clock never advances, so the accidental window never elapses. Reproduced deterministically by widening the descheduling gap; seen once in CI. The three tests that change the result right after awaiting the startup diagnostic now wait for the diagnostic task to finish first, using the single-threaded executor barrier the file already uses elsewhere.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8358410. Configure here.
beekld
left a comment
There was a problem hiding this comment.
AI review: The description says the Android SDK's own DirectEventProcessor reads the legacy flag and keeps its permanent disable unchanged. Android's main branch has no DirectEventProcessor; it exists only on unmerged branches. Android main builds this module's DefaultEventSender and DefaultEventProcessor directly (ComponentsImpl.java), pinned at internal 1.9.0. So Android's next bump of this dependency picks up the cooldown and the ungated diagnostics without a coordinated change.
Relatedly, nothing in either repo reads the dual-written legacy flag. EventSenderWrapper only writes it, so it isn't a consumer that has not migrated.
|
AI review comment: Mirror work pending from the Go PR's human review (launchdarkly/go-sdk-events#68, threads by beekld). The Go side landed these in ff661e0 and fcf7d6f; this PR needs the equivalents before merge. Parking the checklist here rather than pushing now.
|
…fixes (SDK-3375) Diagnostics are now fully out of scope for the event-delivery cooldown, matching the Go implementation. A diagnostic delivery no longer mints a window token or feeds the tracker; it only calibrates the clock from the Date header as before. This removes a blackout mode a reviewer surfaced: a proxy that accepts /bulk but rejects /diagnostic would otherwise open a new window on every periodic diagnostic and discard analytics the service would accept. Backpressure on the diagnostic endpoint itself belongs to the DIAG spec (SDK-2777), not here. Mirrors the fixes made on the Go PR's human review (launchdarkly/go-sdk-events#68): - When a window opens, forget every context key so post-window payloads re-introduce contexts whose index/identify events were lost with the refused batch. The window opens on a flush worker thread while the deduplicator is only safe to touch from the dispatcher thread, so the clear is routed through the inbox as a FLUSH_USERS message. - A delivery failure now breaks the consecutive-success streak even when its token is stale, so success/failure/success no longer resets the ladder across a failure (RETRY 1.8.3). Stale outcomes still cannot open, step, or end a window; unstamped (null) tokens remain fully inert. - Comment trims, and the EventSender.failure() javadoc no longer claims it does not affect the delivery schedule. Tests updated for the window-open wipe: the three diagnostic-coupling tests become decoupling tests (the straddle case is gone with the token), and the dedup tests assert via index-event presence rather than exact flush counts.
…wedge close() (SDK-3375) A flush worker caught only Exception around its send and response handling, so an Error (StackOverflowError, NoClassDefFoundError, a test AssertionError, etc.) escaping the loop body skipped the busy-worker-count decrement. The count stayed elevated and close() blocked forever in waitUntilAllFlushWorkersInactive. Moving the decrement into a finally guarantees it runs even when an Error ends the worker thread. Pre-existing on main; surfaced in review of this change. Adds errorFromEventSenderDoesNotWedgeClose, which times out without the fix.
… failure (SDK-3375) Two review findings from Cursor Bugbot: - The transport-failure warning in DefaultEventSender logged e.toString(), which can include hostnames or URLs. It now logs LogValues.exceptionSummary(e), matching HttpErrors' own transport-failure helper and this module's other exception logging. - A custom EventSender that returns null reported no delivery, but the null was silently ignored, leaving the consecutive-success streak intact. It is now treated as an ordinary failure, the same as an exception from the sender, so it breaks the streak without opening a window. This matches the Go implementation, whose value-typed result has an ordinary-failure zero value and so never had this gap. Adds nullResultBreaksTheSuccessStreak.
… (SDK-3513) A 413 Payload Too Large is locally unrecoverable: DefaultEventSender now drops the payload at the sender call site without retrying it and returns an ordinary failure, so delivery is neither disabled nor put into a cooldown window and a subsequent smaller payload can still succeed (EVTPROC 1.7.3.1). This matches the Go server SDK, which drops immediately (the mobile SDKs retry once before dropping; the server SDKs follow the spec's no-retry reading). The shared HttpErrors classifier is deliberately untouched, since data sources share it. Adds a DefaultEventSenderTest case asserting a 413 is not retried, not unexpected, and not a shutdown.
There was a problem hiding this comment.
Risk: medium.
Automated risk review (dry run): Medium
This PR would not be automatically approved. Human approval required.
Why: This replaces permanent event-delivery shutdown with a concurrent cooldown state machine (and 413 payload drop) in DefaultEventProcessor, DeliveryCooldownTracker, DefaultEventSender, and EventSender, which is a non-trivial shared-SDK behavior change even with extensive tests.
If you disagree with this review, share this PR in #feedback-agent-approvals or submit a changes-requested review explaining why.
Sent by Cursor Approval Agent: Dry-Run PR Approver



What
Implements the EVTPROC event processor spec's Section 1.8 delivery cooldown (launchdarkly/sdk-specs#278) in the shared internal events code: on an unexpected delivery failure (a non-recoverable HTTP status such as 401 or 403), the event processor enters a bounded, self-recovering cooldown window instead of permanently disabling event delivery for the lifetime of the SDK instance.
flushAsync()/flushBlocking(), or the shutdown flush) is a no-op: it makes no network attempt and disposes of the batch exactly as a sent-and-refused payload would be disposed of. Two deliberate departures from pure refusal emulation: the diagnostic events-in-last-batch stat reports 0 (no batch was serialized), and the context deduplicator is flushed when the discarded batch contained an index or identify event, so a later payload can introduce those contexts again.DefaultEventSender(1.7.2) is unchanged.This is a port of the reviewed Go implementation (launchdarkly/go-sdk-events#68), adapted to this module's threading model: delivery outcomes arrive on flush worker threads and the shared executor rather than on the dispatcher thread, so the cooldown tracker is synchronized pure state, and outcomes are correlated to the window era they were dispatched in via an opaque
WindowToken(an outcome from a delivery in flight across a window boundary updates nothing).API (internal module surface)
EventSender.Resultgains theunexpectedclassification with static factoriessuccess(Date),failure(), andunexpectedFailure(), and the accessorisUnexpectedFailure(), which also accepts the legacy flag so results from senders written against the old API still get cooldown behavior.Resultconstructor andisMustShutDown()are deprecated.unexpectedFailure()keeps writing the legacy flag for consumers that have not migrated (the server SDK'sEventSenderWrapperand the Android SDK).DefaultEventSendermigrates off the deprecatedisHttpErrorRecoverable/checkIfErrorIsRecoverableAndLoghelpers toclassifyHttpFailure(first commit, behavior-identical) and then returns the new classification (second commit). Transport-levelIOExceptions keep their always-retry behavior.EventsConfigurationgains a package-privatecooldownNanoTimeFntest seam (deliberately not part of any constructor); the tracker documents that the source must haveSystem.nanoTime()semantics, and all window arithmetic uses overflow-safe difference comparisons.Design notes
Dateheader remains unconditional, exactly as on main.disabledflag remains as a shutdown-only latch; nothing sets it from delivery results anymore.finally, so even anError(not anException) escaping the loop body cannot leaveclose()waiting forever; theErrorstill ends that worker thread. (This last part fixes a pre-existing hazard on main, surfaced in review.)NanoTimeSourceinterface rather thanjava.util.function.LongSupplier, which does not exist on Android below API 24 (this module ships in the Android SDK at minSdk 21); the fix is verified with ad8 --min-api 21dex sweep showing zerojava.util.functionreferences.Testing
DeliveryCooldownTrackerTest(24 tests): window ladder with every rung asserted past the shift-guard bound, reset semantics, boundary-round staleness on both probe success and recoverable probe failure, stale-outcome and in-window-outcome guards, null token, default clock path, boundary instant, the atomic gate, and the once-per-window discard notice.DefaultEventProcessorCooldownTest(17 tests): cooldown entry and resume across a fake clock, legacy shutdown-flag results, contradictory success+shutdown results, in-window disposal with context-introduction recovery through both the index and identify arms, a window open forgetting even already-delivered introductions, empty in-window flushes doing nothing, shutdown during a window, warn-once-then-debug and resume logging, diagnostics being fully decoupled from the cooldown (a diagnostic failure opens no window; diagnostic outcomes do not affect an open window), a null sender result neither stopping delivery nor wedgingclose(), a sender exception breaking the success streak, simultaneous failures across the full worker pool stepping the ladder exactly once, and the batch-size stat zeroing on discards.DeliveryCooldownTrackerTest(24 tests): the window ladder and ceiling clamp, reset semantics, boundary-round staleness on probe success and recoverable probe failure, stale failures breaking the streak without moving a window, null-token inertness, the atomic gate, the once-per-window discard notice, overflow and wrap safety.diagnosticEventsStopAfter401Erroris rewritten asdiagnosticEventsContinueAfter401Error; the permanent-disable test is renamed to describe the cooldown contract; a transport-failure sender test pins the deliberate never-unexpected classification of connection errors.Out of scope (follow-ups)
classifyHttpFailurehas no locally-unrecoverable classification, so a 413 opens a window rather than dropping only the oversized payload. That is a pre-existing classifier gap with its own Jira bug (it also requires unsuppressing the corresponding contract test), deliberately not fixed here.EventSender.Result.STOPdoc reinterpretation, wrapper migration,event-delivery-cooldowncontract-test capability, and the 413 test suppression) ship in a follow-up PR once this module releases.DirectEventProcessorreads the dual-written legacy flag and keeps its current permanent-disable behavior byte-identical before and after this release; migrating it to the cooldown is its own coordinated change.Release notes
Note
Overview
Replaces permanent analytics shutdown on unexpected delivery failures (e.g. HTTP 401/403) with a bounded cooldown: windows start at 5 minutes and double up to 1 hour per failed probe, then resume after a successful probe.
Adds
DeliveryCooldownTrackerwithWindowToken-correlated outcomes so concurrent deliveries and window boundaries don’t double-step the ladder or miscount success streaks.DefaultEventProcessorgates flushes through the tracker: in-cooldown flushes discard buffered batches (no network), log once-per-window warn then debug, flush context deduplication when index/identify would be lost, and keep diagnostics on a separate response path that does not drive cooldown state.EventSender.Resultgainssuccess/failure/unexpectedFailureandisUnexpectedFailure()(legacymustShutDownstill maps to unexpected).DefaultEventSenderusesclassifyHttpFailurefor unexpected vs recoverable, treats 413 as drop-without-retry (not unexpected), and keeps transport errors as ordinary failures with one retry.Hardens flush workers:
nullsender results and sender exceptions count as ordinary failures; response handling errors and busy-worker decrements usefinallysoclose()cannot wedge.Reviewed by Cursor Bugbot for commit 6a9a1b0. Bugbot is set up for automated code reviews on this repo. Configure here.