Skip to content

[Brook S4] Added Stream backends and stream_backends= on the client and Replayer. - #374

Closed
moetemp wants to merge 1 commit into
moe/brook/review/s3-data-modelfrom
moe/brook/review/s4-backends
Closed

moetemp wants to merge 1 commit into
moe/brook/review/s3-data-modelfrom
moe/brook/review/s4-backends

Conversation

@moetemp

@moetemp moetemp commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

What was changed

This PR adds StreamBackend, the base class of the Stream backends, and stream_backends= on the client, the Replayer and WorkflowEnvironment. Each Stream ref keeps its backend's name, so any process can route a ref it gets.

  • A Worker uses its client's backends, and applies them after its plugins, so a plugin that replaces the Workflow runner doesn't drop them.
  • With more than one backend, exactly one must have default=True, since a replayed create must use the same backend.
  • The generated Stream service client and the hand-written writer client start their operations on the crate's runtime. Their create_nexus_client gives a Nexus client that converts each model with the client's converters and codec, as Workflow code does, and calls the runtime in place of an endpoint.
  • The Nexus client runs the codec once for each request and once for each response, on all of their payloads, since a remote codec costs a round trip for each call.

Why?

The Redis and memory backends and the Workflow, Activity and client calls all get their runtime from this registration.

Open questions.

  • StreamBackend subclasses SimplePlugin for its hooks, but plugins= refuses it. Is that OK?
  • The default backend is native, though this SDK has no native backend yet. Is that the right default?
  • ClientConfig.stream_backends is Required, so a hand-built ClientConfig must now set it. Is that OK?

Part of the Streams review series.

Checklist

  1. How was this tested:
    • poe lint and pytest tests/contrib/streams.

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 25, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from b146c64 to ce8bd6d Compare October 26, 2026 00:14
@moetemp moetemp changed the title [Brook S4] Added Stream backends and stream_backends= on the client, Worker and Replayer. [Brook S4] Added Stream backends and stream_backends= on the client and Replayer. Oct 26, 2026
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from ce8bd6d to e453c8e Compare October 26, 2026 02:08
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 9e969ce to 9b54b39 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from e453c8e to 0d0ab14 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 9b54b39 to fda6c46 Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch 2 times, most recently from 9f62387 to 9afffa1 Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 5a0f9bf to 7a7e940 Compare October 28, 2026 01:33
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch 2 times, most recently from 36ed7f5 to c0c0a86 Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 7a7e940 to cbec633 Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 7cba3a9 to a6e385a Compare November 5, 2026 17:19
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch 2 times, most recently from 9cfb80a to 98724fe Compare November 5, 2026 23:10
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from a6e385a to 247c20f Compare November 5, 2026 23:10
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from 98724fe to b7b7ff1 Compare November 6, 2026 09:06
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 247c20f to 8185c1c Compare November 6, 2026 09:06
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 8185c1c to 60b56d3 Compare November 6, 2026 09:22
@moetemp
moetemp force-pushed the moe/brook/review/s4-backends branch from b7b7ff1 to 6668d55 Compare November 6, 2026 09:22
@moetemp

moetemp commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Closed in favor of #381, the combined branch, which has this change.

@moetemp moetemp closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant