Skip to content

[Brook S3] Added Stream refs, Records and errors in temporalio.streams. - #373

Closed
moetemp wants to merge 1 commit into
moe/brook/review/feature-streamsfrom
moe/brook/review/s3-data-model
Closed

moetemp wants to merge 1 commit into
moe/brook/review/feature-streamsfrom
moe/brook/review/s3-data-model

Conversation

@moetemp

@moetemp moetemp commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

This PR adds the data model of the Streams API in the new temporalio.streams module. That's the StreamRef[T] data, the read items, the errors and the generated wire models that they carry. temporalio.common gets no Stream name.

  • A Stream has one payload type, and later calls take it as a type argument. There's no Stream definition.
  • StreamRef[T] is data only. It has the fields of the temporal.api.stream.v1.StreamRef wire model (id, owner and backend) and no calls. [Brook S5] Added the client Stream, its typed creates and Stream reads. #375 adds temporalio.client.Stream and [Brook S7] Added Workflow Streams, which write through Local Activities. #377 temporalio.workflow.WorkflowStream, which make the calls.
  • A ref converts to the proto JSON of StreamRef through nexgen's generated converter, also as a field of a dataclass or a pydantic model, through the nested conversion of [Brook G2] Converted transfer types nested in JSON values. #368. The type argument, such as StreamRef[Turn], is for type checks only. A ref keeps an owner of any type, and only a close checks it.
  • A read item is a Record[T] or a Superseded item. Each has a kind with a Literal type of ItemKind, so a match on r.kind narrows the item's type. There's no close item.
  • Superseded.withdrawn_offsets lists only the Records of this read. Superseded.withdraws() answers for any earlier Record, or for just its Producer, with the crate's withdrawal rule on the oneof epoch.
  • The module exports the generated wire models that refs, Records and Superseded items carry, such as Producer, ProducerEpoch with its cases, WorkflowTaskEpoch, StreamSuperseded and StreamWriter.
  • The errors are written by hand, with the public names, such as StreamAlreadyExistsError, StreamOffsetTooHighError and StreamStorageUnavailableError. Their reason is the proto StreamErrorReason, which the module exports. One helper reads the StreamErrorDetails of a failed call, from a Core error or from the details of an application error, such as a Local Activity or Nexus failure. Only StreamStorageUnavailableError is retryable, and a reason with no class, such as STORAGE_CORRUPT, raises the base StreamError. StreamConfigError isn't a StreamError, since it isn't a Store reason.

Why?

Every later Streams PR builds on these types, and they need no backend or server. Stream handles and big-payload handles discusses the ref's wire form, and why the ref is data with no calls.

Part of the Streams review series.

Checklist

  1. How was this tested:
    • tests/contrib/streams/test_refs.py, and test_vectors.py, which checks the vendored Redis proto against the Redis crate, and the Producer ids, the epoch order and the withdrawal rule against the temporalio-streams crate.

@moetemp moetemp added the skip-changelog Changelog entry rides another PR label Oct 25, 2026
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 9e54468 to 4c0260b Compare October 26, 2026 00:14
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 347fb8d to 9e969ce Compare October 26, 2026 02:08
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch 2 times, most recently from c4ce9d7 to c22d918 Compare October 26, 2026 08:44
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 9b54b39 to fda6c46 Compare October 27, 2026 09:07
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 479a894 to 63423ea Compare October 27, 2026 09:17
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 5a0f9bf to 7a7e940 Compare October 28, 2026 01:33
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 63423ea to 6a7f134 Compare October 28, 2026 01:33
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 7a7e940 to cbec633 Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 6a7f134 to 021a3b7 Compare October 28, 2026 07:54
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 7cba3a9 to a6e385a Compare November 5, 2026 17:19
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 171c473 to 86e492b Compare November 5, 2026 17:19
@moetemp moetemp changed the title [Brook S3] Added Stream definitions, handles, Records and errors to temporalio.common. [Brook S3] Added Stream refs, Records and errors in temporalio.streams. Nov 5, 2026
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 86e492b to 8ce8db1 Compare November 5, 2026 23:10
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch 2 times, most recently from 247c20f to 8185c1c Compare November 6, 2026 09:06
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 8ce8db1 to 411c3e9 Compare November 6, 2026 09:06
@moetemp
moetemp force-pushed the moe/brook/review/s3-data-model branch from 8185c1c to 60b56d3 Compare November 6, 2026 09:22
@moetemp
moetemp force-pushed the moe/brook/review/feature-streams branch from 411c3e9 to 5e64942 Compare November 6, 2026 09:22
@moetemp

moetemp commented Nov 10, 2026

Copy link
Copy Markdown
Owner Author

Closed in favor of #381, the combined branch, which has this change.

@moetemp moetemp closed this Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Changelog entry rides another PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant