Repository navigation
Conversation
482ec01 to
5068d41
Compare
The crate, sdk-python and later the server read one copy of these messages. The gRPC service and its Nexus annotations come with the server.
The Stream protos are A1's files again, and the runtime proto is K10's, so the sample matches the crate. `CreateStreamResponse` has no `status` now. The descriptor set and the bindings are rebuilt with the README's recipe.
The Stream protos are A1's files again, so the sample matches the crate. `CreateStreamResponse` has no `status` now. The descriptor set and the bindings are rebuilt with the README's recipe.
5068d41 to
44519d8
Compare
The sample takes the part of Brook's WIT that nexgen generates, on the Stream protos of temporalio/api#895 at 44519d8a. No SDK has those protos, so the sample projects hold their bindings. The sample is a System Nexus service, so its .NET output builds in its own project. Only .NET generates the service, since Python and TypeScript panic on the read item's oneof.
The failure gives the close status, so the field only let a caller contradict it. An append with no sequence gets the next batch number, so a call that has no writer state still works.
The SDKs copy this WIT to generate their Stream Workflow API and client. The Makefile runs add-message on it, so its records follow the Stream protos. The fork has the directives the WIT uses, until a nexgen release does.
…io (#311) ### What changed? This PR moves the stream sample's Python bindings of the Stream protos into a local package, `temporalio_stream_protos/`, that installs them into `temporalio`. Only the sample changes. nexgen and the generated code stay as they are. - The package adds only the Stream modules under `temporalio/api/`. The sample depends on it through a `uv` `path` source, and `streamprotos/` goes away. - basedpyright now checks `wit/stream_service`. ### Why? The Stream protos of `temporalio/api#895` aren't in a released `temporalio` yet. The generated code already imports them from `temporalio.api`, where a release will have them. When it does, the package goes away. ### How did you test it? Link to a test plan if any - - [x] Unit Tests - [ ] Staging - [ ] End to End Tests `test_stream_service.py` round-trips each read item kind through the `temporalio.api` classes.
The message now has the name of the read item that it is. A StreamRef field is named ref everywhere, as on CreateStreamRequest. The field numbers stay, so the wire format is the same.
The stream sample's WIT is the Stream service WIT of the protos PR, with the sample's own definition, and its protos and bindings are those of the PR. The sample gains the `withdraw_run` helper that the WIT's sources call.
The stream sample's WIT is the Stream service WIT of the protos PR, with the sample's own definition, and its protos and bindings are those of the PR. The sample gains the `withdraw_run` helper that the WIT's sources call.
That head has the sample synced to this WIT. Its nexgen code is the same as at the earlier pin.
A higher epoch fences the lower one, and a reader may hide its Records, for a client too. A client with no name of its own is a new Producer in each process.
An SDK writer is an object now, so no call names a writer by id. A closed writer fails as CLOSED. A Record needs a body, and a timestamp needs both of its fields.
The two comments held for one kind of Store each.
| // The name of the backend that holds the Stream, as the Worker and client config register | ||
| // it, so every process routes a ref it receives. Empty is the default backend. The Store | ||
| // never reads it. | ||
| string backend = 3; |
There was a problem hiding this comment.
Doesn't the SDK already know which backend was selected? The backend wouldn't know which alias it was given, only the SDK knows it.
| // One Workflow Task attempt of a run. It orders by started_event_id, then attempt, then | ||
| // started_time. | ||
| message WorkflowTaskEpoch { | ||
| int64 started_event_id = 1; |
There was a problem hiding this comment.
So you wouldn't know that a producer was superseded in those cases. I don't think that's good enough coverage.
| // Experimental. Might change in the future. | ||
| // The attempt of a Local Activity. It orders by task, then attempt, so a Local Activity of a | ||
| // later Workflow Task supersedes one of an earlier task. | ||
| message LocalActivityEpoch { |
There was a problem hiding this comment.
I see, I'll collapse this discussion into the one above where I mention that you need to reset the epoch on workflow resets and retries, as well as activity resets. You don't have full coverage of superseded events here.
| // Experimental. Might change in the future. | ||
| // The attempt of a client Producer, which the client picks. | ||
| message ClientEpoch { | ||
| int32 attempt = 1; |
There was a problem hiding this comment.
I don't think typically clients will be able to set this so I am inclined to leave it out of the design for now.
| // the server. | ||
| // (-- api-linter: core::0140::prepositions=disabled | ||
| // aip.dev/not-precedent: The floor is the lowest from_offset of the run's reads. --) | ||
| int64 floor_from = 12; |
There was a problem hiding this comment.
I am not sure I understand how this is different from from_offset...
| // Record at or after from_offset exists, when the Stream closes (done), or when wait ends. Its | ||
| // wait may be days, since a parked read adds nothing to History while it waits. Only the | ||
| // server serves it, and a Store refuses it. | ||
| bool wake_only = 9; |
There was a problem hiding this comment.
Can we remove this until we add notification support?
| bytes next_offset = 2; | ||
| // True when the Stream is closed and this read gave its last item. A read of a closed Stream | ||
| // with no item left answers at once. | ||
| bool done = 3; |
There was a problem hiding this comment.
I would add just "done" and not add a special close record.
| bool done = 3; | ||
| // This reader's floor keeps more than half of the Stream's byte cap. The writer fails when the | ||
| // cap fills, and only the reader's continue-as-new frees it. Only set by the server. | ||
| bool continue_as_new_suggested = 4; |
There was a problem hiding this comment.
Drop it for now, let's design this properly once we have reads from workflows.
| oneof position { | ||
| // Read after this offset. It fails with OFFSET_TOO_HIGH if it is past the newest offset. | ||
| // (-- api-linter: core::0140::prepositions=disabled | ||
| // aip.dev/not-precedent: "from" and "to" mark the ends of a range of offsets. --) | ||
| bytes from_offset = 3; | ||
| // Start at the beginning or at the end of the Stream. Unspecified is the beginning. | ||
| temporal.api.enums.v1.StreamReadStart start = 4; | ||
| } |
There was a problem hiding this comment.
This can be simplified as:
| oneof position { | |
| // Read after this offset. It fails with OFFSET_TOO_HIGH if it is past the newest offset. | |
| // (-- api-linter: core::0140::prepositions=disabled | |
| // aip.dev/not-precedent: "from" and "to" mark the ends of a range of offsets. --) | |
| bytes from_offset = 3; | |
| // Start at the beginning or at the end of the Stream. Unspecified is the beginning. | |
| temporal.api.enums.v1.StreamReadStart start = 4; | |
| } | |
| // Read after this offset. It fails with OFFSET_TOO_HIGH if it is past the newest offset. | |
| // Leave empty to read from the beginning. | |
| bytes from_offset = 3; |
A read gives only Records, tells of a close with done, and starts after an offset. A client has no epoch, and a describe gives no Retention, which is a backend setting. The fields that only the server reads come back with the design of Workflow reads on native Streams.
False and not set mean the same for this flag, so an option gave the SDK a third state with no meaning.
That revision makes a list<u8> parameter of a resource method the bytes of its proto field, which the read's from-offset needs.
A caller can pass its own Producer, and the default helper picks one by where the code runs. An append gives back its response, so the API can grow without a breaking change.
The SDKs and the docs say Store for the thing that holds a Stream, so the ref's field says it too.
The resource's `append` takes an optional `sequence`, so a caller that numbers its batches gets each one written once. Two comments no longer name an SDK writer.
Two offsets of one Stream compare bytewise, so the server can order them too. On a Store, Workflow code writes through a Local Activity, so the Workflow Producer comments no longer describe a close there.
The vendored WIT and Stream protos match temporalio/api#895 again. A caller can number an append, the offset comment says how two offsets compare, and no comment names an SDK writer.
The crate builds on the API files of the base branch, which follow temporalio/api#895 again.
The crate's code is the same. Its base now has the API files of temporalio/api#895 with an optional sequence on append.
…#895 (#374) * fix(typescript): infer Long from protobuf resource bindings * feat: let a resource declare its type parameters A resource with `@nexus.type-parameter "<alias>"` is generic in Python, TypeScript and .NET. An operation that returns it, or that its method binds, takes a runtime type for each type parameter of its output, so a response decodes its generic values. * test: drop the low-value type parameter rejection fixtures The kept fixtures cover an undeclared type parameter, a generic resource in a record field or in a result with no proto message, and a language that refuses it. The others test directive syntax, rare name clashes or another copy of a kept check. * docs: shorten the generic resources section of the guide * test: assert the full message of each type parameter rejection * docs: say why a generic handle keeps its runtime types Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: infer the type parameters of a resource A resource that uses a type parameter, also through its records, is generic in it, as a record is. So a resource needs no annotation, and `@nexus.type-parameter` is valid only on a type alias again. * fix(python): decode generic resources from their type arguments (#367) * fix: infer resource type parameters from results in declaration order Only a returned resource gives its type parameters, as the guide says, so a parameter or field that holds one doesn't. The declaration order doesn't change when a method moves. * fix: refuse a generic resource anywhere in a record field A list, map, tuple, result or variant case builds the resource from the record's message too, so it gets no type argument either. * fix: say that a direct generic resource result isn't supported yet The old reason said a proto message holds the type arguments, which it doesn't. * fix: refuse runtime type names that collide with each other or a method Two type parameters such as `body` and `body-t` give the same runtime type name, and a method can take it too. Either one makes a duplicate parameter or member. * fix(python): keep the runtime output type in the System Nexus dispatch The typed hook of a generic response took only the request, so the terminal decoded `ReadRecordsResponse[Turn]` with the plain class and lost `Turn`. * fix(dotnet): send generic System Nexus operations to the generic hook C# generics are invariant, so the `<object>` checks refused a `Stream<Turn>` call. The dispatch now compares generic type definitions and keeps the call's types. * fix(python): read the type argument of a subclassed generic handle A subclass such as `class TurnStream(Stream[Turn])` has no `__orig_class__`, so the handle reads `__orig_bases__` too. A factory tests `body_type is None`, since a class can be falsy. * test: cover a typed TypeScript read and a bare Python handle class The TypeScript stream test now decodes a body with a runtime type. The Python test decodes to the bare handle class, since a generic alias fails for another reason. * chore: fold the generic resource changelog and describe the fixes No release has the Python constructor change, so it belongs with the added entry. The fixed entry now describes the System Nexus dispatch fixes. * docs: reword the generic resource comments and wrap long lines A few comments used "name" as a verb, ran over two lines or over 100 columns. The guide's first paragraph read as one long sentence. * chore: fold the generic resource changelog into one entry * chore: trim the generic resource tests and docs Drop the tests that repeat another test or restate generated code, and keep the ones that guard behavior. Shorten the guide section and the comments. * feat(python): take a resource's runtime types as a subscript Python can't subscript a function, so each generic call is an object whose subscript binds the types, as in `client.get_stream[Turn](ref=ref)`. A factory's optional field defaults to `None`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: close a resource at the end of async with `@nexus.close` marks the method that ends a resource. Python calls it at the end of `async with`, and a call that returns the resource returns a `ResourceCall`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: add @nexus.varargs and flatten a list of one-field records `@nexus.varargs` makes a resource method take a list's items as positional arguments, and a list of a flattened one-field record takes that field's values. The Stream's publish takes values and returns the last offset. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat(python): add a Workflow resource factory Workflow code gets a handle on an existing Stream with `get_stream[T]`, from the fields of its ref, with no call. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat(python): add RPC options to client calls Each client call and client resource method takes `rpc_metadata` and `rpc_timeout`, and passes them to the `start_operation` of its Nexus client. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: flatten the record of one request field `@nexus.flatten-in-api` on a request field makes that call take the fields of the field's record, and the models keep the record. The Stream create takes the id, the owner and the backend of its ref this way, and the other Stream calls keep `ref`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: limit `@nexus.api-omit` to some targets A field can leave the API of one target and take a source in another. A call of a target that omits a required list sends it empty. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): sync the stream sample with StreamWriter temporalio/api#895 moves the Producer and the withdrawal fields of a write into one StreamWriter. The sample WIT also takes the final directives of the open nexgen PRs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): sync the stream sample with the owner and withdrawal removal This syncs the stream sample with the owner and withdrawal removal. The WIT and the stream protos drop the owner, the read-side withdrawal machinery and the `OWNER_*` close statuses, and add the `StreamClose` read item with the kind-oneof `Producer`. The Python, TypeScript and .NET bindings and samples are regenerated, and the hand-written support and tests follow. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): format the stream test with oxfmt Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Omitted an operation's public function with `@nexus.api-omit`. A top-level `func` with `@nexus.api-omit` keeps the operation registered and bound to its resource, but drops the public top-level function and its export. A caller reaches the operation through the resource. On a function the directive omits the workflow top-level function, so its `targets` must include `workflow`. The stream sample omits four operations, so `append-records`, `close-stream`, `read-records` and `describe-stream` have no public function and the `Stream` resource is the way in. `create-stream` stays public, since the Workflow has no other way to create a Stream. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Made a `list<u8>` case of a proto oneof the proto `bytes`. A variant with `@nexus.proto` reads an unnamed `list<u8>` case as the proto `bytes`, the same rule as a field of a proto-backed record. Another width still fails against a `bytes` member. The stream sample's read position is a oneof of a `bytes` offset and a start enum, which showed two more gaps. An enum case kept its proto type, so its WIT enum was pruned. It is now planned as the bound enum, like a record case. A resource method parameter named after the oneof was dropped, since the request plan walked the oneof's members. It now binds to the record's oneof field. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the latest temporalio/api#895. The WIT and protos are those of temporalio/api#895 at b0b2657e. The WIT adds the `@nexus.api-omit` of #405 and the `@nexus.generic` of #399, which the API WIT doesn't have yet. A directive that nexgen doesn't support yet stays a TODO comment, and an offset is an inline `list<u8>`, since nexgen reads only that as `bytes`. * Synced the stream sample with the removal of `create_wait`. temporalio/api#895 at 0ac6b717 drops `create_wait` from a read and renumbers the later fields, so the WIT, the protos and their bindings follow it. * Synced the stream sample with the Stream writes and items reshape. temporalio/api#895 at 23fac086 drops `StreamWriter` for a plain `producer`, moves the offset into the read items, and makes the required fields required. The WIT is the API's as is, and the support helpers are the `producer` sources that it names. * Synced the stream sample with the close offset and namespace comments. * Synced the stream sample with the one fence rule for every Producer. * Synced the stream sample with the unknown writer removal. * Synced the stream sample with the Workflow Producer comments. The API comments now say when a Workflow Producer writes and when it only closes. * Synced the stream sample with the read and Record reshape. The API dropped the read position, the close item and the client epoch. The `read` parameter needs #407 to be `bytes`. * fix: make a `list<u8>` resource method parameter the proto `bytes` of its field (#407) * Typed a `list<u8>` method parameter of a proto `bytes` field as bytes. A proto-backed record reads that field as `bytes`, so a list of integers could not fill it. * Regenerated the stream sample with the bytes parameter. The stream `read` takes its `from_offset` as `bytes` in Python, TypeScript and .NET. * Synced the stream sample with the Store rename and the append result. The API renamed `backend` to `store` in the Stream ref and dropped the `append` result transform. So `append` returns the `AppendRecordsResponse` in every target. The two optional `producer` parameters of the API's WIT are not in this sync, since they need a generator change. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the sequence parameter of append. The API gave `append` an optional `sequence`, so a caller can number a batch and the Store can dedupe a resent one. The optional `producer` parameter still waits for the generator change. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the offset and Workflow Producer comments. The API now says that two offsets of one Stream compare bytewise. It also says that Workflow code on a Store writes through a Local Activity. Only comments changed, so the samples only move in docs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Told why the stream sample offset comment matters. --------- Co-authored-by: Tim Conley <timothy.conley@temporal.io>
…parameter in Python (#408) * fix(typescript): infer Long from protobuf resource bindings * feat: let a resource declare its type parameters A resource with `@nexus.type-parameter "<alias>"` is generic in Python, TypeScript and .NET. An operation that returns it, or that its method binds, takes a runtime type for each type parameter of its output, so a response decodes its generic values. * test: drop the low-value type parameter rejection fixtures The kept fixtures cover an undeclared type parameter, a generic resource in a record field or in a result with no proto message, and a language that refuses it. The others test directive syntax, rare name clashes or another copy of a kept check. * docs: shorten the generic resources section of the guide * test: assert the full message of each type parameter rejection * docs: say why a generic handle keeps its runtime types Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: infer the type parameters of a resource A resource that uses a type parameter, also through its records, is generic in it, as a record is. So a resource needs no annotation, and `@nexus.type-parameter` is valid only on a type alias again. * fix(python): decode generic resources from their type arguments (#367) * fix: infer resource type parameters from results in declaration order Only a returned resource gives its type parameters, as the guide says, so a parameter or field that holds one doesn't. The declaration order doesn't change when a method moves. * fix: refuse a generic resource anywhere in a record field A list, map, tuple, result or variant case builds the resource from the record's message too, so it gets no type argument either. * fix: say that a direct generic resource result isn't supported yet The old reason said a proto message holds the type arguments, which it doesn't. * fix: refuse runtime type names that collide with each other or a method Two type parameters such as `body` and `body-t` give the same runtime type name, and a method can take it too. Either one makes a duplicate parameter or member. * fix(python): keep the runtime output type in the System Nexus dispatch The typed hook of a generic response took only the request, so the terminal decoded `ReadRecordsResponse[Turn]` with the plain class and lost `Turn`. * fix(dotnet): send generic System Nexus operations to the generic hook C# generics are invariant, so the `<object>` checks refused a `Stream<Turn>` call. The dispatch now compares generic type definitions and keeps the call's types. * fix(python): read the type argument of a subclassed generic handle A subclass such as `class TurnStream(Stream[Turn])` has no `__orig_class__`, so the handle reads `__orig_bases__` too. A factory tests `body_type is None`, since a class can be falsy. * test: cover a typed TypeScript read and a bare Python handle class The TypeScript stream test now decodes a body with a runtime type. The Python test decodes to the bare handle class, since a generic alias fails for another reason. * chore: fold the generic resource changelog and describe the fixes No release has the Python constructor change, so it belongs with the added entry. The fixed entry now describes the System Nexus dispatch fixes. * docs: reword the generic resource comments and wrap long lines A few comments used "name" as a verb, ran over two lines or over 100 columns. The guide's first paragraph read as one long sentence. * chore: fold the generic resource changelog into one entry * chore: trim the generic resource tests and docs Drop the tests that repeat another test or restate generated code, and keep the ones that guard behavior. Shorten the guide section and the comments. * feat(python): take a resource's runtime types as a subscript Python can't subscript a function, so each generic call is an object whose subscript binds the types, as in `client.get_stream[Turn](ref=ref)`. A factory's optional field defaults to `None`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: close a resource at the end of async with `@nexus.close` marks the method that ends a resource. Python calls it at the end of `async with`, and a call that returns the resource returns a `ResourceCall`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: add @nexus.varargs and flatten a list of one-field records `@nexus.varargs` makes a resource method take a list's items as positional arguments, and a list of a flattened one-field record takes that field's values. The Stream's publish takes values and returns the last offset. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat(python): add a Workflow resource factory Workflow code gets a handle on an existing Stream with `get_stream[T]`, from the fields of its ref, with no call. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat(python): add RPC options to client calls Each client call and client resource method takes `rpc_metadata` and `rpc_timeout`, and passes them to the `start_operation` of its Nexus client. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: flatten the record of one request field `@nexus.flatten-in-api` on a request field makes that call take the fields of the field's record, and the models keep the record. The Stream create takes the id, the owner and the backend of its ref this way, and the other Stream calls keep `ref`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: limit `@nexus.api-omit` to some targets A field can leave the API of one target and take a source in another. A call of a target that omits a required list sends it empty. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): sync the stream sample with StreamWriter temporalio/api#895 moves the Producer and the withdrawal fields of a write into one StreamWriter. The sample WIT also takes the final directives of the open nexgen PRs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): sync the stream sample with the owner and withdrawal removal This syncs the stream sample with the owner and withdrawal removal. The WIT and the stream protos drop the owner, the read-side withdrawal machinery and the `OWNER_*` close statuses, and add the `StreamClose` read item with the kind-oneof `Producer`. The Python, TypeScript and .NET bindings and samples are regenerated, and the hand-written support and tests follow. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): format the stream test with oxfmt Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Omitted an operation's public function with `@nexus.api-omit`. A top-level `func` with `@nexus.api-omit` keeps the operation registered and bound to its resource, but drops the public top-level function and its export. A caller reaches the operation through the resource. On a function the directive omits the workflow top-level function, so its `targets` must include `workflow`. The stream sample omits four operations, so `append-records`, `close-stream`, `read-records` and `describe-stream` have no public function and the `Stream` resource is the way in. `create-stream` stays public, since the Workflow has no other way to create a Stream. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Made a `list<u8>` case of a proto oneof the proto `bytes`. A variant with `@nexus.proto` reads an unnamed `list<u8>` case as the proto `bytes`, the same rule as a field of a proto-backed record. Another width still fails against a `bytes` member. The stream sample's read position is a oneof of a `bytes` offset and a start enum, which showed two more gaps. An enum case kept its proto type, so its WIT enum was pruned. It is now planned as the bound enum, like a record case. A resource method parameter named after the oneof was dropped, since the request plan walked the oneof's members. It now binds to the record's oneof field. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the latest temporalio/api#895. The WIT and protos are those of temporalio/api#895 at b0b2657e. The WIT adds the `@nexus.api-omit` of #405 and the `@nexus.generic` of #399, which the API WIT doesn't have yet. A directive that nexgen doesn't support yet stays a TODO comment, and an offset is an inline `list<u8>`, since nexgen reads only that as `bytes`. * Synced the stream sample with the removal of `create_wait`. temporalio/api#895 at 0ac6b717 drops `create_wait` from a read and renumbers the later fields, so the WIT, the protos and their bindings follow it. * Synced the stream sample with the Stream writes and items reshape. temporalio/api#895 at 23fac086 drops `StreamWriter` for a plain `producer`, moves the offset into the read items, and makes the required fields required. The WIT is the API's as is, and the support helpers are the `producer` sources that it names. * Synced the stream sample with the close offset and namespace comments. * Synced the stream sample with the one fence rule for every Producer. * Synced the stream sample with the unknown writer removal. * Synced the stream sample with the Workflow Producer comments. The API comments now say when a Workflow Producer writes and when it only closes. * Synced the stream sample with the read and Record reshape. The API dropped the read position, the close item and the client epoch. The `read` parameter needs #407 to be `bytes`. * fix: make a `list<u8>` resource method parameter the proto `bytes` of its field (#407) * Typed a `list<u8>` method parameter of a proto `bytes` field as bytes. A proto-backed record reads that field as `bytes`, so a list of integers could not fill it. * Regenerated the stream sample with the bytes parameter. The stream `read` takes its `from_offset` as `bytes` in Python, TypeScript and .NET. * Filled an unset optional parameter from its field's source in Python. The generated code passed an optional parameter as it was and dropped the source of the field. A caller that passed nothing then sent no value for a required field. * Filled an unset optional parameter from its source in TypeScript. A resource method passed an optional parameter as it was and dropped the source of the field. A caller that passed nothing then sent no value for a required field. * Filled an unset optional parameter from its source in .NET. A resource method passed an optional parameter as it was and dropped the source of the field. The complete request also took a flattened list parameter as it was, so it did not compile. * Gave the stream sample's append and close an optional producer. The sample shows the source fallback in its generated Python, TypeScript and .NET. Its client source helper is `default_producer`, since it picks the Producer by where the code runs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the Store rename and the append result. The API renamed `backend` to `store` in the Stream ref and dropped the `append` result transform. So `append` returns the `AppendRecordsResponse` in every target. The two optional `producer` parameters of the API's WIT are not in this sync, since they need a generator change. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Said in the sample's Producer docs that a call can pass its own. The docs of both `producer` fields match the API's WIT, which has the two optional parameters. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the sequence parameter of append. The API gave `append` an optional `sequence`, so a caller can number a batch and the Store can dedupe a resent one. The optional `producer` parameter still waits for the generator change. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the offset and Workflow Producer comments. The API now says that two offsets of one Stream compare bytewise. It also says that Workflow code on a Store writes through a Local Activity. Only comments changed, so the samples only move in docs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Imported temporalio.client only for type checking in Python clients. (#409) The SDK's `Client` inherits the generated extension, so `temporalio.client` imports the generated package. A run-time import back made importing the package first a circular import. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Filled an unset optional parameter from its field's default in Python. A resource method passed an optional parameter as it was and dropped the default of the field. A source is a helper value and never an argument, so it no longer fills an optional parameter. * Took the source fallback out of TypeScript resource methods. A resource method hands an optional parameter to its operation, which applies the field's default. A source is a helper value and never an argument, so it no longer fills an optional parameter. * Took the source fallback out of .NET resource methods. The stream sample's resource methods go through the options type, which applies the default. A source is a helper value and never an argument, so it no longer fills an optional parameter. * Gave the stream sample's Producer a default instead of a source. A default is an omittable argument, which the optional `producer` of `append` and `close` needs. The guide and the changelog describe the default rule for resource methods. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Told why the stream sample offset comment matters. --------- Co-authored-by: Tim Conley <timothy.conley@temporal.io>
#410) * fix(typescript): infer Long from protobuf resource bindings * feat: let a resource declare its type parameters A resource with `@nexus.type-parameter "<alias>"` is generic in Python, TypeScript and .NET. An operation that returns it, or that its method binds, takes a runtime type for each type parameter of its output, so a response decodes its generic values. * test: drop the low-value type parameter rejection fixtures The kept fixtures cover an undeclared type parameter, a generic resource in a record field or in a result with no proto message, and a language that refuses it. The others test directive syntax, rare name clashes or another copy of a kept check. * docs: shorten the generic resources section of the guide * test: assert the full message of each type parameter rejection * docs: say why a generic handle keeps its runtime types Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: infer the type parameters of a resource A resource that uses a type parameter, also through its records, is generic in it, as a record is. So a resource needs no annotation, and `@nexus.type-parameter` is valid only on a type alias again. * fix(python): decode generic resources from their type arguments (#367) * fix: infer resource type parameters from results in declaration order Only a returned resource gives its type parameters, as the guide says, so a parameter or field that holds one doesn't. The declaration order doesn't change when a method moves. * fix: refuse a generic resource anywhere in a record field A list, map, tuple, result or variant case builds the resource from the record's message too, so it gets no type argument either. * fix: say that a direct generic resource result isn't supported yet The old reason said a proto message holds the type arguments, which it doesn't. * fix: refuse runtime type names that collide with each other or a method Two type parameters such as `body` and `body-t` give the same runtime type name, and a method can take it too. Either one makes a duplicate parameter or member. * fix(python): keep the runtime output type in the System Nexus dispatch The typed hook of a generic response took only the request, so the terminal decoded `ReadRecordsResponse[Turn]` with the plain class and lost `Turn`. * fix(dotnet): send generic System Nexus operations to the generic hook C# generics are invariant, so the `<object>` checks refused a `Stream<Turn>` call. The dispatch now compares generic type definitions and keeps the call's types. * fix(python): read the type argument of a subclassed generic handle A subclass such as `class TurnStream(Stream[Turn])` has no `__orig_class__`, so the handle reads `__orig_bases__` too. A factory tests `body_type is None`, since a class can be falsy. * test: cover a typed TypeScript read and a bare Python handle class The TypeScript stream test now decodes a body with a runtime type. The Python test decodes to the bare handle class, since a generic alias fails for another reason. * chore: fold the generic resource changelog and describe the fixes No release has the Python constructor change, so it belongs with the added entry. The fixed entry now describes the System Nexus dispatch fixes. * docs: reword the generic resource comments and wrap long lines A few comments used "name" as a verb, ran over two lines or over 100 columns. The guide's first paragraph read as one long sentence. * chore: fold the generic resource changelog into one entry * chore: trim the generic resource tests and docs Drop the tests that repeat another test or restate generated code, and keep the ones that guard behavior. Shorten the guide section and the comments. * feat(python): take a resource's runtime types as a subscript Python can't subscript a function, so each generic call is an object whose subscript binds the types, as in `client.get_stream[Turn](ref=ref)`. A factory's optional field defaults to `None`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: close a resource at the end of async with `@nexus.close` marks the method that ends a resource. Python calls it at the end of `async with`, and a call that returns the resource returns a `ResourceCall`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: add @nexus.varargs and flatten a list of one-field records `@nexus.varargs` makes a resource method take a list's items as positional arguments, and a list of a flattened one-field record takes that field's values. The Stream's publish takes values and returns the last offset. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat(python): add a Workflow resource factory Workflow code gets a handle on an existing Stream with `get_stream[T]`, from the fields of its ref, with no call. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat(python): add RPC options to client calls Each client call and client resource method takes `rpc_metadata` and `rpc_timeout`, and passes them to the `start_operation` of its Nexus client. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: flatten the record of one request field `@nexus.flatten-in-api` on a request field makes that call take the fields of the field's record, and the models keep the record. The Stream create takes the id, the owner and the backend of its ref this way, and the other Stream calls keep `ref`. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * feat: limit `@nexus.api-omit` to some targets A field can leave the API of one target and take a source in another. A call of a target that omits a required list sends it empty. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): sync the stream sample with StreamWriter temporalio/api#895 moves the Producer and the withdrawal fields of a write into one StreamWriter. The sample WIT also takes the final directives of the open nexgen PRs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): sync the stream sample with the owner and withdrawal removal This syncs the stream sample with the owner and withdrawal removal. The WIT and the stream protos drop the owner, the read-side withdrawal machinery and the `OWNER_*` close statuses, and add the `StreamClose` read item with the kind-oneof `Producer`. The Python, TypeScript and .NET bindings and samples are regenerated, and the hand-written support and tests follow. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * chore(samples): format the stream test with oxfmt Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Omitted an operation's public function with `@nexus.api-omit`. A top-level `func` with `@nexus.api-omit` keeps the operation registered and bound to its resource, but drops the public top-level function and its export. A caller reaches the operation through the resource. On a function the directive omits the workflow top-level function, so its `targets` must include `workflow`. The stream sample omits four operations, so `append-records`, `close-stream`, `read-records` and `describe-stream` have no public function and the `Stream` resource is the way in. `create-stream` stays public, since the Workflow has no other way to create a Stream. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Made a `list<u8>` case of a proto oneof the proto `bytes`. A variant with `@nexus.proto` reads an unnamed `list<u8>` case as the proto `bytes`, the same rule as a field of a proto-backed record. Another width still fails against a `bytes` member. The stream sample's read position is a oneof of a `bytes` offset and a start enum, which showed two more gaps. An enum case kept its proto type, so its WIT enum was pruned. It is now planned as the bound enum, like a record case. A resource method parameter named after the oneof was dropped, since the request plan walked the oneof's members. It now binds to the record's oneof field. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the latest temporalio/api#895. The WIT and protos are those of temporalio/api#895 at b0b2657e. The WIT adds the `@nexus.api-omit` of #405 and the `@nexus.generic` of #399, which the API WIT doesn't have yet. A directive that nexgen doesn't support yet stays a TODO comment, and an offset is an inline `list<u8>`, since nexgen reads only that as `bytes`. * Synced the stream sample with the removal of `create_wait`. temporalio/api#895 at 0ac6b717 drops `create_wait` from a read and renumbers the later fields, so the WIT, the protos and their bindings follow it. * Synced the stream sample with the Stream writes and items reshape. temporalio/api#895 at 23fac086 drops `StreamWriter` for a plain `producer`, moves the offset into the read items, and makes the required fields required. The WIT is the API's as is, and the support helpers are the `producer` sources that it names. * Synced the stream sample with the close offset and namespace comments. * Synced the stream sample with the one fence rule for every Producer. * Synced the stream sample with the unknown writer removal. * Synced the stream sample with the Workflow Producer comments. The API comments now say when a Workflow Producer writes and when it only closes. * Synced the stream sample with the read and Record reshape. The API dropped the read position, the close item and the client epoch. The `read` parameter needs #407 to be `bytes`. * fix: make a `list<u8>` resource method parameter the proto `bytes` of its field (#407) * Typed a `list<u8>` method parameter of a proto `bytes` field as bytes. A proto-backed record reads that field as `bytes`, so a list of integers could not fill it. * Regenerated the stream sample with the bytes parameter. The stream `read` takes its `from_offset` as `bytes` in Python, TypeScript and .NET. * Filled an unset optional parameter from its field's source in Python. The generated code passed an optional parameter as it was and dropped the source of the field. A caller that passed nothing then sent no value for a required field. * Filled an unset optional parameter from its source in TypeScript. A resource method passed an optional parameter as it was and dropped the source of the field. A caller that passed nothing then sent no value for a required field. * Filled an unset optional parameter from its source in .NET. A resource method passed an optional parameter as it was and dropped the source of the field. The complete request also took a flattened list parameter as it was, so it did not compile. * Gave the stream sample's append and close an optional producer. The sample shows the source fallback in its generated Python, TypeScript and .NET. Its client source helper is `default_producer`, since it picks the Producer by where the code runs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Imported temporalio.client only for type checking in Python clients. The SDK's `Client` inherits the generated extension, so `temporalio.client` imports the generated package. A run-time import back made importing the package first a circular import. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the Store rename and the append result. The API renamed `backend` to `store` in the Stream ref and dropped the `append` result transform. So `append` returns the `AppendRecordsResponse` in every target. The two optional `producer` parameters of the API's WIT are not in this sync, since they need a generator change. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Said in the sample's Producer docs that a call can pass its own. The docs of both `producer` fields match the API's WIT, which has the two optional parameters. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the sequence parameter of append. The API gave `append` an optional `sequence`, so a caller can number a batch and the Store can dedupe a resent one. The optional `producer` parameter still waits for the generator change. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Synced the stream sample with the offset and Workflow Producer comments. The API now says that two offsets of one Stream compare bytewise. It also says that Workflow code on a Store writes through a Local Activity. Only comments changed, so the samples only move in docs. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Left a resource method out of one Python target with `@nexus.api-omit`. A resource method could not be left out of one target, and the stream sample needs a Workflow `Stream` with no `append` and no `close`. Without its close method, a target's resource is not a context manager, and a method that no target keeps is refused. * Left an api-omitted resource method out of the TypeScript Workflow API. A TypeScript resource method calls the Workflow API, so a method omitted for `workflow` goes from the class. Its request helper goes too when no other method used it. * Left an api-omitted resource method out of the .NET Workflow API. A .NET resource method calls the Workflow API, so a method omitted for `workflow` goes from the class. The request overload that only it called becomes private. * Refused a resource method omitted for the Workflow target in Go. Go has no sample that would show its output without the method, so it refuses the directive as it refuses `@nexus.varargs`. A method omitted only for `client` changes nothing in Go. * Left `append` and `close` out of the stream sample's Workflow `Stream`. In this release Workflow code can't append to a Stream or close it, so the Workflow target omits both methods. The client `Stream` keeps them, and the Workflow `create_stream` is a plain call. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Imported temporalio.client only for type checking in Python clients. (#409) The SDK's `Client` inherits the generated extension, so `temporalio.client` imports the generated package. A run-time import back made importing the package first a circular import. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Filled an unset optional parameter from its field's default in Python. A resource method passed an optional parameter as it was and dropped the default of the field. A source is a helper value and never an argument, so it no longer fills an optional parameter. * Took the source fallback out of TypeScript resource methods. A resource method hands an optional parameter to its operation, which applies the field's default. A source is a helper value and never an argument, so it no longer fills an optional parameter. * Took the source fallback out of .NET resource methods. The stream sample's resource methods go through the options type, which applies the default. A source is a helper value and never an argument, so it no longer fills an optional parameter. * Gave the stream sample's Producer a default instead of a source. A default is an omittable argument, which the optional `producer` of `append` and `close` needs. The guide and the changelog describe the default rule for resource methods. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Told why the stream sample offset comment matters. * Rewrote the architecture note on omitted resource methods. The note listed type names and did not say what happens. It says what the directive does and that planning owns the choice. Co-authored-by: Tim Conley <timothy.conley@temporal.io> * Cut the architecture note on omitted resource methods to its gist. --------- Co-authored-by: Tim Conley <timothy.conley@temporal.io>
Workflow code has no built-in append, close or read in this release, so the Workflow target omits the three methods and the client target keeps them. The two `producer` parameters take their value from `@nexus.default`. The nexgen pin moves to `main`, which has both rules.
The vendored Stream WIT matches temporalio/api#895 again. Its Workflow target omits the three methods, and the two `producer` parameters take their value from `@nexus.default`.
The vendored Stream WIT follows temporalio/api#895, whose Workflow target omits append, close and read.
These are the files of the Stream API that temporalio/api#895 adds, byte for byte. An update of the API subtree replaces them once that PR merges.
Workflow code appends through a Local Activity and closes through an Activity, so the Workflow target has both methods again. It still omits read.
The Stream WIT of temporalio/api#895 has append and close on the Workflow target again, and this is its copy.
The Stream WIT of temporalio/api#895 has append and close on the Workflow target again, and this is its copy.
The next commit adds three methods to the Stream WIT that use the directive, and no commit of nexgen `main` has it yet.
Client, Activity and Workflow code read a Stream with these three methods. The SDK implements each one, so no operation binds them, and the Workflow Stream still has no page read.
What changed?
This PR adds the Stream service messages and the Stream service WIT, marked experimental.
Producerand epoch, so a newer attempt fences an older one. A read gives the Records in order and tells of a close withdone, and an offset is opaque, orderedbytes.CreateStreamfails when the Stream exists, unlessid_conflict_policyisUSE_EXISTING. A retry with the samerequest_idgets the Stream it made.Streamhastake,filterandreducefor client, Activity and Workflow code. The SDK implements each one overReadRecords, so no operation binds them. The WorkflowStreamhas no page read.mainand then a release have the directives they use. There's noservice.protoyet.Why?
The SDKs and the Redis Store already use these messages, so they need one home before a server serves them. temporalio/sdk-rust#1733 has a copy of these files only to build, so this PR is the place to review the messages.
Breaking changes
None. These are new files.
Server PR
None needed for now.