Repository navigation
[AI-752] Add Arize tracing samples - #365
Merged
Merged
Conversation
…e, screenshots from live runs
DABH
marked this pull request as draft
September 11, 2026 17:56
This was referenced Sep 11, 2026
The OpenAI Agents plugin now ships as the standalone temporalio-openai-agents package (import path temporalio.openai_agents); temporalio 1.33's openai-agents extra pins an older openai-agents and no longer resolves alongside it. Switch the dependency group and imports, require temporalio 1.33, and raise the OpenInference agents instrumentation floor to 2.x. The single-process span-parenting bug (sdk-python#1852) is fixed in ai-integrations#26 but 1.0.0 predates it, so the --role split stays; reword the notes accordingly. Also bump Phoenix to 20.16.0 and state that the Arize AX exporter path was not exercised live.
DABH
marked this pull request as ready for review
October 1, 2026 16:32
temporalio-openai-agents 1.1.0 carries the span-parenting fix for running workflows and activities in one worker, but it requires openai 3, and the root lock stays on openai 2 for the LiteLLM samples (litellm caps openai<3). Give the sample its own uv project, like openai_agents/mcp_v2, and add it to the standalone CI matrix. With the fix in place, drop the --role split, the workflow-only plugin switch, the detach-context log filter, and the known-issue notes; update the trace diagrams and the screenshot from a single-process run.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The activity heartbeat configuration can cause duplicate LLM calls, and the verifier can report unreliable ingestion and deduplication results.
Review effort: Balanced
Findings: 1
Open (3)
What changed in this PR
Adds a standalone Arize tracing sample demonstrating replay-safe Temporal traces for standard OpenAI activities and the OpenAI Agents SDK.
Changes:
- Adds shared OpenTelemetry/OpenInference configuration and Phoenix verification tooling.
- Adds two ticket-triage scenarios with offline integration tests.
- Integrates the sample into CI, documentation, and ownership configuration.
| File | Description |
|---|---|
.github/CODEOWNERS |
Assigns sample ownership. |
.github/workflows/ci.yml |
Adds standalone sample CI. |
README.md |
Lists the Arize sample. |
uv.lock |
Updates OpenInference packages. |
arize_tracing/.env.example |
Documents environment configuration. |
arize_tracing/README.md |
Provides the main runbook. |
arize_tracing/pyproject.toml |
Defines the standalone project. |
arize_tracing/uv.lock |
Locks standalone dependencies. |
arize_tracing/telemetry.py |
Configures tracing and enrichment. |
arize_tracing/verify_trace.py |
Verifies Phoenix traces. |
arize_tracing/phoenix/docker-compose.yml |
Runs local Phoenix. |
arize_tracing/phoenix-retry-attempts.png |
Shows retry tracing. |
arize_tracing/phoenix-ticket-triage.png |
Shows activity-based tracing. |
arize_tracing/phoenix-ticket-triage-agents.png |
Shows agent tracing. |
arize_tracing/ticket_triage/__init__.py |
Marks the scenario package. |
arize_tracing/ticket_triage/README.md |
Documents the activity scenario. |
arize_tracing/ticket_triage/activities.py |
Implements triage activities. |
arize_tracing/ticket_triage/starter.py |
Starts and traces workflows. |
arize_tracing/ticket_triage/worker.py |
Runs the activity worker. |
arize_tracing/ticket_triage/workflows.py |
Implements the triage workflow. |
arize_tracing/ticket_triage_agents/__init__.py |
Marks the agents package. |
arize_tracing/ticket_triage_agents/README.md |
Documents the agents scenario. |
arize_tracing/ticket_triage_agents/plugin.py |
Configures the Agents plugin. |
arize_tracing/ticket_triage_agents/starter.py |
Starts agent workflows. |
arize_tracing/ticket_triage_agents/worker.py |
Runs the agents worker. |
arize_tracing/ticket_triage_agents/workflows.py |
Implements agent-based triage. |
arize_tracing/tests/__init__.py |
Marks the test package. |
arize_tracing/tests/conftest.py |
Defines standalone fixtures. |
arize_tracing/tests/helpers.py |
Provides span-tree helpers. |
arize_tracing/tests/test_ticket_triage.py |
Tests activity tracing and replay. |
arize_tracing/tests/test_ticket_triage_agents.py |
Tests agent tracing and replay. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
classify_ticket heartbeated only in the --slow-classify loop, so a model response longer than the 10-second heartbeat timeout would have been retried while the first request was still in flight. Run the call through a helper that heartbeats until it completes. verify_trace.py now waits for the root span plus three consecutive polls with an unchanged span count before checking, and no longer claims to detect duplicate exports: Phoenix drops same-id inserts, so only the in-memory tests can observe re-emission.
brianstrauch
reviewed
Oct 6, 2026
moetemp
approved these changes
Oct 7, 2026
|
AX test results and one reset issue are on the docs PR: temporalio/documentation#5306 (comment) |
# Conflicts: # .github/CODEOWNERS
Reuse one lazily created OpenAI client per worker and close it on shutdown instead of building an unclosed client per attempt. Rename the verifier's --expect-attempt flag to --surviving-attempt, validate ACTIVITY=N values, and only finish polling once the root span and one completed-run span per expected run are present, retrying the checks until the deadline. Document what Arize observed against an AX space: a retried activity marks the trace as an error there; stopping a worker mid-workflow logs a harmless "Failed to detach context" traceback; and after a reset that reapplies the update, the reapplied HandleUpdate span reuses the original run's RunWorkflow span id, which Phoenix drops and AX shows as a duplicate subtree. The verifier now warns about the dropped span. Drop the note that the AX path was not exercised.
|
@DABH @justonwestern, could you review or route this sample alongside documentation#5306? I rechecked SDK #2070: it is closed, and the alternative fix #2081 merged, but the latest published Python SDK release I can see predates that fix. Please confirm the intended SDK version/workaround wording before this pair ships. Dat asked us to check review readiness; this sample is already non-draft. The paired docs PR is still partner-owned draft, so please mark it ready when the remaining owner-side requirements are satisfied. No partner branch or runtime claim was changed. |
The single-batch activation default in 1.36.0 applies every job in a workflow task before handlers run, so a reset's reapplied update no longer reuses the original run's RunWorkflow span ID. Drop the README and docs caveat, make the verifier fail when a run's HandleUpdate span is missing, and set plugins on the connect config instead of as a keyword, which 1.36.0's config type now flags. 1.36.0 requires Python 3.11, so the sample does too and CI skips it on 3.10.
Fold the two scenario READMEs into the main one, share the Phoenix helpers between telemetry.py and verify_trace.py, restructure the tests around one runner, trim docstrings and comments, and stop touching the root lock, which the standalone project does not need.
DABH
enabled auto-merge (squash)
November 12, 2026 18:58
The 3.10 Ubuntu root test job has stalled on six runs of this branch, each time right after the Google GenAI streaming test, and a cancelled job leaves no trace. pytest-timeout turns the stall into a failure with every thread's stack, which is what the next run needs to show.
pytest-timeout's thread method, the only one available on Windows, makes the Sentry interceptor test capture a stray third event; the same test passed on Windows before the timeout was added. The hangs this guards against happen on the Linux runner, so the timeout stays on there.
This reverts commit 81cf018.
This reverts commit a8f518b.
ubuntu-latest moved to Ubuntu 26.04 in November 2026. Since then the Python 3.10 root test run stalls between tests on most runs, while main's last run on 24.04 passed. Pin the runner but keep the job name so the required status checks still match. (cherry picked from commit c810e7f)
This reverts commit 5ad49c5.
GeminiTestServer.plugin() builds a google.genai.Client per call and nothing closed it. The client's finalizer schedules aclose() on whatever event loop is running when garbage collection reaches it, which showed up in CI as closes running inside workflow sandboxes and, on Python 3.10, as the root test run stalling right after the streaming test. A fixture now closes every client the tests create, on the test's own loop, and the streaming test closes its subscription instead of leaving the generator to the collector.
DABH
commented
Nov 13, 2026
| project: ${{ fromJSON(needs.detect-standalone-changes.outputs.projects) }} | ||
| exclude: | ||
| # temporalio 1.36.0, which arize_tracing needs, requires Python 3.11+ | ||
| - project: arize_tracing |
Contributor
Author
There was a problem hiding this comment.
TODO (@DABH ): revert this once CI matrix is updated (asked today and PR should be forthcoming from SDK team)
moetemp
approved these changes
Nov 13, 2026
moetemp
disabled auto-merge
November 13, 2026 07:07
moetemp
enabled auto-merge (squash)
November 13, 2026 07:08
tconley1428
approved these changes
Nov 13, 2026
DABH
added a commit
to temporalio/documentation
that referenced
this pull request
Nov 13, 2026
temporalio/samples-python#365 merged, so the five snippets on the Arize page now come from main instead of the pre-filled copies. The sample's condensing pass shortened docstrings and comments; the prose around each snippet still matches.
Duncanma
added a commit
to temporalio/documentation
that referenced
this pull request
Nov 13, 2026
* Document the OpenTelemetry plugin on the Python observability page * Add the Arize integration page for Python * Arize page: refresh snippets, note the single-process worker caveat, use a live trace screenshot * Update the Arize page for temporalio-openai-agents and Phoenix 20.16 Install the agents variant from the standalone package, describe the single-worker span-parenting bug as fixed upstream but unreleased, note that the Arize AX exporter settings were not exercised live, and add Arize's evaluation guides to Resources. * Require temporalio-openai-agents 1.1.0 on the Arize page 1.1.0 ships the span-parenting fix, so the caution about running Workflows and Activities in separate Worker processes and the note about the "Failed to detach context" log filter no longer apply. The install step now asks for 1.1.0 or later, and the agents plugin snippet matches the sample without its worker-role switch. * Refresh the activity snippet and add heartbeat guidance on the Arize page * Fold Arize's AX test results into the Arize page Arize validated the sample against an AX space, so the note that the AX path was not exercised is gone. Add what they observed: AX lists a trace with a failed attempt as an error, a stopping Worker logs a harmless "Failed to detach context" traceback, and after a reset the reapplied update's HandleUpdate span reuses the original RunWorkflow span id, which Phoenix drops and AX shows as a duplicated subtree. * Link the reset span-id issue from the Arize page * Arize page: the reset span-id defect is fixed in temporalio 1.36.0 Require 1.36.0 in the install step, replace the known-defect paragraph with the fixed-version note, and match the worker snippet to the sample. * Sync the Arize snippets from the merged sample temporalio/samples-python#365 merged, so the five snippets on the Arize page now come from main instead of the pre-filled copies. The sample's condensing pass shortened docstrings and comments; the prose around each snippet still matches. * Link the Arize agent-evaluation guide at its canonical URL Dat suggested the current address on the PR; the old one still redirects. * Tighten Arize page wording and pin openai-agents OTel version - Use Workflow ID, drop the validation process note, and shorten the reset paragraph to the version requirement and its symptom - Move the Agents bridge Public Preview note into a caution at the top of its section - Tie the lost-spans window to schedule_delay_millis instead of the sample's 500 ms - Rename the heading to Workflow retries so it doesn't read as Activity retries - Require temporalio-openai-agents 1.1.0 on the OpenAI Agents OpenTelemetry install line, with the span parenting warning --------- Co-authored-by: Duncan Mackenzie <duncanma@duncanmackenzie.net> Co-authored-by: Duncan Mackenzie <duncan.mackenzie@temporal.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds
arize_tracing, a sample that sends Temporal workflow traces to Arize Phoenix or Arize AX. It uses the SDK's existing OpenTelemetry plugin, an OTLP/HTTP exporter, and the OpenInference conventions Arize reads - no new plugin and no Arize SDK.Two scenarios share one telemetry module:
ticket_triagekeeps LLM calls in activities with a human approval update, andticket_triage_agentsruns the same flow on the OpenAI Agents SDK. A small span processor stamps Temporal's spans with an OpenInference kind, the workflow ID as the Arize session, and the activity attempt.verify_trace.pychecks a trace through the Phoenix REST API, and the README walks through replay, restart, retry, and reset.The sample is its own uv project (temporalio 1.36.0, Python 3.11+, openai 3); its tests need no Phoenix or API key. Arize validated the sample against an AX space; the reset defect they found is fixed in temporalio 1.36.0.
The PR also closes the
google.genai.Clientobjects the Gemini tests create. Their finalizer scheduled a close on whatever event loop was running at garbage collection, which CI logged as errors inside workflow sandboxes.