Skip to content

feature: add fabric topology checker and viewer to validation layer - #561

Draft
nrspruit wants to merge 1 commit into
oneapi-src:masterfrom
nrspruit:fabric_toplogy_viewer
Draft

nrspruit wants to merge 1 commit into
oneapi-src:masterfrom
nrspruit:fabric_toplogy_viewer

Conversation

@nrspruit

Copy link
Copy Markdown
Contributor

Add a validation layer checker, enabled with
ZEL_ENABLE_FABRIC_TOPOLOGY_CHECKER=1, that records what the fabric looks like and how a workload moves data across it, for debugging scale-up and scale-out setups.

Each process writes a JSON Lines stream to ZEL_FABRIC_TOPOLOGY_OUTPUT:

  • Topology: once the loader completes zeInitDrivers (or zeDriverGet), the checker enumerates devices and sub-devices, PCI addresses, the P2P matrix and zeFabricVertex/EdgeExp data, with fabric_query diagnostics when no links are reported.
  • Traffic: copies and fills attributed to (executing device, source, destination), counted at append for immediate lists and on every execute for regular lists, aggregated per ZEL_FABRIC_TOPOLOGY_INTERVAL_MS.
  • IPC: export/import records keyed by handle hash and dma-buf inode so streams from several processes and hosts can be joined.

scripts/fabric_topology_viewer.py merges streams from any number of processes and hosts into a text summary and a self-contained HTML report (topology, throughput and shared-memory views), with --watch for live updates.

Add a null-driver test covering topology at init, copy/fill attribution, regular list replay and IPC join keys.

Add a validation layer checker, enabled with
ZEL_ENABLE_FABRIC_TOPOLOGY_CHECKER=1, that records what the fabric
looks like and how a workload moves data across it, for debugging
scale-up and scale-out setups.

Each process writes a JSON Lines stream to ZEL_FABRIC_TOPOLOGY_OUTPUT:
- Topology: once the loader completes zeInitDrivers (or zeDriverGet),
  the checker enumerates devices and sub-devices, PCI addresses, the
  P2P matrix and zeFabricVertex/EdgeExp data, with fabric_query
  diagnostics when no links are reported.
- Traffic: copies and fills attributed to (executing device, source,
  destination), counted at append for immediate lists and on every
  execute for regular lists, aggregated per
  ZEL_FABRIC_TOPOLOGY_INTERVAL_MS.
- IPC: export/import records keyed by handle hash and dma-buf inode so
  streams from several processes and hosts can be joined.

scripts/fabric_topology_viewer.py merges streams from any number of
processes and hosts into a text summary and a self-contained HTML
report (topology, throughput and shared-memory views), with --watch
for live updates.

Add a null-driver test covering topology at init, copy/fill
attribution, regular list replay and IPC join keys.

Signed-off-by: Neil R. Spruit <neil.r.spruit@intel.com>
@github-actions

Copy link
Copy Markdown

Level Zero loader ABI check

last-release (v1.35.0): ✅ no ABI problems. Report

Creating compatibility report ...
Binary compatibility: 100%
Source compatibility: 100%
Total binary compatibility problems: 0, warnings: 0
Total source compatibility problems: 0, warnings: 0

master (master): ✅ no ABI problems. Report

Creating compatibility report ...
Binary compatibility: 100%
Source compatibility: 100%
Total binary compatibility problems: 0, warnings: 0
Total source compatibility problems: 0, warnings: 0

Workflow run

If an ABI change is intended, a reviewer with write access can accept it: leave an approval review on the PR head commit with the keyword abi-failure-approved in the review body. That re-runs the check.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant