Skip to content

pyzes: load the Level Zero loader by SONAME on Linux - #559

Open
weifengpy wants to merge 2 commits into
oneapi-src:masterfrom
weifengpy:fix/pyzes-linux-loader-soname
Open

weifengpy wants to merge 2 commits into
oneapi-src:masterfrom
weifengpy:fix/pyzes-linux-loader-soname

Conversation

@weifengpy

Copy link
Copy Markdown

Problem

On Linux, pyzes loads the Level Zero loader from a hardcoded path:

libName = "/usr/lib/x86_64-linux-gnu/lib" + libName + ".so.1"

When the loader the runtime actually uses lives somewhere else in the standard search path (for example a newer loader installed under /usr/local or shipped with oneAPI), pyzes pins the distro copy instead. A zesInit issued through that mismatched loader copy leaves the process in a state where the oneAPI Unified Runtime no longer enumerates the Level Zero platform at all. Any consumer that queries Sysman before initializing the runtime is then broken for the rest of the process lifetime.

Reproduction

Machine: Intel Arc B580 (Battlemage), NEO 26.40.40196.8, two coexisting loaders:

  • /usr/lib/x86_64-linux-gnu/libze_loader.so.1 — 1.16 (Ubuntu libze1 package)
  • /usr/local/lib/libze_loader.so.1 — 1.35 (built from this repo), the copy the SYCL runtime resolves

PyTorch (both the 2.15.0+xpu wheel and a source build) routes torch.xpu.device_count() through pyzes before XPU initialization:

import torch
torch.xpu.is_available()            # True
torch.xpu.device_count()            # 1 (via pyzes / Sysman)
torch.xpu.get_device_properties(0)  # RuntimeError: No XPU devices are available.
                                    # preceded by "XPU device count is zero!"

Calling get_device_properties first works fine. A minimal SYCL probe confirms the mechanism outside PyTorch: after dlopen of the distro loader plus zesInit/zesDriverGet/zesDeviceGet through it, sycl::platform::get_platforms() no longer lists the Level Zero platform; doing the same through the runtime's loader, or not calling zesInit at all, leaves enumeration intact. Replacing the distro loader file's contents with the newer loader also makes the failure disappear, confirming the trigger is which loader copy serves the Sysman calls.

Fix

Load libze_loader.so.1 by SONAME on Linux so the dynamic linker resolves it through the standard search path (LD_LIBRARY_PATH, the ld.so cache, default directories) — the same copy every other consumer gets. The distro location remains reachable that way, so no supported configuration loses access to the loader.

Testing

On the system above, with the distro 1.16 loader restored so both loaders coexist:

  • Before the fix: the PyTorch sequence above fails as shown.
  • After the fix (patched pyzes on PYTHONPATH): pyzes.gpuLib._name is libze_loader.so.1, the same sequence returns the B580 from get_device_properties, and an XPU matmul runs correctly.

This PR was prepared with the assistance of an AI coding tool and reviewed by the author.

_LoadZeLibrary hardcoded /usr/lib/x86_64-linux-gnu/libze_loader.so.1
on Linux. When a different (e.g. newer) loader is the one the Level
Zero runtime resolves through the standard search path - common when
oneAPI or a locally built loader is installed under /usr/local or
/opt - pyzes pins the distro copy instead. A zesInit issued through
that mismatched loader leaves the process in a state where the
Unified Runtime no longer enumerates the Level Zero platform, which
breaks downstream consumers that query Sysman before initializing
the runtime. torch.xpu.device_count() is one such consumer: after it
runs, torch.xpu.get_device_properties() fails with "No XPU devices
are available" on a healthy Intel Arc B580 system.

Load libze_loader.so.1 by SONAME instead, so the dynamic linker
picks the same loader copy as every other consumer. The distro
location remains reachable through the standard search path.

This commit was authored with the assistance of an AI coding tool.

Signed-off-by: Wei (Will) Feng <134637289+weifengpy@users.noreply.github.com>
@weifengpy

Copy link
Copy Markdown
Author

Downstream report with the full investigation and reproduction details: pytorch/pytorch#204338

@guangyey

Copy link
Copy Markdown

@shubskmr Could you please help review this code change?

@shubskmr shubskmr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@weifengpy

Copy link
Copy Markdown
Author

thanks for taking a look. seems it needs more approval in order to land "At least 1 approving review is required by reviewers with write access."

@guangyey

Copy link
Copy Markdown

@rwmcguir just a gentle ping.

@rwmcguir

Copy link
Copy Markdown
Contributor

Why is the version being hard coded, i.e. the ".so.1", this will break when level-zero goes to Major version 2.

I'd suggest converting this to some kind of dynamic extraction.
A sample of this exists in ./scripts/templates/helper.py get_major_ver(cls, name).

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Level Zero loader ABI check

last-release (v1.35.0): ✅ no ABI problems. Report

Creating compatibility report ...
Binary compatibility: 100%
Source compatibility: 100%
Total binary compatibility problems: 0, warnings: 0
Total source compatibility problems: 0, warnings: 0

master (master): ✅ no ABI problems. Report

Creating compatibility report ...
Binary compatibility: 100%
Source compatibility: 100%
Total binary compatibility problems: 0, warnings: 0
Total source compatibility problems: 0, warnings: 0

Workflow run

If an ABI change is intended, a reviewer with write access can accept it: leave an approval review on the PR head commit with the keyword abi-failure-approved in the review body. That re-runs the check.

The previous change hardcoded the major version in libze_loader.so.1, which would break when Level Zero moves to major version 2, as pointed out in review of oneapi-src#559.

Resolve the SONAME at runtime with ctypes.util.find_library(), which queries the system ldconfig cache and returns the current SONAME (libze_loader.so.1 today, the .so.2 name after a major bump). Fall back to the unversioned libze_loader.so if find_library() returns nothing. Loading still happens by SONAME through the standard search path, so the loader copy the runtime uses still wins over a pinned distro copy.

The suggested helper.py get_major_ver() parses a $X_MAKE_VERSION(...) spec string at code-generation time; that input is not available to pyzes at runtime, so find_library() is the runtime equivalent of dynamic extraction.

This commit was authored with the assistance of an AI coding tool.

Signed-off-by: Wei (Will) Feng <134637289+weifengpy@users.noreply.github.com>
@weifengpy

Copy link
Copy Markdown
Author

@rwmcguir Thanks for catching this — you are right, hardcoding .so.1 would break on a major version 2 bump.

Fixed in 87d9bb9: the Linux path no longer constructs the SONAME from a hardcoded major version. It now resolves it dynamically at runtime:

libName = ctypes.util.find_library(libName) or "lib" + libName + ".so"

ctypes.util.find_library('ze_loader') asks the system (the ldconfig cache) for the loader's current SONAME, so it returns libze_loader.so.1 today and would return the .so.2 name after a major bump, with no code change. If it returns nothing, we fall back to the unversioned libze_loader.so, which the dynamic linker still resolves through the same standard search path. Loading itself is unchanged: still by SONAME through the standard search path, so the loader copy the runtime uses still wins over a pinned distro copy.

On helper.py get_major_ver(cls, name): I looked at it — it extracts the major by regex-parsing a $X_MAKE_VERSION(major, minor) spec string (RE_VERSION) at code-generation time. That spec string is not available to pyzes at runtime, so I used find_library() as the runtime equivalent of the same dynamic-extraction idea rather than copying the parser. Happy to switch approaches if you would prefer a different mechanism.

Verification on the original dual-loader system (distro 1.16 loader in /usr/lib/x86_64-linux-gnu and the self-built 1.35 loader in /usr/local, both present):

  • ctypes.util.find_library('ze_loader') returns 'libze_loader.so.1', resolved dynamically — the patched source contains no .so.1 literal at all.
  • Importing the patched pyzes loads libze_loader.so.1 by that resolved name.
  • Through PyTorch: torch.xpu.device_count() (which routes through pyzes) followed by torch.xpu.get_device_properties(0) returns the Intel Arc B580, and an XPU matmul runs correctly — i.e. the original mismatch failure does not recur.

@rwmcguir

Copy link
Copy Markdown
Contributor

Thanks.... A latent thought before I approve as I am not as familiar with the runtime python bindings.

For normal runtimes the .so package is only installed with the -devel package from Linux, for a Python environment would this be a typical scenario, i.e. to have a -devel package installed? Or only the runtime package which would have the .so.Major and .so.Major.Minor?

If the -devel is not included in a typical installation, then hunting down the .so file (with no versions) will fail. However, I think this is the best 'c' like behavior as you want to pull the most 'up to date' copy which almost always is the -devel package. I am just thinking past this potential scenario. Today clearly .1 is the choice, but tomorrow .2 may be... I am not aware of a Python mechanism to know what version of 'C' library it was built / created against which is a real pickle (C codes in the MAJOR version into the library requesting the runtime, so that is automatic)... So maybe I am jumping the gun and spoke to soon.

I will leave to your expertise here as you've already done as I suggested... Maybe a fall back pathway to the hardcoded is actually a good thing, because it locks you into a 'compatibility' of a known version for fallback... I still think the -devel is the primary <IMHO, but that is just my preference> Just be aware I am heavily C/C++ biased in my knowledge.

Long term to ping this and make it compatible, maybe it might be a good idea that we insert some kind of auto-gen file (similar to a Mako) that gets updated as the L0 Loader version gets updated, so this load stays in sync without human intervention each time? Thoughts? If you like this we can can look at adding something into our release scripting system.

@rwmcguir
rwmcguir requested a review from shubskmr November 13, 2026 07:30
@rwmcguir

Copy link
Copy Markdown
Contributor

Thinking about this more, I want to correct my own earlier point. C does not pick up the newest .so at runtime. It pins the major version at link time. So I think the closest Python match is:

  1. Load libze_loader.so. by name (no path), with the major in one constant, for example _ZE_LOADER_MAJOR = 1.
  2. Optional: fall back to find_library() / the unversioned .so only if step 1 fails.

That keeps the main fix (no hard-coded distro path) and avoids loading a loader with a different major version than pyzes
was written for. Later we can auto-generate that constant from the loader version in our release scripts, so nobody has to
update it by hand. Sorry for the back-and-forth on my first comment.

So maybe our release generation system, can set your _ZE_LOADER_MAJOR =1 someplace that can be picked up by this script?
That would be a later patchset, but making it a variable today would satisfy..
Again sorry for the back and forth... In the middle of a new Spec Release and I hadn't given serous thought to this until now.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants