Repository navigation
Conversation
_LoadZeLibrary hardcoded /usr/lib/x86_64-linux-gnu/libze_loader.so.1 on Linux. When a different (e.g. newer) loader is the one the Level Zero runtime resolves through the standard search path - common when oneAPI or a locally built loader is installed under /usr/local or /opt - pyzes pins the distro copy instead. A zesInit issued through that mismatched loader leaves the process in a state where the Unified Runtime no longer enumerates the Level Zero platform, which breaks downstream consumers that query Sysman before initializing the runtime. torch.xpu.device_count() is one such consumer: after it runs, torch.xpu.get_device_properties() fails with "No XPU devices are available" on a healthy Intel Arc B580 system. Load libze_loader.so.1 by SONAME instead, so the dynamic linker picks the same loader copy as every other consumer. The distro location remains reachable through the standard search path. This commit was authored with the assistance of an AI coding tool. Signed-off-by: Wei (Will) Feng <134637289+weifengpy@users.noreply.github.com>
|
Downstream report with the full investigation and reproduction details: pytorch/pytorch#204338 |
|
@shubskmr Could you please help review this code change? |
|
thanks for taking a look. seems it needs more approval in order to land "At least 1 approving review is required by reviewers with write access." |
|
@rwmcguir just a gentle ping. |
|
Why is the version being hard coded, i.e. the ".so.1", this will break when level-zero goes to Major version 2. I'd suggest converting this to some kind of dynamic extraction. |
Level Zero loader ABI checklast-release ( master ( If an ABI change is intended, a reviewer with write access can accept it: leave an approval review on the PR head commit with the keyword |
The previous change hardcoded the major version in libze_loader.so.1, which would break when Level Zero moves to major version 2, as pointed out in review of oneapi-src#559. Resolve the SONAME at runtime with ctypes.util.find_library(), which queries the system ldconfig cache and returns the current SONAME (libze_loader.so.1 today, the .so.2 name after a major bump). Fall back to the unversioned libze_loader.so if find_library() returns nothing. Loading still happens by SONAME through the standard search path, so the loader copy the runtime uses still wins over a pinned distro copy. The suggested helper.py get_major_ver() parses a $X_MAKE_VERSION(...) spec string at code-generation time; that input is not available to pyzes at runtime, so find_library() is the runtime equivalent of dynamic extraction. This commit was authored with the assistance of an AI coding tool. Signed-off-by: Wei (Will) Feng <134637289+weifengpy@users.noreply.github.com>
|
@rwmcguir Thanks for catching this — you are right, hardcoding Fixed in 87d9bb9: the Linux path no longer constructs the SONAME from a hardcoded major version. It now resolves it dynamically at runtime: libName = ctypes.util.find_library(libName) or "lib" + libName + ".so"
On Verification on the original dual-loader system (distro 1.16 loader in /usr/lib/x86_64-linux-gnu and the self-built 1.35 loader in /usr/local, both present):
|
|
Thanks.... A latent thought before I approve as I am not as familiar with the runtime python bindings. For normal runtimes the .so package is only installed with the -devel package from Linux, for a Python environment would this be a typical scenario, i.e. to have a -devel package installed? Or only the runtime package which would have the .so.Major and .so.Major.Minor? If the -devel is not included in a typical installation, then hunting down the .so file (with no versions) will fail. However, I think this is the best 'c' like behavior as you want to pull the most 'up to date' copy which almost always is the -devel package. I am just thinking past this potential scenario. Today clearly .1 is the choice, but tomorrow .2 may be... I am not aware of a Python mechanism to know what version of 'C' library it was built / created against which is a real pickle (C codes in the MAJOR version into the library requesting the runtime, so that is automatic)... So maybe I am jumping the gun and spoke to soon. I will leave to your expertise here as you've already done as I suggested... Maybe a fall back pathway to the hardcoded is actually a good thing, because it locks you into a 'compatibility' of a known version for fallback... I still think the -devel is the primary <IMHO, but that is just my preference> Just be aware I am heavily C/C++ biased in my knowledge. Long term to ping this and make it compatible, maybe it might be a good idea that we insert some kind of auto-gen file (similar to a Mako) that gets updated as the L0 Loader version gets updated, so this load stays in sync without human intervention each time? Thoughts? If you like this we can can look at adding something into our release scripting system. |
|
Thinking about this more, I want to correct my own earlier point. C does not pick up the newest .so at runtime. It pins the major version at link time. So I think the closest Python match is:
That keeps the main fix (no hard-coded distro path) and avoids loading a loader with a different major version than pyzes So maybe our release generation system, can set your _ZE_LOADER_MAJOR =1 someplace that can be picked up by this script? |
Problem
On Linux,
pyzesloads the Level Zero loader from a hardcoded path:When the loader the runtime actually uses lives somewhere else in the standard search path (for example a newer loader installed under
/usr/localor shipped with oneAPI), pyzes pins the distro copy instead. AzesInitissued through that mismatched loader copy leaves the process in a state where the oneAPI Unified Runtime no longer enumerates the Level Zero platform at all. Any consumer that queries Sysman before initializing the runtime is then broken for the rest of the process lifetime.Reproduction
Machine: Intel Arc B580 (Battlemage), NEO 26.40.40196.8, two coexisting loaders:
/usr/lib/x86_64-linux-gnu/libze_loader.so.1— 1.16 (Ubuntulibze1package)/usr/local/lib/libze_loader.so.1— 1.35 (built from this repo), the copy the SYCL runtime resolvesPyTorch (both the 2.15.0+xpu wheel and a source build) routes
torch.xpu.device_count()through pyzes before XPU initialization:Calling
get_device_propertiesfirst works fine. A minimal SYCL probe confirms the mechanism outside PyTorch: afterdlopenof the distro loader pluszesInit/zesDriverGet/zesDeviceGetthrough it,sycl::platform::get_platforms()no longer lists the Level Zero platform; doing the same through the runtime's loader, or not callingzesInitat all, leaves enumeration intact. Replacing the distro loader file's contents with the newer loader also makes the failure disappear, confirming the trigger is which loader copy serves the Sysman calls.Fix
Load
libze_loader.so.1by SONAME on Linux so the dynamic linker resolves it through the standard search path (LD_LIBRARY_PATH, the ld.so cache, default directories) — the same copy every other consumer gets. The distro location remains reachable that way, so no supported configuration loses access to the loader.Testing
On the system above, with the distro 1.16 loader restored so both loaders coexist:
PYTHONPATH):pyzes.gpuLib._nameislibze_loader.so.1, the same sequence returns the B580 fromget_device_properties, and an XPU matmul runs correctly.This PR was prepared with the assistance of an AI coding tool and reviewed by the author.