Skip to content

Multi-slot Wake-on-Voice (WoV), ECNS, PCAN, and microWakeWord support (wov-001) - #11471

Draft
lgirdwood wants to merge 31 commits into
thesofproject:mainfrom
lgirdwood:wov-d0i3-upstream
Draft

lgirdwood wants to merge 31 commits into
thesofproject:mainfrom
lgirdwood:wov-d0i3-upstream

Conversation

@lgirdwood

Copy link
Copy Markdown
Member

Summary

Squashed and rebased series from intel/wov-d0i3 (wov-001) onto main, organizing the 180 development commits into 31 logical, bisectable patches with primary authorship attributed by surviving code contribution and Co-authored-by trailers for all contributors:

  • Core / Scheduler / IPC4:
    • DP-to-DP component binding via shared ring_buffer (CONFIG_DP_TO_DP_BIND)
    • Priority-inheriting k_mutex for per-core DP scheduler lock
    • Virtual detect sink support in module_adapter, hostless pipeline & dynamic WoV routing
    • Resource teardown NULL guards (dai, dma, copier_dai, ipc-helper), blocking IDC notifier events, sa_set_panic_on_delay() NULL guard
    • NOTIFIER_ID_WOV_DETECT, NOTIFIER_ID_WOV_CTRL, and NOTIFIER_ID_D0IX_STATE notifier events + IPC4 D0iX dispatch
    • UAOL link capability advertisement fix
  • Math & Feature Extraction:
    • Fixed-point PCAN (src/math/pcan/) + Octave references and ztest suite
    • MFCC 8-bit PCAN and soft AGC (AGC8) Mel output modes
  • WoV Components:
    • kpb: multi-client WoV detector triggering, dynamic sample rates, boundary drain fixes (rebased onto Zephyr-native k_queue / CONFIG_KPB_NOTIFIER KPB)
    • wov_arbiter: multi-slot WoV arbiter with first-wins detection, call-state gate, and single active WoV PCM gating
    • ecns: dual-rate 16/48 kHz 20 ms DP ECNS module with built-in IIR HPF and optional test signal
    • detect_test: multi-slot WoV support
    • microwakeword: TFLM submanifest (submanifests/tflm.yml), C++/LLEXT build helpers, mww DP module with D0i3 listening, built-in keyword models, and training/mtrace tuning scripts
  • Topology2, Tools & Docs:
    • EQ-IIR highpass presets, MFCC PCAN/AGC8 configs, WoV component/pipeline includes
    • Dual-rate DMIC, SSP, HDA, nocodec, and SoundWire RT721/RT722 multi-slot WoV topologies
    • tools/wov_capture/ capture app and S0/D0i3 test scripts
    • Developer integration guides and topology2/README.md updates

Draft / Open Review Items

The final commits at the top of the branch are intentionally isolated for discussion/splitting before marking Ready for Review:

  1. app: boards: ace30, ace40: configure PTL, WCL, and NVL for WOV static builds — disables userspace/LLEXT on PTL/WCL/NVL for real-time MWW inference speed; conflicts with os_linux_overlay.conf (CONFIG_SOF_USERSPACE_APPLICATION=y with CONFIG_LLEXT=n fails in zephyr_dp_schedule_application-kernel.c). Likely needs to be moved to an overlay or dropped from default defconfigs.
  2. audio, boards: disable optional modules and UAOL by default in Kconfig — global default-off changes for optional audio components and Intel UAOL.
  3. Optional PR wave split (e.g., Core/Math/MFCC/KPB/Arbiter/ECNS first, then TFLM/MWW, then Topology2/Tools/Docs).

Local Checks Performed

  • PTL (intel_adsp/ace30/ptl) static zephyr.elf build + scripts/check_kernel_includes.py --strict passed after final rebase.
  • TGL (intel_adsp/cavs25) zephyr.elf build + scripts/check_kernel_includes.py --strict passed (prior to final origin/main fast-forward).
  • test/ztest/unit/math/advanced/pcan on native_sim: 5/5 tests passed.

singalsu and others added 30 commits November 16, 2026 22:22
Add CONFIG_DP_TO_DP_BIND to allow connecting two Data Processing (DP)
scheduled modules directly via a shared ring_buffer attached on both
sides of the intermediate comp_buffer without LL-cycle copying.

- Retain a vregion reference on the shared ring_buffer in
  ipc4_comp_connect() and drop it on ring_buffer_free().
- Guard NULL module_data pointers in ipc4_comp_connect() for non-module
  components such as copiers.
- Include <zephyr/llext/symbol.h> explicitly in zephyr/lib/vregion.c for
  non-LLEXT static builds.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…cAVS

Use a k_mutex instead of a k_sem for the per-core DP scheduler lock so
priority inheritance prevents a preempted DP lock holder from stalling LL.
Initialize the DP scheduler in cAVS platform_init() when
CONFIG_ZEPHYR_DP_SCHEDULER is enabled.

Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
…ink params

Update module_adapter_prepare() and period calculation for DP WoV
modules:
- Skip sinks that do not produce audio streams (frame_bytes == 0 or
  rate == 0, such as phrase-detect event sinks) when calculating DP period.
- Propagate mod->stream_params to legacy comp_driver consumers that do not
  set input buffer parameters at bind time.
- Guard sink==NULL period_bytes computation when max_sinks == 0 (single-
  producer fallback).
- Pass SIZE_MAX to audio_buffer_sync_secondary_buffer().

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Support hostless pipelines, dynamic WOV routing, and clean EDF task
teardown:
- Cancel delayed work in schedule_edf_task_free().
- Export pipeline_schedule_cancel() and guard NULL pipe_task checks.
- Log failing component ID when pipeline_comp_prepare() fails.
- Support hostless pipelines and COMP_STATE_PREPARE handling in IPC4
  user-space handler.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…og throttling

Harden resource teardown, cross-core notifications, and host XRUN logs:
- Guard NULL pointers in dai_put(), sof_dma_put(), dai_common_free(),
  dai_free(), and copier_dai_free().
- Stop and reset active/paused components before freeing in ipc_comp_free().
- Use IDC_BLOCKING in notifier_event() to avoid overwriting queued
  notify_data on back-to-back cross-core notifications.
- Guard NULL system agent in sa_set_panic_on_delay().
- Increase SOF_MIN_NO_BYTES_INTERVAL_MS to 1000 ms in host-zephyr.c.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add NOTIFIER_ID_WOV_DETECT, NOTIFIER_ID_WOV_CTRL, and
NOTIFIER_ID_D0IX_STATE notifier IDs along with the D0iX state payload
and WOV arbiter notifier/control ABI header. Dispatch
NOTIFIER_ID_D0IX_STATE from ipc4_module_process_d0ix() so components
such as microWakeWord can switch to low-power D0i3 operation.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Count only UAOL links whose uaol_get_capabilities() call succeeded, pack
link_caps[] from index 0, and size the returned TLV accordingly instead
of reporting unresponsive links with zeroed capabilities.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add fixed-point Per-Channel AGC Normalization (PCAN) math library under
src/math/pcan/ with Octave reference scripts and a Zephyr ztest unit
test suite. Extend mel_filterbank_32.c and auditory.h for 32-bit Mel
output scaling needed by PCAN.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Extend the MFCC component with selectable output data types at setup,
including 8-bit PCAN normalization and 8-bit soft AGC (AGC8) Mel
spectrum output modes used by microWakeWord topologies.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…le rates

Support multiple downstream WOV detector pipelines sharing a single KPB
history buffer:
- Support multi-sink topology binding, dynamic sample rates, and
  synchronous IPC4 reset.
- Fix drain boundary copy, cross-core drain request coherency, and buffer
  parameter configuration races.
- Continue history buffering when a selector sink is full, and disable
  synchronous draining mode blocking.
- Increase KPB_MAX_NO_OF_CLIENTS default to 4 and KPB_MAX_BUFF_TIME to
  6000 ms on ACE 3.0+.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add the WOV arbiter component (CONFIG_COMP_WOV_ARBITER) to arbitrate
between multiple concurrent keyword detector slots sharing a single host
WoV capture PCM:
- First-wins slot arbitration on NOTIFIER_ID_WOV_DETECT, broadcasting
  NOTIFIER_ID_WOV_CTRL pause/resume to sibling detectors.
- Rate-limited PHRASE_DETECTED host notifications (sent once per
  detection) and active slot reset without requiring pipeline teardown.
- Support mixed 16/32-bit sample widths, single active WoV PCM gating, and
  call-state gate control.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Add the ECNS (Echo Cancellation and Noise Suppression) DP component
(CONFIG_COMP_ECNS) supporting dual-rate 16kHz/48kHz processing with a
20 ms period:
- Supports 16kHz single-producer fallback, 16/32-bit formats, built-in
  IIR high-pass filters with +30 dB KPB gain / capture boost switch, and
  optional monotonic test signal generation (CONFIG_ECNS_TEST_SIGNAL).

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Extend the detect_test sample component to support multi-slot WOV
operation alongside wov_arbiter (slot ID configuration, DP thread
batching, and WOV_CTRL pause/resume handling).

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…port

Add submanifests/tflm.yml to track tflite-micro, flatbuffers,
nnlib-hifi4, gemmlowp, and ruy via west (replacing
scripts/tensorflow-clone.sh). Add C++ operator new/delete LLEXT exports,
scripts/mww_shim_flatten.py, Xtensa longcalls/text-section-literals, and
LLEXT C++ linker discovery.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add versioned built-in TFLite Micro keyword model headers and matching
IPC4 large-config control blobs for banana, hey_jarvis, hi_intel,
orange, and strawberry.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add the TensorFlow Lite Micro microWakeWord (CONFIG_COMP_MWW) DP
keyword spotting module:
- Integrates multi-slot MWW with 10 ms DP period, 2 Hz scoring kcontrols,
  D0i3 low-power listening via NOTIFIER_ID_D0IX_STATE, VAD speech gating,
  and versioned ABI model blob loading.
- Uses static storage for the op resolver/interpreter and optional HiFi4
  nnlib acceleration under xt-clang.
- Coordinates with wov_arbiter and KPB for multi-slot detection and
  pre-roll draining.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Add dataset generation, Piper TTS synthesis, PCAN/MFCC training, model
verification, and mtrace feature plotting utilities under
src/audio/microwakeword/tune/.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Recognize the PTL Intel debug signing key in sof_ri_info.py.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Add IIR highpass filter presets (20/30/40/50/100 Hz, 30/40 dB, 16/48 kHz)
and bundle.conf for topology2 pipelines.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add mel40/mel80 PCAN and 8-bit soft AGC MFCC component configuration
blobs and align existing MFCC config parameter headers.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…figs

Add topology2 component and cAVS pipeline building blocks for multi-slot
Wake-on-Voice, ECNS, microWakeWord, KPB, and the WOV arbiter.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add SoundWire DMIC single-slot WoV (sdw-dmic-wov-single.conf) and
PTL nocodec WoV loopback (cavs-nocodec-wov.conf) topologies and enable
the SDCA target.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add SoundWire headset jack ECNS and multi-slot WoV pipeline support
(sdw-jack-wov-multi.conf) and DMIC default definitions for RT722.

Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Add dual-rate DMIC, SSP, HDA, and SoundWire RT721/RT722 multi-slot WoV
topologies, configurable detector counts, and production/development
CMake targets (using PTL symlinks for WCL).

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Add wov_capture_app (ALSA/TinyALSA blocking reader, tinycap-compatible
options, DMIC/Jack phrase-detect ID controls) and automated S0/D0i3
multi-slot WoV test scripts under tools/wov_capture/.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Add developer integration guide for custom WoV and ECNS algorithms,
update top-level README.md, and document multi-slot WoV, dual-rate DMIC,
NHLT, and D0i3 verification in tools/topology/topology2/README.md.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Co-authored-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…_defconfig

Unset GTK_PATH in scripts/host-testbench.sh to avoid Octave library
conflicts, and enable CONFIG_MATH_PCAN in host library_defconfig.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add app/configs/ptl/modules.conf, app/configs/wcl/modules.conf, and
app/wov-d0i3-fake-wake.conf overlay configurations.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Enable WOV arbiter, KPB, microWakeWord, and ECNS configuration on
intel_adsp_cavs25 and intel_adsp_cavs25_tgph.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
… builds

Configure PTL, WCL, and NVL board defconfigs with WOV arbiter, KPB,
ECNS, MFCC, MWW, and open modules using static builds (without
userspace/LLEXT) for real-time MWW inference performance.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Co-authored-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Co-authored-by: Naveen Manohar <naveen.m@intel.com>
Signed-off-by: Naveen Manohar <naveen.m@intel.com>
Disable ARIA, DRC, Multiband DRC, Phase Vocoder, Sound Dose, TDFB, Tone,
and Intel UAOL by default in Kconfig.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants