Repository navigation
WIP macb stall patches - #7667
Draft
nbuchwitz wants to merge 8 commits into
Draft
WIP macb stall patches #7667nbuchwitz wants to merge 8 commits into
nbuchwitz wants to merge 8 commits into
Conversation
nbuchwitz
force-pushed
the
devel/macb-stall
branch
from
September 30, 2026 09:56
88fd650 to
e8ebfe7
Compare
Contributor
Author
|
@pelwell, can you please trigger the builds? Thanks |
nbuchwitz
force-pushed
the
devel/macb-stall
branch
from
October 18, 2026 09:57
e8ebfe7 to
8a4f54b
Compare
Contributor
Author
|
@pelwell can you please re-trigger the build? Thanks! |
nbuchwitz
force-pushed
the
devel/macb-stall
branch
from
October 21, 2026 20:59
8a4f54b to
d9a828b
Compare
nbuchwitz
force-pushed
the
devel/macb-stall
branch
from
November 2, 2026 08:03
bdde328 to
df9bebe
Compare
Contributor
Author
|
@pelwell can you please re-trigger the build? thanks! |
The frames dropped on a TX error are never returned to BQL. Each error shrinks the BQL budget for good until the queue stops with an empty ring. netdev_tx_reset_queue() is not safe here as the task runs without tx_ptr_lock, so macb_start_xmit() may still be queueing. Fixes: e6a5321 ("net: cadence: macb: Implement BQL") Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
The IER write at the end of macb_tx_poll() is posted. The ring check right after it can run before TCOMP is unmasked, and a frame that completes in that window never raises an interrupt. With TSO this happens when TXUBR triggers a poll while the frame is still on the wire. The completed frame is never reaped and BQL keeps the queue stopped. Read IMR back after the write so the check runs with TCOMP unmasked. macb_rx_poll() has the same window. Fixes: 138badb ("net: macb: use NAPI for TX completion path") Fixes: 0bf476f ("net: macb: Fix lost RX packet wakeup race in NAPI receive") Link: https://lore.kernel.org/netdev/20260514215459.36109-3-lukasz@raczylo.com/ Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
gem_shuffle_tx_one_ring() rotates the software TX ring so that the tail sits at index 0 and resets queue->tx_tail to 0, but it never reprograms the hardware transmit buffer queue pointer (TBQP). Other paths that reset tx_tail to the ring base (macb_init_buffers() and macb_tx_error_task()) also reprogram TBQP to queue->tx_ring_dma; this path does not, leaving TBQP pointing at a stale descriptor. gem_shuffle_tx_rings() runs on every link-up from macb_mac_link_up(). After a few link up/down flaps that leave un-completed descriptors in the ring, the stale TBQP keeps pointing at a descriptor whose used bit is set. When TX is re-enabled on link-up, the GEM reads that used descriptor and raises TXUBR. macb_interrupt() schedules the TX NAPI, macb_tx_poll() makes no progress (work_done == 0) and macb_tx_restart() re-issues TSTART, which makes the controller read the same used descriptor again and re-assert TXUBR. As the MAC interrupt is level-triggered, it never deasserts and one CPU is pegged at 100% in the threaded handler, eventually triggering "sched: RT throttling activated" and a dead network interface. Fix it by reprogramming TBQP to the ring base on every path of gem_shuffle_tx_one_ring() that resets tx_tail to 0, mirroring macb_tx_error_task(). The early return for an already-aligned tail is left untouched as TBQP is already consistent there. This is safe because the shuffle runs from macb_mac_link_up() while TE is still disabled, so the transmitter is halted. Fixes: 881a026 ("net: macb: Shuffle the tx ring before enabling tx") Cc: stable@vger.kernel.org Assisted-by: Claude:claude-opus-4-8 Link: https://lore.kernel.org/netdev/20261022-std-macb-v14-send-v1-2-9187bc545bf6@beckhoff.com/ Signed-off-by: Christian Taedcke <christian.taedcke@weidmueller.com> Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
Start the link-up rotation in gem_shuffle_tx_one_ring() at the first frame that still has to be sent instead of at tx_tail, and schedule the TX NAPI poll once transmit is enabled again. macb_mac_link_down() masks TCOMP before disabling transmit. A frame completed in that window keeps the used bit on its first descriptor and is never reaped. The rotation moves it to index 0, where the controller starts after link-up, reads the used bit and stops; macb_tx_restart() then keeps writing TSTART onto the same descriptor, an interrupt storm that on PREEMPT_RT ends in RCU stalls. Keeping the completed frames behind the ring base leaves tx_tail on them for NAPI to reap. Reaping in link-down instead would have to wait for NAPI there, which deadlocks against macb_close() and the PM callbacks disabling it. Fixes: 881a026 ("net: macb: Shuffle the tx ring before enabling tx") Cc: stable@vger.kernel.org Link: https://lore.kernel.org/netdev/20261022-std-macb-v14-send-v1-4-9187bc545bf6@beckhoff.com/ Assisted-by: LLM Signed-off-by: Steffen Dirkwinkel <s.dirkwinkel@beckhoff.com> [nb: context adjusted for 6.18, the NCR write is not under bp->lock there] Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
macb_tx_restart() treats the raw TBQP value divided by the descriptor size as a ring index. With 24-byte descriptors the ring base is never a multiple of the descriptor size, so the index is off and the comparison with the head index means nothing. The restart then kicks a running DMA most of the time and skips the one case where it should kick. Take the index relative to the ring base. Fixes: 5ad7f18 ("net: macb: Restart tx only if queue pointer is lagging") Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
Raspberry Pi 5 users (BCM2712 with the GEM on the RP1 south bridge) reported TX queue stalls of five seconds: macb 1f00100000.ethernet eth0: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5508 ms Further inspection of the registers showed TBQP on the first descriptor of a valid TSO frame with TGO and the used bit clear. So the DMA never fetched the frame. The GEM reads the descriptor behind the frame it fetches. If the next frame lands there a few microseconds later, the DMA has already seen the used bit and halts after the current frame. The TSTART of that xmit is ignored while the transmitter is busy and no TXUBR follows. The TXUBR and tx_pending flags do not help either, the interrupt that clears them often comes before the halt. Restart the transmitter from every TX poll. The completion of the previous frame always comes after the halt. Fixes: 4298388 ("net: macb: restart tx after tx used bit read") Link: raspberrypi#7661 Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
macb_tx_timeout() only restarts the DMA. When the frames went out but their completion was lost, TBQP already matches the head and the restart does nothing. Nothing reaps the frames and the queue stays stopped for good. Schedule the TX poll instead, it reaps what completed and restarts the DMA. Fixes: e438ec3 ("net: macb: add TX stall timeout callback to recover from lost TSTART write") Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
Signed-off-by: Nicolai Buchwitz <nb@tipi-net.de>
nbuchwitz
force-pushed
the
devel/macb-stall
branch
from
November 7, 2026 15:28
df9bebe to
8f15aeb
Compare
Contributor
Author
|
@pelwell, as usual please 🤖 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some patches to try with #7661