[deanq]

What

A queue worker now takes the next job while the previous job's result uploads, instead of holding the slot until the job-done POST returns.

Why

The result POST to job-done takes ~300 ms on every job (GPU and CPU), yet the worker held the job's slot until that response arrived before get_jobs could start the next job-take. Spaced-out requests pay nothing, but every job queued behind another paid the full upload time again: a 20-job burst on a single-slot worker took 14.4 s (~0.72 s/job, ~0.3 s of it waiting on the upload).

Changes

• rp_job.handle_job takes an optional on_result_ready(result) callback, called just before send_result.
• JobScaler.handle_job passes _release_slot as that callback (only for the built-in handler; a custom jobs_handler keeps the old behaviour).
• _release_slot runs the refresh/stopPod check first, so a worker about to restart still never takes another job (preserves the existing refresh guarantee), then marks the job as sending its result and wakes get_jobs.
• run_jobs no longer counts result-sending tasks against current_concurrency.

Measured (4 rounds, 20-job burst on one warm worker)

Metric	main	candidate
QB GPU burst clears	14,424 ms	8,661 ms (~40% faster)
QB CPU burst clears	14,414 ms	8,900 ms
QB GPU median queue delay in burst	6,958 ms	4,072 ms
QB warm delayTime (one at a time)	140 ms	unchanged

Test plan

• New tests/test_serverless/test_result_overlap.py: overlap, no-stacking, refresh-still-stops-intake, custom-handler-keeps-slot. Runs the real JobScaler over HTTP against a fake ai-api whose job-done takes 0.3 s.
• Existing refresh-worker tests still pass.
• make quality-check: 3461 passed, 7 skipped.

Risks

• A job cancelled during the ~300 ms upload now has the upload cancelled; its result may not reach the platform.
• ~300 ms heartbeat gap where an uploading job is not in the in-progress list (heartbeat runs every 10 s).
• The next job-take reports job_in_progress=0 while an upload is in flight; the platform handed out jobs normally in every live run.
• With max_concurrency > 1, occupancy now falls at upload time; not separately benchmarked (bench uses one slot).

[Copilot]

🟡 Changes recommended

Outstanding uploads are not safely coordinated with queue resizing, worker refreshes, or backpressure, risking queue completion-count corruption, undelivered results, and unbounded resource growth.

3 open issues

• Pending result uploads can grow without bound · New
• Worker refresh can interrupt earlier result uploads · New
• Queue resize can race with upload task cleanup · New

What changed in this PR

This PR reduces queued-job latency by allowing serverless workers to take the next job while a completed job’s result uploads.

Changes:

• Adds an optional result-ready callback to the built-in job handler.
• Releases execution slots before uploads finish, while stopping intake on refresh requests and preserving custom-handler behavior.
• Adds HTTP-based tests for upload overlap, burst timing, refresh intake, and custom handlers.

File	Description
tests/​test_serverless/​test_result_overlap.py	Adds slow-upload tests for overlap and handler behavior.
runpod/​serverless/​modules/​rp_scale.py	Tracks uploading jobs separately and frees their execution slots early.
runpod/​serverless/​modules/​rp_job.py	Invokes the optional callback immediately before sending results.

_{Review effort: Balanced}