[deanq]

Two independent performance changes, measured live on Runpod Serverless across all four endpoint kinds (QB/LB × GPU/CPU).

perf(serverless): take the next job while a result uploads

A queue worker held a job's slot until its result POST to job-done returned (300–330 ms on every job), even though the platform records the result well before that. The next job-take only started once the slot freed, so every job queued behind another paid the full upload time again.

• rp_job.handle_job takes an optional on_result_ready(result) callback, invoked just before send_result.
• JobScaler.handle_job passes _release_slot as that callback, only for the built-in handle_job (a custom jobs_handler keeps the old behaviour).
• _release_slot runs the refresh/stopPod check first (preserving the "never take another job while refreshing" guarantee), marks the job as sending its result, removes it from job_progress, and wakes get_jobs.
• run_jobs no longer counts result-sending tasks against current_concurrency, so a newly taken job starts at once.

Measured (4 rounds, a burst of 20 jobs on one warm worker):

Metric	main	candidate	Δ (95% CI)
QB GPU: burst clears	14,424 ms	8,661 ms	−5,763 [−6,325, −5,659]
QB CPU: burst clears	14,414 ms	8,900 ms	−5,514 [−5,786, −5,065]
QB GPU: median queue delay in burst	6,958 ms	4,072 ms	−2,886 [−4,652, −1,112]
QB CPU: median queue delay in burst	6,896 ms	4,003 ms	−2,894 [−4,691, −1,167]
QB warm delayTime (one at a time)	140 / 141 ms	142 / 139 ms	unchanged

A queued burst of 20 drains about 40% faster; queued jobs wait ~2.9 s less.

perf(apps): vendor fastapi without the [standard] extra

Every app with an @app.api resource vendored fastapi[standard], pulling in pygments, opentelemetry, sentry_sdk, google, dns and fastapi_cloud_cli. The runtime (runpod_sdk_runtime.api.server) imports only fastapi and uvicorn, and the API images already install fastapi[standard] through sdk-runtime's own dependency.

API_RUNTIME_REQUIREMENTS = ("fastapi", "uvicorn>=0.30")

uvicorn still finds uvloop and httptools in the API image's site-packages, because the vendored env is placed ahead of the image's packages, not instead of them.

Measured in the same run:

Metric	main	candidate	Δ (95% CI)
Artifact size	31.9 MB	14.2 MB
QB GPU: cold unpack	2,650 ms	900 ms	−1,750 [−2,250, −1,300]
QB GPU: container start → SDK ready	4,745 ms	3,369 ms	−1,376 [−2,881, −86]
LB GPU: cold unpack	1,750 ms	500 ms	−1,250 [−2,400, −800]
CPU (QB and LB): cold unpack	1,100 ms	550 ms	−550 [−1,550, −300]

Risks and things to review

• Cancelling during the upload: a job stopped in the ~300 ms while its result is uploading now has the upload cancelled. The job was finished, but its result may not reach the platform.
• Heartbeat gap: for ~300 ms the heartbeat's in-progress list no longer includes a job whose result is still uploading. The heartbeat runs every 10 s, so the window is tiny.
• job_in_progress=0: the next job-take reports this while an upload is in flight. The platform handed out jobs normally in every live run.
• Batch takes: with max_concurrency > 1, occupancy now falls at upload time rather than after it. Not separately benchmarked; the bench uses one slot.
• fastapi extras: user code relying on fastapi[standard] extras being vendored (e.g. python-multipart for forms) still gets them from the built-in API images. Custom images install runpod-sdk-runtime, which depends on fastapi[standard] itself.

Test plan

• make quality-check: 3462 passed, 7 skipped; coverage 95.12%.
• New tests:
  • tests/test_serverless/test_result_overlap.py: runs the real JobScaler over HTTP against a fake ai-api whose job-done takes 0.3 s: verifies overlap, no stacking, refresh still stops intake, and that a custom handler keeps its slot. All four fail on main.
  • test_api_runtime_vendors_fastapi_without_the_standard_extra.
• Existing refresh-worker HTTP and process tests still pass.

[deanq]

Split into two ticket-scoped PRs so each links its own Linear issue via branch name:

• perf(serverless): take the next job while a result uploads #726 — perf(serverless): take the next job while a result uploads
• perf(apps): vendor fastapi without the [standard] extra #727 — perf(apps): vendor fastapi without the [standard] extra

Same commits (independent), rebased on current main. Closing this one in favor of those.