Skip to content

fix(apps): roll back endpoint changes when deployment fails - #724

Open
KAJdev wants to merge 5 commits into
mainfrom
zeke/con-2174-partial-failure-during-rp-flash-deploy-leaves
Open

KAJdev wants to merge 5 commits into
mainfrom
zeke/con-2174-partial-failure-during-rp-flash-deploy-leaves

Conversation

@KAJdev

@KAJdev KAJdev commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

When reconciling endpoint updates when deploying a flash app, if something fails, will completely roll back any completed operations (updates & creations) and then stop the deploy.

A recovered failure ends with:

Error: Couldn't deploy 'b_bad': quota exceeded
Changes rolled back.

Obsolete endpoints are removed after the deployment succeeds. If that cleanup fails, the deployed app stays in place and the error identifies the endpoint that could not be removed.

Also now runs a parser over everything before attempting the deploy to catch easy validation issues before getting caught mid deploy.

Fixes CON-2174

@KAJdev KAJdev changed the title fix(apps): roll back new endpoints when deployment fails fix(apps): roll back endpoint changes when deployment fails Nov 13, 2026
Comment thread runpod/apps/deploy.py Fixed
Comment thread runpod/apps/spec.py Resolved
Comment thread runpod/apps/deploy.py Resolved
Comment thread runpod/apps/deploy.py
creating = False
log.info("saved endpoint %s (%s)", name, result["id"])

except Exception as exc:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The handler is except Exception, and KeyboardInterrupt and asyncio.CancelledError are BaseExceptions. Under asyncio.run, Ctrl-C after the new build is activated and a couple of endpoints are saved leaves the new build live, endpoints partly updated and new ones created, with nothing rolled back or reported. That matches the behavior before this PR, so it isn't a regression, but the description says completed operations are completely rolled back. Should the rollback also run on cancellation (catch BaseException, run the undo under asyncio.shield, then re-raise), or should the description and the interrupt output point at rp flash env get instead?

Comment thread runpod/apps/deploy.py Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants