Alexanderamiri · Oct 31, 2026 · Oct 31, 2026 · Oct 31, 2026
@@ -170,7 +170,7 @@ terraform/state/
 | `team-provisioner` | Syncs Google Groups, GitHub teams, AWS Budgets from registry team YAML; Atlassian groups from `groups/atlassian.yaml` |
 | `securityhub-summary` | Weekly Security Hub findings summary (Monday 08:00 UTC) |
 | `password-set` | Self-service password-set for new hero accounts (ALB only: password.javazone.no) |
-| `budget-enforcer` | Scales ECS services to zero when team exceeds 200% budget and sets `javabin:desired-count=0` so ecs-deploy keeps them at 0 until a tf-apply |
+| `budget-enforcer` | Suspends a team over 200% of budget: SSM marker, ECS to zero with `javabin:desired-count=0` (so ecs-deploy and tf-apply keep 0 tasks), RDS stopped; reconcile every 6 h re-applies it and lifts it when spend is back under 200% |
 | `resource-tagger` | EventBridge-triggered, auto-tags created-by + commit on new resources |
 | `ci-broker` | Verifies the job's GitHub OIDC token (repo from signed claims), checks repo ∈ app.yaml team, vends short-lived team role credentials |
 | `app-auth-sync` | Creates/deletes Cognito app clients requested by app.yaml `auth:` and `routing.login` (SSM request → client + credentials + allowed groups) |
@@ -250,7 +250,8 @@ EventBridge (Create/Run) ──► compliance-reporter (report to Slack, no auto
   Every 3 hours      ──► team-provisioner sync_access_members (team members into javabin-developer@; Identity Center group memberships for members Google SCIM has provisioned since)
 Registry merge ──► team-provisioner (Google/GitHub/Budget/Cognito/Identity Center sync + member provisioning + access group sync; team and kodesmia members become direct members of javabin-developer@ so Google SCIM provisions them)
 Platform apply ──► team-provisioner sync_team_access (team permission sets once guardrails exist; #javabin-infra-alerts on new access or failure)
-AWS Budgets (200%) ──► budget-enforcer Lambda ──► ECS scale-to-zero + #javabin-cost-alerts
+AWS Budgets (200%) ──► budget-enforcer Lambda ──► suspension marker + ECS scale-to-zero (javabin:desired-count=0) + RDS stop + #javabin-cost-alerts
+  Every 6 hours      ──► budget-enforcer reconcile (keep suspensions in force, lift when spend < 200%)
 EventBridge (Create/Run) ──► resource-tagger Lambda ──► Tag created-by + commit
 ```
 
@@ -279,6 +280,7 @@ All parameters are in `eu-central-1`. Use `--profile javabin --region eu-central
 | `/javabin/apps/{team}/{app}/network` | String | app.yaml `calls:`/`provides:` request (written by app TF, read by app-peers-sync) |
 | `/javabin/apps/{team}/{app}/peer-{t}-{a}-{url,scopes}`, `m2m-client-{id,secret}` | String / SecureString | Written by app-peers-sync, read by ECS execution role |
 | `/javabin/apps/{team}/{app}/peers-status` | String | Written by app-peers-sync: state of the app's calls and consumers |
+| `/javabin/budget-suspensions/{team}` | String (JSON) | Written/deleted by budget-enforcer; read by the ecs-service module (plans the `javabin:desired-count` tag as `0` while it exists) |
 
 ## GCP Connection
 

@@ -18,7 +18,7 @@ Lambda posts to **#javabin-cost-alerts**.
 | Account budget `javabin-account-monthly` | `terraform/platform/cost-analytics/main.tf` | Forecast for the month > 100%; actual > 100%; actual > 150% |
 | Cost Anomaly Detection `javabin-anomaly-monitor` | `terraform/platform/monitoring/main.tf` | Per-service anomaly with total impact >= $10 (IMMEDIATE) |
 | Team budgets `javabin-team-<team>` | `team-provisioner` Lambda (from registry `budget_nok`, default 500 NOK) | Actual > 80% (Slack); actual > 200% triggers `budget-enforcer` |
-| `budget-enforcer` | `terraform/lambda-src/budget_enforcer` | Scales the team's ECS services to `desired_count=0` (nothing is deleted) and sets their `javabin:desired-count` tag to 0, so app pushes keep deploying with 0 tasks. To restore, raise the budget and run the app's pipeline: the plan shows `"javabin:desired-count" = "0" -> "N"`, tf-apply writes it back and ecs-deploy scales the service up |
+| `budget-enforcer` | `terraform/lambda-src/budget_enforcer` | Suspends the team: ECS services to `desired_count=0` and their `javabin:desired-count` tag to 0 (so app pushes keep deploying with 0 tasks), RDS instances stopped (nothing is deleted). See [Budget suspension](#budget-suspension) |
 | `daily-cost-check` / `cost-report` | `terraform/lambda-src/` | Daily spike check; weekly breakdown every Monday |
 
 The account budget limit is `account_budget_usd` in `terraform/platform/main.tf`
@@ -35,6 +35,53 @@ A per-team anomaly monitor was considered and skipped: a tag-based
 `aws_ce_anomaly_monitor` needs every team value listed explicitly and is
 replaced whenever the list changes, and per-team budgets already cover it.
 
+## Budget suspension
+
+When a team's actual spend passes 200% of its monthly budget, AWS Budgets
+notifies `javabin-budget-enforcement` and `budget-enforcer`:
+
+1. Writes the marker `/javabin/budget-suspensions/{team}` (JSON: when, which
+   budget, each service's previous desired count and `javabin:desired-count`
+   tag, the RDS instances it stopped).
+2. Sets every ECS service tagged `team={team}` in `javabin-platform` to
+   `desired_count=0` and its `javabin:desired-count` tag to `0`.
+3. Stops every RDS instance tagged `team={team}` that is `available` (Aurora
+   members and read replicas are skipped; they cannot be stopped this way).
+4. Posts to #javabin-cost-alerts.
+
+**The suspension sticks.** `ecs-deploy` scales each service to its
+`javabin:desired-count` tag, so app pushes keep deploying with 0 tasks. The
+`ecs-service` module reads the marker and plans that tag as `"0"` while it
+exists, so the team's next `terraform apply` does not write
+`compute.desired_count` back either. Every 6 hours a reconcile run re-zeros
+services (and tags) changed by hand and stops the databases again: RDS starts a stopped
+instance by itself after 7 days, and reconcile catches that within 6 hours.
+Terraform changes to a stopped RDS instance fail until it is started.
+
+**How a team gets unsuspended** (any of):
+
+- **Raise the budget.** Raise `budget_nok` in the team's registry YAML. Once
+  team-provisioner has updated the budget, the next reconcile sees spend under
+  200% of the new limit and lifts the suspension (within 6 hours).
+- **Wait for the next month.** Budgets reset on the 1st (UTC); the first
+  reconcile of the month lifts every suspension.
+- **Lift it now.** A platform admin invokes the Lambda:
+  `aws lambda invoke --function-name javabin-budget-enforcer --payload '{"action":"lift","team":"<team>"}' --cli-binary-format raw-in-base64-out /dev/stdout`.
+  AWS Budgets notifies once per threshold per month, so a manual lift holds
+  until next month even if spend stays above 200%.
+
+Lifting deletes the marker, sets each service the enforcer scaled back to its
+previous desired count and `javabin:desired-count` tag (if still at 0), starts the RDS instances it
+stopped, and posts to #javabin-cost-alerts. Services created during the
+suspension come up on their next deploy. If the team's budget cannot be read,
+reconcile keeps the suspension rather than lifting it.
+
+The marker is not tamper-proof: it is untagged so that every team's Terraform
+can read it, which also means a team role could delete it. The developer
+boundary is ~45 bytes under AWS's 6,144-character limit, so a Deny for the
+`budget-suspensions` path does not fit yet. Deleting the marker stops
+reconcile from enforcing the suspension and shows in CloudTrail.
+
 ## Attribution: cost-allocation tags
 
 `team`, `service`, `environment`, `managed-by` and `repo` are activated as

@@ -224,16 +224,21 @@ IdP-synced groups are refused.
 
 ## budget-enforcer
 
-**Trigger:** SNS notification from AWS Budgets (200% threshold)
-**Purpose:** Scales a team's ECS services to `desired_count=0` when spending exceeds 200% of their monthly budget. Does NOT destroy resources — services can be scaled back up after resolution.
+**Triggers:** SNS notification from AWS Budgets (200% threshold, plain-text message); EventBridge every 6 hours with `{"action": "reconcile"}`; manual invoke with `{"action": "lift", "team": "<team>"}`
+**Purpose:** Suspends a team that spends more than 200% of its monthly budget, and keeps it suspended until the suspension is lifted. Nothing is destroyed.
 
-**Flow:** Parse budget name (`javabin-team-{team}`) → list ECS services tagged with team → scale to zero → post Slack alert.
+**Suspend (SNS):** parse `Budget Name:` (`javabin-team-{team}`) → write marker `/javabin/budget-suspensions/{team}` → scale the team's ECS services to `desired_count=0` and set their `javabin:desired-count` tag to `0` (previous counts and tag values recorded in the marker) → stop the team's `available` RDS instances (IDs recorded) → Slack. The `ecs-service` module plans the tag as `0` while the marker exists, so tf-apply and ecs-deploy keep 0 tasks.
 
-| SSM Parameter | Channel |
-|---------------|---------|
+**Reconcile (schedule):** for each marker, read the team budget. Spend still above 200% of the limit (or unreadable) → re-zero services, re-pin their tag and re-stop RDS (RDS restarts stopped instances after 7 days). Otherwise (new month, raised `budget_nok`) → lift.
+
+**Lift:** delete the marker → set the recorded services (still at 0) and their `javabin:desired-count` tag back to their previous values → start the recorded RDS instances → Slack. See [cost-management.md](cost-management.md#budget-suspension).
+
+| SSM Parameter | Use |
+|---------------|-----|
 | `/javabin/slack/platform-cost-alerts-webhook` | #javabin-cost-alerts |
+| `/javabin/budget-suspensions/{team}` | Suspension marker (written and deleted here) |
 
-**Environment vars:** `ECS_CLUSTER` (default: `javabin-platform`)
+**Environment vars:** `ECS_CLUSTER` (default: `javabin-platform`), `COST_WEBHOOK_PARAM`, `BUDGET_NAME_PREFIX`, `SUSPENSION_PATH`, `AWS_ACCOUNT_ID`
 
 ## resource-tagger
 

@@ -76,6 +76,8 @@ Supports `environment` (map) and `secrets` (map of name => ARN) for container co
 
 `health_check_grace_period_seconds` (default `120`) is set on the service only when `target_group_arn` is set (AWS rejects it without a load balancer). `enable_execute_command` (default `false`) turns on ECS Exec; the task role then needs `ssmmessages:CreateControlChannel`, `CreateDataChannel`, `OpenControlChannel` and `OpenDataChannel` (the generator adds them for `compute.exec`).
 
+**Budget suspension:** while the SSM marker `{budget_suspension_path}/{team}` exists (default path `/javabin/budget-suspensions`, written by the `budget-enforcer` Lambda), the service's `javabin:desired-count` tag is planned as `"0"` whatever `desired_count` says, so neither an apply nor the ecs-deploy after it (which scales to the tag) undoes the suspension. See [cost-management.md](cost-management.md#budget-suspension).
+
 **Outputs:** `service_name`, `task_definition_arn`, `log_group_name`
 
 ## service-bucket

@@ -191,7 +191,7 @@ Deploys the `image` input (docker-build's digest-pinned `image` output) to ECS s
 
 1. Copies the family's latest ACTIVE task definition revision (Terraform's newest, or the last one ecs-deploy registered from it) and registers it with the app container's image set to the digest. Every task of the deploy runs exactly this commit's build, also when tf-apply changed the task definition in the same run.
    The revision gets the base revision's tags (or the service's, if the base has none) with `managed-by=ecs-deploy`.
-2. Updates the service to that revision, with the desired count from the service tag `javabin:desired-count` (Terraform writes it from `compute.desired_count`; `budget-enforcer` sets it to 0, which keeps the service at 0 tasks until a tf-apply writes it back).
+2. Updates the service to that revision, with the desired count from the service tag `javabin:desired-count` (Terraform writes it from `compute.desired_count`; `budget-enforcer` sets it to 0 and Terraform plans it as 0 while the team is budget-suspended, so the service stays at 0 tasks until the suspension is lifted; see [cost-management.md](cost-management.md#budget-suspension)).
 3. Waits until that deployment's `rolloutState` is `COMPLETED` on the new revision (default timeout 12 minutes from the start of the step, inside the broker's 15-minute deploy credentials). A circuit-breaker rollback, a `FAILED` rollout, a timeout or an AWS error while waiting fails the job and prints the service events, the stopped tasks' `stoppedReason`, each container's exit code and reason, and the log group. A deployment replaced by a newer revision of a later run passes with a warning; that run reports the result.
 
 The `ecs-service` module ignores the service's `task_definition` and `desired_count` after creation, so this job, not Terraform, decides what runs. A Terraform change to the task definition (env, secrets, cpu) reaches the service at the next ecs-deploy, which follows tf-apply in the same run; if docker-build fails, it waits for the next green run. A new service is created with 0 tasks, so its first deployment starts here, once the image exists, instead of failing in tf-apply.

@@ -12,7 +12,8 @@
 #   3. Point the service at it, with the desired count from the service's
 #      javabin:desired-count tag (Terraform writes it from app.yaml; a new
 #      service is created with 0 tasks and gets them here; budget-enforcer
-#      sets it to 0 and the count stays 0 until a tf-apply writes it back).
+#      sets it to 0 while the team is suspended, and Terraform keeps it at 0
+#      until the suspension is lifted).
 #   4. Wait until that deployment's rolloutState is COMPLETED. A circuit-
 #      breaker rollback, a FAILED rollout, a timeout or an AWS error while
 #      waiting fails the job after printing why the tasks stopped.
@@ -113,7 +114,7 @@ DESIRED=$(jq -r '.tags[]? | select(.key == "javabin:desired-count") | .value' "$
 case "$DESIRED" in
   "") echo "No javabin:desired-count tag on ${SERVICE}; keeping its desired count" ;;
   *[!0-9]*) fail "javabin:desired-count tag on ${SERVICE} is not a number: ${DESIRED}" ;;
-  0) echo "::warning::javabin:desired-count on ${SERVICE} is 0 (budget-enforcer scaled it down, or compute.desired_count is 0): deploying with 0 tasks. tf-apply writes compute.desired_count back to the tag." ;;
+  0) echo "::warning::javabin:desired-count on ${SERVICE} is 0 (budget-enforcer suspended the team, see docs/cost-management.md, or compute.desired_count is 0): deploying with 0 tasks." ;;
 esac
 
 echo "Updating ${SERVICE} on ${CLUSTER}${DESIRED:+ (desired count ${DESIRED})}..."

@@ -257,7 +257,7 @@ def test_keeps_desired_count_without_tag(self):
         self.assertNotIn("--desired-count", calls(st, "update-service")[0])
 
     def test_budget_suspended_service_stays_at_zero(self):
-        # budget-enforcer sets the tag to 0; only a tf-apply writes it back
+        # budget-enforcer sets the tag to 0; lifting the suspension writes it back
         tags = {SERVICE_ARN: [{"key": "team", "value": "cupcake"}, {"key": "javabin:desired-count", "value": "0"}]}
         code, out, st = deploy([COMPLETED], tags=tags)
         self.assertEqual(code, 0, out)