diff --git a/CLAUDE.md b/CLAUDE.md index 22ddf04..652421a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -170,7 +170,7 @@ terraform/state/ | `team-provisioner` | Syncs Google Groups, GitHub teams, AWS Budgets from registry team YAML; Atlassian groups from `groups/atlassian.yaml` | | `securityhub-summary` | Weekly Security Hub findings summary (Monday 08:00 UTC) | | `password-set` | Self-service password-set for new hero accounts (ALB only: password.javazone.no) | -| `budget-enforcer` | Scales ECS services to zero when team exceeds 200% budget and sets `javabin:desired-count=0` so ecs-deploy keeps them at 0 until a tf-apply | +| `budget-enforcer` | Suspends a team over 200% of budget: SSM marker, ECS to zero with `javabin:desired-count=0` (so ecs-deploy and tf-apply keep 0 tasks), RDS stopped; reconcile every 6 h re-applies it and lifts it when spend is back under 200% | | `resource-tagger` | EventBridge-triggered, auto-tags created-by + commit on new resources | | `ci-broker` | Verifies the job's GitHub OIDC token (repo from signed claims), checks repo ∈ app.yaml team, vends short-lived team role credentials | | `app-auth-sync` | Creates/deletes Cognito app clients requested by app.yaml `auth:` and `routing.login` (SSM request → client + credentials + allowed groups) | @@ -250,7 +250,8 @@ EventBridge (Create/Run) ──► compliance-reporter (report to Slack, no auto Every 3 hours ──► team-provisioner sync_access_members (team members into javabin-developer@; Identity Center group memberships for members Google SCIM has provisioned since) Registry merge ──► team-provisioner (Google/GitHub/Budget/Cognito/Identity Center sync + member provisioning + access group sync; team and kodesmia members become direct members of javabin-developer@ so Google SCIM provisions them) Platform apply ──► team-provisioner sync_team_access (team permission sets once guardrails exist; #javabin-infra-alerts on new access or failure) -AWS Budgets (200%) ──► budget-enforcer Lambda ──► ECS scale-to-zero + #javabin-cost-alerts +AWS Budgets (200%) ──► budget-enforcer Lambda ──► suspension marker + ECS scale-to-zero (javabin:desired-count=0) + RDS stop + #javabin-cost-alerts + Every 6 hours ──► budget-enforcer reconcile (keep suspensions in force, lift when spend < 200%) EventBridge (Create/Run) ──► resource-tagger Lambda ──► Tag created-by + commit ``` @@ -279,6 +280,7 @@ All parameters are in `eu-central-1`. Use `--profile javabin --region eu-central | `/javabin/apps/{team}/{app}/network` | String | app.yaml `calls:`/`provides:` request (written by app TF, read by app-peers-sync) | | `/javabin/apps/{team}/{app}/peer-{t}-{a}-{url,scopes}`, `m2m-client-{id,secret}` | String / SecureString | Written by app-peers-sync, read by ECS execution role | | `/javabin/apps/{team}/{app}/peers-status` | String | Written by app-peers-sync: state of the app's calls and consumers | +| `/javabin/budget-suspensions/{team}` | String (JSON) | Written/deleted by budget-enforcer; read by the ecs-service module (plans the `javabin:desired-count` tag as `0` while it exists) | ## GCP Connection diff --git a/docs/cost-management.md b/docs/cost-management.md index 032c8d5..9e187b1 100644 --- a/docs/cost-management.md +++ b/docs/cost-management.md @@ -18,7 +18,7 @@ Lambda posts to **#javabin-cost-alerts**. | Account budget `javabin-account-monthly` | `terraform/platform/cost-analytics/main.tf` | Forecast for the month > 100%; actual > 100%; actual > 150% | | Cost Anomaly Detection `javabin-anomaly-monitor` | `terraform/platform/monitoring/main.tf` | Per-service anomaly with total impact >= $10 (IMMEDIATE) | | Team budgets `javabin-team-` | `team-provisioner` Lambda (from registry `budget_nok`, default 500 NOK) | Actual > 80% (Slack); actual > 200% triggers `budget-enforcer` | -| `budget-enforcer` | `terraform/lambda-src/budget_enforcer` | Scales the team's ECS services to `desired_count=0` (nothing is deleted) and sets their `javabin:desired-count` tag to 0, so app pushes keep deploying with 0 tasks. To restore, raise the budget and run the app's pipeline: the plan shows `"javabin:desired-count" = "0" -> "N"`, tf-apply writes it back and ecs-deploy scales the service up | +| `budget-enforcer` | `terraform/lambda-src/budget_enforcer` | Suspends the team: ECS services to `desired_count=0` and their `javabin:desired-count` tag to 0 (so app pushes keep deploying with 0 tasks), RDS instances stopped (nothing is deleted). See [Budget suspension](#budget-suspension) | | `daily-cost-check` / `cost-report` | `terraform/lambda-src/` | Daily spike check; weekly breakdown every Monday | The account budget limit is `account_budget_usd` in `terraform/platform/main.tf` @@ -35,6 +35,53 @@ A per-team anomaly monitor was considered and skipped: a tag-based `aws_ce_anomaly_monitor` needs every team value listed explicitly and is replaced whenever the list changes, and per-team budgets already cover it. +## Budget suspension + +When a team's actual spend passes 200% of its monthly budget, AWS Budgets +notifies `javabin-budget-enforcement` and `budget-enforcer`: + +1. Writes the marker `/javabin/budget-suspensions/{team}` (JSON: when, which + budget, each service's previous desired count and `javabin:desired-count` + tag, the RDS instances it stopped). +2. Sets every ECS service tagged `team={team}` in `javabin-platform` to + `desired_count=0` and its `javabin:desired-count` tag to `0`. +3. Stops every RDS instance tagged `team={team}` that is `available` (Aurora + members and read replicas are skipped; they cannot be stopped this way). +4. Posts to #javabin-cost-alerts. + +**The suspension sticks.** `ecs-deploy` scales each service to its +`javabin:desired-count` tag, so app pushes keep deploying with 0 tasks. The +`ecs-service` module reads the marker and plans that tag as `"0"` while it +exists, so the team's next `terraform apply` does not write +`compute.desired_count` back either. Every 6 hours a reconcile run re-zeros +services (and tags) changed by hand and stops the databases again: RDS starts a stopped +instance by itself after 7 days, and reconcile catches that within 6 hours. +Terraform changes to a stopped RDS instance fail until it is started. + +**How a team gets unsuspended** (any of): + +- **Raise the budget.** Raise `budget_nok` in the team's registry YAML. Once + team-provisioner has updated the budget, the next reconcile sees spend under + 200% of the new limit and lifts the suspension (within 6 hours). +- **Wait for the next month.** Budgets reset on the 1st (UTC); the first + reconcile of the month lifts every suspension. +- **Lift it now.** A platform admin invokes the Lambda: + `aws lambda invoke --function-name javabin-budget-enforcer --payload '{"action":"lift","team":""}' --cli-binary-format raw-in-base64-out /dev/stdout`. + AWS Budgets notifies once per threshold per month, so a manual lift holds + until next month even if spend stays above 200%. + +Lifting deletes the marker, sets each service the enforcer scaled back to its +previous desired count and `javabin:desired-count` tag (if still at 0), starts the RDS instances it +stopped, and posts to #javabin-cost-alerts. Services created during the +suspension come up on their next deploy. If the team's budget cannot be read, +reconcile keeps the suspension rather than lifting it. + +The marker is not tamper-proof: it is untagged so that every team's Terraform +can read it, which also means a team role could delete it. The developer +boundary is ~45 bytes under AWS's 6,144-character limit, so a Deny for the +`budget-suspensions` path does not fit yet. Deleting the marker stops +reconcile from enforcing the suspension and shows in CloudTrail. + ## Attribution: cost-allocation tags `team`, `service`, `environment`, `managed-by` and `repo` are activated as diff --git a/docs/lambda-functions.md b/docs/lambda-functions.md index a5456cf..949c2a6 100644 --- a/docs/lambda-functions.md +++ b/docs/lambda-functions.md @@ -224,16 +224,21 @@ IdP-synced groups are refused. ## budget-enforcer -**Trigger:** SNS notification from AWS Budgets (200% threshold) -**Purpose:** Scales a team's ECS services to `desired_count=0` when spending exceeds 200% of their monthly budget. Does NOT destroy resources — services can be scaled back up after resolution. +**Triggers:** SNS notification from AWS Budgets (200% threshold, plain-text message); EventBridge every 6 hours with `{"action": "reconcile"}`; manual invoke with `{"action": "lift", "team": ""}` +**Purpose:** Suspends a team that spends more than 200% of its monthly budget, and keeps it suspended until the suspension is lifted. Nothing is destroyed. -**Flow:** Parse budget name (`javabin-team-{team}`) → list ECS services tagged with team → scale to zero → post Slack alert. +**Suspend (SNS):** parse `Budget Name:` (`javabin-team-{team}`) → write marker `/javabin/budget-suspensions/{team}` → scale the team's ECS services to `desired_count=0` and set their `javabin:desired-count` tag to `0` (previous counts and tag values recorded in the marker) → stop the team's `available` RDS instances (IDs recorded) → Slack. The `ecs-service` module plans the tag as `0` while the marker exists, so tf-apply and ecs-deploy keep 0 tasks. -| SSM Parameter | Channel | -|---------------|---------| +**Reconcile (schedule):** for each marker, read the team budget. Spend still above 200% of the limit (or unreadable) → re-zero services, re-pin their tag and re-stop RDS (RDS restarts stopped instances after 7 days). Otherwise (new month, raised `budget_nok`) → lift. + +**Lift:** delete the marker → set the recorded services (still at 0) and their `javabin:desired-count` tag back to their previous values → start the recorded RDS instances → Slack. See [cost-management.md](cost-management.md#budget-suspension). + +| SSM Parameter | Use | +|---------------|-----| | `/javabin/slack/platform-cost-alerts-webhook` | #javabin-cost-alerts | +| `/javabin/budget-suspensions/{team}` | Suspension marker (written and deleted here) | -**Environment vars:** `ECS_CLUSTER` (default: `javabin-platform`) +**Environment vars:** `ECS_CLUSTER` (default: `javabin-platform`), `COST_WEBHOOK_PARAM`, `BUDGET_NAME_PREFIX`, `SUSPENSION_PATH`, `AWS_ACCOUNT_ID` ## resource-tagger diff --git a/docs/reusable-modules.md b/docs/reusable-modules.md index 17e28d3..aa19f0f 100644 --- a/docs/reusable-modules.md +++ b/docs/reusable-modules.md @@ -76,6 +76,8 @@ Supports `environment` (map) and `secrets` (map of name => ARN) for container co `health_check_grace_period_seconds` (default `120`) is set on the service only when `target_group_arn` is set (AWS rejects it without a load balancer). `enable_execute_command` (default `false`) turns on ECS Exec; the task role then needs `ssmmessages:CreateControlChannel`, `CreateDataChannel`, `OpenControlChannel` and `OpenDataChannel` (the generator adds them for `compute.exec`). +**Budget suspension:** while the SSM marker `{budget_suspension_path}/{team}` exists (default path `/javabin/budget-suspensions`, written by the `budget-enforcer` Lambda), the service's `javabin:desired-count` tag is planned as `"0"` whatever `desired_count` says, so neither an apply nor the ecs-deploy after it (which scales to the tag) undoes the suspension. See [cost-management.md](cost-management.md#budget-suspension). + **Outputs:** `service_name`, `task_definition_arn`, `log_group_name` ## service-bucket diff --git a/docs/reusable-workflows.md b/docs/reusable-workflows.md index 4f402e2..c2b0bd7 100644 --- a/docs/reusable-workflows.md +++ b/docs/reusable-workflows.md @@ -191,7 +191,7 @@ Deploys the `image` input (docker-build's digest-pinned `image` output) to ECS s 1. Copies the family's latest ACTIVE task definition revision (Terraform's newest, or the last one ecs-deploy registered from it) and registers it with the app container's image set to the digest. Every task of the deploy runs exactly this commit's build, also when tf-apply changed the task definition in the same run. The revision gets the base revision's tags (or the service's, if the base has none) with `managed-by=ecs-deploy`. -2. Updates the service to that revision, with the desired count from the service tag `javabin:desired-count` (Terraform writes it from `compute.desired_count`; `budget-enforcer` sets it to 0, which keeps the service at 0 tasks until a tf-apply writes it back). +2. Updates the service to that revision, with the desired count from the service tag `javabin:desired-count` (Terraform writes it from `compute.desired_count`; `budget-enforcer` sets it to 0 and Terraform plans it as 0 while the team is budget-suspended, so the service stays at 0 tasks until the suspension is lifted; see [cost-management.md](cost-management.md#budget-suspension)). 3. Waits until that deployment's `rolloutState` is `COMPLETED` on the new revision (default timeout 12 minutes from the start of the step, inside the broker's 15-minute deploy credentials). A circuit-breaker rollback, a `FAILED` rollout, a timeout or an AWS error while waiting fails the job and prints the service events, the stopped tasks' `stoppedReason`, each container's exit code and reason, and the log group. A deployment replaced by a newer revision of a later run passes with a warning; that run reports the result. The `ecs-service` module ignores the service's `task_definition` and `desired_count` after creation, so this job, not Terraform, decides what runs. A Terraform change to the task definition (env, secrets, cpu) reaches the service at the next ecs-deploy, which follows tf-apply in the same run; if docker-build fails, it waits for the next green run. A new service is created with 0 tasks, so its first deployment starts here, once the image exists, instead of failing in tf-apply. diff --git a/scripts/ecs-deploy.sh b/scripts/ecs-deploy.sh index 3d991ac..7e8da77 100644 --- a/scripts/ecs-deploy.sh +++ b/scripts/ecs-deploy.sh @@ -12,7 +12,8 @@ # 3. Point the service at it, with the desired count from the service's # javabin:desired-count tag (Terraform writes it from app.yaml; a new # service is created with 0 tasks and gets them here; budget-enforcer -# sets it to 0 and the count stays 0 until a tf-apply writes it back). +# sets it to 0 while the team is suspended, and Terraform keeps it at 0 +# until the suspension is lifted). # 4. Wait until that deployment's rolloutState is COMPLETED. A circuit- # breaker rollback, a FAILED rollout, a timeout or an AWS error while # waiting fails the job after printing why the tasks stopped. @@ -113,7 +114,7 @@ DESIRED=$(jq -r '.tags[]? | select(.key == "javabin:desired-count") | .value' "$ case "$DESIRED" in "") echo "No javabin:desired-count tag on ${SERVICE}; keeping its desired count" ;; *[!0-9]*) fail "javabin:desired-count tag on ${SERVICE} is not a number: ${DESIRED}" ;; - 0) echo "::warning::javabin:desired-count on ${SERVICE} is 0 (budget-enforcer scaled it down, or compute.desired_count is 0): deploying with 0 tasks. tf-apply writes compute.desired_count back to the tag." ;; + 0) echo "::warning::javabin:desired-count on ${SERVICE} is 0 (budget-enforcer suspended the team, see docs/cost-management.md, or compute.desired_count is 0): deploying with 0 tasks." ;; esac echo "Updating ${SERVICE} on ${CLUSTER}${DESIRED:+ (desired count ${DESIRED})}..." diff --git a/scripts/tests/test_ecs_deploy.py b/scripts/tests/test_ecs_deploy.py index c359423..fa88286 100644 --- a/scripts/tests/test_ecs_deploy.py +++ b/scripts/tests/test_ecs_deploy.py @@ -257,7 +257,7 @@ def test_keeps_desired_count_without_tag(self): self.assertNotIn("--desired-count", calls(st, "update-service")[0]) def test_budget_suspended_service_stays_at_zero(self): - # budget-enforcer sets the tag to 0; only a tf-apply writes it back + # budget-enforcer sets the tag to 0; lifting the suspension writes it back tags = {SERVICE_ARN: [{"key": "team", "value": "cupcake"}, {"key": "javabin:desired-count", "value": "0"}]} code, out, st = deploy([COMPLETED], tags=tags) self.assertEqual(code, 0, out) diff --git a/terraform/lambda-src/budget_enforcer/handler.py b/terraform/lambda-src/budget_enforcer/handler.py index 41bc5af..fbb692f 100644 --- a/terraform/lambda-src/budget_enforcer/handler.py +++ b/terraform/lambda-src/budget_enforcer/handler.py @@ -1,20 +1,30 @@ -"""Budget enforcer — scales ECS services to zero when a team exceeds 200% of budget. - -Trigger: SNS notification from AWS Budgets (200% threshold). -Parses budget name to extract team, finds ECS services tagged with that team, -scales them to desired_count=0, and posts a Slack alert to the cost channel. - -It also sets the service's javabin:desired-count tag to 0. ecs-deploy reads the -desired count from that tag, so app pushes keep deploying with 0 tasks; the -count only comes back when a tf-apply writes compute.desired_count back to the -tag (a reviewed plan shows "javabin:desired-count" = "0" -> "N"). - -Does NOT destroy services — just scales to zero for easy recovery. +"""Budget enforcer — suspends a team that exceeds 200% of its monthly budget. + +Triggers: + - SNS notification from AWS Budgets (200% threshold): suspend the team. + - EventBridge schedule, {"action": "reconcile"}: keep every suspension in + force, and lift the ones whose budget is back under 200% (new month or a + raised budget_nok in the registry). + - Manual invoke, {"action": "lift", "team": ""}: lift early. + +A suspension is an SSM marker at {SUSPENSION_PATH}/{team}. While it exists: + - the team's ECS services stay at desired_count=0, and their + javabin:desired-count tag stays at "0". ecs-deploy sets the desired count + from that tag, so app pushes keep deploying with 0 tasks. The ecs-service + module reads the marker and plans the tag as "0" too, so a terraform apply + does not write compute.desired_count back while the team is suspended; + - the team's RDS instances are stopped. RDS starts a stopped instance again + after 7 days, so reconcile stops it again. + +Lifting deletes the marker, sets the services (and their javabin:desired-count +tag) back to the values they had before, and starts the RDS instances the enforcer stopped. Nothing is +ever deleted. """ import json import logging import os +import re from datetime import datetime, timezone import boto3 @@ -25,16 +35,90 @@ logger.setLevel(logging.INFO) ecs = boto3.client("ecs") +rds = boto3.client("rds") ssm = boto3.client("ssm") +budgets = boto3.client("budgets") CLUSTER = os.environ.get("ECS_CLUSTER", "javabin-platform") COST_WEBHOOK_PARAM = os.environ["COST_WEBHOOK_PARAM"] BUDGET_NAME_PREFIX = os.environ.get("BUDGET_NAME_PREFIX", "javabin-team-") +SUSPENSION_PATH = os.environ.get("SUSPENSION_PATH", "/javabin/budget-suspensions").rstrip("/") +ACCOUNT_ID = os.environ.get("AWS_ACCOUNT_ID", "") DESIRED_COUNT_TAG = "javabin:desired-count" +# Suspension holds while actual spend is above this share of the budget limit. +ENFORCE_RATIO = 2.0 + +# AWS Budgets SNS notifications are plain text, e.g. "Budget Name: javabin-team-x". +_TEXT_FIELD = re.compile(r"^\s*(Budget Name|Budgeted Amount)\s*:\s*\$?\s*(\S+)\s*$", re.MULTILINE) + + +# --------------------------------------------------------------------------- +# Budget notification parsing +# --------------------------------------------------------------------------- +def parse_budget_message(text): + """Return (budget_name, budget_limit) from an AWS Budgets SNS message. + + AWS Budgets sends plain text; JSON ({"budgetName", "budgetLimit"}) is + accepted too for manual tests. + """ + try: + message = json.loads(text) + except (json.JSONDecodeError, TypeError): + message = None + if isinstance(message, dict): + return message.get("budgetName", ""), (message.get("budgetLimit") or {}).get("amount") + + fields = {k: v for k, v in _TEXT_FIELD.findall(text or "")} + limit = fields.get("Budgeted Amount") + return fields.get("Budget Name", ""), limit.replace(",", "") if limit else None + + +# --------------------------------------------------------------------------- +# Suspension marker (SSM) +# --------------------------------------------------------------------------- +def marker_name(team): + return f"{SUSPENSION_PATH}/{team}" + + +def read_marker(team): + try: + resp = ssm.get_parameter(Name=marker_name(team)) + except ssm.exceptions.ParameterNotFound: + return None + try: + return json.loads(resp["Parameter"]["Value"]) + except (json.JSONDecodeError, KeyError): + return {} + + +def write_marker(team, state): + ssm.put_parameter( + Name=marker_name(team), + Value=json.dumps(state, sort_keys=True), + Type="String", + Overwrite=True, + Description=f"Budget suspension for team {team} (budget-enforcer)", + ) + + +def delete_marker(team): + try: + ssm.delete_parameter(Name=marker_name(team)) + except ssm.exceptions.ParameterNotFound: + pass + + +def suspended_teams(): + teams = [] + for page in ssm.get_paginator("get_parameters_by_path").paginate(Path=SUSPENSION_PATH): + for p in page.get("Parameters", []): + teams.append(p["Name"].rsplit("/", 1)[-1]) + return teams + # --------------------------------------------------------------------------- -# ECS helpers +# ECS # --------------------------------------------------------------------------- def list_services_for_team(cluster, team_name): """List ECS services in the cluster that are tagged with the given team.""" @@ -58,45 +142,247 @@ def list_services_for_team(cluster, team_name): return matched -def scale_service_to_zero(cluster, service_name): - """Scale an ECS service to desired_count=0.""" - ecs.update_service( - cluster=cluster, - service=service_name, - desiredCount=0, - ) - logger.info("Scaled %s to 0 in cluster %s", service_name, cluster) +def set_desired_count(cluster, service_name, count): + ecs.update_service(cluster=cluster, service=service_name, desiredCount=count) + logger.info("Set %s to desired_count=%d in cluster %s", service_name, count, cluster) -def pin_desired_count_to_zero(service_arn): - """Set javabin:desired-count to 0 so ecs-deploy does not scale it back up.""" - ecs.tag_resource( - resourceArn=service_arn, - tags=[{"key": DESIRED_COUNT_TAG, "value": "0"}], - ) - logger.info("Set %s=0 on %s", DESIRED_COUNT_TAG, service_arn) +def set_desired_count_tag(service_arn, value): + """Set javabin:desired-count, the count ecs-deploy scales the service to.""" + ecs.tag_resource(resourceArn=service_arn, tags=[{"key": DESIRED_COUNT_TAG, "value": value}]) + logger.info("Set %s=%s on %s", DESIRED_COUNT_TAG, value, service_arn) + + +def _tags(svc): + return {t["key"]: t["value"] for t in svc.get("tags", [])} + + +def scale_team_to_zero(team, previous, previous_tags): + """Scale the team's running services to 0 and pin their javabin:desired-count + tag to "0". Records each service's previous desired count in `previous` and + previous tag value in `previous_tags` (first value wins). Returns the names + scaled.""" + scaled = [] + for svc in list_services_for_team(CLUSTER, team): + name, count = svc["serviceName"], svc.get("desiredCount", 0) + tag = _tags(svc).get(DESIRED_COUNT_TAG) + if tag not in (None, "0"): + # Also before the first deploy, while the service has 0 tasks + try: + set_desired_count_tag(svc["serviceArn"], "0") + previous_tags.setdefault(name, tag) + except Exception as e: + logger.error("Failed to tag service %s: %s", name, e) + if count == 0: + continue + try: + set_desired_count(CLUSTER, name, 0) + except Exception as e: + logger.error("Failed to scale service %s: %s", name, e) + continue + previous.setdefault(name, count) + scaled.append(name) + return scaled + + +def restore_team_services(team, previous, previous_tags): + """Set services the enforcer scaled back to their previous count, and their + javabin:desired-count tag back to its previous value, if still at 0.""" + restored = [] + for svc in list_services_for_team(CLUSTER, team): + name = svc["serviceName"] + tag = previous_tags.get(name) + if tag and _tags(svc).get(DESIRED_COUNT_TAG) == "0": + try: + set_desired_count_tag(svc["serviceArn"], tag) + except Exception as e: + logger.error("Failed to restore tag on service %s: %s", name, e) + count = previous.get(name) + if not count or svc.get("desiredCount", 0) != 0: + continue + try: + set_desired_count(CLUSTER, name, count) + restored.append(name) + except Exception as e: + logger.error("Failed to restore service %s: %s", name, e) + return restored + + +# --------------------------------------------------------------------------- +# RDS +# --------------------------------------------------------------------------- +def list_db_instances_for_team(team_name): + matched = [] + for page in rds.get_paginator("describe_db_instances").paginate(): + for db in page.get("DBInstances", []): + tags = {t["Key"]: t["Value"] for t in db.get("TagList", [])} + if tags.get("team") == team_name: + matched.append(db) + return matched + + +def _stoppable(db): + # Aurora members are stopped per cluster; replicas and their sources can't be stopped. + return not (db.get("DBClusterIdentifier") + or db.get("ReadReplicaSourceDBInstanceIdentifier") + or db.get("ReadReplicaDBInstanceIdentifiers")) + + +def stop_team_databases(team, stopped): + """Stop the team's available RDS instances. Adds their IDs to `stopped`.""" + newly = [] + for db in list_db_instances_for_team(team): + ident = db["DBInstanceIdentifier"] + if db.get("DBInstanceStatus") != "available": + continue + if not _stoppable(db): + logger.info("RDS instance %s cannot be stopped (cluster or replica), skipping", ident) + continue + try: + rds.stop_db_instance(DBInstanceIdentifier=ident) + except Exception as e: + logger.error("Failed to stop RDS instance %s: %s", ident, e) + continue + logger.info("Stopped RDS instance %s", ident) + if ident not in stopped: + stopped.append(ident) + newly.append(ident) + return newly + + +def start_team_databases(stopped): + started = [] + for ident in stopped: + try: + rds.start_db_instance(DBInstanceIdentifier=ident) + started.append(ident) + logger.info("Started RDS instance %s", ident) + except rds.exceptions.InvalidDBInstanceStateFault: + logger.info("RDS instance %s is not stopped, leaving it", ident) + except Exception as e: + logger.error("Failed to start RDS instance %s: %s", ident, e) + return started + + +# --------------------------------------------------------------------------- +# Budget state +# --------------------------------------------------------------------------- +def over_limit(team): + """True while the team's actual spend this month is above 200% of its limit. + + Unknown (no budget, API error) counts as still over, so a suspension is + never lifted by accident. + """ + try: + b = budgets.describe_budget(AccountId=ACCOUNT_ID, BudgetName=f"{BUDGET_NAME_PREFIX}{team}")["Budget"] + limit = float(b["BudgetLimit"]["Amount"]) + actual = float(b["CalculatedSpend"]["ActualSpend"]["Amount"]) + except Exception as e: + logger.error("Could not read budget for team %s: %s", team, e) + return True + return actual > limit * ENFORCE_RATIO + + +# --------------------------------------------------------------------------- +# Actions +# --------------------------------------------------------------------------- +def suspend(team, budget_name, budget_limit): + state = read_marker(team) or {} + state.setdefault("suspended_at", datetime.now(timezone.utc).isoformat(timespec="seconds")) + state["budget"] = budget_name + services = state.setdefault("services", {}) + tags = state.setdefault("desired_count_tags", {}) + stopped = state.setdefault("rds", []) + # Marker first: from here on a terraform apply plans javabin:desired-count=0. + write_marker(team, state) + + scaled = scale_team_to_zero(team, services, tags) + dbs = stop_team_databases(team, stopped) + write_marker(team, state) + + notify(build_suspend_blocks(team, budget_name, scaled, dbs, budget_limit), + f"Budget enforcement: team {team} exceeded 200% — {len(scaled)} service(s) " + f"scaled to zero, {len(dbs)} database(s) stopped") + return {"team": team, "action": "suspended", "services": scaled, "rds": dbs} + + +def lift(team, reason): + state = read_marker(team) + if state is None: + return {"team": team, "action": "not_suspended"} + # Marker first: the restored count matches what terraform now plans. + delete_marker(team) + restored = restore_team_services(team, state.get("services") or {}, + state.get("desired_count_tags") or {}) + started = start_team_databases(state.get("rds") or []) + notify(build_lift_blocks(team, reason, restored, started), + f"Budget suspension lifted for team {team} ({reason})") + return {"team": team, "action": "lifted", "services": restored, "rds": started} + + +def reapply(team): + """Keep a suspension in force: re-zero services and re-stop databases.""" + state = read_marker(team) or {} + services = state.setdefault("services", {}) + tags = state.setdefault("desired_count_tags", {}) + stopped = state.setdefault("rds", []) + before = dict(tags) + scaled = scale_team_to_zero(team, services, tags) + dbs = stop_team_databases(team, stopped) + if scaled or dbs or tags != before: + write_marker(team, state) + logger.info("Re-applied suspension for %s: services=%s rds=%s", team, scaled, dbs) + return {"team": team, "action": "enforced", "services": scaled, "rds": dbs} + + +def reconcile(): + results = [] + for team in suspended_teams(): + try: + if over_limit(team): + results.append(reapply(team)) + else: + results.append(lift(team, "spend is back under 200% of budget")) + except Exception as e: + logger.error("Reconcile failed for team %s: %s", team, e) + results.append({"team": team, "action": "error", "error": str(e)[:200]}) + return results # --------------------------------------------------------------------------- # Slack notification # --------------------------------------------------------------------------- -def build_alert_blocks(team_name, budget_name, services_scaled, budget_limit): - """Build Slack Block Kit message for budget enforcement action.""" +def notify(blocks, fallback): + try: + webhook_url = get_webhook_url(ssm, COST_WEBHOOK_PARAM) + post_to_slack(webhook_url, {"blocks": blocks, "text": fallback}) + except Exception as e: + logger.error("Failed to send Slack notification: %s", e) + + +def _bullets(items, empty): + return "\n".join(f" - `{i}`" for i in items) if items else f" _{empty}_" + + +def _context(): ts = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC") + return {"type": "context", "elements": [{"type": "mrkdwn", "text": f"{ts} | Cluster: `{CLUSTER}`"}]} - service_list = "\n".join( - f" - `{svc}`" for svc in services_scaled - ) if services_scaled else " _(no matching services found)_" - nok_limit = float(budget_limit) * USD_TO_NOK if budget_limit else None +def build_suspend_blocks(team_name, budget_name, services_scaled, dbs_stopped, budget_limit): + """Build Slack Block Kit message for budget enforcement action.""" + try: + nok_limit = float(budget_limit) * USD_TO_NOK if budget_limit else None + except ValueError: + nok_limit = None limit_text = f"${budget_limit}" + (f" (~{nok_limit:.0f} NOK)" if nok_limit else "") - blocks = [ + return [ { "type": "header", "text": { "type": "plain_text", - "text": ":rotating_light: Budget Enforcement — Services Scaled to Zero", + "text": ":rotating_light: Budget Enforcement — Team Suspended", "emoji": True, }, }, @@ -107,114 +393,85 @@ def build_alert_blocks(team_name, budget_name, services_scaled, budget_limit): "text": ( f"Team *{team_name}* has exceeded *200%* of their monthly budget " f"(`{budget_name}`, limit: {limit_text}).\n\n" - f"The following ECS services have been scaled to `desired_count=0`:" + "ECS services scaled to `desired_count=0`:\n" + + _bullets(services_scaled, "(none running)") + + "\n\nRDS instances stopped:\n" + + _bullets(dbs_stopped, "(none running)") ), }, }, + {"type": "divider"}, { "type": "section", "text": { "type": "mrkdwn", - "text": service_list, + "text": ( + f":lock: The suspension holds until it is lifted: the services' " + "`javabin:desired-count` tag is `0`, so deploys and terraform applies keep " + f"them at 0 tasks while `{marker_name(team_name)}` exists.\n" + ":arrow_right: *To lift it*, raise `budget_nok` in the registry (lifted within " + "6 hours), wait for the next month, or ask a platform admin to invoke " + "`javabin-budget-enforcer` with `{\"action\": \"lift\", \"team\": \"" + f"{team_name}\"}}`. Services and databases come back on their own." + ), }, }, - {"type": "divider"}, + _context(), + ] + + +def build_lift_blocks(team_name, reason, restored, started): + return [ + { + "type": "header", + "text": {"type": "plain_text", "text": ":white_check_mark: Budget Suspension Lifted", "emoji": True}, + }, { "type": "section", "text": { "type": "mrkdwn", "text": ( - ":arrow_right: *To restore services*, increase the team budget in " - "the registry, then run the app's pipeline: its tf-apply writes " - "`compute.desired_count` back to the `javabin:desired-count` tag " - "and ecs-deploy scales the service up. App pushes alone keep it at 0." + f"Team *{team_name}* is no longer suspended ({reason}).\n\n" + "ECS services restored:\n" + _bullets(restored, "(none)") + + "\n\nRDS instances started:\n" + _bullets(started, "(none)") + + "\n\nServices created during the suspension come up on the next deploy." ), }, }, - { - "type": "context", - "elements": [ - { - "type": "mrkdwn", - "text": f"Enforced at {ts} | Cluster: `{CLUSTER}`", - } - ], - }, + _context(), ] - return blocks # --------------------------------------------------------------------------- # Lambda entry point # --------------------------------------------------------------------------- def handler(event, context): + action = event.get("action") + if action == "reconcile": + return {"statusCode": 200, "results": reconcile()} + if action == "lift": + team = event.get("team", "") + if not team: + return {"statusCode": 400, "body": "team is required"} + return {"statusCode": 200, "results": [lift(team, "lifted manually")]} + + results = [] for record in event.get("Records", []): - sns_message = record.get("Sns", {}).get("Message", "{}") - - try: - message = json.loads(sns_message) - except json.JSONDecodeError: - logger.error("Failed to parse SNS message: %s", sns_message) + sns_message = record.get("Sns", {}).get("Message", "") + budget_name, budget_limit = parse_budget_message(sns_message) + if not budget_name: + logger.error("No budget name in SNS message: %s", sns_message[:500]) continue - - budget_name = message.get("budgetName", "") if not budget_name.startswith(BUDGET_NAME_PREFIX): - logger.info( - "Ignoring budget notification for %s (not a team budget)", budget_name - ) + logger.info("Ignoring budget notification for %s (not a team budget)", budget_name) continue team_name = budget_name[len(BUDGET_NAME_PREFIX) :] - if not team_name: + if not team_name or "/" in team_name: logger.error("Could not extract team name from budget: %s", budget_name) continue - budget_limit = message.get("budgetLimit", {}).get("amount") - - logger.info( - "Budget enforcement triggered for team %s (budget: %s)", - team_name, - budget_name, - ) - - # Find and scale down team's ECS services - services = list_services_for_team(CLUSTER, team_name) - services_scaled = [] - - for svc in services: - svc_name = svc["serviceName"] - tags = {t["key"]: t["value"] for t in svc.get("tags", [])} - if tags.get(DESIRED_COUNT_TAG, "0") != "0": - # Also before the first deploy, while the service has 0 tasks - try: - pin_desired_count_to_zero(svc["serviceArn"]) - except Exception as e: - logger.error("Failed to tag service %s: %s", svc_name, e) - current_count = svc.get("desiredCount", 0) - if current_count == 0: - logger.info( - "Service %s already at desired_count=0, skipping", svc_name - ) - continue - try: - scale_service_to_zero(CLUSTER, svc_name) - services_scaled.append(svc_name) - except Exception as e: - logger.error("Failed to scale service %s: %s", svc_name, e) - - # Post Slack notification regardless of whether services were found - blocks = build_alert_blocks( - team_name, budget_name, services_scaled, budget_limit - ) - fallback = ( - f"Budget enforcement: team {team_name} exceeded 200% — " - f"{len(services_scaled)} service(s) scaled to zero" - ) - - try: - webhook_url = get_webhook_url(ssm, COST_WEBHOOK_PARAM) - post_to_slack(webhook_url, {"blocks": blocks, "text": fallback}) - except Exception as e: - logger.error("Failed to send Slack notification: %s", e) + logger.info("Budget enforcement triggered for team %s (budget: %s)", team_name, budget_name) + results.append(suspend(team_name, budget_name, budget_limit)) - return {"statusCode": 200, "body": "OK"} + return {"statusCode": 200, "body": "OK", "results": results} diff --git a/terraform/lambda-src/tests/test_budget_enforcer.py b/terraform/lambda-src/tests/test_budget_enforcer.py index bcd4ba5..e72595a 100644 --- a/terraform/lambda-src/tests/test_budget_enforcer.py +++ b/terraform/lambda-src/tests/test_budget_enforcer.py @@ -8,35 +8,151 @@ import unittest LAMBDA_SRC = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") + +PATH = "/javabin/budget-suspensions" ARN = "arn:aws:ecs:eu-central-1:111111111111:service/javabin-platform/" +TAG = "javabin:desired-count" + +BUDGET_TEXT = """AWS Budget Notification November 12, 2026 +AWS Account 111111111111 + +Dear AWS Customer, + +You requested that we alert you when the ACTUAL Cost associated with your javabin-team-cupcake budget is greater than $95.24 for the current month. The ACTUAL Cost associated with this budget is $96.10. + +Budget Name: javabin-team-cupcake +Budget Type: Cost +Budgeted Amount: $47.62 +Alert Type: ACTUAL +Alert Threshold: > $95.24 +ACTUAL Amount: $96.10 +""" + + +class ParameterNotFound(Exception): + pass + + +class InvalidDBInstanceStateFault(Exception): + pass + + +class Paginator: + def __init__(self, fn): + self.fn = fn + + def paginate(self, **kw): + return [self.fn(**kw)] + +class FakeSSM: + exceptions = types.SimpleNamespace(ParameterNotFound=ParameterNotFound) -class FakeEcs: - def __init__(self, services): - self.services = services - self.updated = [] + def __init__(self): + self.params = {"/x/cost": "https://hooks.example/cost"} + + def get_parameter(self, Name, WithDecryption=False): + if Name not in self.params: + raise ParameterNotFound(Name) + return {"Parameter": {"Name": Name, "Value": self.params[Name]}} + + def put_parameter(self, Name, Value, **kw): + self.params[Name] = Value + + def delete_parameter(self, Name): + if Name not in self.params: + raise ParameterNotFound(Name) + del self.params[Name] + + def get_paginator(self, name): + assert name == "get_parameters_by_path" + return Paginator(lambda Path: {"Parameters": [ + {"Name": n} for n in self.params if n.startswith(Path + "/") and "/" not in n[len(Path) + 1:] + ]}) + + +class FakeECS: + def __init__(self, services, desired_tags=None): + # name -> (team, desired); name -> javabin:desired-count tag value + self.services = dict(services) + self.desired_tags = dict(desired_tags or {}) + self.updates = [] self.tagged = [] def get_paginator(self, name): - services = self.services - return types.SimpleNamespace( - paginate=lambda **kw: [{"serviceArns": [s["serviceArn"] for s in services]}]) + return Paginator(lambda cluster: {"serviceArns": [ARN + n for n in self.services]}) def describe_services(self, cluster, services, include): - return {"services": [s for s in self.services if s["serviceArn"] in services]} + out = [] + for arn in services: + n = arn[len(ARN):] + tags = [{"key": "team", "value": self.services[n][0]}] + if n in self.desired_tags: + tags.append({"key": TAG, "value": self.desired_tags[n]}) + out.append({"serviceArn": arn, "serviceName": n, + "desiredCount": self.services[n][1], "tags": tags}) + return {"services": out} + + def tag_resource(self, resourceArn, tags): + for t in tags: + assert t["key"] == TAG + self.desired_tags[resourceArn[len(ARN):]] = t["value"] + self.tagged.append((resourceArn[len(ARN):], tags[0]["value"])) def update_service(self, cluster, service, desiredCount): - self.updated.append((service, desiredCount)) + team, _ = self.services[service] + self.services[service] = (team, desiredCount) + self.updates.append((service, desiredCount)) - def tag_resource(self, resourceArn, tags): - self.tagged.append((resourceArn, tags)) +class FakeRDS: + exceptions = types.SimpleNamespace(InvalidDBInstanceStateFault=InvalidDBInstanceStateFault) + + def __init__(self, dbs): + self.dbs = dbs # list of dicts + self.stopped, self.started = [], [] + + def get_paginator(self, name): + return Paginator(lambda: {"DBInstances": self.dbs}) -def load(ecs): + def _db(self, ident): + return next(d for d in self.dbs if d["DBInstanceIdentifier"] == ident) + + def stop_db_instance(self, DBInstanceIdentifier): + self._db(DBInstanceIdentifier)["DBInstanceStatus"] = "stopping" + self.stopped.append(DBInstanceIdentifier) + + def start_db_instance(self, DBInstanceIdentifier): + db = self._db(DBInstanceIdentifier) + if db["DBInstanceStatus"] != "stopped": + raise InvalidDBInstanceStateFault(DBInstanceIdentifier) + db["DBInstanceStatus"] = "starting" + self.started.append(DBInstanceIdentifier) + + +class FakeBudgets: + def __init__(self, limit, actual): + self.limit, self.actual = limit, actual + + def describe_budget(self, AccountId, BudgetName): + if self.limit is None: + raise RuntimeError("no budget") + return {"Budget": {"BudgetLimit": {"Amount": str(self.limit)}, + "CalculatedSpend": {"ActualSpend": {"Amount": str(self.actual)}}}} + + +def db(ident, team, status="available", **extra): + return {"DBInstanceIdentifier": ident, "DBInstanceStatus": status, + "TagList": [{"Key": "team", "Value": team}], **extra} + + +def load(ecs, rds, ssm, budgets): + clients = {"ecs": ecs, "rds": rds, "ssm": ssm, "budgets": budgets} boto3 = types.ModuleType("boto3") - boto3.client = lambda name, *a, **kw: ecs if name == "ecs" else object() + boto3.client = lambda name, *a, **kw: clients[name] sys.modules["boto3"] = boto3 - os.environ["COST_WEBHOOK_PARAM"] = "/x/cost" + os.environ.update({"COST_WEBHOOK_PARAM": "/x/cost", "SUSPENSION_PATH": PATH, + "AWS_ACCOUNT_ID": "111111111111"}) if LAMBDA_SRC not in sys.path: sys.path.insert(0, LAMBDA_SRC) for name in [m for m in sys.modules if m == "shared" or m.startswith("shared.")]: @@ -45,42 +161,124 @@ def load(ecs): "budget_enforcer_handler", os.path.join(LAMBDA_SRC, "budget_enforcer", "handler.py")) mod = importlib.util.module_from_spec(spec) spec.loader.exec_module(mod) - mod.get_webhook_url = lambda *a: "https://hooks.example/x" - mod.post_to_slack = lambda *a: None + mod.post_to_slack = lambda url, msg: mod._posted.append(msg) + mod._posted = [] return mod -def svc(name, count, desired_tag, team="cupcake"): - tags = [{"key": "team", "value": team}] - if desired_tag is not None: - tags.append({"key": "javabin:desired-count", "value": desired_tag}) - return {"serviceArn": ARN + name, "serviceName": name, "desiredCount": count, "tags": tags} +def sns(message): + return {"Records": [{"Sns": {"Message": message}}]} + + +class BudgetEnforcerTest(unittest.TestCase): + def setUp(self): + self.ecs = FakeECS({"cupcake-web": ("cupcake", 2), "cupcake-idle": ("cupcake", 0), + "other-web": ("other", 1)}, + {"cupcake-web": "2", "other-web": "1"}) + self.rds = FakeRDS([db("cupcake-db", "cupcake"), db("other-db", "other"), + db("cupcake-aurora", "cupcake", DBClusterIdentifier="c")]) + self.ssm = FakeSSM() + self.budgets = FakeBudgets(47.62, 96.10) + self.mod = load(self.ecs, self.rds, self.ssm, self.budgets) + + def marker(self): + return json.loads(self.ssm.params[f"{PATH}/cupcake"]) + + def test_parses_plain_text_budget_notification(self): + self.assertEqual(self.mod.parse_budget_message(BUDGET_TEXT), ("javabin-team-cupcake", "47.62")) + def test_parses_json_notification(self): + msg = json.dumps({"budgetName": "javabin-team-x", "budgetLimit": {"amount": "5"}}) + self.assertEqual(self.mod.parse_budget_message(msg), ("javabin-team-x", "5")) -def breach(team="cupcake"): - msg = {"budgetName": f"javabin-team-{team}", "budgetLimit": {"amount": "50"}} - return {"Records": [{"Sns": {"Message": json.dumps(msg)}}]} + def test_suspend_writes_marker_scales_services_and_stops_team_rds(self): + self.mod.handler(sns(BUDGET_TEXT), None) + self.assertEqual(self.ecs.services["cupcake-web"], ("cupcake", 0)) + self.assertEqual(self.ecs.services["other-web"], ("other", 1)) + self.assertEqual(self.rds.stopped, ["cupcake-db"]) + state = self.marker() + self.assertEqual(state["services"], {"cupcake-web": 2}) + self.assertEqual(state["rds"], ["cupcake-db"]) + self.assertEqual(len(self.mod._posted), 1) -class BudgetEnforcerTests(unittest.TestCase): - def test_scales_to_zero_and_pins_desired_count_tag(self): - ecs = FakeEcs([svc("cupcake-cupcake", 2, "2"), svc("other-app", 1, "1", team="other")]) - load(ecs).handler(breach(), None) - self.assertEqual(ecs.updated, [("cupcake-cupcake", 0)]) - self.assertEqual(ecs.tagged, [(ARN + "cupcake-cupcake", [{"key": "javabin:desired-count", "value": "0"}])]) + def test_suspend_pins_desired_count_tag_so_ecs_deploy_keeps_zero(self): + self.mod.handler(sns(BUDGET_TEXT), None) + self.assertEqual(self.ecs.desired_tags, {"cupcake-web": "0", "other-web": "1"}) + self.assertEqual(self.marker()["desired_count_tags"], {"cupcake-web": "2"}) - def test_pins_tag_of_a_service_not_deployed_yet(self): + def test_suspend_pins_tag_of_a_service_not_deployed_yet(self): # Created with 0 tasks; ecs-deploy would otherwise scale it to the tag - ecs = FakeEcs([svc("cupcake-new", 0, "1")]) - load(ecs).handler(breach(), None) - self.assertEqual(ecs.updated, []) - self.assertEqual(len(ecs.tagged), 1) - - def test_leaves_already_pinned_or_untagged_services_untagged(self): - ecs = FakeEcs([svc("cupcake-a", 1, "0"), svc("cupcake-b", 1, None)]) - load(ecs).handler(breach(), None) - self.assertEqual(ecs.tagged, []) - self.assertEqual(sorted(ecs.updated), [("cupcake-a", 0), ("cupcake-b", 0)]) + self.ecs.desired_tags["cupcake-idle"] = "1" + self.mod.handler(sns(BUDGET_TEXT), None) + self.assertNotIn(("cupcake-idle", 0), self.ecs.updates) + self.assertEqual(self.ecs.desired_tags["cupcake-idle"], "0") + + def test_suspend_leaves_already_pinned_or_untagged_services_untagged(self): + self.ecs.services["cupcake-web"] = ("cupcake", 1) + self.ecs.desired_tags["cupcake-web"] = "0" + self.mod.handler(sns(BUDGET_TEXT), None) + self.assertEqual(self.ecs.tagged, []) + self.assertEqual(self.ecs.services["cupcake-web"], ("cupcake", 0)) + + def test_ignores_non_team_budget(self): + self.mod.handler(sns(BUDGET_TEXT.replace("javabin-team-cupcake", "javabin-account-monthly")), None) + self.assertEqual(self.ecs.updates, []) + self.assertNotIn(f"{PATH}/cupcake", self.ssm.params) + + def test_reconcile_reapplies_while_over_limit(self): + self.mod.handler(sns(BUDGET_TEXT), None) + # Someone scales up by hand; RDS auto-starts after 7 days. + self.ecs.services["cupcake-web"] = ("cupcake", 3) + self.rds._db("cupcake-db")["DBInstanceStatus"] = "available" + + self.mod.handler({"action": "reconcile"}, None) + + self.assertEqual(self.ecs.services["cupcake-web"], ("cupcake", 0)) + self.assertEqual(self.rds.stopped, ["cupcake-db", "cupcake-db"]) + # The original count is what lift restores, not the manual one. + self.assertEqual(self.marker()["services"], {"cupcake-web": 2}) + + def test_reconcile_lifts_when_spend_back_under_limit(self): + self.mod.handler(sns(BUDGET_TEXT), None) + self.rds._db("cupcake-db")["DBInstanceStatus"] = "stopped" + self.budgets.limit = 100 # budget_nok raised in the registry + + out = self.mod.handler({"action": "reconcile"}, None) + + self.assertEqual(out["results"][0]["action"], "lifted") + self.assertNotIn(f"{PATH}/cupcake", self.ssm.params) + self.assertEqual(self.ecs.services["cupcake-web"], ("cupcake", 2)) + self.assertEqual(self.ecs.services["cupcake-idle"], ("cupcake", 0)) + self.assertEqual(self.ecs.desired_tags, {"cupcake-web": "2", "other-web": "1"}) + self.assertEqual(self.rds.started, ["cupcake-db"]) + + def test_reconcile_repins_tag_reset_by_hand(self): + self.mod.handler(sns(BUDGET_TEXT), None) + self.ecs.desired_tags["cupcake-web"] = "3" + self.mod.handler({"action": "reconcile"}, None) + self.assertEqual(self.ecs.desired_tags["cupcake-web"], "0") + # Lift restores the value from before the suspension. + self.assertEqual(self.marker()["desired_count_tags"], {"cupcake-web": "2"}) + + def test_reconcile_keeps_suspension_if_budget_unreadable(self): + self.mod.handler(sns(BUDGET_TEXT), None) + self.budgets.limit = None + self.mod.handler({"action": "reconcile"}, None) + self.assertIn(f"{PATH}/cupcake", self.ssm.params) + + def test_manual_lift(self): + self.mod.handler(sns(BUDGET_TEXT), None) + out = self.mod.handler({"action": "lift", "team": "cupcake"}, None) + self.assertEqual(out["results"][0]["action"], "lifted") + self.assertEqual(self.ecs.services["cupcake-web"], ("cupcake", 2)) + # RDS still stopping: start is skipped, not an error. + self.assertEqual(self.rds.started, []) + + def test_lift_without_suspension_is_noop(self): + out = self.mod.handler({"action": "lift", "team": "cupcake"}, None) + self.assertEqual(out["results"][0]["action"], "not_suspended") + self.assertEqual(self.ecs.updates, []) if __name__ == "__main__": diff --git a/terraform/modules/ecs-service/budget.tf b/terraform/modules/ecs-service/budget.tf new file mode 100644 index 0000000..9e3ae61 --- /dev/null +++ b/terraform/modules/ecs-service/budget.tf @@ -0,0 +1,23 @@ +################################################################################ +# Budget suspension +# +# The budget-enforcer Lambda scales a team's services to 0 when it passes 200% +# of its monthly budget, sets their javabin:desired-count tag to "0" and writes +# {budget_suspension_path}/{team}. Planning the tag as "0" while that marker +# exists keeps the next apply from writing compute.desired_count back, so the +# ecs-deploy after it (which scales to the tag) keeps 0 tasks. The Lambda +# deletes the marker and restores the count and tag when the suspension is +# lifted (docs/cost-management.md). +# +# By path, not a single-parameter lookup: a missing marker returns an empty +# list instead of failing the plan. +################################################################################ + +data "aws_ssm_parameters_by_path" "budget_suspension" { + path = var.budget_suspension_path + with_decryption = false +} + +locals { + budget_suspended = contains(data.aws_ssm_parameters_by_path.budget_suspension.names, "${trimsuffix(var.budget_suspension_path, "/")}/${var.team}") +} diff --git a/terraform/modules/ecs-service/main.tf b/terraform/modules/ecs-service/main.tf index 2f71c59..1e97dca 100644 --- a/terraform/modules/ecs-service/main.tf +++ b/terraform/modules/ecs-service/main.tf @@ -115,7 +115,8 @@ resource "aws_ecs_task_definition" "this" { # The ecs-deploy workflow owns which image runs and how many tasks: each deploy # registers a copy of the family's latest revision pinned to the image digest # built from the commit, and sets the desired count from the -# javabin:desired-count tag below. Terraform sets both only at creation, with +# javabin:desired-count tag below (0 while the team is budget-suspended, see +# budget.tf). Terraform sets both only at creation, with # 0 tasks, so a new service does not start (and fail) before its first image # is pushed. A task definition change here reaches the service on the next # deploy, which follows tf-apply in the same pipeline run. @@ -168,8 +169,10 @@ resource "aws_ecs_service" "this" { enable_ecs_managed_tags = true propagate_tags = "SERVICE" + # "0" while the team is budget-suspended (budget.tf), so neither this apply + # nor the deploy after it scales the service back up. tags = { - "javabin:desired-count" = tostring(var.desired_count) + "javabin:desired-count" = local.budget_suspended ? "0" : tostring(var.desired_count) } lifecycle { diff --git a/terraform/modules/ecs-service/variables.tf b/terraform/modules/ecs-service/variables.tf index e734948..230cfe6 100644 --- a/terraform/modules/ecs-service/variables.tf +++ b/terraform/modules/ecs-service/variables.tf @@ -142,3 +142,9 @@ variable "readonly_root_filesystem" { type = bool default = true } + +variable "budget_suspension_path" { + description = "SSM path of budget suspension markers ({path}/{team}, written by the budget-enforcer Lambda). While the team's marker exists the service is planned at desired_count=0." + type = string + default = "/javabin/budget-suspensions" +} diff --git a/terraform/platform/lambdas/main.tf b/terraform/platform/lambdas/main.tf index 3cfb888..54cc502 100644 --- a/terraform/platform/lambdas/main.tf +++ b/terraform/platform/lambdas/main.tf @@ -1109,6 +1109,44 @@ resource "aws_iam_role_policy" "budget_enforcer" { Action = ["ecs:UpdateService", "ecs:TagResource"] Resource = "arn:aws:ecs:${var.region}:${var.aws_account_id}:service/${var.project}-platform/*" }, + { + # Suspension markers: read by the ecs-service module in app Terraform + Sid = "SuspensionMarkers" + Effect = "Allow" + Action = [ + "ssm:GetParameter", + "ssm:GetParametersByPath", + "ssm:PutParameter", + "ssm:DeleteParameter", + ] + Resource = [ + "arn:aws:ssm:${var.region}:${var.aws_account_id}:parameter/${var.project}/budget-suspensions", + "arn:aws:ssm:${var.region}:${var.aws_account_id}:parameter/${var.project}/budget-suspensions/*", + ] + }, + { + Sid = "RDSDescribe" + Effect = "Allow" + Action = "rds:DescribeDBInstances" + Resource = "arn:aws:rds:${var.region}:${var.aws_account_id}:db:*" + }, + { + # Only team-tagged instances; the handler matches the team tag + Sid = "RDSStopStart" + Effect = "Allow" + Action = ["rds:StopDBInstance", "rds:StartDBInstance"] + Resource = "arn:aws:rds:${var.region}:${var.aws_account_id}:db:*" + Condition = { + Null = { "aws:ResourceTag/team" = "false" } + } + }, + { + # Lift a suspension once spend is back under 200% + Sid = "BudgetsRead" + Effect = "Allow" + Action = "budgets:ViewBudget" + Resource = "arn:aws:budgets::${var.aws_account_id}:budget/${var.project}-team-*" + }, ] }) } @@ -1133,6 +1171,8 @@ resource "aws_lambda_function" "budget_enforcer" { ECS_CLUSTER = "${var.project}-platform" COST_WEBHOOK_PARAM = "/javabin/slack/platform-cost-alerts-webhook" BUDGET_NAME_PREFIX = "${var.project}-team-" + SUSPENSION_PATH = "/${var.project}/budget-suspensions" + AWS_ACCOUNT_ID = var.aws_account_id } } } @@ -1151,6 +1191,30 @@ resource "aws_lambda_permission" "budget_enforcer_sns" { source_arn = aws_sns_topic.budget_enforcement.arn } +# Keeps suspensions in force (re-stops RDS, which restarts itself after 7 +# days; re-zeros services scaled up by hand) and lifts them once the team's +# spend is back under 200% (new month or raised budget_nok). +resource "aws_cloudwatch_event_rule" "budget_enforcer_reconcile" { + name = "${var.project}-budget-enforcer-reconcile" + description = "Reconcile team budget suspensions every 6 hours" + schedule_expression = "rate(6 hours)" +} + +resource "aws_cloudwatch_event_target" "budget_enforcer_reconcile" { + rule = aws_cloudwatch_event_rule.budget_enforcer_reconcile.name + target_id = "reconcile-budget-enforcer" + arn = aws_lambda_function.budget_enforcer.arn + input = jsonencode({ action = "reconcile" }) +} + +resource "aws_lambda_permission" "budget_enforcer_reconcile" { + statement_id = "AllowEventBridgeReconcile" + action = "lambda:InvokeFunction" + function_name = aws_lambda_function.budget_enforcer.function_name + principal = "events.amazonaws.com" + source_arn = aws_cloudwatch_event_rule.budget_enforcer_reconcile.arn +} + ################################################################################ # SNS Subscriptions — slack-alert subscribes to both topics ################################################################################