AWS Setup Runbook

Purpose: the day-to-day operator's guide to the AskFlorence AWS environment. How we authenticate, how we deploy, how we read logs, how we roll things back, and which AWS surface area belongs to which environment. Paired with the deeper cloudtrail-setup.md, guardduty-setup.md, and phase-specific network docs under this directory.

Account topology

Account name	Account ID	Purpose	SSO permission sets available
`askflorence-management`	778477254880	AWS Organizations root; billing; Terraform state backend; the pre-existing `askflorence-data` S3 bucket	`AdminAccess`, `PowerUserAccess`, `BillingReadOnly`, `SecurityAudit`
`askflorence-prod`	039624954211	Production workloads (ECS, ALB, CloudFront, WAF, prod Secrets Manager). Provisioned at Phase 8; Phase 1/2/2.5 baseline only as of Phase 5	`AdminAccess` (break-glass), `PowerUserAccess`, `SecurityAudit`
`askflorence-staging`	549136075525	Pre-prod validation; Phase 5 staging stack reachable at `stage.askflorence.health`	`AdminAccess`, `PowerUserAccess`, `SecurityAudit`
`askflorence-log-archive`	754660694122	Immutable audit trail destination (CloudTrail, Config, WAF, VPC Flow Logs). Object-lock COMPLIANCE 7-year retention on tfstate + CloudTrail buckets	`SecurityAudit`, `PowerUserAccess` (limited — no workload resources live here)

Details + provenance: account-inventory.md.

Access + profiles

Day-to-day human access is via AWS IAM Identity Center (SSO). Long-lived IAM users, access keys, and static credentials are not used anywhere. CI/CD uses GitHub Actions OIDC federation to short-lived STS role assumptions per environment.

Local `~/.aws/config` profiles

These are the canonical profile names referenced throughout the runbooks:

ini

[profile askflorence]
sso_session     = askflorence
sso_account_id  = 778477254880
sso_role_name   = AdministratorAccess
region          = us-east-1

[profile askflorence-prod]
sso_session     = askflorence
sso_account_id  = 039624954211
sso_role_name   = AdministratorAccess
region          = us-east-1

[profile askflorence-staging]
sso_session     = askflorence
sso_account_id  = 549136075525
sso_role_name   = AdministratorAccess
region          = us-east-1

[profile askflorence-log-archive]
sso_session     = askflorence
sso_account_id  = 754660694122
sso_role_name   = AdministratorAccess
region          = us-east-1

[sso-session askflorence]
sso_start_url   = https://askflorence.awsapps.com/start
sso_region      = us-east-1
sso_registration_scopes = sso:account:access

Authenticating

bash

aws sso login --profile askflorence
# Browser opens → approve → tokens cached for ~8h
aws sts get-caller-identity --profile askflorence-staging   # sanity check

Tokens expire. If you see Error loading SSO Token or credentials not found, re-run aws sso login. The rest of this doc assumes you're logged in and have set AWS_PROFILE=askflorence-<env> before each command.

Deploy a change to prod

Prod is on a manual workflow_dispatch trigger — no deploys happen without an explicit click. GitHub Team plan doesn't support required-reviewers on private-repo environments, so workflow_dispatch is the approval surrogate.

bash

# Browser:
#   https://github.com/askflorencehealth/ask-florence/actions/workflows/deploy-prod.yml
#   → Run workflow → leave ref as `main` → Run workflow

# CLI equivalent:
gh workflow run deploy-prod.yml --ref main

The workflow:

OIDC-assumes arn:aws:iam::039624954211:role/GitHubActionsDeployRole
Builds + pushes image to prod ECR (immutable tags, GHA cache backend — no :latest on prod)
Renders a fresh task-def revision with the new :<sha> image
aws ecs deploy-task-definition with wait-for-service-stability (up to 15 min)
On first deploy only: scales desired 0 → 2
Smoke GET /api/health against origin.askflorence.health (direct ALB, bypassing CloudFront + WAF — avoids GitHub runner IP false-positives in WAF's AnonymousIpList/AmazonIpReputationList)

Watch live:

bash

gh run watch $(gh run list --workflow=deploy-prod.yml --limit 1 --json databaseId -q '.[0].databaseId')

Rollback: re-run with ref: <previous-sha> via CLI — gh workflow run deploy-prod.yml --ref <sha>. The workflow checks out that ref, builds its image, and deploys.

Prod-only caveats:

Immutable tags: every tag pushed to prod ECR is permanent. No :latest — each deploy pins its own :<sha>. Rollback is a separate build from the same commit, not a tag move.
:latest is NOT pushed on prod. If anything references :latest for prod, it's looking at a stale image. Always reference the SHA.
Deletion protection ON: the prod ALB cannot be deleted by terraform destroy without first setting enable_deletion_protection = false and re-applying. This is intentional.

Before any invocation: confirm the canary hostname prod-canary.askflorence.health + origin.askflorence.health resolve through Cloudflare correctly. If DNS breaks, the workflow's smoke step will fail even with a healthy deploy.

Atlas gotcha re-run from Phase 7: if the prod Atlas IP allowlist gets tightened and an already-running ECS task holds stale DNS state, force-new-deployment. Not applicable during normal deploys — only comes up after networking changes.

Deploy a change to staging

bash

# 1. Land the code change on main, then fast-forward staging to main.
git checkout main && git pull
# ... make changes, commit ...
git push origin main
git checkout staging
git merge --ff-only main
git push origin staging         # triggers .github/workflows/deploy-staging.yml

The deploy-staging.yml workflow does: OIDC-assume GitHubActionsDeployRole in staging → ECR login → docker buildx build --push (amd64, with PostHog build args + DEPLOY_ENV=staging) → pull current task def → render with new image via amazon-ecs-render-task-definition → register + deploy via amazon-ecs-deploy-task-definition (wait-for-stability, 10-min cap) → smoke test GET https://stage.askflorence.health/api/health.

Watch a run:

bash

gh run watch --exit-status $(gh run list --workflow=deploy-staging.yml --limit 1 --json databaseId -q '.[0].databaseId')

Prod deploy flow will be identical but against askflorence-prod via a deploy-prod.yml workflow that requires a manual approval on a protected GitHub environment. Lands in Phase 8.

Reading application logs

bash

export AWS_PROFILE=askflorence-staging
aws logs tail /aws/ecs/askflorence-staging-app --region us-east-1 --since 5m --format short --follow

Per-container streams named app/app/<task-id>. Retention 14 days (staging). Prod retention will be 90 days hot + long-term archive to log-archive S3.

Updating secrets

Secrets live in the workload account (staging/* and prod/* namespaces), CMK-encrypted (staging: alias/askflorence-staging-data; prod will be alias/askflorence-prod-data). Never pass secret values on the command line — always via a mode-600 temp file:

bash

export AWS_PROFILE=askflorence-staging
TMP=$(mktemp -t secret-XXXX)
trap 'rm -f "$TMP"' EXIT
printf '%s' "$NEW_VALUE" > "$TMP"
chmod 600 "$TMP"
aws secretsmanager put-secret-value \
  --secret-id staging/<secret-name> \
  --secret-string "file://$TMP" \
  --region us-east-1

Secret values are injected into ECS task containers via the task execution role's secretsmanager:GetSecretValue + kms:Decrypt on the specific ARNs. Secrets are only re-read when a new task starts. For a running secret change to take effect, force a deployment:

bash

aws ecs update-service --cluster askflorence-staging --service askflorence-staging-app --force-new-deployment --region us-east-1

The end-to-end script for populating all staging Mongo URIs from .env.staging.local lives at scripts/aws/populate-staging-secrets.sh.

Registering a task definition revision out of band

The Terraform ecs-service module sets lifecycle { ignore_changes = [container_definitions] }, which means terraform apply alone will not push env-var changes onto the running task. GitHub Actions CI/CD is the intended path. For one-off env changes between deploys, register a fresh revision manually:

bash

export AWS_PROFILE=askflorence-staging
aws ecs describe-task-definition \
  --task-definition askflorence-staging-app-task \
  --region us-east-1 \
  --query taskDefinition > /tmp/td.json

# Mutate /tmp/td.json (e.g. add/remove env vars), then:
python3 -c '
import json, sys
td = json.load(open("/tmp/td.json"))
for k in ("taskDefinitionArn","revision","status","requiresAttributes","compatibilities","registeredAt","registeredBy"):
    td.pop(k, None)
json.dump(td, open("/tmp/td-new.json","w"))
'

aws ecs register-task-definition --cli-input-json file:///tmp/td-new.json --region us-east-1
aws ecs update-service --cluster askflorence-staging --service askflorence-staging-app \
  --task-definition askflorence-staging-app-task \
  --region us-east-1 --force-new-deployment

Next GH Actions deploy picks up the new revision as its base and layers the new image on top.

Rollback

Code revert — git revert <sha>, push to staging (or main + fast-forward staging), GH Actions redeploys the prior image.

Image revert — point the ECS service at an older task def revision:

bash

aws ecs update-service --cluster askflorence-staging --service askflorence-staging-app \
  --task-definition askflorence-staging-app-task:<old-revision> --region us-east-1

Infra revert — terraform apply against a prior commit under infra/envs/staging/. State backend handles locking via askflorence-tfstate-locks DynamoDB.
DNS revert (if Phase 10 cutover is live and something breaks on AWS prod) — in Cloudflare, flip the askflorence.health CNAME back from the prod CloudFront distribution to the preserved Vercel deployment. TTL on that record is kept low (300s) for the first 72h after cutover for exactly this reason.

What lives where

Concern	Source of truth
Infra definitions	`infra/envs/<env>/` + `infra/modules/`
Terraform state	S3 `askflorence-tfstate-778477254880` (mgmt), DynamoDB locks `askflorence-tfstate-locks`
Secrets	AWS Secrets Manager, `staging/` in 549136075525, `prod/` in 039624954211
CI/CD	`.github/workflows/deploy-staging.yml` (Phase 5) + `deploy-prod.yml` (Phase 8)
Organizations baseline	`aws-organizations.md`, `cloudtrail-setup.md`, `guardduty-setup.md`, `security-hub-setup.md`, `config-setup.md`
Networking	`phase-4-staging-dns-records.md` + per-env `infra/envs/<env>/network.tf`