CTN ASR Deployment Procedures

This document describes how the Association Register is built, deployed, configured and operated. It is cloud-agnostic by intent — Azure is the current reference implementation, but every step has a documented equivalent on AWS, GCP, or self-hosted Kubernetes. Cloud-specific examples are labelled accordingly.

Stack reminder. ADR-00002 selects Quarkus + PostgreSQL + Keycloak, packaged as an OCI image and deployed onto a managed container runtime. The procedures below assume that stack.

Prerequisites

• Java 25 LTS (Temurin / Eclipse Adoptium recommended)
• Maven 3.9+ or the Quarkus CLI
• Docker / Podman for building the OCI image and running local Dev Services
• kubectl (Kubernetes), az (Azure), aws (AWS) — only the CLI for your target environment
• Access to the target container registry (e.g. crctnasrdev.azurecr.io, ECR, GitHub Container Registry)

Production Deployment

Build the OCI image

The standard Quarkus container build produces a portable OCI image; the same artefact runs on every supported container runtime.

# Build the JVM image (fast cold-start is typically good enough for the ASR's traffic profile)
./mvnw clean package -Dquarkus.container-image.build=true \
                     -Dquarkus.container-image.group=ctn-asr \
                     -Dquarkus.container-image.tag=$(git rev-parse --short HEAD)

# Or build a native binary for smallest memory and fastest startup
./mvnw clean package -Pnative -Dquarkus.native.container-build=true \
                     -Dquarkus.container-image.build=true

Push to the registry:

docker push <registry>/ctn-asr:<tag>

Deploy the API (reference: Kubernetes / AKS via Helm + ArgoCD)

Automated (GitOps) via the CI pipeline that watches api/. The pipeline:

1. Builds the OCI image (as above).
2. Pushes to the container registry.
3. Bumps the image tag in the Helm values; ArgoCD reconciles the release onto the cluster.
4. Smoke-tests the resulting endpoint.

Manual — Helm (any conformant Kubernetes cluster, incl. AKS):

helm upgrade --install ctn-asr ./infrastructure/deploy/helm/ctn-asr \
  --namespace ctn-asr-dev --create-namespace \
  --set image.repository=<registry>/ctn-asr \
  --set image.tag=<tag>
kubectl rollout status deployment/ctn-asr-api -n ctn-asr-dev

Manual — force an ArgoCD sync:

argocd app sync ctn-asr-dev
argocd app wait ctn-asr-dev --health

Deploy the portals (frontend)

The frontend stack is not yet decided (see Coding Standards §Open Questions). When chosen, document the build command and the static-asset deployment target here. The reference implementation uses a CDN-fronted static host (Azure Static Web Apps in the prototype; AWS CloudFront + S3 or Cloudflare Pages are equally valid).

Verify Deployment

After deployment, the standard Quarkus health endpoints are available:

• Liveness: GET /q/health/live
• Readiness: GET /q/health/ready
• Info: GET /q/info (build metadata)
• OpenAPI: GET /q/openapi

Smoke-test:

curl https://<api-fqdn>/q/health/ready

Database Migrations

The application uses Liquibase (built into Quarkus, TDR-0001). Changesets are YAML files under src/main/resources/db/changelog/, included in order by changelog-master.yaml.

dev/test apply the changelog at application startup (quarkus.liquibase.migrate-at-start=true). Production never migrates from the app process — a CI-built migrations image runs liquibase update as a Kubernetes init container before the app container starts.

# Dev mode applies the changelog (dev context, incl. demo seed) at startup:
./mvnw quarkus:dev

# Out-of-band via the Liquibase CLI against any environment.
# --contexts is MANDATORY: a context-less run also applies context-tagged
# changesets, i.e. the dev demo seed.
liquibase update \
  --url=jdbc:postgresql://<host>:5432/<db> \
  --username=<user> --password=<password> \
  --changelog-file=db/changelog/changelog-master.yaml \
  --contexts=prod

To verify applied changesets:

SELECT orderexecuted, id, author, dateexecuted, exectype
FROM databasechangelog
ORDER BY orderexecuted DESC
LIMIT 10;

Configuration

Environment variables

Quarkus resolves configuration in this order: profile-specific properties → application.properties → environment variables → command-line. Secrets are injected at runtime from the configured secret store; properties refer to them indirectly (${...}).

Typical environment variables (names are stack-stable; the secret store binds them):

Variable	Purpose
QUARKUS_DATASOURCE_JDBC_URL	PostgreSQL JDBC URL
QUARKUS_DATASOURCE_USERNAME	DB username
QUARKUS_DATASOURCE_PASSWORD	DB password (from secret store)
QUARKUS_OIDC_AUTH_SERVER_URL	Keycloak issuer URL
QUARKUS_OIDC_CLIENT_ID	OIDC client ID
QUARKUS_OIDC_CREDENTIALS_SECRET	OIDC client secret (from secret store)
KVK_API_KEY	KvK API key (from secret store)
JWT_SIGNING_KEY_REF	Reference to the VAD signing key in HSM / KMS

Secret stores (cloud-specific examples)

Azure Key Vault:

az keyvault secret set \
  --vault-name kv-ctn-demo-asr-dev \
  --name POSTGRES-PASSWORD \
  --value "<new-password>"

# Restart the workload to pick up the new secret value
kubectl rollout restart deployment/ctn-asr-api -n ctn-asr-dev

AWS Secrets Manager:

aws secretsmanager update-secret \
  --secret-id ctn-asr/postgres-password \
  --secret-string "<new-password>"

HashiCorp Vault:

vault kv put secret/ctn-asr postgres-password=<new-password>

In every case the application picks up the new value on the next restart (or via dynamic reloading if configured).

Monitoring

Logs

Quarkus emits structured JSON logs. Ship them to the central log platform of your environment:

Kubernetes (reference, AKS): pods stream stdout/stderr to the cluster’s log pipeline (Container Insights → Log Analytics on AKS):

kubectl logs deploy/ctn-asr-api -n ctn-asr-dev -f

Metrics

Quarkus exposes Prometheus-format metrics on /q/metrics via quarkus-micrometer-registry-prometheus. Scrape with the cloud’s metric platform:

• Azure: Azure Monitor managed Prometheus, scraping the /q/metrics endpoint.
• AWS: CloudWatch Container Insights or AMP (Managed Prometheus).
• Self-hosted: Prometheus + Grafana stack.

Standard alerts

Alert	Threshold
API 5xx errors	> 10 in 5 min — critical
Request rate	> 1000 / 5 min — warning
Memory usage	> 80% of limit for 15 min — warning
P95 response time	> 5s — warning

Troubleshooting

API returns 5xx

1. Check liveness: curl /q/health/live.
2. Check readiness: curl /q/health/ready — failed readiness usually means a downstream (database, Keycloak, KvK API) is unreachable.
3. Pull the recent logs for stack traces.
4. Verify the running pods and rollout (kubectl get pods -n ctn-asr-dev; kubectl rollout history deployment/ctn-asr-api -n ctn-asr-dev).

Workload not scaling

Check the configured autoscaling rule (CPU, request rate, or queue depth) against current metrics. Quarkus apps with native binaries scale up faster (sub-second cold-start) than JVM images — if cold-start is the issue, consider the native build.

Database connection issues

# Connect from inside the workload (Kubernetes):
kubectl exec -n ctn-asr-dev deploy/ctn-asr-api -- \
  /bin/sh -c 'pg_isready -h $QUARKUS_DATASOURCE_JDBC_URL_HOST -p 5432'

Common causes: secret rotation not propagated to the workload, firewall rule missing for the workload subnet, SSL mode mismatch.

New route not working

1. Verify the resource class exists and is annotated (@Path, HTTP-verb annotation).
2. Verify the deployed image contains the change (compare the image digest against the local build).
3. Verify the new route appears in /q/openapi — if not, the build did not include the change.

Production still serving an old version

A deploy that “succeeds” in CI but does not change runtime behaviour almost always means the old pods are still serving (image tag not rolled), not that the build was wrong. Diagnose in order:

1. Compare the deployed build/version (e.g. the git SHA exposed at /q/info or in the image tag) against the commit the pipeline built.
2. Confirm the new pods are actually running the new image and Ready (kubectl rollout status deployment/ctn-asr-api -n ctn-asr-dev; kubectl get pods -n ctn-asr-dev -o wide); check that ArgoCD shows the app Synced/Healthy.
3. If old pods still serve, force a fresh rollout (kubectl rollout restart deployment/ctn-asr-api -n ctn-asr-dev) or re-sync ArgoCD.
4. Rule out CDN/Front Door caching of API responses — purge if necessary.

Rollback

Application

Helm / ArgoCD (GitOps):

# Roll back to the previous Helm release
helm rollback ctn-asr -n ctn-asr-dev

# Or revert the image tag in Git and let ArgoCD reconcile
argocd app sync ctn-asr-dev

Kubernetes (direct):

kubectl rollout undo deployment/ctn-asr-api -n ctn-asr-dev

Database migration

The policy is roll-forward only (TDR-0001): changesets carry no rollback blocks, and liquibase rollback is not used. Roll back by applying the inverse change as a new forward changeset. Never delete or edit an already-applied changeset.

-- Verify the schema state after the corrective changeset
SELECT id, author, dateexecuted, exectype
FROM databasechangelog
ORDER BY orderexecuted DESC
LIMIT 5;

Infrastructure as Code

Cloud resources are provisioned with OpenTofu; the Kubernetes workload is delivered via Helm + ArgoCD (GitOps). Both live next to the application code:

• infrastructure/tofu/main.tf — root module (AKS, PostgreSQL, networking, monitoring)
• infrastructure/tofu/modules/aks/ — cluster + node pools
• infrastructure/deploy/helm/ctn-asr/ — Helm chart for the ASR API
• infrastructure/deploy/argocd/ — ArgoCD Application manifests per environment

Provision the cloud resources:

cd infrastructure/tofu
tofu init
tofu plan  -var-file=env/dev.tfvars
tofu apply -var-file=env/dev.tfvars

ArgoCD then reconciles the Helm release onto the cluster (GitOps); see “Deploy the API” above. AKS is the Azure reference; the same OpenTofu modules and Helm chart target any Kubernetes (EKS, GKE, self-hosted).

Additional Resources

• Reference environment dashboards / URLs are environment-specific and live in the operations team’s runbook, not in this repo.
• CI/CD pipelines are defined under .github/workflows/ (GitHub Actions); they build the OCI image, run tofu apply, and let ArgoCD sync the Helm release.

Quick Reference

# Health
curl https://<api-fqdn>/q/health/ready
curl https://<api-fqdn>/q/health/live

# OpenAPI
curl https://<api-fqdn>/q/openapi

# Prometheus metrics
curl https://<api-fqdn>/q/metrics