Production Setup Notes

Production setup notes for ekx. Keep this doc updated when prod paths, networks, or deploy steps change.

Quick Facts

Host: 64.227.103.35 (Ubuntu 24.10)
Primary domain: joinekx.com (www.joinekx.com)
App container: ekx_prod_app (Next.js)
Pandapower container: ekx_prod_pandapower
PDF container: ekx_prod_pdf
Database: ekx_prod_postgres (PostgreSQL 16)
Reverse proxy: ekx_reverse_proxy (nginx)

Key Paths (Server)

App compose: /var/www/ekx/docker-compose.yml
App env file: /var/www/ekx/.env.production
Proxy compose: /var/www/ekx/proxy/docker-compose.yml
Proxy nginx config: inside container at /etc/nginx/conf.d/*.conf
View active config: docker exec ekx_reverse_proxy nginx -T

Networks and Volumes

App compose defines networks: app_network and shared_web (external)
shared_web must exist for reverse proxy -> app routing
App compose uses external named volume: ekx_postgres_data
On a new host, create the volume before the first compose deploy:
docker volume inspect ekx_postgres_data >/dev/null 2>&1 || docker volume create ekx_postgres_data

Deploy Flow (GitHub Actions)

Workflow: .github/workflows/build-deploy.yml

1) Detect changed deployment areas. 2) Validate deploy shell scripts with bash -n and shellcheck. 3) Build and push only changed images to GHCR. 4) Upload docker-compose.yml and scripts/prod-deploy.sh to prod. 5) Pull changed images by digest, tag local :latest. 6) Run migrations and seed the system cable catalog only when migrator inputs changed. 7) Recreate only changed services with compose (not docker restart), or all app-side services when compose changed.

The app and migrator images are separate Docker targets. The migrator target is based on installed dependencies and checked-in source/migration files, but it does not run the Next.js production build. Keep this split intact so migration and cable-catalog deploys do not pay for an app bundle build.

The workflow uses:

COMPOSE_FILE_PATH=/var/www/ekx/docker-compose.yml
env file preference: /var/www/ekx/.env.production (fallback: .env)

Codex Production Debug Access

Codex debugging should use the restricted codex-debug account rather than a human root/admin SSH key. The local private key lives at ~/.ssh/ekx_codex_debug and the SSH alias is ekx-prod-debug.

Install or refresh the server-side account from an approved admin SSH session:

scripts/install-prod-debug-access.sh

That installer creates:

Linux user: codex-debug
SSH key auth only, password locked
No Docker group membership
Passwordless sudo only for /usr/local/bin/ekx-prod-debug

Run production diagnostics through the local wrapper:

scripts/prod-debug.sh status
scripts/prod-debug.sh health
scripts/prod-debug.sh logs app 200
scripts/prod-debug.sh errors app 500
scripts/prod-debug.sh migrations
scripts/prod-debug.sh daily-activity "2026-06-22 00:00:00" "2026-06-23 00:00:00"

The wrapper may read .env.codex-prod.local for local non-secret pointers such as EKX_PROD_DEBUG_SSH_TARGET. Do not store SSH private keys in .env files.

The daily-activity command is a fixed read-only production database report for admin summaries. It accepts an explicit half-open local timestamp window and returns JSON with new sign-ups, active users, product activity counts, support reports, subscription changes, and user names/emails for the reporting window.

Manual Deploy (Server)

Recreate services with the latest pulled/tagged images:

# Ensure shared network exists
docker network inspect shared_web >/dev/null 2>&1 || docker network create shared_web

# Ensure the external database volume exists
docker volume inspect ekx_postgres_data >/dev/null 2>&1 || docker volume create ekx_postgres_data

# Recreate pdf-service + pandapower + app
docker compose -f /var/www/ekx/docker-compose.yml --env-file /var/www/ekx/.env.production \
  up -d --no-build --force-recreate --no-deps pdf-service pandapower app

Verify container image ID matches local tag:

docker image inspect ekx-app-prod:latest --format '{{.Id}}'
docker inspect ekx_prod_app --format '{{.Image}}'

Common Issues

502 Bad Gateway after deploy

Cause: nginx cached old container IP for ekx_prod_app.

Fix:

# Reload to re-resolve docker DNS
docker exec ekx_reverse_proxy nginx -s reload

# If still 502, restart proxy
docker restart ekx_reverse_proxy

Check upstream connectivity from the proxy container:

docker exec ekx_reverse_proxy wget -qO- http://ekx_prod_app:3000/ || echo "proxy cannot reach app"

Migrations fail with "type already exists"

Drizzle migration attempted to create an enum/type that already exists. This does not block container recreation, but should be fixed by making migrations idempotent or reconciling the migrations table with DB state.

Compose warnings about missing env vars

Make sure /var/www/ekx/.env.production is present and includes required keys:

DATABASE_URL
MIGRATION_DATABASE_URL or DRIZZLE_DATABASE_URL for deploy-time migrations, or POSTGRES_PASSWORD so the deploy can derive the internal Postgres superuser URL. If POSTGRES_PASSWORD or POSTGRES_DB are missing, the deploy falls back to the running ekx_prod_postgres container environment and exports the recovered Postgres settings for the later compose command.
PANDAPOWER_SERVICE_TOKEN, PDF_SERVICE_TOKEN, and PDF_RENDER_TOKEN for private service-to-service auth. If missing, the deploy generates matching random values and appends them to the production env file before service recreation.
AUTH_SECRET
AUTH_GOOGLE_ID / AUTH_GOOGLE_SECRET
Stripe, Turnstile, Resend, etc.

Stripe Billing Product IDs

Production Stripe account: acct_1RpuZEGvbjJF0eeu (Bremen Engineering Inc.).

Canonical public products:

Starter: prod_TnxlgGLkt63uvI
Monthly: price_1TFyHqGvbjJF0eeuZ0rHZqMo ($27/mo)
Yearly: price_1TFyHqGvbjJF0eeuQXjfr2KP ($270/yr)
Professional: prod_Tnxl6hGCGjwWXL
Monthly: price_1TFyHrGvbjJF0eeuGfdjVmUM ($149/user/mo)
Yearly: price_1TFyHsGvbjJF0eeudYSOGmav ($1,490/user/yr)

Team is legacy-only and should remain inactive in Stripe-hosted surfaces:

Product: prod_Tnxl0YXxR1eoqf
Monthly: price_1SqLpqGvbjJF0eeus6Wy20Gn
Yearly: price_1SqLpqGvbjJF0eeunGvapoNe
Lifetime: price_1SqLprGvbjJF0eeunwaulrP1

Reverse Proxy Notes

Proxy allows only Cloudflare IP ranges (deny all others).
Nginx config uses: proxy_pass http://ekx_prod_app:3000;
Duplicate server_name warnings indicate multiple server blocks for joinekx.com.

Clean up duplicates to avoid undefined behavior.