Wait, readiness, retry helpers

Bootstrap wait First line of every setup

Wait for the agent's bootstrap sentinel before any work.

until [ -f /opt/instruqt/bootstrap/host-bootstrap-completed ]; do sleep 1 done

Image-specific sentinel: on the cloud-client base image, wait for gcp-bootstrap-completed in the same directory instead. Different file name, same place.

HTTP readiness Per-service polling

Accept every status code that means "server is responsive" — 200, 302 (auth redirect), 401 (auth required but listening).

Avoids the brittle "expected 200, got 302" false-not-ready failure when the service is up but redirecting to login.

until curl -s -o /dev/null \ -w "%{http_code}" http://localhost:8000 \ | grep -qE "^(200|302|401)$"; do sleep 3 done

Flat retry Local services

For processes on localhost (Flask, uvicorn, in-VM dashboards) where exponential backoff is overkill.

for i in $(seq 1 30); do if curl -fsS \ http://127.0.0.1:5000/health then echo "OK"; break fi sleep 2 done

Exponential backoff Shared external services

Five attempts, doubling timeout each failure (1s / 2s / 4s / 8s / 16s). For SaaS APIs and cloud control planes.

retry_command() { local n=0 max=5 delay=1 while [ $n -lt $max ]; do "$@" && return 0 sleep $delay delay=$((delay * 2)) n=$((n + 1)) done return 1 } retry_command curl -fsS "$API/init"

Retry with jitter Concurrent SaaS pressure

Random sleep avoids thundering herd when many learners boot simultaneously and all hit the same shared SaaS tenant.

tries=0 until provision_account; do ((tries++)) [[ $tries -eq 5 ]] && exit 12 sleep $(shuf -i 3-20 -n 1) done

Combine with set +e/set -e if your script uses set -euo pipefail and the until body's intermediate failures would otherwise abort.

Standalone retry wrapper Ergonomics

Ship a /usr/local/bin/retry wrapper for clean per-call usage.

# /usr/local/bin/retry (chmod +x): #!/bin/bash RETRY_MAX=${RETRY_MAX:-10} RETRY_DELAY=${RETRY_DELAY:-5} for i in $(seq 1 $RETRY_MAX); do "$@" && exit 0 sleep $RETRY_DELAY done exit 1 # In setup: retry curl -fsS "$API/healthz"

Kubernetes API Cluster readiness

Don't trust kubectl cluster-info — it can return OK before the API is fully ready.

kubectl proxy --port=8001 & until curl -s \ http://localhost:8001/healthz \ | grep -q ok; do sleep 2; done pkill -f "kubectl proxy"

Cloud CLI retry Just-provisioned credentials

Service principal login can transiently fail right after a subscription is provisioned. Tenant edge propagation lags credential injection.

For per-challenge cloud-side setup, extend to 12 iterations — propagation can take longer than 50 seconds.

for i in {1..5}; do az login --service-principal \ -u "$ARM_CLIENT_ID" \ -p "$ARM_CLIENT_SECRET" \ --tenant "$ARM_TENANT_ID" \ && break || sleep 10 done

Notes

Sleep is not readiness. sleep N as a substitute for actual polling is brittle. If provisioning takes longer than expected, the script breaks; if shorter, time is wasted. Always poll a real signal — file existence, HTTP status, port responsiveness — when one is available.
dpkg lock guard for Ubuntu 22+ / 24+: unattended-upgrades may hold the dpkg lock at first boot. Wait it out: while fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 2; done before any apt-get.
timeout for inherently flaky steps: wrap with timeout --kill-after=30 300 cmd || { echo "Timed out"; exit 1; }. The --kill-after=30 sends SIGKILL 30 seconds after SIGTERM if the process refuses to die.
nohup doesn't survive across challenges. Background processes die at the challenge boundary. If you need an async warmup that the next challenge depends on, run it synchronously in the prior challenge's setup.
Choose the right tool for the wait. Bootstrap = sentinel file. Local service = flat retry. Shared SaaS = exponential backoff. Concurrent SaaS = jitter. K8s API = healthz endpoint. Cloud CLI just after provisioning = explicit retry loop.