Preface
This book is the recipe form of a working library. Every pattern here has been used in a production track, hit a problem, been refined, and come back to be used again. The names are the names that authors use when they describe what they built to other authors. The shapes are the shapes that survive contact with real learners on real labs.
The book is organized for the moment you have a problem in front of you. Each recipe is self-contained. You can read one and act on it without reading the rest of the book. The pieces are small on purpose: when something is broken at four o'clock on a Friday before a Monday demo, you do not have time to read three chapters.
The companion book is The Instruqt track builder's field guide, which walks the lifecycle of a track end to end and explains the reasoning behind the patterns. The Field Guide teaches you how to think about a track. This Cookbook tells you exactly what to type. The two are best read together — Field Guide first, Cookbook on the desk afterwards — but each works on its own. Decision Trees v2 cross-references this Cookbook by recipe name; the Cheat Sheets v2 pack is for glance-and-go reference.
Three constraints shape every recipe. No real track names appear anywhere. The patterns are described concretely enough that the reader can implement them from the description alone. Code is like-for-like, not verbatim. Hostnames, image references, and identifiers are generic placeholders — image registries, customer org slugs, and platform-vendor namespaces are stripped — but the shapes mirror real production code. Voice is direct. When a pattern has a known failure mode, the failure mode is named and the symptom is named. When an opinion is earned, the opinion is given without softening.
This is the v2 edition. It is built against a larger curated source than v1, and it expands the catalog from forty-seven recipes to fifty-five. Two recipes — Recipe 14 and Recipe 27 — preserve their v1 numbering as carry-forward seeds. The rest are reorganized, retuned, and extended.
How to Read
The Cookbook is structured as recipes. Each recipe is a single, complete, copy-friendly answer to a specific design problem. You can read it cover to cover, but it is built to be opened to one page when you have a problem in front of you.
Every recipe follows the same six-section format:
- Use when — a one-line guidance test. If this line does not match your situation, skip the recipe.
- Problem — the situation that prompts the pattern. What you tried first, why it did not work, what the failure looked like.
- Solution — the conceptual shape of the answer. The mental model, in prose, before any code.
- Skeleton — copy-friendly code or YAML. Generic names; safe to lift and adapt.
- Gotchas — the failure modes other authors have hit. Specific symptoms, specific fixes.
- Variations — related recipes, alternative shapes, or extensions worth knowing about.
Patterns are categorized by scope.
Infrastructure patterns describe how a track's hosts, networks, and resources are wired together. They are decisions that live in config.yml and shape what every other pattern can do.
Technique patterns describe a single self-contained move — a trick, a wrapper, a setup convention — that a track might or might not use. Most are bash patterns from setup or check scripts. They compose freely.
Author patterns describe how content is structured and presented to the learner. They live in assignment.md files and in the surrounding decisions about pacing, hinting, and feedback.
Recipes are numbered globally and continuously. Recipe 14 in Part II is the fourteenth recipe in the book; Recipe 27 in Part III is the twenty-seventh. Cross-references in the text use the recipe number.
Each recipe carries a Difficulty marker — Basic, Intermediate, or Advanced. Basic means the pattern is one or two moves and the failure modes are obvious. Intermediate means there are several moving parts and at least one non-obvious failure mode. Advanced means the pattern coordinates multiple hosts, services, or runtime states and is easy to wire up wrong on the first try.
The recipes assume working knowledge of the Instruqt platform — what a track is, what track.yml and config.yml do, what a lifecycle script is. If you need that grounding, start with the Field Guide's Part I.
Single container track
A track for a CLI tool, a scripting walkthrough, or a self-contained tutorial does not need a database, a VM, or a cloud account. The standard temptation is to reach for a bigger shape than the problem requires — multi-container, container-plus-VM, full sandbox — because the bigger shapes feel more "real." They are slower to boot, they cost more lab-session compute, and they add failure modes the lesson does not teach. Single-container is almost always the right shape when one host is enough.
Declare one container in config.yml, give it a sensible name, point it at one of the platform's base images, and stop. Use a single terminal tab. Put any tooling the lesson needs into the track-level setup script.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash#!/bin/bash
set -euxo pipefail
apt-get update -qq
apt-get install -y -qq jq treetabs:
- title: Terminal
type: terminal
hostname: workstationThe platform ships two common base images. The lighter one is minimal — no cloud CLIs preinstalled. The heavier cloud-client image ships with aws, gcloud, az, kubectl, and helm already installed. Use the lighter image for pure-bash and pure-CLI lessons; the cost of having it slim is worth it. Use the heavier image when even one cloud CLI would otherwise need to be installed in setup.
The default shell when no cmd: is set on the terminal tab is /bin/bash running as root. That is fine for most labs. If the lesson teaches non-root workflows, see Recipe 29.
The memory: field is in megabytes. 1024 is enough for almost any single-container track that does not run a database, a JVM, or a model. Bump to 2048 only after measuring; bigger memory does not make a track snappier.
For tracks that demonstrate a tool the platform's base images do not ship, build a custom container image from one of the platform's base images and reference it from config.yml. Cache pulls with Recipe 20 to keep boot times low.
For tracks that need no custom infrastructure at all — a pure introductory tutorial, a documentation walkthrough — see Recipe 6 (sandbox preset adoption), which lets you skip config.yml entirely.
Single VM track
Containers are fast to boot, cheap to run, and sufficient for most tracks. They are not sufficient when the lesson involves systemd unit files, kernel-module loading, real network namespaces beyond what a container can simulate, or any GUI workload that requires a display server. Trying to teach those things inside a container is a long fight with workarounds; teaching them on a VM is the path of least resistance.
Declare one VM in config.yml, use a stock Ubuntu image (or a custom image if you have one), and provision tooling in the VM's track-level setup script. Surface a terminal tab pointing at the VM.
version: "3"
virtualmachines:
- name: workstation
image: ubuntu-2204
machine_type: n1-standard-2
shell: /bin/bashtabs:
- title: Terminal
type: terminal
hostname: workstationVM boot takes 60–120 seconds compared to a container's 5–15. Set the user expectation accordingly: the first challenge's loading message should mention that the environment is provisioning. See Recipe 21 (bootstrap wait) for the pattern that protects setup scripts from running before the host is fully up.
VMs cost more per lab-session than containers. Default to a container; reach for a VM only when the track genuinely needs one.
The machine_type: field accepts standard GCP machine type strings. n1-standard-2 (2 vCPU, 7.5 GB RAM) is a sensible default for a single-VM workstation. Step up to n1-standard-4 only when you have measured a workload that needs it. Prefer machine_type: over the older memory: + cpus: pair — machine_type: is the canonical knob across modern tracks.
If the lesson involves Docker or kind running inside the VM, set nested_virtualization: true and pick a machine type with enough headroom. Drop nested virtualization when the track does not need it — it slows boot.
For tracks that need a desktop UI on the VM, see Recipe 17 (noVNC) or Recipe 16 (Guacamole). For tracks that need a single GUI application without a full desktop, see Recipe 18 (KasmVNC).
For tracks that need a VM running Docker — a meta-pattern where the VM hosts containers — see Recipe 19.
For tracks that need a pre-baked k3s cluster, see Recipe 38.
Multi-container composition
A workstation container is enough for command-line teaching. Once the lesson involves a real application — an API the learner runs, a frontend they edit, a worker they restart — and that application talks to a database or a queue, a single container starts fighting itself. Running both processes in one container means juggling supervisord or background processes, and the failure modes leak into the lesson.
The clean shape is one process per container, wired together by hostname.
Declare each service as its own container in config.yml. Use container names as the hostname for inter-service communication. Set environment: keys on each container to pass connection strings or service URLs. The platform creates a Docker network that all declared containers share, so any container can reach any other by its declared name.
version: "3"
containers:
- name: workstation
image: cloud-client
ports: [8080]
memory: 1024
shell: /bin/bash
environment:
DB_HOST: db
QUEUE_URL: redis://queue:6379
- name: db
image: postgres:16
ports: [5432]
memory: 1024
private: true
environment:
POSTGRES_USER: app
POSTGRES_PASSWORD: app
POSTGRES_DB: app
- name: queue
image: redis:7-alpine
ports: [6379]
memory: 256
private: trueContainers boot in parallel, not in dependency order. The workstation's setup script may start before postgres or redis is ready to accept connections. Always poll for readiness — pg_isready -h db for postgres, redis-cli -h queue ping for redis — before any setup that depends on the service. See Recipe 22 for the general wait-for-port pattern.
The learner's browser cannot resolve internal Docker hostnames. If the workstation serves a web UI on port 8080 and that UI's JavaScript calls fetch('http://api:5000/data'), the fetch fails because the browser is talking to the platform's traffic proxy, not the Docker network. Always use relative paths in client-side code; route everything through one service tab.
The ports: field declares which ports the platform should be aware of for service-tab purposes. It does not restrict which ports the container can use internally; containers can talk to each other on any port without declaring it. Mark backend-only services private: true so they do not appear as terminal tabs to the learner.
For a workstation paired specifically with postgres and a live web dashboard, see Recipe 14. For a workstation paired with a FastAPI service, see Recipe 12. For a workstation paired with a mock REST API and a branded CLI that talks to it, see Recipe 13.
When the second container is a dependency of the application rather than a peer (e.g., a sidecar log collector), still declare it as a peer container. Sidecar shapes are not a first-class concept on the platform.
Multi-VM composition
Two VMs is a meaningfully different problem from one VM. The platform boots both in parallel, IPs are not stable in advance, and any setup that depends on one VM finding the other has to handle the case where the other is still coming up. Naive multi-VM tracks have setup scripts that fail intermittently because they assume hostname resolution works the moment the script starts.
Declare each VM in config.yml with a stable name. Use the environment: block on each VM to inject the hostnames of the other VMs. In setup, wait for the bootstrap sentinel file before doing anything that depends on the network being up. Coordinate cross-VM state via one of three patterns: agent variable for single-string values, a JSON broker sidecar for structured payloads, or cross-VM env passthrough by sourcing curl output (see Recipe 40).
version: "3"
virtualmachines:
- name: leader
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
- name: follower-1
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
environment:
LEADER_URL: http://leader:8081
- name: follower-2
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
environment:
LEADER_URL: http://leader:8081#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/host-bootstrap-completed ]; do
sleep 1
done
# Wait for the leader's broker, then source its env
until curl -fsS "$LEADER_URL/env" > /tmp/leader.env; do
sleep 2
done
source /tmp/leader.env
# Now safe to join the cluster
join-cluster --leader "$LEADER_HOST" --token "$CLUSTER_TOKEN"VMs in the same sandbox resolve each other by name, but only after the sandbox network is up. Always gate cross-VM calls on the bootstrap sentinel file plus a readiness probe against the target service.
VM-to-VM env-var injection via config.yml's environment: block is one-way at provisioning time. The leader does not know follower hostnames unless you hardcode them. Use the broker pattern for anything dynamic.
Tracks that provision external resources from one VM (cloud accounts, SaaS tenants) and consume them from another need a cross-VM hand-off. See Recipe 34 for the agent variable form and Recipe 40 for the broker form. They are not mutually exclusive.
For a multi-node Kubernetes cluster pre-baked into the platform's k3s VM image, see Recipe 38 — the image's auto-join logic removes most of the cross-VM coordination work.
For a workstation that needs a SaaS tenant pre-provisioned by a separate VM, JSON-broker the credentials (Recipe 40) rather than encoding them in agent variable strings.
Container plus managed VM
The learner needs a sane shell with cloud CLIs, the right kubectl version, the right helm version, and a pre-cloned repo of lab content. They also need to operate a real Linux service that requires a VM — a database engine, a kernel-tracing tool, a systemd-managed daemon. Putting both in one container is awkward; putting both on a VM is slow and gives the learner a less polished workstation experience.
The clean shape is a container as the workstation and a VM as the operand.
Declare the container with the workstation tooling and a VM with the service under study. Wire the container's environment: to expose the VM's hostname. In the container's setup, wait for the VM's service port before continuing. Surface a terminal tab on the container as the primary workspace; surface a service tab pointing at the VM if the lesson involves a UI.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash
environment:
TARGET_HOST: target-vm
virtualmachines:
- name: target-vm
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/gcp-bootstrap-completed ]; do
sleep 1
done
# Wait for the VM service port
until nc -z "$TARGET_HOST" 22; do sleep 2; doneThe cloud-client container uses a different bootstrap sentinel than other images. On cloud-client, wait on gcp-bootstrap-completed. On the lighter shell image and on VMs, wait on host-bootstrap-completed. Mixing them up causes the wait to either return immediately (false ready) or never (hang).
The container's setup runs in parallel with the VM's setup. The workstation can be ready in 10 seconds while the VM is still 90 seconds out. Always poll for the VM's service port on the workstation side; do not assume the VM is up when the workstation's bootstrap completes.
If the VM's setup script needs a value from the container (rare but real), use Recipe 34 to write it via agent variable set in the container and read it on the VM.
For tracks where the VM is in fact a Kubernetes cluster (k3s, kind, kubeadm), the workstation pattern is identical — the container holds kubectl, helm, and a pre-merged kubeconfig pointing at the VM. See Recipe 38 for the pre-baked k3s shape.
For multi-VM stacks (e.g., a database cluster), pair the container with multiple VMs and use Recipe 40 for cross-VM coordination.
Sandbox preset adoption
For a click-through tutorial against Kibana, a managed-cluster console, or a temporal sandbox, every minute spent declaring and provisioning hosts is wasted. The lesson is the UI; the infrastructure underneath is overhead the learner does not see and does not benefit from. Authors who default to config.yml here pay startup time the lesson never claims back.
Use sandbox_preset: instead of config.yml. The preset gives a pre-configured set of hosts with relevant tooling already installed. Reference the preset's hostnames in your tab declarations. Skip track_scripts/ entirely.
slug: my-managed-ui-track
title: Click-through workshop
sandbox_preset: managed-vm-elastic-9
timelimit: 3600tabs:
- title: Kibana
type: service
hostname: kubernetes-vm
port: 30001
- title: Terminal
type: terminal
hostname: lab-host-1The preset's hostnames are fixed by the preset, not by you. Look up the preset's hostname cheat sheet before writing your tab declarations — referring to a hostname the preset does not expose silently breaks the tabs without an obvious error.
Presets lock you out of track_scripts/, custom images, multi-VM topologies, secrets injection, and per-track networking. The moment the lesson needs any of those, you have outgrown the preset and should switch to config.yml. Trying to layer setup work onto a preset usually fails awkwardly — start with config.yml if there is any doubt.
Some presets pre-deploy a full software stack (a Kibana stack, an MCP environment) before the first challenge. Tracks built on those presets often need zero setup scripts at all — pure click-through against a pre-warmed UI. Check what your preset has already done before you write any setup work.
For a track that needs most of the preset's environment but with one custom tweak (a specific package, a per-learner credential), the answer is usually to abandon the preset and use config.yml. Trying to bolt a custom setup onto a preset is more work than starting from scratch.
For tracks where the preset gets you 90% of the way and the last 10% is challenge-level state, per-challenge setup scripts on a preset's host work — but only inside what the preset already exposes.
Container plus AWS account
Teaching AWS in a fake environment is unconvincing. Localstack covers parts of the surface, but the moment the lesson involves the AWS console, IAM roles, billable services, or any "click here in your account" step, the learner needs a real account. Provisioning one per learner ahead of time is impractical at scale; provisioning one inside the lab is what the platform exists for.
Declare an aws_accounts: resource in config.yml with the services and regions the lesson needs and the IAM policy the learner should hold. The platform mints a fresh account at sandbox start, injects credentials and an account ID into the workstation container, and reaps the account at cleanup.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash
aws_accounts:
- name: lab-account
services: [s3, ec2, iam]
regions: [us-east-1]
iam_policy: |-
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["s3:*", "ec2:Describe*"],
"Resource": "*"
}
]
}#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/gcp-bootstrap-completed ]; do
sleep 1
done
# Surface the account ID into assignment-body interpolation
agent variable set ACCOUNT_ID "$INSTRUQT_AWS_ACCOUNT_LAB_ACCOUNT_ACCOUNT_ID"
agent variable set USERNAME "$INSTRUQT_AWS_ACCOUNT_LAB_ACCOUNT_USERNAME"The account name in config.yml becomes a prefix on the injected env vars, uppercased with hyphens replaced by underscores. A name of lab-account becomes INSTRUQT_AWS_ACCOUNT_LAB_ACCOUNT_*. Forgetting to uppercase or mishandling the dashes is the single most common reason cloud creds appear empty in setup.
If the lesson needs both narrowly-scoped learner credentials and broader admin credentials for setup work, declare both iam_policy and admin_iam_policy in the resource block. Setup runs as admin; the learner's session uses the narrower role.
Referencing INSTRUQT_AWS_ACCOUNT_* env vars without an aws_accounts: block in config.yml returns empty strings. The platform only injects these when the resource is declared. Symptom: AWS CLI calls fail with empty-credential errors with no clear pointer to the missing block.
The lighter shell base image does not ship with the AWS CLI. If you use it with aws_accounts:, install the CLI in setup before any aws command runs. The cloud-client image already has it.
For "bring your own credentials" — a track where the learner connects to a customer-owned AWS account rather than a sandbox-provisioned one — declare AWS credentials as secrets: instead of aws_accounts: and reference them as standard AWS env vars. You lose automatic IAM scoping, SCP policies, and account cleanup, so use this only when the customer relationship requires it.
For tracks that need a managed Kubernetes cluster (EKS) on top of an AWS account, see Recipe 38's principles applied to a runtime-provisioned cluster: kick off eksctl create cluster in the background during setup and gate the first cluster-touching challenge on completion.
Container plus Azure subscription
Azure has the same fundamental constraint as AWS: meaningful lessons require a real subscription. Service principals, role assignments, the portal, and most managed services do not have credible offline equivalents. Per-learner subscription provisioning is what the platform manages.
Declare an azure_subscriptions: resource with the resource-provider namespaces the lesson needs and the roles the learner and the admin should hold. The platform mints a subscription, registers the providers, and injects ARM credentials into a workstation container.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash
azure_subscriptions:
- name: lab-sub
services:
- Microsoft.Compute
- Microsoft.Network
- Microsoft.Resources
- Microsoft.Storage
regions: [eastus]
roles: [Reader, Contributor]
admin_roles: [Owner]#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/gcp-bootstrap-completed ]; do
sleep 1
done
# az login can be transient immediately after subscription provisioning
for i in {1..12}; do
az login --service-principal \
-u "$ARM_CLIENT_ID" -p "$ARM_CLIENT_SECRET" --tenant "$ARM_TENANT_ID" \
&& break || sleep 10
done
az account set --subscription "$ARM_SUBSCRIPTION_ID"
agent variable set ACCOUNT_USER "$INSTRUQT_AZURE_SUBSCRIPTION_LAB_SUB_USERNAME"When the subscription resource is present, the platform also injects the Terraform AzureRM-style aliases (ARM_CLIENT_ID, ARM_CLIENT_SECRET, ARM_TENANT_ID, ARM_SUBSCRIPTION_ID) on the cloud-client container. Use these directly in setup and Terraform without manual aliasing. AZURE_LOCATION is also injected with the configured region.
Service principal login can transiently fail in the first 30–60 seconds after the subscription is provisioned. Wrap az login in a retry loop with at least five attempts; for per-challenge setups extend to twelve. Tenant edge propagation is the cause and there is nothing you can do besides wait it out.
Always provision a cleanup script that calls az rest --method POST against the subscription cancel endpoint. Without this, the subscription persists after the lab ends and accrues cost. The "skip cleanup for costed resources" anti-pattern hits hardest with Azure subscriptions because they are large and expensive.
If your iam_policy block references a region not in the regions: list, the role assignment silently fails. Match regions to roles.
For tracks that need a fresh AKS cluster, provision it in setup using the auto-injected ARM credentials and add a wait loop for the cluster to be ready. For long boot times, gate the first cluster-touching challenge on a kubectl get nodes readiness probe.
For tracks delivered via the platform's Custom Resources mechanism (a Terraform module attached to the sandbox via the web UI), the credential injection pattern looks different — secrets appear as named variables rather than INSTRUQT_AZURE_SUBSCRIPTION_*. The decision rule: if your config.yml shows a secrets block with client_id, tenant_id, subscription_id, and billing-related fields but no azure_subscriptions: block, a Custom Resource is doing the provisioning behind the scenes.
Container plus GCP project
GCP teaching requires a real project for the same reasons AWS and Azure teaching does. The console, IAM, billing-aware services, and most managed offerings have no fake equivalent worth using.
Declare a gcp_projects: resource with the API services the lesson needs and the IAM roles the learner and admin should hold. The platform creates the project, enables the APIs, and injects the project ID and a service account key.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash
gcp_projects:
- name: lab-project
services:
- compute.googleapis.com
- storage.googleapis.com
- iam.googleapis.com
regions: [us-central1]
roles:
- roles/storage.objectUser
- roles/compute.viewer
admin_roles:
- roles/owner#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/gcp-bootstrap-completed ]; do
sleep 1
done
# Service account key arrives base64-encoded; decode and clean up on exit
SA_JSON=$(mktemp)
trap "rm -f $SA_JSON" EXIT
echo "$INSTRUQT_GCP_PROJECT_LAB_PROJECT_ADMIN_SERVICE_ACCOUNT_KEY" \
| base64 -d > "$SA_JSON"
gcloud auth activate-service-account --key-file="$SA_JSON"
gcloud config set project "$INSTRUQT_GCP_PROJECT_LAB_PROJECT_PROJECT_ID"
agent variable set PROJECT_ID "$INSTRUQT_GCP_PROJECT_LAB_PROJECT_PROJECT_ID"The service account key is base64-encoded. You must decode before use. Forgetting to decode produces opaque "invalid credentials" errors from gcloud.
Trap-clean the credentials file on exit. A SA key sitting on disk after setup is a low-tier security smell at minimum; on shared images it is worse.
roles: accepts only valid GCP IAM role strings. roles/writer is not a real role; the correct write role for Cloud Storage is roles/storage.objectUser or roles/storage.objectCreator. Invented role names silently fail at sandbox provisioning. Always validate role strings against the GCP IAM predefined roles list.
When the same setup script must serve multiple tracks with different project names, resolve the SA key variable name dynamically with bash indirect expansion: SA_KEY="INSTRUQT_GCP_PROJECT_${PROJECT_NAME}_ADMIN_SERVICE_ACCOUNT_KEY"; echo "${!SA_KEY}" | base64 -d > "$SA_JSON". The ${!VAR} form reads the variable whose name is in VAR.
For tracks that need GKE, provision in setup with gcloud container clusters create. Gate the first cluster-touching challenge on a kubectl get nodes readiness probe.
For tracks that expose a private VM behind IAP, see Recipe 11 — the GCP project is the foundation; the IAP tunnel is the surface.
For cost-bounded GPU labs, set --provisioning-model=FLEX_START, --instance-termination-action=DELETE, and --max-run-duration=7200s on gcloud compute instances create. FLEX_START is GCP's preemptible mode; significantly cheaper but instances may be reclaimed.
Multi-cloud track wiring
Multi-cloud lessons are rare and mostly unnecessary. Most "we need both clouds" requests resolve to "we need one cloud and a believable mock of the other." But some lessons — connecting an on-prem-style network across providers, federating IAM, demonstrating cross-cloud data movement — fundamentally need both. Writing those tracks requires juggling two sets of credentials, two cleanup paths, and one workstation that holds them all.
Declare both cloud resource blocks in config.yml. Use a single workstation container with the cloud-client image, which holds the CLIs for all major providers. In setup, log into each provider in sequence, surface useful identifiers via agent variable set for assignment-body interpolation, and wait for both cloud accounts to be ready before completing setup.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 2048
shell: /bin/bash
aws_accounts:
- name: aws-side
services: [vpc, ec2, route53]
regions: [us-east-1]
iam_policy: |-
{ "Version": "2012-10-17", "Statement": [
{ "Effect": "Allow", "Action": "*", "Resource": "*" } ] }
azure_subscriptions:
- name: azure-side
services:
- Microsoft.Network
- Microsoft.Compute
- Microsoft.Resources
regions: [eastus]
roles: [Contributor]
admin_roles: [Owner]#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/gcp-bootstrap-completed ]; do
sleep 1
done
# AWS side
export AWS_ACCESS_KEY_ID="$INSTRUQT_AWS_ACCOUNT_AWS_SIDE_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$INSTRUQT_AWS_ACCOUNT_AWS_SIDE_AWS_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION=us-east-1
aws sts get-caller-identity > /dev/null
# Azure side, with retry against tenant-edge transient failures
for i in {1..12}; do
az login --service-principal \
-u "$ARM_CLIENT_ID" -p "$ARM_CLIENT_SECRET" --tenant "$ARM_TENANT_ID" \
&& break || sleep 10
done
az account set --subscription "$ARM_SUBSCRIPTION_ID"Cleanup must handle both clouds. Forgetting to cancel the Azure subscription in cleanup leaks cost; forgetting to delete AWS resources leaks cost. Make cleanup explicit and idempotent for each provider.
Workstation memory needs to be larger than for single-cloud tracks. Two clouds means two CLIs, two contexts, larger Terraform state, and more concurrent processes during setup. 2048 MB is a reasonable starting point; 1024 MB will run out during multi-cloud Terraform applies.
Cross-cloud Terraform with explicit dependencies between providers can require phased applies. If Terraform's dependency graph cannot order resources correctly across clouds (e.g., AWS Transit Gateway must exist before Azure VNet peering), use sequential -target= applies to force the order. This is a workaround; restructure the modules to express dependencies correctly when possible.
For tracks that need a third cloud, the same shape extends — add a gcp_projects: resource and decode the SA key in setup. The pattern scales but the cleanup discipline becomes critical.
For tracks where one cloud is fully provisioned and the other is "bring your own credentials" (e.g., the customer's existing AWS account, but a fresh Azure subscription), mix the resource block with secrets: injection. Setup logs into both as appropriate.
Container plus private VM behind IAP tunnel
A track that teaches GPU work or runtime-provisioned VMs cannot use the platform's standard virtualmachines: block — that block is for VMs whose lifecycle is owned by the sandbox, not VMs the lesson itself provisions. A learner-launched VM also should not be exposed to the public internet; doing so is a security smell and requires firewall churn the lesson does not teach. The clean answer is IAP: the VM has no public IP, the workstation tunnels into it through Google's Identity-Aware Proxy.
Use a workstation container with a GCP project. From setup, provision the VM with gcloud compute instances create and no external IP. Open IAP tunnels mapping remote ports to local ports on the workstation. Surface terminal tabs that ssh -p <local-port> root@localhost and service tabs pointing at the local tunnel ports.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash
gcp_projects:
- name: lab-project
services: [compute.googleapis.com, iap.googleapis.com]
regions: [us-central1]
roles:
- roles/iap.tunnelResourceAccessor
- roles/compute.viewer
admin_roles: [roles/owner]#!/bin/bash
set -euxo pipefail
# (Decode SA key, gcloud auth — see Recipe 9.)
# Provision a private VM with no external IP
gcloud compute instances create lab-vm \
--zone=us-central1-a \
--machine-type=n1-standard-2 \
--no-address \
--image-family=ubuntu-2204-lts \
--image-project=ubuntu-os-cloud
# Allow IAP CIDR through the firewall
gcloud compute firewall-rules create allow-iap \
--direction=INGRESS \
--source-ranges=35.235.240.0/20 \
--allow=tcp:22,tcp:80
# Open IAP tunnels in the background
nohup gcloud compute start-iap-tunnel lab-vm 22 \
--local-host-port=localhost:2222 \
--iap-tunnel-disable-connection-check \
> /tmp/ssh-tunnel.log 2>&1 & disown
nohup gcloud compute start-iap-tunnel lab-vm 80 \
--local-host-port=localhost:8888 \
--iap-tunnel-disable-connection-check \
> /tmp/http-tunnel.log 2>&1 & disown
# Wait until the SSH tunnel is reachable
until nc -z localhost 2222; do sleep 2; donetabs:
- title: VM Terminal
type: terminal
hostname: workstation
cmd: ssh root@localhost -p 2222 -o StrictHostKeyChecking=no
- title: Web App
type: service
hostname: workstation
port: 8888--iap-tunnel-disable-connection-check is mandatory when the target VM may still be booting. Without it, gcloud performs a synchronous connectivity probe that exits non-zero before the VM finishes startup, and your setup script aborts.
The IAP role must be on both the user role and the admin role for the project. Missing it from one of them produces opaque permission errors deep inside gcloud.
nohup ... & disown is the right pattern for tunnels within a single setup script. Do not rely on nohup to keep tunnels alive across challenge boundaries — the runner does not preserve nohup'd processes between challenges. If a tunnel must survive challenge transitions, install it as a systemd service on a real VM rather than backgrounding from a container.
For GPU labs that should be cost-bounded, provision the VM with --provisioning-model=FLEX_START --instance-termination-action=DELETE --max-run-duration=7200s. FLEX_START is significantly cheaper than standard but instances may be reclaimed.
For tracks that need the VM ports to be reachable from the learner's browser (not from the workstation container), expose the VM with a sandbox-public URL pattern instead of IAP. See Recipe 39.
Container plus FastAPI service
A REST API lesson where the learner only reads the spec is forgettable. A lesson where the learner can hit the API live, see real responses, and edit the running code is what closes the loop. Spinning up a FastAPI service in a separate container — with autogenerated /docs Swagger UI — gives the learner a hands-on surface with no extra teaching cost.
Declare a workstation container with the cloud-client image and a separate container running a FastAPI application. Wire the workstation to the API via environment:. Expose FastAPI's autogenerated Swagger UI as a service tab; expose the workstation terminal as a terminal tab.
version: "3"
containers:
- name: workstation
image: cloud-client
memory: 1024
shell: /bin/bash
environment:
API_URL: http://api:8000
- name: api
image: example/fastapi-app:latest
ports: [8000]
memory: 512tabs:
- title: Terminal
type: terminal
hostname: workstation
- title: API Docs
type: service
hostname: api
port: 8000
path: /docsFastAPI's /docs page is interactive; the learner can fire requests from the browser. That is exactly the point — but it also means the API's CORS configuration must allow the platform's traffic-proxy origin. If the lesson involves the learner pasting a curl command into the terminal and clicking "Try it out" in the docs page, both code paths must work; CORS misconfiguration breaks the click-through path silently.
FastAPI's reload mode (uvicorn --reload) is convenient during authoring but expensive at runtime — it watches the filesystem and restarts on every change, including any file the learner edits. Disable reload in track-shipped images.
If the API depends on a database, wire the database as a third container and gate setup on database readiness (Recipe 22). The workstation should never assume the API is ready; always poll the API's health endpoint before any challenge that depends on it.
For a track where the API and a portal frontend live in one container on one port, see Recipe 25.
For a track where the API also needs a custom CLI wrapper (so the learner can use a branded acme-cli rather than raw curl), see Recipe 13.
For a track where the API is the lesson's entire surface and there is no terminal, drop the workstation container and surface the docs tab as the only working interface.
Container plus mock REST API and custom CLI
When the lesson teaches a product's CLI ergonomics — "use acme-cli policy create to do X" — the cleanest experience is the real CLI talking to the real backend. The real backend is often unavailable, expensive, or stateful in ways the lesson does not want to manage. A mock REST API plus a branded CLI wrapper gives the same learner experience with predictable backend behavior.
Declare a workstation container that holds the branded CLI and a separate container running a mock REST API (FastAPI is convenient). The CLI is a small Python script using httpx and rich that points at the mock API via an env var. The mock API returns canned-but-realistic responses keyed by request shape.
version: "3"
containers:
- name: workstation
image: example/branded-workstation:latest
memory: 1024
shell: /bin/bash
environment:
ACME_API_URL: http://api:8000
PATH: /opt/acme-cli/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
- name: api
image: example/acme-mock-api:latest
ports: [8000]
memory: 512# /opt/acme-cli/bin/acme — packaged into the workstation image
import argparse, os, sys
import httpx
from rich.console import Console
from rich.table import Table
console = Console()
def cmd_policy_list(args):
r = httpx.get(f"{os.environ['ACME_API_URL']}/policies")
r.raise_for_status()
table = Table()
table.add_column("ID", style="cyan")
table.add_column("Name", style="green")
for p in r.json():
table.add_row(p["id"], p["name"])
console.print(table)
def main():
parser = argparse.ArgumentParser(prog="acme")
sub = parser.add_subparsers(required=True, dest="resource")
p = sub.add_parser("policy"); ps = p.add_subparsers(required=True, dest="verb")
pl = ps.add_parser("list"); pl.set_defaults(func=cmd_policy_list)
args = parser.parse_args()
args.func(args)
if __name__ == "__main__":
main()A two-level argparse (<resource> <verb>) reads naturally and matches what most SaaS CLIs do. Avoid one-level argparse for anything beyond a toy CLI — learners struggle to discover commands without the resource grouping.
When the mock API returns an error, the CLI must surface it cleanly. Catch httpx.HTTPStatusError and render the response body in a colored panel rather than letting Python dump a traceback. A traceback-on-error CLI feels half-finished.
Bake the CLI into the workstation image rather than installing it via setup. Setup-time installs of Python packages are slow and add a class of failure at sandbox start. The image is the right boundary.
For tracks where the CLI also exposes interactive subcommands (Q&A flows, certification answer submission), the CLI grows but the structure stays the same. See Recipe 30 for the larger CLI-wrapper shape.
For tracks where the API and a portal page live together, route the CLI traffic and the portal traffic through one port (Recipe 25). Two ports for related surfaces is usually unnecessary.
Container plus Postgres with a live web dashboard
A pure terminal track teaching SQL is functional but flat. The learner runs CREATE VIEW, types a query, and trusts that something happened. There is no feedback loop that closes the gap between "I ran a SQL statement" and "I changed something visible."
Pair a workstation container (with psql and any other tooling the learner needs) with a postgres container as the database, then add a small web application — Flask or FastAPI — that polls the database every few seconds and renders the current state. Surface the dashboard as a service tab so it lives next to the terminal.
The dashboard becomes the feedback loop. When the learner creates a view, populates a table, or runs a stored procedure, the dashboard updates and the learner sees their work materialize.
A two-container config.yml:
version: "3"
containers:
- name: workstation
image: cloud-client
ports: [5000]
memory: 1024
shell: /bin/bash
environment:
DATABASE_URL: postgres://app:app@db:5432/app
- name: db
image: postgres:16
ports: [5432]
memory: 1024
private: true
environment:
POSTGRES_USER: app
POSTGRES_PASSWORD: app
POSTGRES_DB: appA track-level setup script on workstation that waits for postgres, seeds the schema, and launches the dashboard:
#!/bin/bash
set -euxo pipefail
# Wait for postgres to accept connections
until pg_isready -h db -U app -d app; do sleep 1; done
# Seed the schema
psql "$DATABASE_URL" < /opt/seed/schema.sql
# Launch dashboard in background
cd /opt/dashboard
nohup python3 app.py > /var/log/dashboard.log 2>&1 &
# Poll for readiness so the service tab does not 502
until curl -fsS http://localhost:5000/healthz > /dev/null; do sleep 1; doneThe assignment frontmatter exposes the dashboard alongside the terminal:
tabs:
- title: Terminal
type: terminal
hostname: workstation
- title: Dashboard
type: service
hostname: workstation
port: 5000The postgres image takes longer to be ready than to be running. Use pg_isready (which ships with the postgres client) rather than checking that the container is up — the container accepts TCP connections seconds before the database accepts queries. If your workstation image lacks pg_isready, fall back to a Python loop that catches psycopg2.OperationalError and retries.
The dashboard's JavaScript must use relative paths for any fetch calls. The learner's browser is talking to the platform's traffic proxy, not your Docker network, and any absolute URL pointing at an internal hostname will fail with no usable error message.
The setup script should poll the dashboard's own port for HTTP readiness before exiting. Without that final wait, the service tab loads while the Flask app is still binding the port and the learner sees a 502 on first render.
When the dashboard's purpose is to demonstrate a REST API rather than a UI, swap Flask for FastAPI and let the auto-generated /docs Swagger page serve as the interactive surface. The pattern collapses to "container + postgres + FastAPI service tab" with no separate frontend code at all (Recipe 12).
When the dashboard needs to host both a portal page and a backend API, serve them from the same hostname and the same port, routing by URL prefix on the server side. The proxy addresses one endpoint per service tab; collapsing two surfaces into one removes a class of cross-origin headaches (Recipe 25).
VM plus virtual browser
When the lesson is a click-through against a real product UI, a terminal tab is the wrong surface. A real browser running inside the lab — pointed at the right URL, with the right credentials surfaced — is what the lesson needs. Authors who build this by giving the learner an external link out of the lab break the embedded experience and lose all the contextual goodness the platform provides.
Declare the VM that hosts (or backs) the application. Declare a virtualbrowsers: entry pointing at the URL. Surface the virtual browser as a tab. Credentials are surfaced separately in the assignment body via agent variable interpolation.
version: "3"
virtualmachines:
- name: app-vm
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
provision_ssl_certificate: true
allow_external_ingress: [http, https]
virtualbrowsers:
- name: app-ui
url: https://app-vm.${_SANDBOX_ID}.instruqt.iotabs:
- title: Terminal
type: terminal
hostname: app-vm
- title: Application
type: browser
hostname: app-uiThe browser tab type targets the name: of a virtualbrowsers: entry, not a hostname directly. Mismatching produces a tab that loads but does not point anywhere obvious.
Internal-hostname virtualbrowsers: URLs work — https://app-vm.${_SANDBOX_ID}.instruqt.io resolves through the platform's traffic proxy. Pair with provision_ssl_certificate: true on the VM. Useful when one VM serves multiple SaaS-style URLs/paths.
For external SaaS targets (Google Cloud console, AWS console, ServiceNow), a virtualbrowsers: entry can point at the product's URL directly. Credentials go in the assignment body via agent variable interpolation; do not include credentials in the URL query string — they will appear in browser history and server logs.
For a website tab pointed at a sandbox-public URL, use type: website with an explicit HTTPS URL. The website tab supports ${_SANDBOX_ID} interpolation in url:; the URL must be HTTPS or instruqt track push validation fails.
For tracks that need to expose multiple paths on the same VM (e.g., one URL for the admin UI, another for the user UI, a third for a metrics endpoint), add multiple virtualbrowsers: entries with the same hostname and different paths.
For a true Linux desktop rather than a SaaS UI, see Recipe 16 (Guacamole) or Recipe 17 (noVNC).
VM plus Guacamole desktop
Web-based desktops (noVNC, KasmVNC) cover most needs. They do not cover Windows; they do not cover scenarios that require RDP-specific features (clipboard sharing, printer redirection, multi-monitor); and they do not cover the multi-VM pattern where the learner switches between several remote desktops in one session. Guacamole is the right tool for those cases.
Run a Guacamole container next to the target VM(s). Configure Guacamole's user-mapping.xml with one connection per target. Surface Guacamole as a service tab whose path deeplinks the learner straight into a connection.
version: "3"
containers:
- name: guac
image: guacamole
ports: [8080]
memory: 1024
virtualmachines:
- name: target-windows
image: windows-server-2022
machine_type: n2-standard-4
shell: powershell#!/bin/bash
set -euxo pipefail
mkdir -p /config/guacamole
cat > /config/guacamole/user-mapping.xml <<'EOF'
<user-mapping>
<authorize username="guac_user" password="guac_password">
<connection name="windows-target">
<protocol>rdp</protocol>
<param name="hostname">target-windows</param>
<param name="port">3389</param>
<param name="username">Administrator</param>
<param name="password">${WIN_PASSWORD}</param>
<param name="ignore-cert">true</param>
<param name="resize-method">display-update</param>
<param name="enable-font-smoothing">true</param>
</connection>
</authorize>
</user-mapping>
EOF
# Wait for the Windows VM's RDP listener
while ! nc -z target-windows 3389; do sleep 5; donetabs:
- title: Windows Desktop
type: service
hostname: guac
port: 8080
path: /guacamole/#/client/<connection-name>?username=guac_user&password=guac_passwordGuacamole v2 path format differs from v1. v1 used /guacamole/#/?username=...&password=... (hash-fragment query string). v2 requires the connection name in the path: /guacamole/#/client/<connection-name>?username=...&password=.... Upgrading the Guacamole image from v1 to v2 without updating the path lands the learner on the connection-selection screen rather than the desktop.
Windows passwords from aws ec2 get-password-data may contain XML special characters. Escape before embedding in user-mapping.xml: a small escape_xml() shell helper that runs the password through sed for &, <, >, ", and '. Without escaping, the XML fails to parse and all connections appear broken.
Polling for RDP readiness (nc -z target 3389) is the right gate before declaring setup done. Windows VMs take longer than Linux VMs to expose RDP — 90+ seconds is normal — and a Guacamole tab that loads before RDP is ready shows a permanent connection error in the desktop pane.
For Linux desktops, swap RDP for VNC and point the Guacamole connection at port 5901 with <protocol>vnc</protocol> and the appropriate VNC password parameters.
For multi-VM scenarios where the learner switches between desktops, list multiple <connection> blocks under one <authorize> block. The Guacamole sidebar shows all connections; the learner picks.
For tracks that need a Windows-style desktop without a real Windows VM, run a Linux VM with nested virtualization and a Windows-desktop Docker container exposing RDP on port 3389. Cheaper than a real Windows VM but slower to boot and resource-constrained.
VM plus noVNC Linux desktop
For a single Linux desktop session — XFCE, GNOME, KDE — Guacamole works but is heavy. It runs a separate container, requires user-mapping.xml, and adds a layer between the learner and the desktop. Sometimes you just want the desktop on the VM, accessible directly from a browser tab.
Install tightvncserver and a desktop environment on the VM. Wrap with noVNC for browser access over WebSocket. Expose via a website tab with a sandbox-public URL.
version: "3"
virtualmachines:
- name: ubuntu-desktop
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
provision_ssl_certificate: true
allow_external_ingress: [http, https, high-ports]#!/bin/bash
set -euxo pipefail
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y -qq xfce4 xfce4-goodies tightvncserver wget git
# Install noVNC
cd /opt
wget -q https://github.com/novnc/noVNC/archive/refs/tags/v1.3.0.tar.gz
tar xzf v1.3.0.tar.gz
git clone https://github.com/novnc/websockify noVNC-1.3.0/utils/websockify
# VNC password
mkdir -p /root/.vnc
echo "password" | vncpasswd -f > /root/.vnc/passwd
chmod 600 /root/.vnc/passwd
# Certs in BOTH locations
cp /root/self.crt /root/noVNC-1.3.0/utils/websockify/self.crt
cp /root/self.key /root/noVNC-1.3.0/utils/websockify/self.key
# Start VNC
vncserver :1 -geometry 1280x800 -depth 24
# Start noVNC over TLS
/opt/noVNC-1.3.0/utils/novnc_proxy \
--vnc localhost:5901 \
--cert /root/self.crt --key /root/self.key \
--ssl-only --listen 8443 \
> /var/log/novnc.log 2>&1 &tabs:
- title: Desktop
type: website
url: https://ubuntu-desktop.${_SANDBOX_ID}.instruqt.io:8443/vnc.html?host=ubuntu-desktop.${_SANDBOX_ID}.instruqt.io&port=8443Certs must exist in two places — /root/ for whatever provisioned them, and inside noVNC-1.3.0/utils/websockify/ for the websockify binary. Forgetting the second copy yields an SSL handshake failure with no obvious symptom on the learner's side beyond "the page does not load."
provision_ssl_certificate: true mints a real cert via the platform — do not run certbot certonly at runtime. The Let's Encrypt rate limit (5 certs per registered domain per week) silently breaks tracks under load, and provision_ssl_certificate: true has no such limit.
The ?host=&port= query string on the URL is what makes noVNC auto-connect. Without it, the learner lands on the noVNC home page and has to fill in the form by hand.
For tracks that need a single GUI app rather than a full desktop, swap to KasmVNC with the app baked into a Docker container (Recipe 18). Lighter setup, simpler tab.
For tracks where the desktop is Windows, swap to Guacamole RDP (Recipe 16). noVNC for Windows is technically possible but rarely worth the friction.
VM plus KasmVNC single-app
A full desktop is overkill when the lesson needs only one app. Booting XFCE just to run a single IDE is slow, pulls in window-manager UI the lesson does not teach, and gives the learner a desktop they do not need. KasmVNC-based Docker images bundle one app with a streaming-VNC server, exposing the app directly through a browser tab.
Run a Linux VM with nested virtualization and Docker. Pull a KasmVNC-based image of the target app onto the VM in setup. Run the container with the app bound to the KasmVNC port. Surface a service tab pointing at the VM's KasmVNC port.
version: "3"
virtualmachines:
- name: ide-vm
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
nested_virtualization: true
provision_ssl_certificate: true
allow_external_ingress: [http, https, high-ports]#!/bin/bash
set -euxo pipefail
# Install Docker
apt-get update -qq
apt-get install -y -qq docker.io
systemctl enable --now docker
# Pre-pull the IDE image (also do this in image baking — see Recipe 20)
docker pull example/kasm-ide:latest
# Project workspace; container UID may differ from root
mkdir -p /workspace
chown -R 1000:1000 /workspace
# Run the IDE
docker run -d --restart=unless-stopped \
-p 8080:6901 \
-v /workspace:/home/kasm-user/workspace \
example/kasm-ide:latest
# Wait for HTTP readiness
until curl -s -o /dev/null -w "%{http_code}" http://localhost:8080 \
| grep -qE "^(200|302|401)$"; do sleep 3; donetabs:
- title: IDE
type: service
hostname: ide-vm
port: 8080KasmVNC images run as a non-root user inside the container. The default UID is image-specific — common values are 911 and 1000. Run docker run --rm <image> id -u to discover it, or check the image's Dockerfile for USER. Mounting /workspace without chown-ing it to the right UID means the container cannot write, and the IDE silently fails to load.
Display server constraints: containers on the platform do not support virtual display servers (X11, Xvfb). KasmVNC images that need an X server will not run in a container — they require a VM with nested_virtualization: true. A web-based IDE like a browser-based code editor is fine in a container because it is a web server, not a GUI app.
Pre-pull the KasmVNC image during VM image baking, not at first-challenge setup. KasmVNC images are large (1–3 GB), and a first-challenge pull takes the lab past the start-up budget.
For a track that needs to patch an internal Chromium binary inside the KasmVNC image (e.g., add --no-sandbox), use Recipe 27's idempotent binary wrapper, applied via docker exec -u root against the running container.
For multiple GUI apps in one session, swap to Recipe 17 (noVNC desktop) or Recipe 16 (Guacamole). KasmVNC excels at one app per tab.
VM plus Docker plus ttyd terminal
Terminal tabs are great for shell sessions. They are not great for programs that own the screen — ncurses interfaces, terminal-based games, REPLs with custom keybindings. The Instruqt terminal tab uses xterm.js with platform-specific input handling and may not pass every keystroke through cleanly. The clean answer is ttyd, a small program that wraps a terminal session in a WebSocket and serves it over HTTP, which the platform can frame as a service tab.
On a VM with Docker, run the target program inside a container with ttyd --writable wrapping it. Set the base path so multiple ttyd instances can coexist behind a reverse proxy. Add an nginx config on the VM that proxies each path to its ttyd instance with WebSocket support. Surface each as a service tab.
version: "3"
virtualmachines:
- name: workstation
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
provision_ssl_certificate: true
allow_external_ingress: [http, https, high-ports]#!/bin/bash
set -euxo pipefail
apt-get update -qq && apt-get install -y -qq docker.io nginx
systemctl enable --now docker
# Run the target program with ttyd
docker run -d --restart=unless-stopped --name=game \
-p 7681:7681 \
example/dungeon:latest \
ttyd --writable --port 7681 --base-path /game/ \
--client-option fontSize=20 \
/usr/local/bin/dungeon
# nginx reverse proxy with WebSocket directives
cat > /etc/nginx/conf.d/ttyd.conf <<'NGX'
server {
listen 8443 ssl;
ssl_certificate /root/self.crt;
ssl_certificate_key /root/self.key;
location /game/ {
proxy_pass http://127.0.0.1:7681;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400s;
proxy_send_timeout 86400s;
}
}
NGX
systemctl reload nginxtabs:
- title: Game
type: service
hostname: workstation
port: 8443
path: /game/WebSocket termination through nginx requires proxy_http_version 1.1, proxy_set_header Upgrade $http_upgrade, and proxy_set_header Connection "upgrade". Missing any of these breaks the upgrade handshake silently — the page loads but the terminal never appears.
The default nginx 60-second timeout closes idle WebSocket connections mid-session. Set proxy_read_timeout and proxy_send_timeout to a long value (24 hours is common) — without this, learners get kicked out of the program after one minute of idle.
ttyd's --writable flag is required for any session the learner will interact with. Without it, the terminal is read-only and keystrokes do not reach the underlying program.
For multiple ttyd instances on one VM (a multi-pane retro arcade, a multi-REPL lesson), run each as its own container with its own port and add one location /name/ block per program in the nginx config. Pre-pull all the images in setup (Recipe 20).
For a single program that must run as a privileged process and cannot live in a container, skip Docker and run ttyd directly against the program on the host VM. The nginx reverse proxy is identical.
Pre-pull Docker images in setup
docker compose up (or the setup script's first docker run) would otherwise pull large images at lab start.
The first docker compose up in a track that uses six images and is not pre-pulled takes longer than the rest of the lesson. Learners hit a long pause they cannot diagnose, the demo loses momentum, and the lab clock is eating budget while nothing happens.
Pre-pull every image the lesson uses in track-level setup, in parallel. Background each pull with &, then wait for all of them. The setup script bears the wait so the first interactive moment is fast.
#!/bin/bash
set -euxo pipefail
docker pull example/web:1.4 &
docker pull example/api:2.1 &
docker pull example/worker:2.1 &
docker pull postgres:16 &
docker pull redis:7-alpine &
docker pull nginx:1.27-alpine &
waitWithout wait, setup completes before the pulls finish. The first docker compose up still pays the full cost. The & backgrounds the pulls; the lone wait joins them.
Pre-pulling does not help when the image gets re-pulled later because it changed upstream. Pin tags to specific versions when the lesson should be deterministic; use mutable tags (:latest) only for content that genuinely should track upstream.
For very large images (multi-GB IDE-in-browser, ML stacks), bake them into the VM image rather than pulling at setup time. Image bake is a longer feedback loop for the author but a much shorter cold-start for the learner.
For tracks that pull from a private registry, declare the credentials in secrets: and docker login before the pull. Do not bake credentials into the setup script body.
For tracks with extremely large images and a tight start-up budget, pre-pull at image bake time. The Packer/image-bake pipeline does the work once for every learner who uses the image.
Bootstrap wait
Setup scripts can begin before the host's bootstrap pipeline has finished writing the helper binaries (fail-message, set-status, agent variable) onto disk. Calls to those binaries fail with cryptic "command not found" errors. The race is timing-dependent and intermittent.
Block on the bootstrap sentinel file at the top of every track-level setup script.
#!/bin/bash
set -euxo pipefail
# On VMs and the lighter shell container image:
until [ -f /opt/instruqt/bootstrap/host-bootstrap-completed ]; do
sleep 1
done# On the cloud-client image, the sentinel is different:
until [ -f /opt/instruqt/bootstrap/gcp-bootstrap-completed ]; do
sleep 1
doneThe sentinel file path differs by image. On VMs and on the lighter shell base image, the file is host-bootstrap-completed. On the cloud-client base image, it is gcp-bootstrap-completed. Both live under /opt/instruqt/bootstrap/. Mixing them up causes the wait to either return immediately (false ready) or never (hang).
For Hashi-style hardened setups, pair the file wait with a port check on the local agent: ss -tuln | grep -q ':15779'. The port check catches the case where the file exists but the agent has not fully come up.
Per-challenge setup does not need this wait — by the time challenge N runs, the host has been up for a while. Reserve the wait for track-level setup.
For tracks where setup may run before some other prerequisite (a peer VM is reachable, an external API has provisioned a tenant), layer additional waits on top of the bootstrap sentinel. They are not substitutes for each other.
Wait-for-port readiness probe
A naive "this should be ready by now" assumption fails the moment the service takes longer to start than usual. CI is flaky, lab cold-starts surface the timing, and the failure mode is a setup script that fails on a randomly timed first run and works on retry.
Poll the target's port (or a specific health endpoint) until it answers, with a sane sleep and a sane retry budget. Distinguish between local-service readiness (flat retry, short total budget) and shared SaaS readiness (exponential backoff with longer budget).
# Local service — flat retry
for i in $(seq 1 30); do
if curl -fsS http://127.0.0.1:8080/health > /dev/null; then
echo "Ready"
break
fi
echo "Retry $i/30"
sleep 2
done# Postgres — accept TCP plus accept queries
until pg_isready -h db -U app -d app; do sleep 1; done
# As a fallback when pg_isready is unavailable
until PGPASSWORD=app psql -h db -U app -d app -c 'SELECT 1' > /dev/null 2>&1; do
sleep 2
done# Generic HTTP — accept any "responsive" status
until curl -s -o /dev/null -w "%{http_code}" http://localhost:8080 \
| grep -qE "^(200|302|401)$"; do sleep 3; doneContainer readiness is not service readiness. A container can accept TCP connections seconds before the service inside it accepts requests. Always poll the service-level signal (pg_isready, /health, an authenticated probe), never just the port being open.
A "readiness" check that accepts only HTTP 200 misses serving-but-redirecting and serving-but-auth-required cases. Accept 200, 302, and 401 as "the server is responding." That is what the regex ^(200|302|401)$ is for.
For shared external SaaS APIs (a vendor service many learners share), use exponential backoff to avoid thundering-herd retries. For local services on localhost, flat retry is simpler and adequate.
For tracks where the target is a Kubernetes API server, use kubectl proxy and poll /healthz. More reliable than kubectl cluster-info, which can return success before the API is fully ready.
For tracks where readiness depends on a Kubernetes Event rather than a port (Helm chart provisioning), poll for the event with kubectl wait --for=condition=... rather than tailing logs.
Background process launch with log redirection
A background process launched naively from setup leaves stdout and stderr attached to the script's session. Once the script exits, the process can lose its file descriptors and die. Even when it survives, debugging its behavior later is hard because the output disappeared.
Redirect stdout and stderr to a log file, run with nohup, and disown. The process keeps running, the logs are inspectable, and the script can exit cleanly.
#!/bin/bash
set -euxo pipefail
cd /opt/dashboard
nohup python3 app.py > /var/log/dashboard.log 2>&1 &
disown
# Wait until the dashboard responds before completing setup
until curl -fsS http://localhost:5000/healthz > /dev/null; do sleep 1; donenohup keeps the process alive across the script's exit, not across challenge boundaries. The platform's runner does not preserve nohup-ed processes between challenges. If a process must survive a challenge transition, install it as a systemd service.
disown removes the job from the script's job table so the script can exit cleanly. Without it, the script may wait at exit for the backgrounded job, wasting boot budget.
Polling for HTTP readiness after launch is the difference between a service tab that works on first render and one that 502s for the first 5–10 seconds of the lab.
For services that must survive challenge boundaries, write a systemd unit (Restart=always, RestartSec=5s) and systemctl enable --now. Survives reboots, restarts on crashes, and is what you want for any persistent service.
For Windows VMs, use Register-ScheduledTask with New-ScheduledTaskTrigger -AtStartup instead of nohup. Bash conventions do not apply on Windows.
Heredoc-based project generation
External content repos break in two predictable ways: the GitHub clone fails because the network blip ate the request, or the upstream changed in a way that breaks the lesson. A small project that lives entirely inside the setup script avoids both.
Write the project tree from setup using bash heredocs. One cat > path block per file. mkdir -p directories beforehand. Use single-quoted heredoc terminators ('EOF') for files where bash should not expand variables (Dockerfiles, sample code that contains $VAR literally).
#!/bin/bash
set -euxo pipefail
mkdir -p /workspace/myapp/{src,tests}
cat > /workspace/myapp/src/app.py <<'EOF'
from flask import Flask
app = Flask(__name__)
@app.route("/")
def home():
return "hello, world"
EOF
cat > /workspace/myapp/Dockerfile <<'DOCKERFILE'
FROM python:3.11-slim
WORKDIR /app
COPY src/ /app/
RUN pip install flask
CMD ["python", "app.py"]
DOCKERFILE
cat > /workspace/myapp/docker-compose.yml <<EOF
services:
web:
build: .
ports:
- "5000:5000"
environment:
ENV_NAME: ${ENV_NAME:-dev}
EOF
# Pre-build so the first compose up is a no-op
cd /workspace/myapp
docker compose buildSingle-quoted terminators ('EOF') disable variable expansion in the heredoc body. Use them for anything that contains $VAR literally — Dockerfiles, code that interpolates at its own runtime, environment-variable examples. Use unquoted terminators when bash should expand variables before writing.
Pre-build images in setup so the learner's first docker compose up is a near-instant no-op. A learner who waits 30 seconds for the first build loses momentum.
For codebases over a hundred files, switch to a clone-from-public-repo pattern (Recipe 41 for private repos, plain git clone for public). Heredocs scale poorly past that size.
For tracks that need the project to be re-creatable on each challenge (a "reset" pattern between challenges), wrap the heredoc block in a per-challenge setup script and add a rm -rf of the prior state. The reset must be idempotent.
For tracks where the heredoc would be very long, bake the project into the custom VM image instead. Image bake is a one-time cost; setup is paid every lab session.
Co-hosted portal plus API
Splitting a portal page and its API into two service tabs means two places where the URL might be wrong, two places to authenticate, and a guarantee of cross-origin pain when the portal's fetch calls go to the API's URL. The clean answer is one service tab, one port, and routing inside the server.
Serve both the portal and the API from the same hostname and the same port. Route by URL prefix on the server side: /api/* goes to the API handler, everything else is the portal HTML. The platform addresses one endpoint per service tab; collapsing two surfaces into one removes the cross-origin class of bugs.
# /opt/portal/main.py — single FastAPI app exposing portal + API on one port
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from fastapi.responses import JSONResponse
app = FastAPI()
@app.get("/api/healthz")
def healthz():
return {"ok": True}
@app.get("/api/items")
def items():
return JSONResponse([{"id": "1", "name": "alpha"}])
# Everything else: serve the portal SPA
app.mount("/", StaticFiles(directory="/opt/portal/static", html=True))tabs:
- title: Portal
type: service
hostname: workstation
port: 8080JS fetch() calls in the portal must use relative paths (/api/items), not absolute URLs pointing at internal hostnames. The platform's traffic proxy proxies the one declared port; absolute URLs to internal Docker hostnames fail with no usable error.
Routing order matters in single-app frameworks. Mount static files last (or with a lower priority than API routes); otherwise the static-file handler swallows /api/* requests.
When the API and portal need different memory or runtime characteristics, do not collapse them onto one process — split the containers and accept the cross-origin work. The collapse is for cases where one process can serve both cleanly.
For a track where the portal is purely static and only one API endpoint is needed, an Nginx server with a location /api/ proxy block is simpler than a single Python app.
For a track where multiple APIs back the portal (e.g., REST plus WebSocket), keep them in one app or one Nginx config — the goal is one declared port to the platform.
Helper Python module for tests
Copy-pasted assertion code in five check scripts will diverge in five different ways within two weeks. The fix is to put the helpers in one place and import them from each script.
In track-level setup, write helper Python modules into the system site-packages directory. Import them from any check or solve script.
cat > /usr/lib/python3/dist-packages/instruqt_tests.py <<'PY'
import os, re, subprocess
def test_file_exists(path):
assert os.path.isfile(path), f"missing: {path}"
def test_string_in_file(path, needle):
with open(path, encoding="utf-8") as f:
assert needle in f.read(), f"{needle!r} not in {path}"
def test_command(cmd, expected_pattern):
out = subprocess.run(cmd, shell=True, capture_output=True, text=True).stdout
assert re.search(expected_pattern, out), f"output of `{cmd}` did not match {expected_pattern!r}"
PY#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from instruqt_tests import test_file_exists, test_command
test_file_exists("/workspace/output.txt")
test_command("psql -At -c 'SELECT count(*) FROM users'", r"^[3-9]$")
PYWriting into /usr/lib/python3/dist-packages/ requires root, which setup has by default. Do not write into /usr/local/lib/... for shared helpers — it works but is harder to discover.
Keep helpers small and assertion-shaped. The point is to remove duplication across check scripts; the moment the helper grows test orchestration logic, switch to a real test runner (pytest, mocha) instead.
Solve scripts can use the same module — run_as_user(), add_to_bash_history(), "do the lesson in five lines" helpers — without duplicating the bash behind them.
For tracks in JavaScript-heavy ecosystems, the same idea applies with mocha + chai, with the helper bundle written into a per-track tests directory and each check script invoking it via a single mocha invocation. Pick the language that matches the ecosystem.
For tracks where check scripts must run inside a heredoc'd Python block (no separate file), inline the helper code rather than importing — only when the check logic is small enough to inline cleanly.
Idempotent binary wrapper
The standard pattern for wrapping a binary is to rename the original out of the way, then write a shell script in its place that calls the renamed original with extra arguments. For example, a track that needs Chromium to run with --no-sandbox inside a constrained container will rename /usr/bin/chromium to /usr/bin/chromium-real and write a new /usr/bin/chromium that exec's chromium-real --no-sandbox "$@".
This works perfectly the first time. On the second run, the rename moves the wrapper to chromium-real, and the new wrapper ends up calling itself. The result is infinite recursion the first time anyone invokes the binary, often manifesting as a process explosion that fills the host's process table before anything else gets a chance to run.
Guard the rename. Only move the original out of the way if it has not already been moved.
#!/bin/bash
set -euo pipefail
# Only rename if the real binary hasn't already been moved
if [ ! -f /usr/bin/chromium-real ]; then
mv /usr/bin/chromium /usr/bin/chromium-real
fi
# Write (or overwrite) the wrapper
cat > /usr/bin/chromium <<'EOF'
#!/bin/bash
exec /usr/bin/chromium-real --no-sandbox "$@"
EOF
chmod +x /usr/bin/chromiumThe wrapper itself is safe to rewrite on every run — the danger is only in the rename. Putting the rename behind the [ ! -f ] guard is the entire fix.
If the original binary is a symlink rather than a file, [ ! -f ] still works because -f follows symlinks, but mv will move the symlink itself. If you need the underlying target moved instead, resolve first with readlink -f.
This pattern composes with anything that needs an idempotent overwrite-once: replacing a config file, swapping a default editor, or substituting a library. The same [ ! -f <backup> ] guard pattern applies in each case.
When the wrapping is conditional on runtime context — for example, only patching the binary inside a specific container — combine the guard with a feature-detection check rather than running the patch unconditionally.
When the binary lives inside a container that is already running and needs to be patched post-start, use docker exec -u root <container> bash -c '...' and apply the same idempotency guard inside the inner shell.
When multiple wrappers chain (a binary wrapped, then re-wrapped by a higher-level tool), maintain a separate -real suffix per layer (chromium-real, chromium-real-real) and document the chain in a comment at the top of the setup script. The guard logic still applies per layer.
Profile.d plus .bashrc dual-write
/etc/profile.d/*.sh is sourced by login shells. ~/.bashrc is sourced by interactive non-login shells. A learner whose terminal opens with cmd: su - <user> gets a login shell; a learner whose terminal opens without that gets an interactive non-login shell. Putting the env var in only one place means it is missing in the other context — usually discovered when the lesson does not work and the env var turns out to be unset.
Write to both. /etc/profile.d/<lesson>.sh for login shells, and the user's .bashrc for plain interactive shells. Same content, two destinations.
#!/bin/bash
set -euxo pipefail
# Login shells (when the terminal opens via su -)
cat > /etc/profile.d/lab-env.sh <<'EOF'
export ACME_API_URL="http://api:8000"
export PS1='acme:\w$ '
EOF
chmod +x /etc/profile.d/lab-env.sh
# Interactive non-login shells (terminal tab without su -)
cat >> /root/.bashrc <<'EOF'
export ACME_API_URL="http://api:8000"
export PS1='acme:\w$ '
EOFA custom PS1 belongs in .bashrc rather than profile.d. The login-shell flow already handles PS1, but plain interactive shells will inherit the default unless .bashrc overrides it.
Avoid ANSI color codes inside PS1. Color works fine in the terminal tab's xterm.js, but when setup echoes $PS1 for diagnostics, the codes corrupt the platform's debug log.
For a cleaner branded prompt, omit username and hostname: \u@\h adds noise and makes copy-pasted commands ugly. \w for the current directory plus $ (or # if you must signal root) is enough.
For tracks that create a non-root user (Recipe 29), write to both /etc/profile.d/ and the user's home .bashrc, not root's. Mismatching the user's home directory leaves the env vars unset when the user logs in.
For tracks that need a function (not just a variable) in both contexts, define the function in the profile.d file and serialize it into .bashrc with type <funcname> | tail -n +2.
Non-root user pattern
sudo, file permissions, multi-user awareness — and running everything as root would distort the lesson.
Running setup as root is fine; running the learner's shell as root makes most "real Linux" lessons feel staged. The lesson teaches sudo apt-get install; the lab gives the learner a shell where sudo is invisible. Things you can do without thinking are not the things the lesson is about.
In setup, create a non-root user, copy whatever dotfiles and env vars they need into the user's home, and either set the terminal tab to cmd: su - <user> or set set-workdir to the user's home so plain bash opens there.
#!/bin/bash
set -euxo pipefail
useradd -m -s /bin/bash iggy
usermod -aG sudo iggy
echo "iggy ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/iggy
# Carry env over from root
cp /etc/profile.d/lab-env.sh /home/iggy/.bashrc-lab
echo 'source ~/.bashrc-lab' >> /home/iggy/.bashrc
chown -R iggy:iggy /home/iggytabs:
- title: Terminal
type: terminal
hostname: workstation
cmd: su - iggycmd: su - iggy opens a login shell for iggy. Env vars set only in root's .bashrc will not appear. Use Recipe 28's dual-write pattern, or copy the env into the user's home explicitly.
For a stock Debian or Ubuntu image whose default user is ubuntu (not iggy) and the lesson actually wants root, set cmd: sudo su on the terminal tab. Minimal ceremony, escalates cleanly.
VM-level shell: and per-tab cmd: are not equivalent. On VMs, shell: su - <user> at the VM level applies to every terminal tab without per-tab repetition. On containers, use per-tab cmd: su - <user>. Mixing them is allowed but noisy; pick one and stick with it within a track.
For tracks where multiple users matter (a lesson on file permissions across owners), create each user explicitly and let the learner switch between them as part of the lesson rather than via cmd: magic.
For tracks where the non-root user must exist on a fresh boot in a Windows VM, use PowerShell with Rename-Computer and account-creation cmdlets — the bash idiom does not apply.
Custom CLI wrapper
A real branded CLI talking to a mock backend is the cleanest learner experience for a SaaS-CLI lesson. You get the brand, the verbs, the panels, the colored output — without the real backend's latency or rate limits. The trick is making the wrapper feel like a real CLI rather than a thin shim.
Write a Python CLI that uses httpx for HTTP and rich for tables and panels. Two-level argparse (<resource> <verb>). Catch HTTP errors and connection errors with specific handlers; render the response body as a colored panel rather than letting Python dump a traceback. Bake into the workstation image; do not install at setup time.
#!/usr/bin/env python3
import argparse, os, sys
import httpx
from rich.console import Console
from rich.panel import Panel
from rich.table import Table
console = Console()
def client():
return httpx.Client(
base_url=os.environ["ACME_API_URL"],
headers={"Authorization": f"Bearer {os.environ.get('ACME_TOKEN', '')}"},
timeout=10.0,
)
def run(fn):
try:
fn()
except httpx.HTTPStatusError as e:
console.print(Panel(e.response.text, title=f"[red]API {e.response.status_code}[/red]"))
sys.exit(1)
except httpx.ConnectError:
console.print("[red]Could not reach the API. Is the service tab open?[/red]")
sys.exit(1)
except Exception:
console.print_exception()
sys.exit(1)
def policy_list(_):
def f():
with client() as c:
r = c.get("/policies"); r.raise_for_status()
t = Table(title="Policies")
t.add_column("ID", style="cyan")
t.add_column("Name", style="green")
for p in r.json():
t.add_row(p["id"], p["name"])
console.print(t)
run(f)
def main():
p = argparse.ArgumentParser(prog="acme")
sub = p.add_subparsers(dest="resource", required=True)
pol = sub.add_parser("policy")
psub = pol.add_subparsers(dest="verb", required=True)
psub.add_parser("list").set_defaults(func=policy_list)
args = p.parse_args()
args.func(args)
if __name__ == "__main__":
main()Single-file CLIs scale further than people expect — five resources × four verbs in one file is fine. Splitting into a package adds a layer of overhead the lab does not need.
Rich's Console().print_exception() formats unhandled errors clearly. Reach for it when an unexpected error surfaces; the default Python traceback wall is hostile to a learner who sees it for the first time.
Bake the CLI into the workstation image and prepend its bin directory to PATH in config.yml's environment: block. Setup-time pip install is slow and adds cold-start failure modes the lesson does not need.
For interactive CLIs that capture multi-line input (Q&A flows, certification answer submission), pair the CLI with a read -p loop that writes to a JSON file consumed by check scripts.
For tracks where the vendor publishes a Python SDK, the CLI can be a thin wrapper over the SDK rather than over raw HTTP. Same shape; different inner module.
Companion code-tab pair
A code lesson where the learner gets stuck at minute 30 has two bad outcomes: they give up, or they look up the answer outside the lab and break the experience. Providing a completed reference inside the lab — but not by default — gives them a way out without taking the lesson away.
Two code tabs in the assignment frontmatter. The first points at the learner's editable file. The second points at a completed reference file. Setup writes both files. The reference is for self-recovery only; the assignment body should not encourage opening it before attempting the work.
tabs:
- title: Terminal
type: terminal
hostname: workstation
- title: Your code
type: code
hostname: workstation
path: /workspace/acme_report.py
- title: Hint (reference)
type: code
hostname: workstation
path: /workspace/acme_report_complete.py#!/bin/bash
set -euxo pipefail
cat > /workspace/acme_report.py <<'EOF'
# TODO: implement summarize_orders(orders) returning a dict of region -> total
def summarize_orders(orders):
pass
EOF
cat > /workspace/acme_report_complete.py <<'EOF'
def summarize_orders(orders):
out = {}
for o in orders:
out[o["region"]] = out.get(o["region"], 0) + o["total"]
return out
EOFOrder matters in the tab list. Put the editable file before the reference. Learners scan tabs left-to-right; the editable one being first signals which is "yours."
If the reference is a complete spoiler of the lesson, the lesson is teaching the wrong shape. Refactor the lesson into smaller steps, each with its own reference, rather than one big reveal.
Per-challenge setup overwrites the editable file at challenge start. Learners who navigate away and back lose their work — usually fine for short challenges, painful for long ones. For long challenges, write the file only on first run ([ -f /workspace/acme_report.py ] || cat > ...).
For tracks where the reference lives in a documentation tab rather than a code tab, use type: code with a directory path; the editor renders a folder tree. Useful when the reference is a multi-file project.
For tracks where the hint should be progressive (one step at a time), see Recipe 51 (layered hint) — the hint tab is just one form factor of the broader pattern.
Defensive challenge-level setup
Tracks with skipping_enabled: true are useful: a learner who is short on time can jump to the part they care about. But the moment Challenge 3 assumes that Challenge 2 created a database row, a skip breaks the lesson silently. Equally, a learner who restarted the lab partway through may have lost the prior state without noticing.
In each challenge's setup-{hostname}, query for the expected state and re-create only what is missing. Idempotent and fast on the common case (state is already there); recovers transparently when it is not.
#!/bin/bash
set -euxo pipefail
# Make sure the customer the learner was supposed to create in challenge 1 exists
EXISTS=$(psql -At "$DATABASE_URL" -c \
"SELECT 1 FROM customers WHERE email='alice@example.com'")
if [ -z "$EXISTS" ]; then
psql "$DATABASE_URL" <<'SQL'
INSERT INTO customers (name, email) VALUES ('Alice', 'alice@example.com');
SQL
fiThe check must be cheap. Setup runs every time the challenge opens; an expensive defensive check makes the lab feel slow without the learner understanding why.
Defensive setup is for correcting missing state, not for normalizing it. If the learner did Challenge 2 correctly but slightly differently, the defensive setup should leave that alone, not reset it. Otherwise the learner watches the lab undo their work.
Pair with full-chain solve scripts (Recipe 33). Defensive setup helps when the learner is moving forward; full-chain solve scripts are the safety net when an auto-skip happens.
For tracks where defensive setup needs to query an external SaaS rather than a local database, use the same shape with the SaaS's API. Wrap with retries and a sane timeout — external dependencies can be slow in ways internal ones are not.
For tracks where the defensive setup is large (re-creates dozens of records), consider whether the lesson's challenge boundaries are in the right place. Large defensive setup often signals that the boundary is too far from the natural seam.
Idempotent solve script
A solve script that assumes "Challenge 2 already ran" breaks the moment a learner skipped Challenge 2. The skip generates a hidden lab error: the auto-solve hits a missing prerequisite and exits non-zero, and the lab tells the learner the lesson is broken when in fact the lesson is fine and the test infrastructure is not.
Each solve script re-creates the entire prerequisite chain from the beginning, idempotently, before doing the current challenge's work. If state already exists, skip; if not, create.
#!/bin/bash
set -euxo pipefail
# Rebuild prerequisites from challenge 1
psql "$DATABASE_URL" <<'SQL'
CREATE TABLE IF NOT EXISTS customers (id serial, name text, email text);
INSERT INTO customers (name, email) SELECT 'Alice', 'alice@example.com'
WHERE NOT EXISTS (SELECT 1 FROM customers WHERE email='alice@example.com');
SQL
# Now do challenge 3's actual solve work
psql "$DATABASE_URL" <<'SQL'
CREATE OR REPLACE VIEW recent_orders AS
SELECT * FROM orders WHERE created_at > NOW() - INTERVAL '7 days';
SQLCREATE TABLE IF NOT EXISTS and INSERT ... WHERE NOT EXISTS are the standard idempotent shapes for SQL. Use them everywhere; a solve script that fails because a table is already there is hostile to the test pipeline.
Idempotent solve scripts are large. That is fine. Their job is to be the safety net, not the elegant lesson. Optimize for "this works on every entry point" over "this is short."
When the prerequisite chain is genuinely long, factor the rebuild into a helper script that every solve script calls. One canonical rebuild beats five copies of the same logic.
For tracks with cloud-side state, the rebuild must use the cloud API and tolerate "resource already exists" errors. Wrap each create with || true only when the create's failure mode is genuinely "already there"; do not blanket-suppress errors.
For tracks where the rebuild is expensive (provisioning a full sandbox tenant), the right answer may be to forbid skipping rather than to write a heavy solve script.
Agent variable cross-host hand-off
Cross-host coordination tempts authors to invent fancy mechanisms — shared file mounts, custom REST endpoints, environment-variable gymnastics. For single-string values, the platform already has the right primitive: agent variable set/get. Use it before reaching for anything heavier.
agent variable set KEY VALUE on one host writes the value into the platform's per-sandbox key/value store. agent variable get KEY on any other host (or any other lifecycle script) reads it. The same value is also available in assignment.md via [[ Instruqt-Var key="KEY" hostname="..." ]].
#!/bin/bash
set -euxo pipefail
CLUSTER_TOKEN=$(generate-cluster-token)
agent variable set CLUSTER_TOKEN "$CLUSTER_TOKEN"
# Surface for assignment-body interpolation
agent variable set DASHBOARD_URL "https://leader.${_SANDBOX_ID}.instruqt.io:8443"#!/bin/bash
set -euxo pipefail
CLUSTER_TOKEN=$(agent variable get CLUSTER_TOKEN)
join-cluster --token "$CLUSTER_TOKEN"<!-- 01-first-challenge/assignment.md body -->
Open the dashboard at [[ Instruqt-Var key="DASHBOARD_URL" hostname="leader" ]]._SANDBOX_ID is a shell variable, not an agent variable. To surface it via [[ Instruqt-Var ]], write it explicitly: agent variable set SANDBOX_ID "$_SANDBOX_ID". The most common "Instruqt-Var renders nothing" symptom traces back to this.
The [[ Instruqt-Var ]] form supports interpolation in essentially every string field — cmd:, url:, path:, frontmatter teaser, body markdown (including inside fenced code blocks), notes text, button labels, table cells. No escaping needed inside backticks or code blocks.
For multi-field structured values, agent variable's single-string limit forces ugly serialization. Switch to a JSON broker sidecar (Recipe 40) when the payload has structure.
For credentials that should be surfaced to the body (passwords, tokens) use the same pattern with care: the [[ Instruqt-Var ]] will render the value into the rendered HTML, where the learner's browser sees it. That is fine for per-sandbox credentials; do not use it for long-lived shared secrets.
For values that need to flow from cleanup back into a cleanup-of-another-host, the same agent variable get works during cleanup — useful for retrieving the cloud-side resource ID set during setup so cleanup can delete it.
Cloud SDK auth credential override
gcloud auth activate-service-account switches the active context globally. If setup activates the admin SA, runs an admin operation, and then forgets to switch back, every subsequent gcloud call (including those the learner sees in the terminal) runs as admin. That is wrong on its face and dangerous if the learner is supposed to hit permission errors as part of the lesson.
Use CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE to scope the admin credentials to a single command's environment, rather than activating them globally. The override is per-process; the global gcloud context is never disturbed.
#!/bin/bash
set -euxo pipefail
SA_JSON=$(mktemp)
trap "rm -f $SA_JSON" EXIT
echo "$INSTRUQT_GCP_PROJECT_LAB_PROJECT_ADMIN_SERVICE_ACCOUNT_KEY" | base64 -d > "$SA_JSON"
# Admin operation, scoped via env var
CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE="$SA_JSON" \
gcloud projects add-iam-policy-binding "$PROJECT_ID" \
--member="user:$LEARNER_EMAIL" \
--role="roles/storage.objectViewer"
# Learner-facing operations continue with the default (learner) credentials
gcloud config list accountThe override env var is a per-process scope. Setting it once and running multiple commands inside that exported scope is fine; setting it inline as VAR=val gcloud ... keeps the scope to one invocation. Pick the form that matches what you need.
Trap-clean the temp credentials file. A SA JSON sitting on disk after setup is a smell at minimum and worse on shared images.
The same pattern applies for AWS (AWS_ACCESS_KEY_ID=... inline before a command) and Azure (sourcing a per-context profile). The mechanism varies; the principle — scope admin credentials narrowly — is the same.
For tracks where the admin SA must persist across multiple setup steps but should still be cleared at end of setup, set the override at the script's top, do all admin work, then unset and rm before exit.
For tracks where setup never needs to switch back (the learner does not interact with gcloud directly), global activation is fine. The override pattern is for the case where the contexts must coexist.
Granular check messaging
A check script that combines five assertions into one [ X && Y && Z && ... ] test gives the learner one bit of information when they fail: the whole thing is broken. They have no idea which of the five things is wrong. Frustration follows.
Split into several independent assertions. After each, call fail-message with a specific remediation hint and exit 1. The learner gets actionable feedback at the first thing that breaks.
#!/bin/bash
set -euxo pipefail
# Check 1: user exists
if ! psql -At "$DATABASE_URL" -c \
"SELECT 1 FROM pg_user WHERE usename='reader'" | grep -q '^1$'; then
fail-message "Create the 'reader' user with: CREATE USER reader WITH PASSWORD '...';"
exit 1
fi
# Check 2: user has SELECT on the table
if ! psql -At "$DATABASE_URL" -c \
"SELECT has_table_privilege('reader', 'orders', 'SELECT')" | grep -q '^t$'; then
fail-message "Grant the reader user SELECT on orders: GRANT SELECT ON orders TO reader;"
exit 1
fi
# Check 3: user does NOT have DELETE
if psql -At "$DATABASE_URL" -c \
"SELECT has_table_privilege('reader', 'orders', 'DELETE')" | grep -q '^t$'; then
fail-message "The reader user must not have DELETE — REVOKE it from the role."
exit 1
fi
set-status "All three permission checks passed."fail-message and set-status availability is image-dependent. The platform's published images always have them; raw upstream Docker images may not. Guard with || true if you cannot be sure: fail-message "..." || true.
For VMs using stock cloud images (Ubuntu 22.04, 24.04, Rocky 9), fail-message is available — the platform's agent installs during boot. For very old custom images or for raw container images with stripped PATH, fall back to echo plus exit 1.
When running multiple checks under set -euo pipefail, an unguarded subprocess substitution that fails will silently exit the script. Wrap subprocess assignments with explicit handlers: OUT=$(... ) || { fail-message "..."; exit 1; }.
For long check scripts, count the assertions automatically and prefix each message with [Check N/$TOTAL]. The learner sees their progress as they fix things.
For self-paced workshops where any failed check should not block progress, swap the pattern for "always pass, surface a set-status describing what was checked." Recipe 37 covers the graded-rubric variant.
Scored certification rubric
A certification check that exits 0 or 1 with no detail is hostile to learners who almost passed. They get no signal on what to study. They retry blindly. The right shape is a rubric: each criterion is worth points, the script tallies, and the result is presented with what was earned and what was missed.
In the check script, build an array of criteria with point values and assertion logic. Tally as the script runs. Emit a per-criterion pass/fail line plus a final score line. Color-code the total. Exit 0 only at full marks (or whatever the pass threshold is); otherwise fail-message with the missing items.
#!/bin/bash
set -uo pipefail
TOTAL=100
SCORE=0
assertion() {
local label=$1; local points=$2; local cmd=$3
if eval "$cmd" > /dev/null 2>&1; then
echo "[PASS] $label (+$points)"
SCORE=$((SCORE + points))
else
echo "[FAIL] $label"
fi
}
assertion "Reader user exists" 10 \
"psql -At \"$DATABASE_URL\" -c \"SELECT 1 FROM pg_user WHERE usename='reader'\" | grep -q '^1$'"
assertion "Reader has SELECT on orders" 30 \
"psql -At \"$DATABASE_URL\" -c \"SELECT has_table_privilege('reader','orders','SELECT')\" | grep -q '^t$'"
assertion "Reader lacks DELETE on orders" 30 \
"! psql -At \"$DATABASE_URL\" -c \"SELECT has_table_privilege('reader','orders','DELETE')\" | grep -q '^t$'"
assertion "Audit table exists" 30 \
"psql -At \"$DATABASE_URL\" -c \"SELECT 1 FROM pg_tables WHERE tablename='audit_log'\" | grep -q '^1$'"
echo "----"
echo "Score: $SCORE / $TOTAL"
if [ "$SCORE" -eq "$TOTAL" ]; then
set-status "Score: $SCORE/$TOTAL — all criteria passed."
exit 0
else
fail-message "Score: $SCORE/$TOTAL — review the [FAIL] criteria above and try again."
exit 1
fiThe point values should map to the lesson's relative weight, not be uniform. A criterion that is "the whole point of the challenge" should be worth more than a setup detail.
For partial credit at scale (10+ criteria), wrap the rubric in a single Python heredoc rather than chaining bash assertions. The Python version is easier to read and easier to extend.
Reserve scored rubrics for tracks that are explicitly graded. Putting a rubric on a workshop track frustrates learners who skipped a section deliberately.
For tracks where partial credit is also captured externally (a learning platform's gradebook), POST the score from the check script to the platform's API at the end of the run. Use INSTRUQT_AUTH_TOKEN to authenticate.
For tracks where the rubric should also produce a per-criterion remediation list, append failed criteria to a file the learner can read and reference. The check script becomes both grader and study guide.
Pre-baked k3s VM with control-plane auto-join
Bootstrapping a Kubernetes cluster from scratch in setup — install kubeadm, init the control plane, generate join tokens, distribute, run kubeadm join on each worker — takes minutes the lab does not have. It also adds five places where setup can fail. For a track where the cluster is the foundation, not the lesson, that is wasted complexity.
Use a platform-published k3s VM image. The image ships with a built-in bootstrap that auto-joins the control plane from an env var: set K3S_CONTROL_PLANE_HOSTNAME on each worker VM in config.yml's environment: block, and the worker joins on boot with no setup script.
version: "3"
virtualmachines:
- name: control-plane
image: platform-published-k3s
machine_type: n2-standard-4
shell: /bin/bash
- name: worker-1
image: platform-published-k3s
machine_type: n2-standard-4
shell: /bin/bash
environment:
K3S_CONTROL_PLANE_HOSTNAME: control-plane
- name: worker-2
image: platform-published-k3s
machine_type: n2-standard-4
shell: /bin/bash
environment:
K3S_CONTROL_PLANE_HOSTNAME: control-plane#!/bin/bash
set -euxo pipefail
until [ -f /opt/instruqt/bootstrap/host-bootstrap-completed ]; do sleep 1; done
# Wait for all workers to join
TARGET=2
until [ "$(kubectl get nodes --no-headers | grep -c worker-)" -ge "$TARGET" ]; do
sleep 5
done
kubectl get nodesThe control-plane VM's hostname must match the value of K3S_CONTROL_PLANE_HOSTNAME on every worker. A typo silently produces a single-node cluster and the workers spin trying to join an unreachable name.
Do not call kubeadm on these images. The image's bootstrap is not kubeadm — running kubeadm on top fights the existing daemon. If you need a kubeadm-flavored cluster instead, do not use the pre-baked k3s image.
The control plane is ready before the workers. If the first challenge depends on workers being present, gate on the node count, not on kubectl version or cluster-info.
For tracks that need a single-node cluster (control plane plus workloads on the same node), declare only the control plane and skip the worker VMs. The image runs as a single-node cluster by default.
For tracks needing kind or k3d (Kubernetes-in-Docker), Recipe 19's Docker-on-VM shape is the substrate. Pre-baked k3s images are the right choice when you want a real multi-VM cluster without the kubeadm dance.
Sandbox-public URL exposure
A service tab embeds the target URL in an iframe served from the platform's traffic proxy. That works for most service UIs, but it does not work when the target sets strict frame-blocking headers that cannot be stripped, when the service must be opened in a new browser window for OAuth-style flows, or when the lesson explicitly demonstrates the URL the learner would use in production. The right answer is a sandbox-public URL: the VM's port is reachable on a public DNS name with TLS provisioned out-of-band.
Set provision_ssl_certificate: true and allow_external_ingress: [http, https, high-ports] on the VM. The platform mints a TLS cert and routes <vm-name>.${_SANDBOX_ID}.instruqt.io:<port> to the VM's port. Surface as a type: website tab with the URL templated; the URL is also paste-friendly into the assignment body for "share this link" flows.
version: "3"
virtualmachines:
- name: app-vm
image: ubuntu-2204
machine_type: n2-standard-4
shell: /bin/bash
provision_ssl_certificate: true
allow_external_ingress: [http, https, high-ports]tabs:
- title: App
type: website
url: https://app-vm.${_SANDBOX_ID}.instruqt.io:8443<!-- assignment body -->
Share this URL with a colleague: `https://app-vm.[[ Instruqt-Var key="SANDBOX_ID" hostname="app-vm" ]].instruqt.io:8443`type: website URLs must be HTTPS. instruqt track push validates this and rejects HTTP URLs.
_SANDBOX_ID is a shell variable, not an agent variable. To interpolate it into the assignment body, run agent variable set SANDBOX_ID "$_SANDBOX_ID" in setup and reference as [[ Instruqt-Var key="SANDBOX_ID" ... ]]. URL fields like tabs[].url: accept the ${_SANDBOX_ID} form directly because the platform substitutes shell-style placeholders before the field is consumed.
The cert is provisioned for standard ports (80/443). Custom high ports get the same cert but you must explicitly add high-ports to allow_external_ingress for the route to be open. Forgetting that produces "site unreachable" with no obvious cause.
For tracks that need a sandbox-public URL on a non-standard subdomain (e.g., kibana.${_SANDBOX_ID}... rather than <vm-name>.${_SANDBOX_ID}...), the same pattern applies — the VM name in the subdomain must match the name: field in config.yml.
For OAuth flows that require the redirect URL to be reachable from the learner's browser, the sandbox-public URL is what the OAuth client should be configured with. Set the URL via agent variable set and inject it into the OAuth client config in setup.
JSON broker sidecar for cross-VM payload
agent variable set is one-string-per-key. Stuffing a multi-field payload into one string by serializing JSON works but is ugly to read on the consuming side and impossible to extend. Cross-VM environment: injection is one-way at provisioning time; it cannot carry a value computed on one VM at runtime to another VM. When the payload is structured, the right answer is a tiny HTTP sidecar that returns JSON.
On the leader VM, run a small Python HTTP server that serves a single JSON document on a high port. On consumer VMs, curl the leader's URL and parse with jq. The leader is the source of truth; consumers re-fetch as needed.
#!/bin/bash
set -euxo pipefail
# Provision the external resource
ENDPOINT=$(provision-tenant)
TOKEN=$(provision-token)
# Write the payload
cat > /tmp/broker-payload.json <<EOF
{
"endpoint": "$ENDPOINT",
"token": "$TOKEN",
"username": "lab-user",
"password": "$(openssl rand -hex 16)"
}
EOF
# Serve on port 8081
nohup python3 -m http.server 8081 \
--directory /tmp > /var/log/broker.log 2>&1 &
disown# config.yml — make the leader hostname known to followers
virtualmachines:
- name: leader
image: ubuntu-2204
machine_type: n2-standard-2
shell: /bin/bash
- name: follower-1
image: ubuntu-2204
machine_type: n2-standard-2
shell: /bin/bash
environment:
LEADER_URL: http://leader:8081/broker-payload.json#!/bin/bash
set -euxo pipefail
until curl -fsS "$LEADER_URL" > /tmp/payload.json; do sleep 2; done
ENDPOINT=$(jq -r .endpoint /tmp/payload.json)
TOKEN=$(jq -r .token /tmp/payload.json)
join-cluster --endpoint "$ENDPOINT" --token "$TOKEN"python3 -m http.server is fine for small private payloads inside a sandbox. Do not expose the broker outside the sandbox network, and do not put long-lived secrets in it — the sandbox boundary is the security boundary. agent variable for sensitive single-string values, broker for structured payloads.
Followers must wait for the broker. The leader's broker comes up after some setup work; the follower's setup may run faster. A retry loop on the follower side is the right gate.
When the structure of the payload changes between v1 and v2 of the track, version the URL path or add a version: key to the JSON. Older follower setups break silently if the payload structure changes underneath them.
For tracks where the payload should be cross-cloud (one cloud's leader broadcasting to another cloud's follower), use a small managed bucket (S3, GCS) instead of an HTTP sidecar. The shape is the same; the transport is different.
For sandbox-internal use where security matters, add a token check on the broker and pass the token via agent variable set/get. Keep the broker dumb otherwise.
Sparse-checkout from a private repo with a short-lived token
Cloning a whole repo for one lesson is wasteful and slow. Putting a Personal Access Token in git clone https://<token>@github.com/... lives forever in repo history and works around a security control rather than respecting it. The clean answer is a GitHub App installation token minted at setup time, used once, and discarded.
In setup, mint a short-lived GitHub App installation token using a private key delivered via Instruqt secrets. Use the token in a sparse-checkout against the private repo. Discard the token before exit; do not write it to disk past the clone.
#!/bin/bash
set -euxo pipefail
##### START WORKSHOP_DEPLOYMENT_PARAMETERS #####
# DO NOT MODIFY THIS SECTION — track-test automation scrapes these markers.
agent variable set TRACK_REPO "lesson-content"
agent variable set TRACK_PATH "lessons/foo"
agent variable set TRACK_BRANCH "main"
##### END WORKSHOP_DEPLOYMENT_PARAMETERS #####
get_github_token() {
local app_id="111111"
local install_id="22222222"
local pem="$GH_APP_PRIVATE_KEY" # delivered via secrets:
local now=$(date +%s)
local exp=$((now + 540))
local header=$(printf '{"alg":"RS256","typ":"JWT"}' | base64 -w0 \
| tr -d '=' | tr '/+' '_-')
local payload=$(printf '{"iat":%d,"exp":%d,"iss":"%s"}' "$now" "$exp" "$app_id" \
| base64 -w0 | tr -d '=' | tr '/+' '_-')
local sig=$(echo -n "$header.$payload" \
| openssl dgst -sha256 -sign <(echo "$pem") -binary \
| base64 -w0 | tr -d '=' | tr '/+' '_-')
local jwt="$header.$payload.$sig"
curl -sS -X POST \
-H "Authorization: Bearer $jwt" \
-H "Accept: application/vnd.github.v3+json" \
"https://api.github.com/app/installations/$install_id/access_tokens" \
| jq -r .token
}
TOKEN=$(get_github_token)
mkdir -p /opt/lesson
cd /opt/lesson
git init -q
git remote add origin "https://x-access-token:${TOKEN}@github.com/your-org/${TRACK_REPO}"
git config core.sparsecheckout true
echo "${TRACK_PATH}/" > .git/info/sparse-checkout
git pull --depth=1 origin "$TRACK_BRANCH"
unset TOKENThe bash JWT signer relies only on openssl, base64, tr, jq, and curl — all present on standard workstation images. The signing pattern is reusable for any RS256-JWT API (GCP service accounts, Auth0, custom OIDC); GitHub App is the most common case but not the only one.
The deployment-parameters comment block is not just a comment. Some track-test pipelines scrape these markers to discover the source repo. If you copy this pattern, keep the START/END comments verbatim.
For public repos, skip all of the above. git init; git remote add ... https://github.com/<org>/<repo>; git config core.sparsecheckout true; ... is enough. The JWT machinery is only needed when the repo is private.
For lesson content that lives in a private GitLab rather than GitHub, the same JWT pattern applies with GitLab's token-minting API; the URL and headers differ but the pattern is identical.
For tracks where the same content is fetched by many parallel labs, consider hosting the content as a tarball in a managed bucket and downloading at setup. Faster cold-start and one less moving part.
Trap-based error trace and check cleanup
set -e alone, or a check script provisions per-check state that must be cleaned up regardless of pass/fail.
set -e aborts on any non-zero exit but does not say which line. Diagnosing a 200-line setup that exits non-zero with no other output is a slog. Equally, a check script that creates temporary state to validate something can leak that state on failure paths if cleanup is only in the happy-path tail.
Use trap '... ${BASH_SOURCE[0]}:${LINENO}: ${BASH_COMMAND}' ERR for line-numbered error reporting. Use trap cleanup EXIT to guarantee cleanup runs on every exit path — normal, error, signaled. Pair with set -E so the ERR trap fires for failures inside functions and subshells too.
#!/bin/bash
set -Eeuo pipefail
trap 'echo "ERR at ${BASH_SOURCE[0]}:${LINENO}: ${BASH_COMMAND}" >&2' ERR
# (long setup script)
deploy-stuff
configure-stuff
validate-stuff# Check script that provisions and tears down temp state
#!/bin/bash
set -Eeuo pipefail
cleanup() {
ctm env delete check-env > /dev/null 2>&1 || true
}
trap cleanup EXIT
# Idempotent re-add in case prior run left it behind
ctm env delete check-env > /dev/null 2>&1 || true
ctm env add check-env "$AAPI_ENDPOINT" "$AAPI_TOKEN"
ctm env set check-env
# (validation logic)set -E is the difference between an ERR trap that fires for failures inside functions and one that does not. Without -E, a failure in a called function bypasses the trap and you are back to "set -e ate my error message."
For check scripts, set -o history; export HISTFILE=/root/.bash_history; history -r is needed if the check uses history to validate engagement. Non-interactive bash defaults to history off. Without the explicit enable, history returns nothing regardless of what the learner typed.
PS4='+$0[$LINENO]+' plus set -x makes the platform's debug log readable. Every traced line shows the script name and line number. Without it, a long setup script's xtrace looks like a wall of opaque commands.
For check scripts on raw upstream Docker images that lack fail-message, replace the helper calls with echo plus exit 1. The error-trap pattern is identical; the messaging primitive differs.
For tracks with a private build pipeline that runs all check scripts in CI, the trap-based error trace is what makes those CI failures actionable. Keep it on by default.
Service-tab CSP and header overrides
X-Frame-Options, Content-Security-Policy) and you want to embed it inside a service tab.
Many SaaS UIs ship X-Frame-Options: DENY or a strict CSP. Embedded in an iframe, they refuse to render. Two layered remedies exist: the platform's service tab supports per-tab header overrides, and a local reverse proxy (Caddy, nginx) can strip the headers before the platform sees the response. socat is the wrong tool here — it is a TCP-level proxy and cannot touch HTTP headers.
Try the per-tab override first. It is one block in assignment.md and zero lines of code. If the upstream does not respect the request override (some SaaS UIs ignore client hints), add a Caddy or nginx reverse proxy on the workstation and point the service tab at the proxy port.
# 01-first-challenge/assignment.md frontmatter — per-tab override
tabs:
- title: Admin UI
type: service
hostname: app-vm
port: 8443
custom_request_headers:
x-strip-frame-options: "1"
custom_response_headers:
content-security-policy: "frame-ancestors *"
x-frame-options: ""# track_scripts/setup-workstation — Caddy fallback
cat > /etc/caddy/Caddyfile <<'EOF'
{
auto_https off
}
:8080 {
reverse_proxy https://upstream:8443 {
header_down -X-Frame-Options
header_down -Content-Security-Policy
transport http {
tls
tls_insecure_skip_verify
}
}
}
EOF
systemctl restart caddy# Repointed service tab
tabs:
- title: Admin UI
type: service
hostname: workstation
port: 8080The Caddy auto_https off block is mandatory at the top of the Caddyfile. Without it, Caddy attempts Let's Encrypt provisioning on internal-only hostnames and silently fails to start.
socat does not work for stripping HTTP headers. Several teams hit this blind alley before reaching for Caddy or nginx. socat has a legitimate use for pure TCP forwarding; the anti-pattern applies only to the HTTP-header case.
For nginx, the equivalent directives are proxy_hide_header Content-Security-Policy and proxy_hide_header X-Frame-Options. The CSP rewrite is also one option — set add_header Content-Security-Policy "frame-ancestors *" after stripping.
For self-hosted UIs you control, fix the headers at the source (the application's config) rather than rewriting in transit. The Caddy/nginx pattern is for upstreams you cannot touch.
For tracks where the iframe-embed is OAuth-related (Kibana with SSO), the auth flow may also need a header rewrite — the proxy pattern still works, but the OAuth redirect URLs must point at the proxy hostname, not the upstream.
Step-based body structure
A challenge body that is one wall of prose loses learners. Where does step 2 begin? Did I do step 3? A long block of running text without visual seams is a usability failure even when the prose itself is good.
Break the body into numbered steps with a horizontal-rule separator between each. Render the step header in a colored <h2> so it stands out. Close the challenge with a single completion line marked with a checkmark.
---
[Optional pre-work: prerequisites, downloads, links]
---
<h2 style="color: #37C980;">Step 1: Configure the database connection</h2>
Open the connection settings and set `DATABASE_URL` to the value below.echo "DATABASE_URL=postgres://app:app@db:5432/app" >> /workspace/.env
---
<h2 style="color: #37C980;">Step 2: Run the migration</h2>
Run the migration script and verify the schema is up.cd /workspace && ./scripts/migrate.sh
---
<h2 style="color: #37C980;">Step 3: Verify connectivity</h2>
Connect with `psql` and list the tables.psql "$DATABASE_URL" -c '\dt'
---
✅ The database is configured. Click **Check** to continue.Open the body with a horizontal rule only if there is a pre-work section before Step 1. Otherwise start directly with the first step header. A leading rule with nothing before it looks like a stray separator.
Steps are sequential and do not skip numbers. Skipping (jumping from Step 2 to Step 4 because Step 3 was deleted) confuses learners and signals careless authoring.
The closing line uses ✅. Reserve ❌ for inline "wrong path" markers within step content. Mixing the two erodes the convention.
Setext-style headings (Title\n=== for H1, Title\n--- for H2) are the alternative to ATX-style (## Title) and render identically. Pick one and use it consistently within a track. Be careful: Title\n--- with no blank line above makes a heading; \n---\n with blank lines makes a horizontal rule. Both appear in well-formed bodies; the distinction matters only when editing separators.
For challenges that are all one short instruction, drop the step structure entirely. The "step" pattern earns its place once there are three or more.
Pre-challenge notes overlay
Putting context inside the challenge body forces the learner to read it before they can start. Embedding it as a video that plays inline distracts from the working tabs. A pre-challenge overlay shows the context once, the learner dismisses it, and the working tabs are clean.
Use the notes: array in the challenge frontmatter. Each entry is a slide; types are text, video, or image. The overlay shows before the challenge starts; the learner clicks through to begin.
notes:
- type: text
contents: |
<style>img { display: block; margin: 0 auto; max-width: 80%; }</style>
- type: text
contents: |
## What we will build
In this challenge you will configure a postgres connection,
run a migration, and verify connectivity from the workstation.
- type: video
url: https://www.youtube.com/embed/abc123?start=0&end=45&autoplay=1&modestbranding=1&rel=0enhanced_loading: false must be set at the track level for notes: slides to render as the pre-challenge overlay. Setting it true (or omitting it) changes the loading flow and turns notes into a clickable tab rather than a full-screen overlay.
The first challenge can leave enhanced_loading: null (inherit the track value); subsequent challenges should set enhanced_loading: false explicitly. Without the explicit override, the platform may render notes inconsistently across challenges.
YouTube embeds support start= and end= query params. Use them when only a clip of a longer video is the right context for this challenge.
For locally hosted videos (avoiding external dependency), reference an .mp4 in ../assets/ rather than a YouTube URL. Keep videos short — 30 to 60 seconds — so the pre-challenge gate does not become a wait.
For multi-slide intros, use multiple - type: blocks. Each renders as one slide; the learner clicks through.
Tab buttons for inline navigation
"Now switch to the Terminal tab" is a stale instruction the moment a track grows past three tabs. The learner has to scan the tab bar, identify the right one by label, and click. Worse, the lesson author later reorders the tabs and the prose now points at the wrong place.
Use the platform's tab-button shortcode in the assignment body. The button appears inline, the learner clicks, the right tab opens. Tab references are zero-indexed; (tab-0) is the first tab.
Run the migration:
[button label="Open Terminal" variant="success"](tab-0)
Then verify the schema:
[button label="Open Database UI" variant="success"](tab-1)For navigation between challenges:
[button label="Next Section" variant="outline"](section--02-run-the-migration)Indexing is by tab order in assignment.md's frontmatter, not by label. Reorder the tabs and every tab-N reference shifts. Audit when you reorder.
The variant= parameter handles standard semantics — success (green), warning (amber), danger (red, for destructive-action warnings), outline (outlined, for nav-only buttons). Use these rather than custom hex codes; consistency across tracks matters.
Hash-anchor variants like (#) and (#null) exist for "no-op" buttons in some templates. Use them only when the button is a visual cue, not a navigation target.
For inter-challenge navigation (Next Section, Previous Section), use (section--<challenge-slug>) rather than a tab anchor. The slug is the directory name minus the leading number.
For tracks that use the block parameter, the button renders full-width rather than inline. Use sparingly — full-width buttons compete with the body for attention.
Variable interpolation in body
Hardcoding "the URL is x" works for a static demo. For a real lab where each learner's sandbox has its own URL, account ID, or credentials, the body must show the right values for this learner without the author maintaining ten variants.
Set the value via agent variable set in setup. Reference in the body via [[ Instruqt-Var key="..." hostname="..." ]]. The platform substitutes at render time.
agent variable set DASHBOARD_URL "https://workstation.${_SANDBOX_ID}.instruqt.io:8443"
agent variable set ACCOUNT_ID "$INSTRUQT_AWS_ACCOUNT_LAB_AWS_ACCOUNT_ID"
agent variable set TEMP_PASSWORD "$(pwgen --ambiguous 12 1)"Open the dashboard at [[ Instruqt-Var key="DASHBOARD_URL" hostname="workstation" ]].
Sign in with the account ID `[[ Instruqt-Var key="ACCOUNT_ID" hostname="workstation" ]]`
and the temporary password `[[ Instruqt-Var key="TEMP_PASSWORD" hostname="workstation" ]]`.Interpolation works inside fenced code blocks, table cells, button labels, and most other string fields. No escaping required inside backticks.
_SANDBOX_ID is a shell variable, not an agent variable. Run agent variable set SANDBOX_ID "$_SANDBOX_ID" to surface it in the body.
The hostname: parameter must point at a host where the agent variable is defined. Variables set on the workstation are not visible to lookups against db unless explicitly set on db too.
For credential dashboards rendered as HTML pages (rather than markdown body), see Recipe 55 — the same primitives work behind a templated page.
For tracks with many variables, group them in a single setup step at the bottom of the script. Hunting for "where is FOO set?" across a 200-line setup is unpleasant.
Agent-as-teacher
A track for an AI coding tool, an LLM-driven workflow, or any agent-based product has a different rhythm from a normal track. The learner spends most of their time prompting the agent and reading responses, not running commands. A standard "type these commands; check pass/fail" structure misfires.
Structure challenges around prompts. Each challenge presents a prompt the learner pastes into the agent and a description of the response they should observe. Challenges may not need check scripts at all — the agent's response is the validation. Progression goes from "observe what the agent does" to "review what the agent suggests" to "edit alongside the agent."
<h2 style="color: #37C980;">Step 1: Ask the agent to explore the codebase</h2>
In the agent panel, paste this prompt and press Enter:Explore the codebase in /workspace/myapp and tell me:
Read the agent's response. You should see a summary of the framework, entry point, and dependencies.
---
<h2 style="color: #37C980;">Step 2: Ask the agent to find a bug</h2>There is a bug in the function summarize_orders in src/app.py. Find it and explain what it does wrong.
The agent should identify that empty input lists raise a KeyError when no orders are present.
---
✅ The agent has done the exploration. Move on to **Challenge 2** to start editing.Challenges without check scripts are valid here. The platform's "Check" button can pass automatically (exit 0) for a workshop-format track where the agent's interaction is the work.
Pasted prompts should use copy (copy button, no run button). The learner pastes the prompt into the agent panel, not into a terminal.bash,run` is wrong here; it would offer to execute the prompt in a shell.
Pacing matters. An Agent-as-Teacher track is more relaxed than a "type a command and check" track. Build that pacing into the body — fewer steps per challenge, more whitespace, longer transitions.
For tracks where the agent is paired with light terminal work, intersperse bash,run blocks with copy blocks. Make the boundary visible: "Run this in the terminal" versus "Paste this to the agent."
For tracks with a final certification challenge, reintroduce a real check script at the end. The agent guides discovery; the check verifies the final state.
File-based quiz answers
type: quiz is too narrow for the validation you need — case-insensitive matching, multi-part answers, fuzzy normalization — and you want the answer captured from terminal input.
type: quiz is great for multiple-choice with a fixed answer. It does not handle "the answer is roughly this string with whitespace tolerance" or "any of three equivalent forms." When the lesson needs a freeform answer, the right shape is a type: challenge with a check script that reads a file the learner writes.
Ask the learner to write the answer to a known file (echo "answer" > /tmp/answer). The check script reads, normalizes (strip whitespace, lowercase), and compares to a list of acceptable answers. Fail with a specific message; pass with set-status summarizing what they got.
<!-- assignment.md body -->
Write your answer to `/tmp/quiz-answer`:echo "<your answer>" > /tmp/quiz-answer
# check-workstation
#!/bin/bash
set -uo pipefail
if [ ! -f /tmp/quiz-answer ]; then
fail-message "Write your answer to /tmp/quiz-answer first."
exit 1
fi
ANSWER=$(tr -d '[:space:]' < /tmp/quiz-answer | tr '[:upper:]' '[:lower:]')
case "$ANSWER" in
productcatalog|product-catalog|catalog) ;;
*)
fail-message "Try again. Hint: it's the service that owns product metadata."
exit 1
;;
esac
set-status "Correct — $ANSWER."Normalization is the whole point. Whitespace, case, and trivial punctuation should not break a correct answer. Strip and lowercase before comparing.
A plain case with several alternatives is more readable than nested ifs when there are 3+ acceptable forms. For 10+, switch to a Python heredoc with a list and in.
For multi-part answers (affected service; affected product id; exception text), split on ;, normalize each part, and validate each independently.
For a branded answer-submission CLI (submit 1 "answer" rather than echo > /tmp/...), see Recipe 30 plus a small wrapper script. The CLI is more discoverable than the file-based form.
For tracks where the answer is computed against live data (a query result), have the check script re-run the same query against the live data and compare to the learner's submission. More robust than hardcoding the expected answer.
Failure-message anatomy
"FAIL" is not an error message. "Something is wrong" is not an error message. Generic check failures send learners into a tailspin: they do not know what they got wrong and the obvious troubleshooting paths are all wrong.
Every fail-message should name the missing thing, name the fix, and (where possible) include the literal command or query the learner should run. The fail-message is teaching, not punishment.
#!/bin/bash
set -uo pipefail
if ! psql -At "$DATABASE_URL" -c \
"SELECT 1 FROM pg_user WHERE usename='reader'" | grep -q '^1$'; then
fail-message "The reader user does not exist. Create it with: CREATE USER reader WITH PASSWORD 'lab-pass';"
exit 1
fi
if ! psql -At "$DATABASE_URL" -c \
"SELECT has_table_privilege('reader','orders','SELECT')" | grep -q '^t$'; then
fail-message "The reader user lacks SELECT on orders. Run: GRANT SELECT ON orders TO reader;"
exit 1
fiThe fix should be runnable, not a description of how to derive the fix. "Make sure the user has access" is bad; "Run GRANT SELECT ON orders TO reader;" is good. Specificity beats generality.
Long fail-messages compete with the body for attention. Keep each under two sentences; if the message wants to be longer, that is a sign the lesson should split into smaller checks.
The fail-message is rendered in the platform's UI, not in the terminal. Markdown formatting will not work; plain text only.
For self-paced workshop tracks, the fail-message can be more permissive — "expected to see X but did not, did you skip the previous step?" When the learner is in control of progression, the message is a nudge, not a hard stop.
For graded certification tracks, the fail-message must be more diagnostic — list the criterion that failed, the expected state, and the observed state. The learner is being scored; precision matters more than warmth.
Layered hint
A learner who is stuck has two options: stay stuck, or open a hint that gives away the answer. The first wastes time; the second wastes the lesson. A layered hint gives them a nudge first, then a stronger nudge, then the answer — opt-in at each layer.
Three nested <details> collapsibles: nudge, hint, answer. The learner expands as much as they need. Closing all three is the default state; the lesson is intact for learners who do not need help.
Find the configuration option that controls cluster auto-join.
<details>
<summary>🤔 Nudge</summary>
Look in the file at `/etc/lab/cluster.conf`.
</details>
<details>
<summary>💡 Hint</summary>
The setting starts with `auto_` and takes a boolean.
</details>
<details>
<summary>✅ Answer</summary>
Set `auto_join = true` in `/etc/lab/cluster.conf`.sed -i 's/auto_join = false/auto_join = true/' /etc/lab/cluster.conf
</details>The <details> element is interactive HTML, not markdown. It is supported in assignment bodies. Inline styles (<details style="...">) work for further visual customization but are usually unnecessary.
The order matters: nudge, hint, answer. A learner who opens "answer" first short-circuits the layering. The sequence has to feel like progressive escalation.
The completion-providing <details> should still leave the learner with something to do — even the "answer" reveal can be a copy block they execute, not a pre-applied state.
For tracks where hints should be persistent (a hint tab the learner returns to), see Recipe 31 (companion code-tab pair) — the reference file pattern is the persistent form.
For tracks where the hint is video, embed a short clip inside the deepest <details> rather than text. The video plays only when the learner opens the collapsible.
Mid-track recap challenge
A long track without seams becomes a slog. The learner is moving forward but losing track of why. A short recap challenge in the middle — "here is what we have built so far; here is what comes next" — gives them a moment to consolidate.
Insert a content-only challenge at the midpoint. No setup script changes, no check that requires action. The body summarizes what was learned, points forward, and ends with a check that auto-passes.
<h2 style="color: #37C980;">What we have built</h2>
Over the first five challenges you provisioned a postgres database, seeded it with sample data, wrote a migration, configured a read-only user, and built a small API on top.
<h2 style="color: #37C980;">What comes next</h2>
The next five challenges focus on observability: surfacing query latency, alerting on slow queries, and capturing audit logs.
<h2 style="color: #37C980;">Take a breath</h2>
Click **Check** when you are ready to continue.
✅ Recap complete.# check-workstation
#!/bin/bash
echo "Recap acknowledged."
exit 0The recap challenge has zero setup work. Do not add a setup-{hostname} script that does anything visible — the recap is for reflection, and any visible state change distracts.
Mid-track recaps that try to be useful by reviewing material with prompts and exercises miss the point. A recap is a pause, not a quiz. Reserve actual review for end-of-track quiz challenges.
For tracks under eight challenges, the recap is overhead. Reach for it only when the track length earns it.
For tracks with multiple thematic sections, use multiple recaps. One per section seam works; one per challenge does not.
For tracks where the lesson is genuinely a single arc with no seams, do not add a recap. Inserting one to "be thorough" weakens the existing momentum.
Read-only intro challenge
A first challenge that drops the learner into an action without context loses the "why." A first challenge that opens with two paragraphs of body and then asks for an action splits attention. Neither shape is what an intro should be.
Make the first challenge body-only. No commands, no files to edit, no tabs to interact with. The body summarizes the lab, points at the agenda, and ends with a single Check button that auto-passes.
<h2 style="color: #37C980;">Welcome</h2>
This lab teaches the fundamentals of running postgres in production: provisioning, schema migration, role-based access control, and observability.
<h2 style="color: #37C980;">What you will do</h2>
- Stand up a postgres instance with sample data.
- Migrate the schema with a versioned migration tool.
- Configure read-only and admin roles.
- Surface query latency and slow-query alerts.
<h2 style="color: #37C980;">Time</h2>
About 90 minutes. The lab is paced for following along, not racing.
✅ Click **Check** to continue to Challenge 2.# check-workstation
#!/bin/bash
echo "Intro acknowledged."
exit 0The intro challenge should not include a quiz, a confirmation prompt, or a "type yes to continue" interaction. It is a context-setter; gating progression on a trivial action makes the lab feel padded.
If the intro feels like it needs to do real setup work, that work belongs in track_scripts/setup-{hostname}, not in the intro challenge's setup. Track-level setup runs once before the first challenge; intro challenges do not need a per-challenge setup.
For tracks with a pre-challenge notes overlay (Recipe 45), the overlay is a different layer from the body. Use both: the overlay shows once, the body persists.
For tracks with strict skipping (no skipping enabled), an intro challenge ensures every learner sees the context. For tracks with skipping enabled, the intro is optional and learners may jump past it.
For very short tracks (3–5 challenges total), an intro challenge is overhead. The first real challenge can carry the context inline.
Opt-in details collapsible
Inline reference material slows everyone down. A learner who knows the syntax has to read past it; a learner who does not know it has to stop and read. Tucking the material behind a click resolves the conflict: present by default for learners who need it, invisible for those who do not.
Use a <details> collapsible with a clear summary. The body of the details is the supplementary content. Default state is closed; opening it does not change the lesson state.
Set the `auto_join` configuration in `/etc/lab/cluster.conf`.
<details>
<summary>🤔 Reminder: how to edit a config file</summary>sed -i 's/auto_join = false/auto_join = true/' /etc/lab/cluster.conf
Or open the file in a code tab and change line 14 manually.
</details><summary> content should be scannable. "Click here for more" is content-free; "Reminder: how to edit a config file" tells the learner whether they want to expand it.
Avoid stacking multiple collapsibles in a row. Two next to each other compete for attention; three or more turn the body into a click-everything game. If you have three pieces of supplementary material, the body might be too long.
The collapsible is interactive HTML, not markdown — but standard markdown inside the body works (fenced code blocks, lists, links). Inline styles work but are usually unnecessary.
For collapsibles that contain progressive hints rather than reference material, see Recipe 51 (layered hint) — same primitive, different framing.
For supplementary material that should be present in every challenge of a track, factor it into a notes: slide on the first challenge rather than repeating in every body.
Branded credential dashboard
A track that surfaces "here is your URL, your username, your password, your tenant ID, your account ID" by repeating them in every challenge body is noisy. The learner re-reads the same boilerplate before each step. A single dashboard tab — credentials in one place, branded to match the track's visual style — is the right shape.
Render an HTML page from a template at setup time using gomplate (or a similar Go-template renderer). The template is a small HTML file with placeholders. Setup runs gomplate -f against the template and writes the result to a path served by the workstation's local web server. The dashboard appears as a service tab pointing at the workstation's port 80.
<!-- /opt/dashboard/index.html.tmpl -->
<!doctype html>
<html><head><title>Lab credentials</title>
<style>
body { font-family: system-ui, sans-serif; padding: 2rem; }
table { border-collapse: collapse; }
td { padding: 0.4rem 1rem; border-bottom: 1px solid #eee; }
td.k { font-weight: bold; }
td.v { font-family: monospace; }
</style></head>
<body>
<h1>Lab credentials</h1>
<table>
<tr><td class="k">URL</td><td class="v">{{ .Env.DASHBOARD_URL }}</td></tr>
<tr><td class="k">Username</td><td class="v">{{ .Env.LAB_USER }}</td></tr>
<tr><td class="k">Password</td><td class="v">{{ .Env.LAB_PASS }}</td></tr>
<tr><td class="k">Account ID</td><td class="v">{{ .Env.ACCOUNT_ID }}</td></tr>
</table>
</body></html># track_scripts/setup-workstation (cloud-client image, which serves /var/www/html on port 80)
#!/bin/bash
set -euxo pipefail
export DASHBOARD_URL="https://workstation.${_SANDBOX_ID}.instruqt.io:8443"
export LAB_USER="lab-user"
export LAB_PASS=$(pwgen --ambiguous 12 1)
export ACCOUNT_ID="$INSTRUQT_AWS_ACCOUNT_LAB_AWS_ACCOUNT_ID"
gomplate -f /opt/dashboard/index.html.tmpl -o /var/www/html/index.htmltabs:
- title: Lab info
type: service
hostname: workstation
port: 80
path: /
- title: Terminal
type: terminal
hostname: workstationThe cloud-client image ships nginx serving /var/www/html/. Confirm before assuming — on the lighter shell image, you would need to install nginx in setup, which adds a class of failure the dashboard does not need.
gomplate reads from .Env for environment variables. Other data sources (files, JSON URLs) are also supported but rarely needed for a credential dashboard.
Do not put long-lived secrets on the dashboard. The dashboard is convenient for per-sandbox values that are sandbox-scoped; permanent credentials still belong in the platform's secrets store, not on a learner-visible page.
For tracks where the dashboard should also include a real-time status indicator (cluster ready, services healthy), pair the static dashboard with a small JS poll against /healthz endpoints. The result is a dashboard that updates as the lab progresses.
For tracks with multiple user personas (admin, operator, viewer), render multiple sections on the dashboard — one per persona — so the learner can see all credentials at a glance.
Closing Note
The recipes in this book are answers, not laws. Every track is different; the pattern that fits one will be the wrong shape for another. Use the recipes as starting points, lift the parts you need, and adapt them to the lesson in front of you.
When in doubt:
- Start small. The smallest infrastructure shape that supports the lesson is almost always the right one.
- Make failure modes visible. A check that fails with a specific message teaches; a check that fails generically frustrates.
- Be ruthless about cleanup. Every cloud resource you provision must have a path to deletion when the lab ends.
- Optimize for the learner's first thirty seconds. Boot time, first-render correctness, and the moment they realize the lab is real are the moments that decide whether they keep going.
Build well. Test honestly. Ship deliberately. Maintain with patience.