42 Kubernetes Deployment and Runtime Validation

RES509 — Day 1 (session 4 overall). In this lab, you will deploy your containerized application to Kubernetes and validate that it behaves correctly under real orchestration conditions.

You will move from single-host Docker Compose to cluster-managed execution. PostgreSQL is optional today and required through a guided setup in Lab 52. Lab 42 is assessed in RES509; outstanding Lab 30 work remains part of RES507.

Context and goals

So far, you have:

• Built a Node.js application
• Containerized it with Docker
• Orchestrated services with Docker Compose
• Automated validation using CI

In this lab, you will:

• Deploy your application to Kubernetes
• Expose it via a Service
• Validate runtime behavior
• Simulate failures
• Explore scaling and resilience

For this four-hour session, prioritize the core work: deploy the app, access /health, scale to three replicas, observe Pod replacement, and deliberately break and recover readiness.

Start an extension when the core work is verified. You do not need to finish the extension during this session. The final assessment takes place later; its timing will be announced in class. One verified extension is needed for full marks under the rubric below.

Check your progress

Use these checkpoints as you work. If an expected result is missing, inspect status, events, and logs before continuing. You do not need to write an answer for every step.

Save the failure output before fixing it; the failed Pod and its events may disappear after recovery. Keep only the short evidence listed near the end of this lab.

Restart checklist

Returning for RS509 session 2? Reopen your existing VM, fork, and manifests. Start at the first checkpoint you cannot demonstrate. Keep working files and the existing cluster; you do not need to reinstall k3s or start the lab again.

Before class, have your VM and repository accessible and identify your last successful checkpoint. Bring the exact error if blocked; you are not expected to solve every blocker before class.

Run these checks inside your Ubuntu VM, from your own course repository:

1
git status
2
git branch --show-current
3
kubectl get nodes -o wide
4
kubectl -n quote-lab get deployments,pods,services

Expect your existing work branch, a Ready node, and the resources you previously created. NotFound for quote-lab means you have not reached the namespace checkpoint yet. If the API is unreachable, use the troubleshooting sheet below before running more Kubernetes commands. Preserve uncommitted work before switching branches.

• Locate your completed application in labs/20-containerized-node-app and any manifests in labs/42-kubernetes.
• Identify your first missing result in the progress table above.
• Show your instructor that checkpoint, its expected result, and the actual symptom.
• Resume from the corresponding lab section; save failure events before fixing the problem.

Ready to move forward?

Ask for a short instructor check-in when you can show:

• Your Deployment and Service, plus successful /health requests through port-forwarding and the in-cluster Service.
• Three Ready app replicas matching the saved manifest, and evidence of a deleted Pod being replaced.
• The deliberate readiness failure, its HTTP 404 event, and a recovered rollout with /health restored.
• Your working manifests and concise evidence saved in labs/42-kubernetes.

Explain one observation in your own words. These checks establish the core handoff to Lab 52. An extension is not required to move forward; one verified extension remains part of the later Lab 42 assessment. Once the instructor releases Lab 52, begin its guided PostgreSQL section. If you already verified the database extension, reuse it.

Troubleshooting sheet

Choose the symptom you actually see. Run the checks below inside the VM. Replace POD_NAME with a real Pod name. After the API works, use the explicit quote-lab namespace so an old terminal context does not hide your resources.

Cluster access or missing resources

1
sudo systemctl status k3s --no-pager
2
sudo journalctl -u k3s -n 50 --no-pager
3
kubectl config current-context
4
kubectl config view --minify -o jsonpath='{..namespace}'
5
kubectl get namespaces

Expected: k3s is active, your kubeconfig reaches the course cluster, and quote-lab exists once you have created it. A blank namespace means the default namespace, not missing workloads.

For permission errors, check the home-directory kubeconfig instructions in “Verify your environment”. For NotFound, confirm the namespace and whether you created that resource. If the service is failed or the node stays unready, show the service logs to the instructor. Reinstalling k3s is not the recovery step.

Image cannot be pulled

1
kubectl -n quote-lab get pods
2
kubectl -n quote-lab describe pod POD_NAME
3
sudo k3s ctr images list | grep 'quote-app'

Expected: the Deployment’s image tag exists in k3s’s image store. The baseline tag is docker.io/library/quote-app:local; keep your newer tag if you deliberately built and imported one.

For ErrImagePull or ImagePullBackOff, compare the exact image name in events with the imported tag. Return to “Build and import your image for Kubernetes” on this same VM. After correcting the import, delete only the failing app Pod and verify its replacement. If app code changed, build/import a new tag and update the saved manifest. A Docker image listing alone does not verify the k3s import.

Pod is Pending, restarting, or not Ready

1
kubectl -n quote-lab get pods -l app=quote-app -o wide
2
kubectl -n quote-lab describe pod POD_NAME
3
kubectl -n quote-lab logs POD_NAME
4
kubectl -n quote-lab get events --sort-by=.metadata.creationTimestamp

Expected: each settled app Pod shows 1/1 Ready. Events identify scheduling, configuration, or probe problems; logs show application errors. For a restarting container, also inspect its previous instance:

1
kubectl -n quote-lab logs POD_NAME --previous

Use --previous only when a previous container instance exists. For Pending, inspect scheduling events and VM capacity. For app crashes, check the startup error and configuration. A Running Pod with 0/1 and readiness HTTP 404 is the intended bad-probe symptom: save its event, restore /health in the manifest, apply, and wait for recovery. Keep the readiness probe.

Service or health request fails

1
kubectl -n quote-lab get svc quote-app -o yaml
2
kubectl -n quote-lab get pods -l app=quote-app --show-labels
3
kubectl -n quote-lab get endpointslices -l kubernetes.io/service-name=quote-app -o yaml
4
kubectl -n quote-lab get pod service-check

Expected: the Service selector matches app: quote-app, Service port 80 targets application port 3000, and settled healthy Pods have ready endpoints. Reuse the Ready service-check client from “Verify the in-cluster Service path”, or create it there if missing. Then check:

1
kubectl -n quote-lab exec service-check -- wget -qO- http://quote-app/health

Expect JSON containing "ok": true. If the client image cannot download, inspect its events first. If the app Pod is Ready but the Service has no ready endpoints, compare selectors and labels. Port-forward success alone does not verify this Service path.

VM access works but laptop access fails

Restart this command in a VM terminal and leave it running:

1
kubectl -n quote-lab port-forward svc/quote-app 8080:80

In a second VM terminal, check:

1
curl --fail --show-error --max-time 5 http://127.0.0.1:8080/health

Expected: "ok": true inside the VM. Your laptop’s localhost is a different machine. Use the SSH tunnel or VS Code Ports instructions in “Access the health endpoint” for laptop access. If the selected Pod was replaced, restart the VM port-forward. If port 8080 is already in use, check for an existing forward before starting another.

Health works but the quote page fails

Expected for the Lab 42 baseline: /health can work before PostgreSQL is configured. Quote retrieval and insertion require the app’s database handlers, DATABASE_URL, and an initialized database. Finish the Lab 42 core, then use Lab 52’s guided database setup when released. A successful health check does not prove database functionality.

If you spend 10 minutes without a new diagnostic finding, request a check-in. Show the command, expected result, actual result, and the latest relevant event or log excerpt. Keep credentials out of shared output.

Verify your environment

Your goal is to end up with a working Kubernetes cluster and kubectl access. Whenever a command contains POD_NAME, replace it with an actual name from kubectl get pods; do not paste the placeholder literally.

Use k3s on your course Ubuntu Server VM, reusing the VM and your own GitHub fork from RES507. Run all shell commands in this lab inside that VM, including Docker builds and image imports.

Use the branch containing your completed Session 2 application. Do not replace it with a fresh copy of the scaffold. If it is missing, fetch your fork, find that branch with git branch -a, and ask for help before switching with uncommitted changes. Verify /health in your existing app before troubleshooting Kubernetes.

Check capacity before installing:

1
nproc
2
free -h
3
df -h / /tmp
4
cat /etc/os-release

K3s documents a server minimum of two CPU cores and 2 GB RAM, before your workloads. For this lab, 4 GB RAM and 10 GB free disk are planning recommendations for the builds, images, and database work, not a measured guarantee. Use the course Ubuntu Server VM; ask for help if capacity is low. See the k3s requirements.

If your node is not Ready after 20 minutes of setup work, request an instructor check-in. If it is already Ready, continue; you do not need to wait for the rest of the class.

Set up k3s and kubectl access

1
command -v k3s

1
sudo apt update
2
curl -sfL https://get.k3s.io -o /tmp/install-k3s.sh
3
sudo env INSTALL_K3S_CHANNEL=stable sh /tmp/install-k3s.sh

1
sudo systemctl status k3s --no-pager

1
sudo journalctl -u k3s -n 50 --no-pager

1
mkdir -p "$HOME/.kube"
2
sudo install -m 600 -o "$(id -u)" -g "$(id -g)" \
3
  /etc/rancher/k3s/k3s.yaml "$HOME/.kube/config"
4
echo 'export KUBECONFIG="$HOME/.kube/config"' >> ~/.bashrc
5
export KUBECONFIG="$HOME/.kube/config"

1
kubectl wait --for=condition=Ready node --all --timeout=120s
2
kubectl get nodes -o wide
3
kubectl get pods -A

Tools checklist

1
kubectl version --client
2
docker version

Create a dedicated namespace

Create a namespace for this lab:

1
kubectl create namespace quote-lab --dry-run=client -o yaml | kubectl apply -f -

A namespace groups this lab’s resources. Set it as the default in your current kubeconfig context (this persists across terminals and future sessions):

1
kubectl config set-context --current --namespace=quote-lab

Verify:

1
kubectl config view --minify -o jsonpath='{..namespace}'
2
kubectl get pods

Expect quote-lab; an empty Pod list is normal before deploying. Use kubectl -n quote-lab ... if another exercise changes your default namespace.

Build and import your image for Kubernetes

In your Ubuntu VM terminal, return to the root of your own lab repository, then enter the existing application directory:

1
cd "$(git rev-parse --show-toplevel)"
2
git status
3
git branch --show-current
4
cd labs/20-containerized-node-app
5
ls app/package.json app/package-lock.json docker/Dockerfile

Build the image using the lab directory as its context:

1
docker build -t quote-app:local -f docker/Dockerfile .

Import the image before deploying it. Default k3s uses containerd, whose image store is separate from Docker’s. A successful Docker build alone does not make the image available to Kubernetes.

1
docker save quote-app:local -o /tmp/quote-app.tar
2
sudo k3s ctr images import /tmp/quote-app.tar
3
sudo k3s ctr images list | grep 'quote-app:local'

Continue only when the image list contains docker.io/library/quote-app:local. All three operations must happen on the same Ubuntu VM that runs your single-node cluster.

If you change application code later, build and import a new tag, such as quote-app:v2, then update the Deployment’s image to that tag. Applying unchanged YAML does not deploy a rebuilt image automatically.

1
kubectl delete pod POD_NAME
2
kubectl get pods -w

1
kubectl create deployment nginx-test --image=nginx
2
kubectl rollout status deployment/nginx-test --timeout=120s

1
kubectl delete deployment nginx-test

Prepare your manifest directory

Keep your Kubernetes manifests separate from the existing application files. In the same VM terminal:

1
cd "$(git rev-parse --show-toplevel)"
2
mkdir -p labs/42-kubernetes
3
cd labs/42-kubernetes

Create deployment.yaml and service.yaml here, and run the following manifest commands from this directory. Commit these files in your own fork.

Create a Deployment

The baseline deploys the application without PostgreSQL. Its /health route returns {"ok": true} without querying the database. Your completed quote page and quote insertion may fail until you add database integration; that does not invalidate this health-only baseline.

Create a file named deployment.yaml with the following content:

1
apiVersion: apps/v1
2
kind: Deployment
3
metadata:
4
  name: quote-app
5
spec:
6
  replicas: 1
7
  selector:
8
    matchLabels:
9
      app: quote-app
10
  template:
11
    metadata:
12
      labels:
13
        app: quote-app
14
    spec:
15
      containers:
16
        - name: quote-app
17
          image: quote-app:local
18
          imagePullPolicy: IfNotPresent
19
          ports:
20
            - containerPort: 3000
21
          readinessProbe:
22
            httpGet:
23
              path: /health
24
              port: 3000
25
            initialDelaySeconds: 3
26
            periodSeconds: 5

Before applying, identify replicas, the Pod template’s app: quote-app label, and the Deployment selector matching that label. A Deployment manages ReplicaSets, which maintain the requested Pods.

Readiness determines eligibility for normal Service traffic. Liveness can trigger container restart. This baseline uses readiness only. Neither probe validates database behavior unless its endpoint actually checks the database.

Apply it:

1
kubectl apply -f deployment.yaml
2
kubectl rollout status deployment/quote-app --timeout=120s

Observe:

1
kubectl get pods
2
kubectl describe pod POD_NAME

Continue when the rollout completes and your application Pod shows 1/1 in the READY column. Running alone is not enough.

Expose the application with a Service

Create service.yaml:

1
apiVersion: v1
2
kind: Service
3
metadata:
4
  name: quote-app
5
spec:
6
  selector:
7
    app: quote-app
8
  ports:
9
    - port: 80
10
      targetPort: 3000
11
  type: ClusterIP

Identify the Service selector matching the Pod label. port: 80 is the Service port; targetPort: 3000 is the application’s listening port. A containerPort declaration alone does not make the application listen there.

Apply it:

1
kubectl apply -f service.yaml

Verify:

1
kubectl get svc

Access the health endpoint

In a VM terminal, start port-forwarding and leave it running:

1
kubectl port-forward svc/quote-app 8080:80

Wait for the Forwarding from 127.0.0.1:8080 message. In a second terminal inside the same VM, check the endpoint:

1
curl --fail --show-error --max-time 5 http://127.0.0.1:8080/health

Expect JSON containing "ok": true. You are checking application health; quote retrieval and insertion are part of the database extension.

ClusterIP is internal to the cluster. This port-forward listens on the VM’s loopback interface, not your laptop’s. To use your laptop browser, keep the VM port-forward running and open another terminal on your laptop, using Bash or zsh (on Windows, use WSL Bash):

1
printf "VM SSH username: "
2
read -r vm_user
3
printf "VM IP address: "
4
read -r vm_host
5
ssh -N -L 8080:127.0.0.1:8080 "$vm_user@$vm_host"

Use the same SSH destination and connection options you normally use to reach the course VM. Leave this tunnel running, then open the health endpoint in your laptop browser. If using VS Code Remote SSH, you can instead forward the VM’s port 8080 through its Ports panel.

A port-forward to a Service selects one Pod and forwards directly to it. It does not demonstrate Service load balancing or readiness-based traffic selection. If that Pod disappears, restart the VM port-forward; other Ready replicas do not keep that particular forwarding session alive.

Verify the in-cluster Service path

From your VM, create a temporary client Pod in quote-lab:

1
kubectl run service-check --image=busybox:1.37 --restart=Never --command -- sleep 3600
2
kubectl wait --for=condition=Ready pod/service-check --timeout=120s
3
kubectl exec service-check -- wget -qO- http://quote-app/health

Expect {"ok":true}. This request uses cluster DNS and the Service path, unlike port-forwarding. If it fails, inspect the client Pod’s events, the Service selector, and its EndpointSlices. An image download error is a client setup problem, not proof that the Service is broken. If service-check already exists from an earlier attempt, reuse it if Ready or delete that diagnostic Pod and recreate it.

Keep the client for the readiness exercise. Save this result beside the port-forward result.

Scale the Deployment

In deployment.yaml, change replicas: 1 to replicas: 3. Apply the file so the saved desired state matches the cluster:

1
kubectl apply -f deployment.yaml
2
kubectl rollout status deployment/quote-app --timeout=120s
3
kubectl get pods -l app=quote-app -o wide

Confirm that three Pods have READY equal to 1/1.

Discuss:

• What does scaling change?
• Does scaling the application also create a database or validate quote operations?

Keep replicas: 3 in the file for the following exercises. Reapplying a file with replicas: 1 would undo the scaling change.

Observe Pod replacement

Predict: which object will create the replacement, and will its name be the same? Three Pods on this single-node cluster do not protect against losing the VM.

In one VM terminal, watch the Pods:

1
kubectl get pods -l app=quote-app -w

In another VM terminal, list the Pods and delete one. Replace POD_NAME below with a name from the list:

1
kubectl get pods -l app=quote-app
2
kubectl delete pod POD_NAME

Observe a replacement Pod appearing with a new name and becoming Ready. The Deployment’s ReplicaSet reconciles the replica count back to three. Press Ctrl+C to stop watching once the replacement is Ready.

The remaining Ready Pods can remain eligible for Service traffic while the replacement starts. This exercise demonstrates reconciliation, not a guarantee of zero downtime. If the deleted Pod was selected by your port-forward, restart that command and repeat the health request.

Introduce a controlled readiness failure

This step is required for grading. Predict first: will the container restart, and can old healthy replicas still serve requests?

In deployment.yaml, change only the readiness probe’s path from /health to /does-not-exist. Keep the image, application port, and three replicas unchanged. Apply the change:

1
kubectl apply -f deployment.yaml
2
kubectl rollout status deployment/quote-app --timeout=60s
3
kubectl get pods -l app=quote-app
4
kubectl get rs -l app=quote-app

The rollout command is expected to time out because the new Pods cannot become Ready. The timeout stops your wait; Kubernetes continues trying to reconcile the Deployment.

Find a new Pod showing 0/1 in the READY column. Replace POD_NAME with that Pod’s name and inspect it:

1
kubectl describe pod POD_NAME
2
kubectl logs POD_NAME
3
kubectl get events --sort-by=.metadata.creationTimestamp

Look for an Unhealthy event reporting a readiness probe failure with HTTP status 404. The container can be Running while its Pod is not Ready. A failing readiness probe does not itself restart the container.

Inspect the Service’s endpoint readiness:

1
kubectl get endpointslices -l kubernetes.io/service-name=quote-app -o yaml

In each endpoint, compare targetRef.name with your Pod list and find conditions.ready. Expect false for the new failing Pod and true for healthy Pods.

The new unready Pods are not eligible for normal Service traffic. With the default rolling-update strategy, old healthy replicas can remain available while the rollout is stalled. The Service may therefore still work; a total outage is not the expected proof of this failure. Do not use a successful port-forward request as proof that the new Pods are Ready.

An invalid image tag would produce a different failure (ErrImagePull or ImagePullBackOff), rather than a readiness probe failure. Use the bad probe path for this exercise so you can distinguish the two.

Restore the probe path to /health in the file, then verify recovery:

1
kubectl apply -f deployment.yaml
2
kubectl rollout status deployment/quote-app --timeout=120s
3
kubectl get pods -l app=quote-app

Confirm the rollout completes and three Pods are Ready once the rollout settles. Restart port-forwarding if needed, and repeat the /health request.

Compare your diagnostic signals

Compare your observations verbally: status showed 0/1, events identified the HTTP 404, and logs showed the application’s activity. Which source explained the cause most directly?

Repeat the in-cluster request after recovery, then remove the diagnostic client:

1
kubectl exec service-check -- wget -qO- http://quote-app/health
2
kubectl delete pod service-check

Extension lanes

If unsure, start with Lane B. Time estimates below are planning guides after the core, not deadlines. Lanes A/B overlap with Lab 52; carry that work forward rather than repeating it.

Choose one lane for the extension part of the final assessment. Complete working changes, show the verification result, and explain what it demonstrates. Adding YAML without checking its effect or writing an explanation alone is insufficient.

Begin as soon as the core work is verified, including while waiting for an instructor review. Continue unfinished extension work before the later assessment. Extra lanes are optional; focus on one well-understood result rather than the number completed.

Reflect on orchestration

Answer the following verbally during review:

• What does Kubernetes automate?
• What remains your responsibility?
• What signals determine application health?
• How does this differ from Docker Compose?

Prepare your review evidence

Keep these items in labs/42-kubernetes in your own fork:

• Your final working deployment.yaml and service.yaml, plus any extension manifests or code.
• A short verification.md with selected terminal output or linked screenshots showing: the forwarded and in-cluster health requests, three Ready replicas, the deleted and replacement Pod names, the deliberate readiness failure, and the recovered rollout. Include the probe failure event; a screenshot of 0/1 alone does not explain the cause.
• Three brief bullets in that same file: symptom, cause, and fix. Name the probe path you changed and explain why restoring it recovered readiness.
• For an extension, include the change and its verification result. Identify what the result demonstrates.

Use short excerpts with labels; do not paste whole logs or write a report for each step. The conceptual questions above are for the oral review.

Commit and push your final working files and evidence. Before review, open your fork and have the VM ready to show the current Deployment, Pods, and Service. Explain what your evidence proves and what it leaves untested.

Grading rubric

The final assessment uses four broad categories. In-session check-ins provide feedback; you are not expected to complete the final assessment in this first session.

Points reflect demonstrated behavior and reasoning, not the number of completed steps. Partial work can earn partial credit when you can show and explain it. A working baseline alone does not earn extension or oral-understanding points.

Wrap-up

You have:

• Deployed an application to Kubernetes
• Exposed it with a Service
• Validated runtime behavior
• Simulated failures
• Explored scaling and resilience

This establishes the orchestration baseline for RES509. Next, Lab 52 adds a required, guided PostgreSQL deployment and connects this system to VMs, isolation, and production architecture. Keep your cluster, manifests, and evidence for that work.

In larger systems, this foundation expands to:

• Ingress controllers
• Service meshes
• Observability stacks
• GitOps workflows
• Infrastructure automation

You now understand the core control loop that powers modern cloud-native systems.