52 From Kubernetes Recovery to Persistent Data

Today’s route

RES509 — Session 2 (session 5 overall). Finish the Kubernetes core, connect your app to persistent storage, and explain the layers underneath it.

Start with the Lab 42 restart checklist. If you are blocked, use its troubleshooting sheet and show your first missing checkpoint at the opening check-in.

Keep your own fork and existing work. A partner may help diagnose a problem, but demonstrate the result on your own VM. The Lab 42 extension remains part of its later assessment; the database work here can satisfy that lane when its verification is complete.

Our four hours

These are work blocks, not a deadline to complete every task. Join the architecture discussion at your current checkpoint. If the class needs more recovery time, the instructor will demonstrate database persistence; record it as observed, with your own database work still pending. A demonstration is not individual completion evidence.

Session evidence and stopping points

Keep the evidence small. Your target after verifying the Lab 42 core is:

• PostgreSQL has a Bound PVC and a Ready Pod; manifests reference a Secret without committing its values.
• A distinct quote inserted through your app is still retrievable after database Pod replacement using the same PVC.
• A diagram of your actual setup is linked or embedded in labs/52-architecture/architecture-notes.md.
• Three brief notes explain the shared kernel, persistence boundary, and next unresolved task or main limitation.
• Working manifests and selected evidence are committed and pushed in your own fork.

Keep Lab 42 failure/recovery evidence where it is and link to it. You do not need a second readiness failure, a five-row comparison table, or a production design report today. Resource limits, liveness experiments, and production rollouts are later work or optional extensions after today’s checks. Final production architecture design belongs with Lab 62.

If you finish only part of this route, save the last successful result, the blocker, and one next action. Agree the next checkpoint with the instructor before leaving; do not label unfinished work complete.

Verify your handoff

Show the instructor the core handoff listed in Lab 42’s restart checklist: both health paths, three Ready replicas, Pod replacement, and readiness failure/recovery with saved manifests and evidence. Explain one observation. Move forward after that check-in; an extension is not an entry requirement.

Run all commands below inside your course Ubuntu VM, from your own fork. Replace POD_NAME in diagnostic commands with an actual Pod name. Select the lab namespace:

1
kubectl config set-context --current --namespace=quote-lab
2
kubectl get deployments,pods,services

Reopen your port-forward and verify /health as in Lab 42. If a previously working deployment has broken, return to its troubleshooting sheet before adding database configuration.

Connect the application to PostgreSQL

PostgreSQL is required in this lab, even if you chose another extension in Lab 42. Allow 45–75 minutes for this guided setup. If your database extension already works, keep it and demonstrate the same insertion and persistence checks below; do not replace its credentials, initialization, or PVC. Adapt the verification commands to your existing resource names.

Prepare the namespace and files

Run these commands inside the course Ubuntu VM, from your own fork. Keep the app manifests in their existing directory:

1
cd "$(git rev-parse --show-toplevel)"
2
kubectl config set-context --current --namespace=quote-lab
3
mkdir -p labs/52-architecture
4
cp labs/20-containerized-node-app/db/init.sql labs/52-architecture/init.sql
5
cd labs/52-architecture
6
kubectl get storageclass

The manifest below uses k3s’s local-path StorageClass. If it is missing, ask for help before continuing. This is storage on the course VM, not replicated storage or a backup.

Create a ConfigMap containing the existing initialization SQL:

1
kubectl create configmap quote-db-init --from-file=init.sql \
2
  --dry-run=client -o yaml > db-init-configmap.yaml
3
kubectl apply -f db-init-configmap.yaml

The SQL creates the quotes table and example data. Initialization runs only when PostgreSQL starts with an empty data directory; changing this ConfigMap does not migrate an existing database. Do not delete a PVC to apply a schema change.

Create database configuration

For a new database, create a lab-only password and a Secret. Check for an existing Secret first:

1
kubectl get secret quote-db-secret

If it exists, reuse its configuration. If the result is NotFound and you are creating a new database, run:

1
db_password=$(openssl rand -hex 16)
2
kubectl create secret generic quote-db-secret \
3
  --from-literal=POSTGRES_USER=quote \
4
  --from-literal=POSTGRES_PASSWORD="$db_password" \
5
  --from-literal=POSTGRES_DB=quotes \
6
  --from-literal=DATABASE_URL="postgres://quote:$db_password@quote-db:5432/quotes"
7
unset db_password

Keep credentials out of Git and screenshots. Commit manifests that reference the Secret, not an export of its contents. The generated hex password is safe to include in this connection URL. Do not regenerate credentials for an existing PVC: PostgreSQL’s initialization environment variables do not change an existing database user’s password.

Deploy storage, PostgreSQL, and its Service

Create database.yaml:

1
apiVersion: v1
2
kind: PersistentVolumeClaim
3
metadata:
4
  name: quote-db-data
5
spec:
6
  accessModes:
7
    - ReadWriteOnce
8
  storageClassName: local-path
9
  resources:
10
    requests:
11
      storage: 1Gi
12
---
13
apiVersion: apps/v1
14
kind: Deployment
15
metadata:
16
  name: quote-db
17
spec:
18
  replicas: 1
19
  strategy:
20
    type: Recreate
21
  selector:
22
    matchLabels:
23
      app: quote-db
24
  template:
25
    metadata:
26
      labels:
27
        app: quote-db
28
    spec:
29
      containers:
30
        - name: postgres
31
          image: postgres:18
32
          envFrom:
33
            - secretRef:
34
                name: quote-db-secret
35
          ports:
36
            - containerPort: 5432
37
          readinessProbe:
38
            exec:
39
              command:
40
                - sh
41
                - -c
42
                - pg_isready -h 127.0.0.1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"
43
            initialDelaySeconds: 5
44
            periodSeconds: 5
45
          volumeMounts:
46
            - name: data
47
              mountPath: /var/lib/postgresql
48
            - name: init
49
              mountPath: /docker-entrypoint-initdb.d
50
              readOnly: true
51
      volumes:
52
        - name: data
53
          persistentVolumeClaim:
54
            claimName: quote-db-data
55
        - name: init
56
          configMap:
57
            name: quote-db-init
58
---
59
apiVersion: v1
60
kind: Service
61
metadata:
62
  name: quote-db
63
spec:
64
  selector:
65
    app: quote-db
66
  ports:
67
    - port: 5432
68
      targetPort: 5432
69
  type: ClusterIP

The official PostgreSQL image documentation describes its initialization and storage behavior. PostgreSQL 18 uses the parent mount /var/lib/postgresql; do not copy an older image’s mount path without checking its documentation. Recreate avoids overlapping database Pods during a Deployment update. This single-instance teaching deployment has downtime during replacement and is not a production database HA design.

Apply and inspect:

1
kubectl apply -f database.yaml
2
kubectl rollout status deployment/quote-db --timeout=180s
3
kubectl get pvc quote-db-data
4
kubectl get pods -l app=quote-db
5
kubectl logs deployment/quote-db

Expect a Bound PVC and one Ready database Pod. If the rollout fails, inspect Pod events and the PVC before continuing. pg_isready checks whether PostgreSQL accepts connections; it does not prove that the app credentials or table queries work.

Configure the application connection

Edit ../42-kubernetes/deployment.yaml. Add this env entry under the existing quote-app container, alongside image, ports, and readinessProbe:

1
env:
2
  - name: DATABASE_URL
3
    valueFrom:
4
      secretKeyRef:
5
        name: quote-db-secret
6
        key: DATABASE_URL

Merge it into an existing env list if you already have one. Keep three replicas and the /health readiness probe. The app consumes DATABASE_URL; setting only POSTGRES_USER and POSTGRES_PASSWORD in the app container does not configure its connection.

1
kubectl apply -f ../42-kubernetes/deployment.yaml
2
kubectl rollout status deployment/quote-app --timeout=120s

Verify quote operations and persistence

In another VM terminal, start or restart the app port-forward:

1
kubectl port-forward svc/quote-app 8080:80

From your working VM terminal, insert a uniquely named quote through the application and read it back:

1
quote_marker="res509-$(date +%s)"
2
curl --fail --show-error --max-time 10 \
3
  --data-urlencode "author=RES509" \
4
  --data-urlencode "text=$quote_marker" \
5
  http://127.0.0.1:8080/quotes
6
curl --fail --show-error --max-time 10 http://127.0.0.1:8080/ -o /tmp/res509-quotes.html
7
grep -F "$quote_marker" /tmp/res509-quotes.html

Expect the marker in the returned page. If not, check that your RS507 containerized-app SELECT and INSERT handlers are implemented, inspect app logs, and verify the connection and table initialization. A passing /health is insufficient.

Replace the database Pod while keeping its PVC:

1
kubectl get pods -l app=quote-db
2
kubectl rollout restart deployment/quote-db
3
kubectl rollout status deployment/quote-db --timeout=180s
4
kubectl get pods -l app=quote-db
5
curl --fail --show-error --max-time 10 http://127.0.0.1:8080/ -o /tmp/res509-quotes-after.html
6
grep -F "$quote_marker" /tmp/res509-quotes-after.html

Expect a new database Pod and the same quote. Keep the marker value in this terminal until the check is complete. Save the two Pod names and the before/after quote result. Do not delete the PVC. Explain why this proves persistence across Pod replacement but not recovery after VM or disk loss.

Map the current architecture

Create labs/52-architecture/architecture-notes.md in your fork. Draw your current system on paper or digitally; add a photo or image and link or embed it in that file.

Include:

• Browser, laptop-to-VM access, and the temporary port-forward.
• App Service, app Deployment, and its Pods and containers.
• Database Service, PostgreSQL Pod, and PVC when implemented.
• The Ubuntu VM as the Kubernetes node, its guest kernel, the hypervisor, and host hardware.

Distinguish traffic paths from “runs inside” boundaries. A port-forward to the app Service selects one Pod; it does not demonstrate Service load balancing. Add the in-cluster client path you verified in Lab 42 separately. Mark PostgreSQL as planned if you have not deployed it; label any instructor demonstration as observed.

Which kernel do your containers share?

Discuss with a partner: app and database containers on this node share the Ubuntu VM’s kernel. The hypervisor provides the VM boundary; Kubernetes manages workloads inside it. Add one sentence explaining that boundary to your notes.

What survives replacement, and what survives VM loss?

Use your Lab 42 Pod replacement and the database persistence observation. Explain why an app Pod can be replaced, why the quote survives database Pod replacement with its PVC, and why three replicas plus local-path storage on one VM do not provide recovery from VM or disk loss.

Add one sentence about that storage/failure boundary. Keep production backup and multi-node design as future work, rather than claiming your teaching cluster already provides them.

Save concise evidence and your next action

In architecture-notes.md, keep:

• A link to your existing Lab 42 verification evidence.
• The database check results: Bound PVC, old/new database Pod names, and the quote marker retrieved before and after replacement. If incomplete, record the last successful checkpoint and actual blocker instead.
• Your diagram and the two short boundary notes above.
• One next action, or your main remaining limitation if today’s checks pass.

Keep database manifests and the initialization ConfigMap in labs/52-architecture; keep the updated app manifest in labs/42-kubernetes. Commit Secret references, never Secret exports, kubeconfig, or passwords. Secret base64 encoding is not encryption.

Review the selected files and Git diff before committing and pushing. Explain the results orally; you do not need an answer to every discussion prompt.

Check out with the instructor

Show what works on your VM, identify anything still pending, and agree your next checkpoint. If the database worked already, reuse that verification for Lab 42’s database extension; link to the same evidence rather than producing a second report.

Continue when ready

After today’s database and diagram checks, choose one continuation if time permits:

• Verify or complete Lab 42’s resource management lane. Show requests/limits in the saved manifest and live Pod configuration.
• Verify or complete its liveness probe lane. Show restart counts and events for the deliberately bad liveness path, then restore /health and verify recovery. Keep database availability out of this health endpoint.
• If those are already verified, annotate your diagram with one single point of failure and discuss a possible improvement and its tradeoff.

Reuse completed extensions. You do not have to complete every lane. Follow the detailed instructions in Lab 42; no new extension requirement is added here.

Controlled version changes and rollback continue with lesson 60 when released. The larger production architecture challenge belongs to Lab 62. The instructor will confirm the remaining scope after today’s check-ins; deferred activities are not an additional end-of-session checklist.