Troubleshooting Playbook (BTS, Airflow/K8s, DB)

# Troubleshooting Playbook (BTS, Airflow/K8s, DB)

AI Summary

Purpose:

• Generalized, reusable troubleshooting lessons distilled from real Labradorlabs incidents, grouped by domain (BTS/data-transfer, Airflow/Kubernetes, DB/host). Each entry: symptom → root cause → fix → prevention.

Key points:

• BTS/transfer: missing read timeouts cause infinite hangs and CLOSE_WAIT pile-up; large-file transfers fail when an idle proxy timeout fires before the first bytes; DNS round-robin + missing certbot config silently skips cert renewal.
• Airflow/K8s: cluster re-creation invalidates kubeconfig certs; PVC finalizers and selector mismatches block binding; container-runtime data-root misconfig fills the root partition; runtime UID changes break NFS file ownership; path migrations require pod restarts; du double-counts bind mounts.
• DB/host: Docker bridge network throttles large MySQL import; redo-log too small causes checkpoint stalls; gunicorn memory leak + SSH login storm + NFS drop cascade into host hang.

Relevant when:

• Diagnosing a hang, CrashLoopBackOff, disk-pressure, cert-expiry, slow import, or transfer failure in this environment.

Do not read full document unless:

• You need the specific commands, config keys, or the exact failure chain for one incident.

Linked documents:

• [[mysql-binlog.md]]
• [[../people/career-timeline.md]]

Open Questions

• Specific IPs, hostnames, and account names from the original incident reports are intentionally omitted as PII/infra-sensitive. Use placeholders.

Details

Source: author's troubleshooting notes (Obsidian vault, 700. Trouble Shooting). PII (IPs, hostnames, usernames) removed; lessons generalized.

Group A — BTS / data transfer

#### A1. Infinite download hang + CLOSE_WAIT pile-up

• Symptom: binlog download intermittently stalls; CLOSE_WAIT connections accumulate on the download-proxy; container restart temporarily fixes it but it recurs.
• Root cause: the HTTP read had no timeout (urlopen(url) with a blocking read()), so any unreceived bytes (e.g. from 0.1–0.5% packet loss on a long path) block forever; no retry logic. Application bug amplified by intermittent network loss.
• Fix: add a socket/read timeout (e.g. timeout=300); add bounded retries (e.g. 3 attempts with backoff); chunked reads with progress logging; longer-term, migrate from urllib.request to httpx for better connection management.
• Prevention: never do unbounded blocking I/O on network reads; always set timeouts + retries; on the OS side, tune TCP keepalive/tcp_fin_timeout so dead connections and CLOSE_WAIT are reclaimed faster. Diagnose loss with mtr -r -c 100 <host> or ping -c 1000.

#### A2. Large file fails while small files succeed (proxy idle timeout)

• Symptom: large binlog (~317MB) yields HTTP 500 or a 0-byte file; files under ~220MB are fine; the same URL downloads fine via curl.
• Root cause: a too-short idle timeout (10s). The Updater requests through a proxy that must first fetch the file from the origin; for a large file the origin takes >10s to start delivering, the Updater sees no response and aborts, the proxy detects the abort and stops mid-download, leaving a created-but-empty (0-byte) file.
• Fix: increase the timeout (e.g. timeout=300) and stream to disk in a read/write loop so bytes flow periodically and the proxy never sees the connection as idle.
• Prevention: size timeouts to the largest expected payload, not the typical one; prefer streaming writes over buffering whole responses; that curl succeeds while the app fails is a strong signal the bug is in the app's I/O settings, not the network.

#### A3. SSL certificate silently not renewed (DNS round-robin)

• Symptom: intermittent NET::ERR_CERT_DATE_INVALID. Two servers behind DNS round-robin; one healthy, one expired with no renewal log for months.
• Root cause (primary): the expired server's certbot renewal config (/etc/letsencrypt/renewal/*.conf) was missing/corrupt, so certbot dropped that domain from its renewal list and never even tried — "0 attempts", not "1000 failed attempts". Root cause (structural): HTTP-01 challenge fails under DNS round-robin because the challenge request can land on the other node that lacks the token file (404).
• Fix: switch from per-node "each renews itself" to centralized issuance + sync — one master renews via certbot, then rsynces /etc/letsencrypt to the other node and reloads Nginx on both, via a daily cron.
• Prevention: in load-balanced setups, designate a single cert master and replicate; monitor renewal logs (absence of logs is itself a red flag); HTTP-01 + round-robin is fragile — prefer DNS-01 or centralized issuance.

Group B — Airflow / Kubernetes

#### B1. Cluster re-creation breaks everything downstream

• Symptom: after kubeadm reset + cluster re-create (done because k8s certs expired), Airflow webserver unreachable, DAG pods Pending, webserver/etcd restart loops, log permission errors.
• Root cause (bundle): (1) old ~/.kube/config certs invalid after re-create (x509: certificate signed by unknown authority); (2) PVCs stuck Terminating due to kubernetes.io/pvc-protection finalizer; (3) PVC selector / PV label mismatch blocked binding; (4) etcd restart loop + disk I/O bottleneck; (5) Helm chart default runAsUser: 50000 mismatched existing nobody-owned files; (6) DAG pods required a node label that no longer existed.
• Fix: copy fresh /etc/kubernetes/admin.conf to every user's ~/.kube/config; remove stuck PVC finalizers (kubectl patch pvc ... -p '{"metadata":{"finalizers":[]}}') and clear PV claimRef; drop the PVC selector and re-apply; restart kubelet to settle etcd; set securityContext UID to match existing file ownership; re-add the required node label.
• Prevention: keep a cluster-recreate checklist — reset, clean /etc/cni/net.d + iptables, recreate, renew kubectl certs for all accounts (mandatory), recreate PV/PVC, reinstall Airflow Helm. Use PV Retain to preserve NFS data across resets.

#### B2. Node DiskPressure → Airflow DB connection refused

• Symptom: node root / at >82%, DiskPressure; airflow-scheduler/webserver stuck Init:CrashLoopBackOff; wait-for-airflow-migrations logs Connection refused.
• Root cause: storage was physically moved to /data but the container-runtime configs were never updated, so data kept piling onto root: containerd config.toml still root = "/var/lib/containerd" and Docker daemon.json had no data-root. Disk pressure took down the Docker service, killing the PostgreSQL container Airflow depended on.
• Fix: stop runtime, rsync data to /data, update config.toml root and daemon.json data-root, restart (stop docker.socket too, since it can reactivate the service). Verify with docker info | grep "Docker Root Dir".
• Prevention: when relocating storage, update the runtime config in the same change — moving files alone does nothing if the daemon still points at the old path. Delete *.bak copies only after a stable period.

#### B3. After path migration: logs unreachable, scheduler crash, web unreachable

• Symptom: Airflow UI log error No host supplied; scheduler CrashLoopBackOff from repeated git-sync-init failures; webserver unreachable on NodePort.
• Root cause: (1) Kubelet data-dir path change left running worker pods with stale host info → log fetch fails; (2) the git-sync SSH key belonged to a user whose repo access was revoked (team move) — auth succeeds but git fetch is denied; (3) Service externalTrafficPolicy: Cluster plus kube-proxy restart caused endpoint routing mismatch.
• Fix: restart worker/webserver/scheduler pods to refresh host info; re-issue the SSH key for an account that still has repo access and recreate the *-git-sync-ssh-key secret (then restart scheduler so the new secret loads); set externalTrafficPolicy: Local for stable NodePort access.
• Prevention: any Kubelet/path migration requires restarting affected pods; treat SSH auth success but fetch failure as a permissions (not key) problem; diagnose git-sync via the git-sync-init container logs — they are the authoritative signal. Never rotate airflow-fernet-key (loses all encrypted data).

#### B4. Disk-usage illusion from bind mounts (du vs df)

• Symptom: du on /var/lib/kubelet (~400GB) plus /home (~371GB) sums to more than the physical disk, yet the system runs fine.
• Root cause: Kubelet bind-mounts /home into /var/lib/kubelet/...; du walks the file tree and double-counts the same physical data via both paths. df reports physical blocks and shows the true ~371GB.
• Fix/diagnosis: use du -x (don't cross filesystem/mount boundaries) to find the real consumer; check for ghost files with lsof / | grep '(deleted)'; trust df for physical usage.
• Prevention: when auditing disk on a k8s host, prefer du -x and df; understand bind-mount double-counting before "freeing" space that isn't actually used twice.

Group C — DB / host

#### C1. Slow large MySQL import (network + checkpoint stalls)

• Symptom: import crawls; docker stats shows sender sent ~300MB but MySQL received only ~10KB; NVMe write speed spikes then drops to ~0 repeatedly.
• Root cause: Docker bridge-network overhead throttles throughput; redo log too small → checkpoint storm causes periodic stalls. (Low memory use vs config is normal lazy allocation, not a problem.)
• Fix: set network_mode: host on the containers (and change the DB host to 127.0.0.1, since host mode loses Docker DNS); set NVMe I/O scheduler to none; in my.cnf for bulk import set innodb_flush_log_at_trx_commit=0, sync_binlog=0 (temporary speed boosters), and permanently innodb_flush_method=O_DIRECT, high innodb_io_capacity/_max, larger innodb_redo_log_capacity (e.g. 4G), max_allowed_packet=1G.
• Prevention: for bulk loads, enlarge redo capacity to avoid checkpoint stalls and bypass the bridge network; after the load, restore innodb_flush_log_at_trx_commit=1 and sync_binlog=1 for durability before going live. Verify by watching that NET I/O on both sides rises together and Block I/O stays high without stalling.

#### C2. Host hang from compounded resource exhaustion

• Symptom: server console/SSH unreachable, screen frozen (system hang); recovered only by hard reset, then resource instability recurred.
• Root cause: three factors compounded — (1) a web app (gunicorn) leaking memory and being OOM-killed in a crash loop; (2) an automation account doing a per-second SSH login/logout "login storm" wasting sessions and overloading journald; (3) the final trigger, an NFS disconnect that put I/O-waiting processes into uninterruptible D-state. With no spare headroom, a normally-minor NFS blip caused deadlock.
• Fix/prevention: set a memory limit on the app pod (e.g. 2GB) to contain a single runaway process; fix the automation to hold a persistent SSH connection instead of reconnecting every second; add NFS mount options soft,timeo=50,retrans=2 so storage outages don't freeze the host; fix container images missing curl that fail health checks and spam logs.
• Prevention principle: cap per-process resources, harden against storage-dependency stalls (soft NFS mounts), and treat any tight reconnect loop as a latent DoS on the host.