<!-- markdownlint-disable MD013 -->
# User-confirmed systems engineering history for the Toss application
Source: Discord attachment message.txt captured on 2026-07-14. This note preserves the claims that the user explicitly confirmed; it does not copy confidential credentials, host addresses, or customer identifiers.
Confirmed facts
- The shared Airflow/Kubernetes platform runs more than 80 crawlers created by interns and employees. Hyunwook owns the common scheduling, rate-limit, compute, network, DB-access and monitoring design; each crawler owner handles crawler-specific logic.
- The original on-premise Kubernetes cluster had two IDC servers. Resource concentration created roughly 70–100 Pod-allocation failures per week. Four idle in-house servers were joined through WireGuard/NAT, making six nodes total and reducing failures to fewer than 1–5 per week.
- Kubernetes operational issues included token/key expiry and server overuse. Expiry alerts were added so renewals could happen before an outage.
- A general-purpose PC with M.2 storage froze under heavy DB load and heat. Syslog, network conditions and workload timing were checked. Schedule spreading reduced monthly incidents from about 10–20 to fewer than five; moving the workload to IDC servers eliminated current DB operational incidents.
- More than 80 parallel crawlers originally accessed one DB directly. A shared broker/queue boundary reduced DB lock and connection errors from about three or four per day to zero.
- Metadata, binaries and Git repositories from many open-source ecosystems required different data-lake/storage designs. SeaweedFS/XFS and ZFS/Forgejo are used for different mutation patterns.
- A monthly DB backup task that required a person to run a script, wait, and move files to NAS was automated end to end and connected to messenger notifications.
- New employee Linux account creation across team servers was automated with a server user-management script.
- Prometheus and Grafana monitor roughly 10–20 team servers, DB read/write and replica health, collection gaps and volume, and customer data-delivery status. PostgreSQL time-series data is used for data-flow dashboards.
- For portfolio wording, Hyunwook described the common infrastructure work as project-lead/technical ownership. Do not imply that he originally designed the pre-existing Kubernetes cluster or personally fixed every crawler's internal logic.
Corroborating internal records
- Confluence 4026335502: WireGuard/NAT hybrid Kubernetes runbook and incident history.
- Confluence 4079321125: Airflow collection workflow design.
- Confluence 4142465026: DML Broker design and bounded queue/connection pool policy.
- Confluence 4160618647: GatheringDB backup and monitoring history.
- Confluence 4160389371: team Linux account policy and user-management script.