binlog-transfer-system

# binlog-transfer-system

AI Summary

Purpose:

• Store repository-specific architecture, verification commands, and review

constraints for the BTS v4 binlog transfer system.

Key points:

• download-server is a Python/FastAPI encrypted-binlog HTTP server with

direct, proxy, and relay modes.

• updater is the Go client that downloads, verifies, decrypts, and applies

binlog files to the target database.

• Review of feature/request-variant-in-post at 94ae92a against main

cf904ad ended in Request changes; the branch was not modified.

• The POST transport preserves the fixed client URL, JSON status fields,

Smart Polling header, and core 200/202/304 flow, but misses direct POST operational status logs, uses coercive identifier parsing, and has no behavioral regression coverage.

• The three customer containers (lab-master-db, download-proxy-server,

updater) all run the same image with different command values, so image-level changes apply to all three.

• v4.1.1 replaces the base image's el9 curl/libcurl 7.76.1 with an upstream

8.21.0 build for BT26-707. mysqld never linked libcurl; only microdnf/librepo/libdnf and mysql-shell did.

Relevant when:

• Reviewing or modifying download request transport, status reporting,

proxy/relay behavior, updater configuration, or release packaging.

• Patching base-image packages, or building any native library into the image.

Do not read full document unless:

• Exact commands, review findings, deployment constraints, or known pitfalls

are required.

Linked documents:

• ai/workspace/repos.md
• ai/wiki/projects/bts.md
• ai/worklog/2026/2026-W32.md

Repo Info

• Repo ID: binlog-transfer-system
• Main branch: main
• Local path: ~/labrador/platform/binlog-transfer-system
• Remote: bitbucket.org:labradorlabs/binlog-transfer-system.git

Common Commands

git diff origin/main...HEAD
git diff --check origin/main...HEAD

cd updater
go test -race -shuffle=on -count=1 ./...
go vet ./...

cd ../download-server
PYTHONDONTWRITEBYTECODE=1 uv run --no-project \
  --with-requirements requirements.txt python -c \
  'from app.routers.cdbvdb_v4 import BtsDownloadBody; print(BtsDownloadBody)'

Architecture Notes

• download-server/app/routers/cdbvdb_v4.py owns GET/POST download routing,

direct encrypted streaming, proxy pass-through, relay status forwarding, Smart Polling, and active-file protection.

• download-server/app/main.py owns access logging. As of cf904ad, its

updater success/error log reads status, reason, and msg only from query parameters.

• updater/internal/client/downloader.go constructs download/status requests,

verifies X-Original-Hash, and updates persisted download state.

• updater/internal/config/config.go loads environment-based updater settings.
• UnifiedDockerfile builds both the Go updater/security tools and the

PyInstaller download-server binary into the unified MySQL-based image.

• One image, three customer containers. lab-master-installer/ runs

lab-master-db, download-proxy-server and updater from the same labradorlabs/master-bts:<ver> image; entrypoint.sh dispatches on the command (mysqld default / server / updater). An image-level fix applies to all three at once.

• Base image is mysql:8.4.0 (Oracle Linux 9), which already ships

curl/libcurl, mysql-shell, microdnf and openssl.

• Measured libcurl.so.4 consumers inside the image: microdnf,

librepo.so.0, libdnf.so.2, mysqlsh, mysql-secret-store-login-path, curl. mysqld does not link libcurl, and no MySQL plugin or /app binary references it (including via dlopen). Nothing in the repo invokes curl or mysqlsh.

• As of v4.1.1 the image carries an upstream curl 8.21.0 build instead of the

el9 RPM. curl-builder is a stage FROM mysql:8.4.0 and both stages run microdnf update -y openssl-libs zlib so the runtime libs match.

Known Pitfalls

• Moving updater status fields from query parameters to a POST body bypasses

the query-only operational status logger unless logging is moved to a shared parsed-request seam.

• Non-strict Pydantic int | str accepts JSON booleans and integral floats as

integers. Use strict types when GET/POST identifier semantics must match.

• A JSON body removes ordinary URI-size limits. Apply an ingress/ASGI body cap,

field/list bounds, and bounded relay concurrency before exposing POST to a reachable network.

• download-server/requirements.txt uses lower bounds rather than a lock, so

FastAPI/Pydantic behavior can drift between image builds.

• download-server/app/main.py creates /app/data/log at import time. Local

read-only imports can fail even when router-level syntax/model checks pass; this is pre-existing and was not attributed to the POST branch.

• v4.0.0 is already customer-facing. Do not silently rebuild or replace it;

define a new image version and migration/release plan, especially because security-tool keys are embedded at build time.

• A release build needs the production SECRET_KEY. build.sh "$KEY" embeds it

into decryptor (wrapped), ops_connect and the server-side encryptor; with no argument it silently generates a random key ("development only"), and a mismatched key breaks customer-side decryption. /config/ and /bin/ are gitignored and absent from the checkout.

• build.sh also writes generated RSA keys into ./config when they are

missing, and reuses whatever is there on the next run. Never run build.sh for a throwaway/validation build — call docker build directly with a temp --secret, or a later production build will silently pick up the dev keys.

• Removing a base-image package by editing the microdnf install line is not

enough. curl/libcurl come from the base too; rpm -q --last distinguishes base packages (2024 build date) from ones our layer installed or upgraded.

• Any library rebuilt for this image must be compiled on the same base as the

final stage. Building curl on oraclelinux:9 produced libssl.so.3: version 'OPENSSL_3.2.0' not found because that base's OpenSSL was newer than the target's. Keep the DT_NEEDED set minimal for the same reason.

• rpm -e in a derived layer does not reclaim space: the files remain in lower

layers and the rewritten rpmdb is added on top (an in-place removal patch of 4.0.0 measured 1.37GB → 1.39GB).

• On labM, run db-infra/up.sh and down.sh **as the labrador account

(uid 1000)**. scripts/lib/common.sh:119 sets UID_GID=$(id -u):$(id -g), so the container inherits the invoking user; running as another account makes mysqld fail on /var/log/mysql/error.log and /data/lab-master (both owned by 1000:1000) and the container crash-loops. Recovery without that account: pass UID_GID=1000:1000 in the environment to docker compose up -d — shell env outranks --env-file in Compose v2.

• up.sh prints STARTUP-MYSQL DONE and sends a Slack "인스턴스 시작" notice

based only on the compose exit code. It does not verify mysqld stayed up, so always check docker ps and the error log after a start.

• The lab-master DB image is set by DOCKER_IMAGE in the instance .env, not

in docker-compose.yml. The updater pins its tag in docker-compose.yml directly. A version rollout has to touch both.

Current Work

• 2026-08-13: v4.1.1 curl/libcurl remediation for BT26-707 (신한투자증권).

main is at 3aca7f1 (v4.1.0 released; the POST branch was merged in 36acaa0). Uncommitted on top: UnifiedDockerfile (curl-builder stage + el9 curl/libcurl replacement + build-time assertions), VERSION → 4.1.1, new RELEASE/v4.1.1.md. Validated with a linux/amd64 build using a throwaway key; a release image still needs the production SECRET_KEY.

• Earlier: branch feature/request-variant-in-post at 94ae92a was reviewed

against cf904ad on 2026-08-03 with Request changes. It was merged anyway as part of v4.1.0 (36acaa0, 85f8fb2); the review findings were not re-checked against the merged code.

Decisions

• Do not approve the POST branch until direct status-log parity, strict body

validation, body/relay resource bounds, and cross-boundary tests are added.

• Keep GET as the default and treat fixed-URI propagation across proxy/relay

hops as an explicit topology decision rather than an assumption.

Open Questions

• Does the customer's fixed-URI restriction apply only to updater-to-BTS, or

also to BTS proxy/relay-to-upstream hops?

• What request-body, rate, and authentication limits are already enforced by

the actual ingress or network relay?

• What new image version and rollout path will carry the POST feature?
• Who holds the production SECRET_KEY and config/*.pem for a release build?

Without them 4.1.1 can only be patched in place on top of the 4.1.0 image, which would leave the embedded version string reading 4.1.0.

• Does 신한투자증권's v3 (image 2.0.2) → v4 upgrade land before the 8월 말 due

date, or does 2.0.2 need the same curl patch as a fallback?