Labrador Platform (k8s, data-platform, scrapers, sqlmodel)

# Labrador Platform (k8s, data-platform, scrapers, sqlmodel)

AI Summary

Purpose:

• Capture durable knowledge about the Labrador data platform: how its four

repos interact, the labrador-sqlmodel queue refactor (the most reusable engineering result), and the AI automated PR-review system.

Key points:

• Four repos: labrador-sqlmodel (ORM base: BaseScraper, BaseRecord),

labrador-scrapers (ETL components built on that base, packaged as Docker images), labrador-data-platform (Airflow DAGs that run those images via DockerOperator/KubernetesPodOperator and load GatheringDB/DistributionDB), and k8s (Helm/scripts to install Airflow on Kubernetes with gitSync of DAGs).

• Pipeline: scraper image → registry (devlabrador) → Airflow DAG → MySQL DB,

on a kubeadm + Weave Net + Metrics Server cluster, dev/prod selected by a single PROFILE (namespaces airflow-dev vs airflow).

• labrador-sqlmodel queue refactor (most durable result): the old

RecordUpdater did one SELECT per record (N+1), had unbounded recursion, per-batch schema inspection, and a qsize/empty race. It was replaced by a single INSERT ... ON DUPLICATE KEY UPDATE UPSERT path (RecordUpdater removed, ~220 lines deleted), bounded recursion (MAX_RECURSION_DEPTH=10), and atomic get_nowait() queue draining.

• AI auto-PR review: Bitbucket Pipelines runs a Python script that sends the

git diff (filtered, chunked for context limits) to Claude 3.5 Sonnet and posts review comments via Bitbucket REST API v2, as a first-pass filter to cut human-reviewer load.

Relevant when:

• Working in any of the four repos, on Airflow DAGs / K8s install, on the

sqlmodel queue/UPSERT behavior, or on CI PR automation.

Do not read full document unless:

• You need the K8s install step order or the exact before/after queue code.

Linked documents:

• [[vulnerability-collection.md]] (crawlers built on sqlmodel, run as DAGs)
• [[license-collection.md]] (shares the ON DUPLICATE KEY UPDATE write standard)
• [[home-server.md]] (personal GitOps homelab mirrors these patterns)
• [[../concepts/crawler-collection-count-log.md]] (per-batch product/version

insert·update count log shared by the crawler-lib-* library crawlers)

• [[index.md]]

Open Questions

• AI PR-review system: architecture done; Python script + bitbucket-pipelines.yml

implementation and inline-comment support were still pending in the source note. Needs confirmation of production state.

• VCS appears to be Bitbucket here; the personal homelab uses GitHub. Both are

in use for different contexts.

Details

Repo interactions

• Developers build scrapers in labrador-scrapers on top of BaseScraper /

BaseRecord from labrador-sqlmodel.

• Scrapers are built into container images (with run command, env, DB options)

and pushed to the devlabrador registry.

• labrador-data-platform Airflow DAGs (git-managed) run those images via

DockerOperator / KubernetesPodOperator on schedule and load results into GatheringDB / DistributionDB (MySQL family); data-quality checks happen here.

• k8s defines the Airflow-on-Kubernetes infra as code (Helm values, secrets,

PV/PVC, Docker-in-Docker). gitSync pulls DAGs from Bitbucket.

K8s install flow

• setup/system_setup.sh: install containerd, kubeadm/kubelet/kubectl, CNI,

Helm on the node.

• setup/create_cluster.sh: kubeadm init + Weave Net + Metrics Server.
• install_apps.sh (PROFILE=dev|prod) → apps/airflow/install.sh: secrets →

PV/PVC → (dev only) local-path-provisioner → dind service → Helm chart (KubernetesExecutor, gitSync, NodePort). dev uses airflow-dev namespace + local storage; prod uses airflow.

labrador-sqlmodel queue refactor (durable engineering result)

Old design (queueing.py): a TransactionManager fed RecordInserter (bulk_save_objects) and RecordUpdater consumer threads. Problems found:

• N+1 queries: RecordUpdater called select_matching_record() per record →

one SELECT each → network round-trips and connection-pool pressure.

• Per-batch schema inspection: inspect(engine).get_columns() ran every flush.
• Unbounded recursion in _consume_records chunk splitting → stack-overflow

risk.

• Race: reading qsize() then looping while not empty could miscount.

Refactor:

• Removed RecordUpdater entirely (priority_columns conditional-update logic

dropped); all records go through one path using insert(Model).values(...).on_duplicate_key_update(...) — a single native MySQL/MariaDB UPSERT (DB-specific, intentional).

• Bounded recursion: MAX_RECURSION_DEPTH = 10, fail-safe on exceed.
• Atomic draining: drain with get_nowait() until Empty instead of

qsize/empty checks.

• Result: ~220 fewer lines, N+1 eliminated, clearer UPSERT intent, safer

multithreading.

AI automated PR review

• Trigger: Bitbucket PR open/update → Bitbucket Pipelines (serverless).
• Preprocess (Python): extract git diff origin/main...source, filter noise

(*.lock, images, dist/), chunk per file for the Claude context window.

• Inference: Claude 3.5 Sonnet via Anthropic API, prompted for critical issues

(bugs, security, inefficient queries) — not summaries — answering in Korean.

• Action: post comments via Bitbucket REST API v2 (General Comment first;

Inline Comment is a harder follow-up).

• Goal: first-pass filter so human reviewers focus on architecture/business

logic; sends only the diff to cut API cost.