2026-W34 Worklog

# 2026-W34 Worklog

AI Summary

Purpose:

• Record cross-repository development and review work for ISO week 34 of 2026.

Key points:

• 2026-08-20 (3): Built the Kmong Data Engineer tailored application package

from Remember posting 334810. The evidence audit maps 52/52 resume claims, 21/21 numeric claims, and 13/13 portfolio-only claims. The final A4 artifacts are a two-page resume and six-page portfolio, with CEO/CTO/Tech Lead reviews at 96/97/98 and both visual-QA lanes passing. Deterministic quality is 94.7/100 because verified experience covers 8/10 must-have and 2/7 preferred rows; Spark, production recommendation/ranking serving, and formal DW remain honest gaps. The package is therefore DONE_WITH_CONCERNS and was not copied into the final-submission archive.

• 2026-08-20 (2): Reconciled the current Confluence state with the durable wiki

and Korean portfolio. Added source captures for DT/4209377355 and DT/4211146807 + DT/4212457542, corrected the stale RHEL summary from the old dimension/view and delete-reinsert draft to the current three-table + row-diff design, and added two evidence-bounded portfolio entries: RHEL VEX collection redesign and crawler-source legal/operating governance. Both are explicitly labeled as design/pre-review work, not deployed impact or completed legal approval.

• 2026-08-20: The published one-page crawler-source briefing now separates

public/OSS sources, conditional sources requiring rate/API/robots controls, and restricted sources requiring a supported interface, alternative data, or written agreement. The customer-operated collection project is distinct from LabradorLabs' own third-party redistribution; the latter remains unresolved.

• 2026-08-19: Created Confluence DT/4211146807 (child of "26. 미래아카데미"

4211605514) for the 2026-08-20 14:00 customer meeting on crawler legal review. Per user, page v2 carries ONLY the two prep items from 조학수 CTO's announcement: ① crawl-site inventory refreshed against the live crawler fleet (file components excluded), ② per-source legal pre-review with risk grades (Snyk / GitLab gemnasium-db flagged high). v3 merged the site table and the legal table into one, turned URLs into links, and dropped in-progress/ops-status remarks. libraries.io was removed from the page — user states it is no longer used by the PyPI crawler (2026-08-19); the code on this host still references it, so verify on next pypi_crawler work. The scope-proposal table, IP/know-how boundary framing (meeting conclusions 5–6) and prep checklist were removed from the page and preserved in this worklog's 2026-08-19 details. Base doc DT/3010625669 left untouched.

• 2026-08-18 (4): Finalized CJ ENM, CJ Olive Young, and Vroong as rejected

based on the user's result update. Updated the Drive final-archive index, the Korean application tracker and web page, and the durable career-transition wiki. Recorded a source-backed retrospective that separates confirmed status from inferred causes and carries concrete next-resume actions.

• 2026-08-18 (6): Simplified the schema after re-reading the source document

once more. The surrogate PRODUCT_KEY / MODULE_KEY dimensions are gone; the CPE string is the key and MODULE is a single column. Five tables plus a view became three. The source models the data as CPE -> package -> fixed version, so keying on the CPE reproduces that model directly and the source's own bubblewrap example is literally three rows differing only in CPE — which is also the clearest way to explain the table to a reviewer. Cost is the CPE repeated across ~4.9M rows (~200 MB) against 14.7 GB today. Confluence at v7, rewritten in 개조식 with tables instead of prose.

• 2026-08-18 (5): Reframed the whole plan so the source Confluence document

(4149674090) is the spine rather than an appendix, at the user's instruction. Confluence DT/4209377355 v5 now runs: what the source fixed → how it is implemented → measured answers to what it left open → what is outside its scope. Also corrected the matching key: the host CPE string is NOT the join key. Measured, a literal CPE join matches 27.3% of fixed rows and drops the 38.2% under AppStream, while MAIN and the entitled channel disagree on the fixed EVR in 96.9% of comparable cases — so the key is (OS_MAJOR, SUPPORT_CHANNEL) derived from the CPE, expanded to the repo streams the host has. That vindicates the source document's own line that "CPE가 직접적으로 필요하지는 않음".

• 2026-08-18 (4): Refinement pass found four defects in my own design and fixed

them against measurement rather than intuition — the EVR tie-break was rationalized, not verified; a primary-key collision would have aborted the load; the match-time rule for FIXED-vs-AFFECTED was missing entirely; and the serving DB turned out to be the binlog master shipping to customer on-premise servers, which makes write volume a product constraint. Confluence page at v3.

• 2026-08-18 (3): Published the design as Confluence DT/4209377355

[os 패키지 취약점 - v4] Red Hat, a child of the empty v4 hub page 4209180746, following the v3 naming convention. Confluence Cloud auto-converts ```mermaid code blocks into native Mermaid diagram macros, so the three diagrams (data flow with table placement, ERD, decision flow) render rather than sit as code.

• 2026-08-18 (2): After the user confirmed the existing crawler is unused and

will be rebuilt, rewrote the page as a greenfield design spec grounded in the full 63,152-document VEX corpus. Scope A vs B row counts, CPE resolution coverage, EVR-collapse rate, module inventory, parse throughput all measured.

• 2026-08-18: Investigated the RHEL OS-package vulnerability false positives

reported for ServerCare, ahead of the table-structure 협의 agreed in the 2026-08-04 AI engine team meeting. Added ai/wiki/projects/rhel-vex-vuln-collection.md.

• Finding that reframes the task: the format migration is already done. The raw

crawler in labrador-scrapers/etl_components/os_pkg_vuln/rhel_crawler has been pulling security.access.redhat.com/data/csaf/v2/vex/ (full .tar.zst archive + changes.csv incremental) since at least 2025-10. The problem is the parse stage, which still interprets VEX with OVAL-era assumptions.

• Seven defects confirmed against live data, not inferred:

1. _get_os_versions() fans one product's fix across every minor of the major; OS_VERSION_EDITION = 'generated' rows dominate the table. 2. build_thresholds_from_fixed_nevra() keeps only the major of the first fixed NEVRA. CVE-2024-42472 flatpak has fixes for majors 7/8/9; only 7 is written. 3. Module conditions (::mysql:8.0) break NEVRA_RE — parsed architecture becomes "0" — and are never applied at match time. 75 such product_ids in CVE-2022-21607 alone. 4. PRODUCT_RE truncates package names at the first hyphen: gstreamer-plugins is stored as gstreamer, compat-glibc as compat. 5. No CPE filter, so 5Client-OracleJava-*, CRB-*, NFV-*, 4AS-SAP and other non-OS products land in the OS table. 6. red_hat_enterprise_linux_ai_(rhel_ai)_3 is parsed as RHEL 3. 7. ARCHITECTURE is copied from an arbitrary first NEVRA and sits in the PK: src 1.90M / aarch64 1.30M / x86_64 0.42M of 3.71M rows.

• Measured volumes on gatheringdb: TB_OS_PKG_VULN_RHEL 3,709,299 rows /

10.4 GB data / 4.3 GB index; TB_OS_PKG_VULN_RHEL_RAW_STATUS 723,247 rows. Debian is 261K rows and Alpine 782K by comparison.

• Proposed direction: resolve product_id through product_tree.relationships

to the product_name node and take version + subscription channel + repo stream from its CPE only; store one row per (CVE, product, package, module) with no minor fan-out; move host resolution to match time using agent-reported CPE, enabled dnf modules and kernel-rt presence. Draft two-table structure and a W34–W39 phase plan against the end-of-September feature deadline are in the wiki page.

• Also flagged for the rewrite: no delete/tombstone path (so a full re-parse is

required, not optional), known_not_affected discarded entirely, a per-batch list-reuse bug that lets one CVE's fix suppress another CVE's known_affected row, a runtime dependency on endoflife.date, and a hardcoded production DB URI with credentials as a default argument in both crawler classes.

Relevant when:

• Picking up the RHEL VEX work, or preparing the OS-package table structure and

API definition for the AI engine team.

Do not read full document unless:

• You need the per-day detail.

Linked documents:

• [[../../wiki/projects/rhel-vex-vuln-collection.md]]
• [[../../wiki/projects/vulnerability-collection.md]]
• [[../../repo-notes/labrador-scrapers.md]]

Open Questions

• ServerCare support scope for CRB / NFV / SAP / Client streams and for RHEL

4–5 is unresolved; it gates the CPE reject list.

• Whether the agent can report the subscription CPE on unregistered or

RHEL-derivative hosts (no subscription-manager, so no /etc/pki/product/*.pem).

Details

2026-08-19 — 미래아카데미 크롤러 법적 이슈 사전 검토 페이지

Context: the customer (미래아카데미, ISMP 2차년도) asked to self-collect component/vulnerability/license data to cut external data-license cost, and a 2026-08-20 14:00 meeting (담당관·과장) covers "크롤러 수집 시 법적 이슈 검토". The plan under discussion delivers RAW collection only, excluding know-how and patented tech (CENTRIS/VUDDY/XVDB) — meeting minutes 4199350485 conclusions 5–6.

Created Confluence page DT/4211146807 (크롤러 수집 사이트 현행화 및 법적 이슈 사전 검토 (2026-08-20 미팅)) under "26. 미래아카데미" (4211605514):

• Scope proposal table: include library components / vulnerabilities /

licenses; exclude file, function, binary components, code-level vuln (VUDDY/XVDB-adjacent), LPP/CTI; EOL, AI-model, AI license extraction marked 결정 필요. All framed as (안) — decisions left to the meeting (review-docs rule).

• Site inventory refreshed from the live fleet (labrador-scrapers

etl_components + labrador-data-platform DAGs + in-code URLs), base doc DT/3010625669 (2025-12) untouched. New vs base doc: OSV (osv-vulnerabilities GCS bucket), NVD JSON feed crawler, OpenSSF malicious-packages, SPM/Swift Package Index, server OS-package vuln DAGs (alpine/debian/ubuntu/rhel; RHEL = CSAF/VEX v4 redesign), Conan switched to conan-center-index clone. PHP/NuGet/RubyGems have no data-platform DAG — marked 운영 상태 확인 필요.

• Legal pre-review: common issues (ToS/data licenses + attribution, Korean DB

producer rights, collecting-party identity for ToS/API keys — customer-name collection is safer than collect-and-redistribute, robots/rate-limit compliance, maintainer PII in registry metadata) + per-source table with risk grades. High-risk: Snyk web crawling (commercial curated DB), GitLab gemnasium-db (license restricts commercial use outside GitLab). libraries.io (CC-BY-SA share-alike) was initially flagged from code inspection, but the user states it is no longer used by the PyPI crawler (2026-08-19) — removed from the page in v3; host code still references it, verify on next pypi_crawler work. All "known conditions" labeled as pre-check requiring original-text verification.

• IP/know-how boundary table (RAW collection vs normalization/mapping know-how

vs patents) + contract clauses proposal, grounded in meeting conclusions 5–6.

• Meeting-prep task list matching the announcement (data samples, site

materials, legal pre-review, collecting-party question, legal-team review decision).

Sources: Confluence 3010625669 (base URL doc), 4199350485 (2026-08-09 회의록), 4197941878 (고객 요구사항 정리), 4198629558 (ISMP CDBVDB 방안), host fleet inspection 2026-08-19.

Update (same day): user found v1 too complex — the Confluence page (v2) now carries ONLY the two prep items from 조학수 CTO's meeting announcement (site materials + legal pre-review). The sections below were REMOVED from the page and are preserved here instead. If needed later, they can be re-published as a separate internal page (do not put them back on the meeting page).

Update 2 (v3–v4, user corrections for the customer build):

• Snyk excluded from the collection scope (like libraries.io) — removed

from the vulnerability rows; noted in the page's 제외 line.

• RHEL will use OVAL, not VEX, for this customer build (user decision;

page now lists redhat.com/security/data/oval/v2). Note the base URL doc's remark says Red Hat OVAL stopped exposing data and internal collection moved to CSAF/VEX — flag this tension before implementation.

• Java (Maven) current sources are only 3, verified in code:

Maven Central (repo1.maven.org + search.maven.org), Cloudera (repository.cloudera.com), Atlassian (packages.atlassian.com). The 9-repo list (WSO2/Liferay/Jenkins/Eclipse/Clojars/Google 등) in the base doc is historical; Google Maven belongs to the excluded binary crawler.

• Page format: site table and legal table merged into one (one row per source:

항목/수집처/방안·주기/리스크 및 사유/확인 필요), URLs as links (not code spans), multi-URL cells as bullet lists, per-row risk grade now carries a brief reason, ops-status remarks ("운영 중", "재설계 진행 중" 등) dropped.

• v5: the 확인 필요 column now links each source's terms/license original

(NVD developers/terms-of-use, cve.org TermsOfUse, KEV catalog page — cisa.gov blocks bots so the license.txt path was not linked, first.org/epss, gemnasium-db LICENSE.md, github advisory-database, npm/PyPI/NuGet/RubyGems terms, vendor legal hubs). Verified live before linking: NVD ToU, nuget.org Terms, gemnasium-db LICENSE.md blob; PyPI ToU redirects to policies.python.org/pypi.org/Terms-of-Use/ (final URL used).

Update 3 — vendor ToS actually read (2026-08-19; page NOT edited — user is editing it themselves and asked for a report only):

• AWS Site Terms (aws.amazon.com/terms, LICENSE AND SITE ACCESS):

prohibits "any use of data mining, robots, or similar data gathering and extraction tools" and any commercial reproduction/exploitation without express written consent. Only docs.aws.amazon.com is CC-BY-SA-4.0, and ALAS is not under docs. → ALAS HTML crawling is textually prohibited: risk should move 중→상. Mitigation to investigate: Amazon Linux repo metadata (updateinfo) meant for machine consumption, or written consent.

• Oracle Terms of Use (oracle.com/legal/terms.html, fetched via curl —

WebFetch gets 403): "You agree not to use any robot, spider, scraper or other automated means to access the Site … without Oracle's express written permission"; Materials clause limits use to personal/informational/ noncommercial. The OVAL directory itself (linux.oracle.com/security/oval) is a bare file listing with no license text. → textually 상, though the OVAL feed exists precisely for scanner consumption; written confirmation recommended.

• Cloudera ToS (cloudera.com/legal/terms-and-conditions): prohibits

"robot, spider, crawler, scraper or other automated means … to extract data" and collecting/copying/distributing Materials; open-source components keep their own licenses. → 중~상; the repo exists for build-tool access, so written inquiry is the clean path.

• Atlassian: no Website Terms of Use exists on the legal hub (the old

/legal/website-terms* URLs 404); Customer Agreement covers products, not the Maven repo. No explicit crawl prohibition found. → stays 중.

• npm — good news: official Crawlers policy (docs.npmjs.com/policies/

crawlers) allows website crawling at ≤1 req/s and explicitly recommends replication; Open Source Terms explicitly permit "replicate data from the Public Registry using the Public APIs" (our crawler's replicate._changes + registry API path), with a stated unreasonable-volume line of 5M req/month. Caveat: "data about the security of Packages" (audit data) may not be redistributed — we collect metadata only. → npm can move 중→하 (via the replication path, within rate limits).

• Swift Package Index: robots.txt allows User-agent: * on everything

except /api/ (AI-training bots are blocked; Content-Signal ai-train=no), but the site actively bot-blocks via Cloudflare in practice (FAQ fetch 403s, matches our crawler's 403 history). → robots-permitted but block-in-practice; the GitHub PackageList repo remains the safe universe source.

• HuggingFace ToS: no explicit crawling/bulk-download clause; states OSS

license terms are not overridden by the ToS; model licenses vary per model. → stays 중 (no prohibition found, per-model license handling needed).

• SUSE Terms of Use: website Materials may not be reproduced without

written permission; no explicit crawler clause; no license text found for the CVRF/CSAF feeds themselves. → stays 중 (feed-specific license still unverified).

Update 5 — Atlassian properly verified (page v14): Atlassian Acceptable Use Policy (atlassian.com/legal/acceptable-use-policy) applies to "all products and websites owned or operated by Atlassian, and any related websites, sub-domains and pages" and prohibits both automated request rates beyond human speed and "accessing or searching any part of the services by any means other than our publicly supported interfaces (for example, 'scraping')". Both packages.atlassian.com and its successor robots.txt are a full Disallow: /. Also discovered: packages.atlassian.com/mvn/ maven-external 301-redirects to maven.artifacts.atlassian.com (2026-08-19) — crawler target URL needs updating. Atlassian moved 중→상 (alongside Cloudera); nuance noted that build-tool protocol access is arguably the "publicly supported interface".

Update 6 — final sweep of remaining 미확인 items (page v15), per user "확인해서 없으면 하로": EPSS — first.org/epss states data is "freely and openly accessible" (no restriction found) → 하. SUSE — the CSAF feed directory carries a LICENSE file: CC-BY-4.0 (changed from CC-BY-NC-4.0 on 2021-05-12) → 하, linked in the page. OSADL — checklist raw data is provided "without access restrictions" and "may be incorporated into scanning tools, companies' internal utilities" (always-use-current-version caveat) → 하. OLIS — no prohibition found; note: no 공공누리(KOGL) mark, footer says "All Rights Reserved" → marked 하 per user rule with the KOGL absence noted in the cell. OSV remaining sources — Bitnami vulndb Apache-2.0 (LICENSE.md), Haskell security-advisories CC0 public domain (LICENSE.txt), both verified via raw.githubusercontent → OSV row now entirely 하. Result: the only remaining flags are 상 = GitLab gemnasium-db (community MIT repo as the fix), Amazon ALAS, Oracle OVAL, Cloudera Maven, Atlassian Maven; no 중 rows remain.

Update 7 (page v17): user then asked to keep 중 for sources with usage constraints (rate limits etc.) rather than eliminating the grade. Re-graded to 중: Maven Central (rate limit), npm (1 req/s, 5M req/month), NVD API (rate limit + API key; JSON feed exempt — shown as 하+중 dual chips), GitHub GraphQL API (rate limit + token identity; data itself stays 하 CC-BY), SPI (robots-permitted but Cloudflare bot-block in practice), OSADL (must-use-current-version condition). Grading convention now: 하 = data nature free, 중 = usable with operational conditions, 상 = textual prohibition; dual chips per row where nature and access constraints differ.

Update 8 (page v18): OS-package row broken down per distro (user request), with two more licenses verified: 하 = Red Hat (CC-BY 4.0), SUSE (feed LICENSE CC-BY-4.0 verified), Mariner (CBL-MarinerVulnerabilityData LICENSE = MIT, verified), Ubuntu (public git, clone/rsync instructions in README, no prohibition), Alpine·Debian·Alma·Rocky·Arch·Fedora (community public data, no prohibition found); 중 = Photon (GitHub wiki data is public — photon repo COPYING is GPLv3 — but packages.vmware.com vendor (Broadcom) site terms unverified); 상 = Amazon ALAS, Oracle OVAL (unchanged). Photon added to the summary 중 line and the OS row's 확인 필요 cell.

Update 10 (2026-08-20, page v23): reviewer supplied three legal position statements ("2. 보안데이터 수집 체계의 법적 고려사항") to merge into the page: (1) this is a build-service (용역 구축) project — the customer collects, so redistribution concerns don't apply to the vendor; (2) scraping-restricted sites must be accessed only within permitted limits; (3) rate-limit-style constraints must be honored. Chapter 2 restructured accordingly (retitled to the reviewer's heading), with each of the three items explicitly marked "(검토 의견 반영)" and an intro line "최종 확정은 계약 전 법무 검토를 전제" — deliberate attribution so the engineering document carries facts while legal judgments remain attributed to their source (user's role-boundary concern: ToS fact-finding is engineering work, legal position statements are not). Kept the caveat that licenses restricting USE (gemnasium-db) still bind the customer regardless of who collects. Item 1's premise (약관 동의·API 키 고객 명의, 계약서 명시) folded in from the old 수집 주체 item; 저작권/DB권, attribution, 개인정보 items retained below.

Update 9 — AI-model crawler verified in code + DB (page v22): ai_huggingface_crawler has 3 scrapers chained in one daily-20:00 DAG (ALL_SUCCESS serial): ① hugging_model — HfApi.list_models + model_info via huggingface_hub → TB_COMP_MODEL (3.1M rows; MODEL_ID/TASK/AUTHOR/MODIFIED, LICENSE_IDS mapped from license: tags, MODEL_CONTENT = model card, MODEL_MAL_YN); ② hugging_vuln — parses huggingface.co/{id}/tree/main HTML for HF's own security-scan unsafe files → TB_COMP_MODEL_VULN_INFO (1.1M; FILE_NAME/IS_UNSAFED/IS_PICKLED/MAL_INFO); ③ hugging_pipeline_model — clones huggingface/transformers, AST-parses pipelines SUPPORTED_TASKS → TB_HF_PIPELINE_MODEL (43 rows: task×framework default model). DB freshness: ① daily through 2026-08-18; ② no new rows since 2026-04-10 (grid shows frequent model-crawler failures cascading upstream_failed to ②③, and green runs of ② still produce nothing — parsing selector likely broken); ③ unchanged since 2026-03-24 (may be legitimate no-change). Page AI-model row updated with the HTML path (모델 페이지 웹 파싱) and cycle corrected to 매일 (was 7일). Incidental: component contains ad-hoc library_license.py (npm LICENSE_AI backfill, references mvnrepository.com) — not a registered scraper.

Update 4 — reviewer feedback applied (page v13): per-registry grades for Maven (Central 하 / Cloudera 상 / Atlassian 중); SPI graded on its stated policy (robots.txt allows all but /api/) = 하, with the Cloudflare bot block noted as a CDN-level technical measure separate from the terms; HuggingFace graded on Hub content ToS (no crawling prohibition, verified) = 하, per-model-license caveat dropped; OSV broken down by source with verified licenses — GitHub Advisory / PyPA advisory-database / golang/vulndb / google/oss-fuzz-vulns all CC-BY 4.0 (verified via each repo README), RustSec advisory-db public domain except GHSA-derived entries (verified), distro sources 하, remaining sources (Bitnami·Haskell 등) 중; GitLab — 공개판 gitlab-org/advisories-community exists and is MIT-licensed (verified), added as the migration alternative to gemnasium-db in the source cell, risk cell, 확인필요 cell and summary. Summary 상 now: GitLab gemnasium-db (or switch to community MIT repo), Amazon ALAS · Oracle OVAL · Cloudera Maven.

#### Removed section — 수집 범위(안) (per-category include/exclude proposal)

Basis: 협상 전략 (수집 대상은 ISMP SBOM 분석에 필요한 구성요소·버전·CVE 중심 취약점·라이선스로 한정, CENTRIS·VUDDY·XVDB 특허 배제) + 2026-08-09 회의 결론.

#### Removed section — 지적자산·노하우 경계 (논의 안건)

Follow-up framing of 회의 결론 5 (고객 제공 소스에 특허·고유 기술 배제) and 결론 6 (소유권·사용권 조건 별도 검토 — 당사 재사용 제약 방지). Presented as proposal only; decisions belong to the meeting/협의.

계약서 반영 필요 사항(안): ① 고객 제공 산출물 범위를 위 계층으로 명시, ② 제공 소스코드에 대한 당사의 계속 사용·재사용권 보장(결론 6), ③ 수집 데이터의 원출처 라이선스 조건이 고객 측 의무로 승계됨을 명시, ④ 수집 주체(약관 동의·API 키 명의) 지정.

#### Removed section — 미팅 준비 체크리스트

• 일정 공유 — 영환님 (공지 기준)
• 크롤링 데이터 샘플 준비 — 출처별 RAW 원본 예시(NVD CVE JSON, OSV JSON,

Red Hat VEX 문서, Alpine secdb JSON, npm 메타 JSON 등)

• 사이트 관련 자료 — 페이지 1장 공유
• 법적 이슈 사전 검토 자료 — 페이지 2장 공유 + 고위험 3건(Snyk·GitLab

Advisory·libraries.io) 처리 방침 내부 입장 정리

• 수집 주체 질의 정리 — 약관 동의·API 키(NVD, GitHub 등) 명의를 고객 기관으로

할지

• 소유권·사용권 논의 안건 내부 사전 협의
• "확인 필요" 약관 항목의 법무 검토 의뢰 여부 결정

2026-08-18 (4) — 방안 정련: 자체 설계 결함 4건 수정

Re-reviewed the design critically instead of building the prototype. Four things were wrong or missing, each now settled by measurement.

1. EVR 충돌 규칙이 근거 없이 정해져 있었다. The draft said "keep the max EVR because a MAIN host tracks the newest Z-stream". Inspecting the 20,176 conflict rows (scope B; 12,578 in scope A) shows that rationale describes a minority case: 80.2% are same-epoch same-version with only the release differing, and the dominant shape is a single legacy product listing successive builds (3AS: 0:2.4.21-20.0.1.EL and 0:2.4.21-27.0.4.EL for CVE-2004-1073). For that shape max EVR manufactures a false positive on an already-patched host. Design now stores FIXED_EVR (min, used for the decision), FIXED_EVR_LATEST (max, shown as the target) and FIXED_EVR_COUNT, so the choice is auditable and switchable without re-collecting.

2. The primary key could not hold the data. Dropping architecture collapses rows VEX reported under different statuses: 356 fixed+known_not_affected and 45 known_affected+known_not_affected at the exact grain key. 401 hard PK collisions that would abort the load. Added an explicit parse-time precedence (fixed > known_affected > known_not_affected > under_investigation, because the usual cause is one arch fixed and another never vulnerable) plus a STATUS_CONFLICT flag.

3. The matching spec had no rule for multiple matching rows. Measured 27,079 (major, package, module) combos in the MAIN family carrying both a FIXED row (repo-stream CPE) and an AFFECTED row (whole-major CPE). Without a rule every one would report "패치 없음" while a patch exists. Added: channel precedence first (minor-pinned > MAIN), then status (FIXED > AFFECTED).

4. The serving DB is the binlog master for customer on-premise servers. labradordb on dist-m feeds BTS; binlog growth has already caused low-spec on-prem servers to fall behind. So delete-and-reinsert per document was the wrong physical apply. Measured on 120 republished documents (archive version vs current): only 30 of 120 change any fact row at all, and only 22.7% of rows change — 268 docs/day × 114.9 × 2 ≈ 61,600 row events for replace versus ≈ 7,000 for a diff apply, an 8.8x difference replicated to every customer. The document remains the unit of recomputation; only the physical apply changed.

Also settled: column widths from measured maxima (package 110, EVR 62, CPE 47, module:stream 25); the product dimension now stores every Red Hat CPE with IS_TARGET so new products are visible; deletions.csv needs its own watermark; bootstrap catch-up is ~2,909 documents (1,031 new + 1,878 republished), not the 1,031 stated earlier; and known_not_affected's suppression value is measurably negligible (45 / 356 overlaps against 2.3M rows), which turns decision 4 from taste into arithmetic.

2026-08-18 (3) — 수집/분석 테이블 분리 + Confluence 게시

Three further measurements settled open design points:

• Serving DB (labradordb) already carries TB_OS_PKG_VULN_RHEL at 4,167,167

rows / 9.9 GB plus TB_CONTAINER_OS_VULN_V2_RHEL (1,555,420 / 464 MB). The v3 convention of annotating columns "수집DB에만 필요한" is what the v4 split formalizes: _DOC never leaves gatheringdb; the two dimensions, the slim fact table and a view go to labradordb.

• deletions.csv exists and is large: 280,797 entries over 2024-08-20 ~

2026-08-17, only 7 of which still exist in the archive. Genuine withdrawals, not churn — this is the "CVE가 발급됐다가 없어지는 케이스" from v3.

• 5.3% of in-scope known_affected references (71,843 of 1,368,320) are SRPM

names. Instead of a binary/source mapping table, the agent reports rpm -qa --qf '%{NAME}|%{SOURCERPM}' and the fact table carries PKG_KIND.

Also found while enumerating CPEs: RHEL 7 realtime is rhel_extras_rt:7, not enterprise_linux:7::realtime, so it falls outside the product allowlist — added as an open question rather than silently included.

2026-08-18 (2) — Greenfield design after the rebuild decision

The user decided the existing rhel_crawler is not in use and will be replaced outright, so the wiki page was rewritten from a defect analysis into a design specification. To ground it, the full VEX corpus was pulled and measured rather than sampled: csaf_vex_2026-08-09.tar.zst (273 MB) -> 16 GB, 63,152 CVE documents, 36,130 of which carry a RHEL OS product.

Measurements that drove design decisions:

• product_tree.relationships resolves every product_id appearing in

product_status — 0 unresolved out of ~7.15M entries. So the pipeline needs no string parsing of the product half at all.

• Dropping architecture collapses 7.15M product_status entries to 4.82M grain

rows (32.6%); 3.93M of those already had a single architecture.

• Scope A (Confluence stream allowlist: none/baseos/appstream/realtime) yields

4,866,082 rows. Scope B (A + ::server/::computenode/::as/::es) yields 5,177,949. Scope A silently drops all of RHEL 4–7, which use ::server, so the allowlist in the Confluence note needs extending — this is now the first open question for 협의.

• MAIN CPEs carry no minor, so many Z-stream product_ids collapse onto one CPE

(enterprise_linux:8::appstream <- 27 product_ids). 12,578 grain rows (1.05% of fixed) get more than one candidate EVR; the design keeps the max EVR by rpm comparison and flags the row with EVR_COLLAPSED.

• changes.csv and deletions.csv are snapshots, not logs. Verified: 64,176

rows / 64,176 unique paths and 280,797 rows / 280,797 unique paths, zero duplicates in either, both sorted newest-first. So changes.csv is "every currently published document with its last-modified time", and its oldest timestamp (2025-08-22) is just the least-recently-touched document — not a retention boundary. Incremental recovery works from any watermark; a snapshot simply cannot express deletion, which is why deletions.csv is mandatory. (An earlier reading of this as a rolling 1-year window was wrong and has been corrected in the wiki page and Confluence DT/4209377355 v2.)

• 96 distinct module:stream pairs, 663,205 references (virt:rhel 112,459,

mysql:8.0 70,737, container-tools:rhel8 56,086). Module gating is the single largest false-positive control.

• Parse throughput: 786 docs/s single process, 4,526 docs/s on 8 — the whole

corpus parses in ~1.3 minutes on one core. Re-collection is cheap enough to repeat whenever a scoping decision changes.

Design output: 4 tables (_DOC, _PRODUCT, _MODULE, fact) with surrogate PRODUCT_KEY/MODULE_KEY so the clustered PK is (PRODUCT_KEY, PACKAGE_NAME, MODULE_KEY, CVE_ID) and the host-match query is a covered range scan. Staleness is handled by document-scoped replace rather than tombstones. Full DDL, normalization vocabulary, matching spec and W34–W39 phase plan are in the wiki page.

Working corpus left at <scratchpad>/vex/ (16 GB) with analyze.py, scope.py, bench.py for the W34 prototype. Delete when done.

2026-08-18 — RHEL VEX collection investigation

Sources read: Confluence 4149674090 (VEX 수집 방법), 4093706274 (OVAL 분석 / per-distro capability matrix), 4191846604 (AI엔진팀 협의 2026-08-04).

Code read: etl_components/os_pkg_vuln/rhel_crawler/* in labrador-scrapers and dags/os_pkg_rhel_vuln.py in labrador-data-platform (schedule 40 */4 * * *, raw task then parse task, max_active_runs=1).

Verification method: downloaded CVE-2024-42472 and CVE-2022-21607 from the public CSAF VEX endpoint and replayed the crawler's own regexes against them, then cross-checked the resulting rows in gatheringdb.TB_OS_PKG_VULN_RHEL. Every defect above has either a parsed-output example or a live row behind it.

No code was changed. Output is the design/analysis page and this log entry.

2026-08-20 (2) — Confluence-to-wiki and portfolio parity

Read the current rendered Confluence pages rather than relying on the 2026-08-18 worklog summary:

• DT/4209377355 [os 패키지 취약점 - v4] Red Hat was updated again on

2026-08-20. The current document has three tables (_CPE, fact, _DOC), a fact key of (CPE, PACKAGE_NAME, MODULE, CVE_ID), no serving-side view, and a row-diff physical apply. The full wiki body already contained those changes, but its AI Summary still described the earlier dimensions/view and DELETE+INSERT draft. The summary now matches the body and Confluence.

• DT/4211146807 크롤러 수집 사이트 현행화 및 법적 이슈 사전 검토 and its

one-page child DT/4212457542 were published/updated on 2026-08-20. The durable wiki now records the source-routing model and its evidence limits; private comments and personal identifiers were not copied.

Durable outputs:

• ai/sources/confluence/2026-08-20-rhel-vex-redesign.md
• ai/sources/confluence/2026-08-20-crawler-source-legal-review.md
• ai/wiki/projects/crawler-source-governance.md
• updated ai/wiki/projects/rhel-vex-vuln-collection.md
• updated ai/repo-notes/labrador-scrapers.md

Human outputs:

• human/portfolio/items/rhel-vex-collection.html
• human/portfolio/items/crawler-source-governance.html
• matching index cards and reproducible SVG/Mermaid diagrams

Portfolio verification:

• Recomputed all filter counts (52 total; 41 LabradorLabs) and checked every

relative link.

• Captured all five changed pages at 375, 768, and 1280 px. Browser checks

confirmed no body overflow or broken images, a 19/52 security-filter result, visible keyboard focus, and focusable horizontal exploration for dense mobile diagrams.

• Preserved Korean word units in narrow headings and raised the shared purple

accent used on the four changed detail pages to WCAG-AA contrast.

• Visual evidence is stored under

.omo/evidence/confluence-portfolio-sync-2026-08-20/.

Evidence boundaries kept in the portfolio:

• The RHEL page reports corpus measurements and design estimates, not a

production cutover or measured false-positive reduction.

• The crawler-source page reports a technical pre-review, not legal advice or a

confirmed removal of every restricted source.

2026-08-18 (4) — CJ-family and Vroong rejection status + retrospective

• User-confirmed outcome: CJ ENM Mnet Plus and CJ Olive Young are rejected;

Vroong must be treated as rejected rather than presumed silent rejection.

• CJ Olive Young remains a verified 2026-08-04 document-screen rejection. CJ

ENM and Vroong notification dates, channels, and selection stages are Needs confirmation because the user did not supply those details.

• Updated all active status surfaces: human/career/2026-지원-현황.md,

human/career/index.html, and career/최종 이력서/README.md. Submitted final PDFs remain under their dated folders; deprecated/ continues to hold only no-application decisions.

• Preserved the user update in

ai/sources/career/2026-08-18-cj-vroong-application-results.md and compiled the durable result and decision rules into ai/wiki/projects/2026-career-transition.md.

• Added the Korean retrospective at

human/reports/2026-08-18-cj-vroong-rejection-retrospective.md. No employer- specific reason was available, so it labels causality as inference. Strongest shared signal: central production Kafka/streaming, distributed/data-lake, and DW/Data Mart requirements were not directly evidenced, while verified depth is batch pipelines, data integrity, MySQL, Airflow/Kubernetes, and batch CDC.

• Next resume revision is scoped, not implemented: rebuild the first page around

three direct operating signals, remove repeated CDC narration, and use the recovered space for stakeholder, decision, and observed-result evidence.

2026-08-20 (3) — Kmong Data Engineer tailored application package

Captured Remember posting 334810 for Kmong's Data Engineer role, due 2026-08-30, and built the working package under career/맞춤이력서/크몽-Data-Engineer/.

Verified alignment:

• Supported must-have rows: 8/10. The package directly evidences Python,

Airflow, Kubernetes, batch collection, data extraction/reconciliation, data infrastructure, EC2-scoped AWS operations, and a ChatGPT API classification stage.

• Supported preferred rows: 2/7. AI/LLM product integration and a security data

product are supported.

• Explicit gaps: Spark, production recommendation/ranking training and serving,

formal data warehouse/data mart design, Terraform, Personalize/SageMaker, Feature Store, real-time streaming, and recommendation/advertising domain experience.

Package evidence and QA:

• evidence-audit.md: 52/52 resume claims, 21/21 numeric claims, and 13/13

portfolio-only claims mapped to durable sources.

• resume.pdf: A4, two pages, canonical ATS template, extractable single-column

reading order.

• portfolio.pdf: A4, six pages, with the DML control/data path, source

reconciliation, bounded LLM classification scope, and MySQL operating cases.

• qa/round12/: eight final page captures, all manually inspected. Both the

design-system/functional and CJK/fidelity lanes passed.

• Independent final reviewers passed at CEO 96, CTO 97, and Tech Lead 98.
• All direct public-detail links in the PDFs return HTTP 200 and match their

linked claims. Direct links to locally corrected but not yet deployed AI license and crawler pages were removed.

• Strict human-voice check: BAN 0 / STRUCT 0 / WARN 0 across nine files.

The deterministic pipeline score is 94.7/100. Evidence, narrative, ATS, and artifact integrity are full-score; posting alignment is 19.74/25 because the verified job-fit gaps cap the result below the required 95. The package remains in the working directory and was not copied to career/최종 이력서/2026-08-20/크몽/.