Vulnerability Collection (OSV + OS Package Trackers)

# Vulnerability Collection (OSV + OS Package Trackers)

AI Summary

Purpose:

• Capture durable design knowledge for collecting open-source and OS-package

vulnerabilities from multiple upstream sources (OSV, Alpine secdb, Debian Security Tracker, Ubuntu CVE Tracker) and normalizing them into the company vulnerability database.

Key points:

• Two collection families: (1) library/OSS vulnerabilities via OSV, and (2)

OS-package vulnerabilities per distro (Alpine, Debian, Ubuntu).

• Shared strategy: raw-data-first + SHA256 hash check + diff analysis. Store the

raw upstream artifact dated, hash it, and only re-store + diff when the hash changes. The diff yields inserted / updated / deleted, which is essential because a flat "download the whole file" approach cannot detect deletes (rejected/withdrawn CVEs) and leaves stale rows.

• Critical cross-source insight: fixed_version = "0" (in Alpine secdb and

Debian tracker) means NOT AFFECTED / false positive, NOT "version 0.x" and NOT "unfixed". Misreading this caused data-integrity problems; it must map to a distinct not_affected status.

• Status model (per source, roughly): fixed, not_affected,

open/potentially_vulnerable, and a possibly_fixed grace state (7 days) before deciding fix vs delete.

• OSV-specific: a two-stage pipeline (raw crawler → scraper) with a normalized

relational schema (TB_VULN_OSV* tables), PURL-based package dedup, and commit-ID-to-git-tag resolution with multi-token GitHub/GitLab rate-limit rotation.

Relevant when:

• Building or debugging any vulnerability crawler, designing the vuln DB schema,

handling CVE rejection/deletion, version comparison, or git-tag mapping.

Do not read full document unless:

• You need the full OSV table-by-table schema or per-distro status tables.

Linked documents:

• [[license-collection.md]] (sibling data-collection pipelines)
• [[labrador-platform.md]] (crawlers run as Airflow-orchestrated images using

labrador-sqlmodel)

• [[index.md]]

Open Questions

• Implementation status differs by source: Ubuntu and Debian crawlers report

completed; Alpine was at design/POC stage as of the source notes. Needs confirmation of current production state.

Details

Shared raw-diff + hash strategy

1. Download the upstream artifact (JSON or crawled HTML, or a git repo).
2. Compute SHA256 over the raw content.
3. If hash equals the previous stored hash → skip (no storage, no diff).
4. If different → store the dated raw file ({YYYY}/{mm}/... layout) and diff

against the previous raw to classify inserted / updated / deleted.

1. Apply only the changed records to the DB.

Benefits: dedup storage, dated history for rollback/audit, and reliable delete detection (rejected CVEs) — the main weakness of full-snapshot loading.

fixed_version = "0" = Not Affected

Both Alpine secdb (secfixes) and Debian tracker use "0" to mean the CVE does NOT affect the package (false positive), not an unfixed/0.x version. Map it to status = not_affected (is_false_positive = true); real versions → status = fixed. Existing rows had to be re-classified.

OSV (library / OSS) collection

• Two-stage: OsvRawCrawler pulls raw JSON into TB_VULN_OSV_RAW

(PROCESSED=0) and the filesystem; OsvCrawlerScraper reads unprocessed rows, normalizes, and sets PROCESSED=1. Separating fetch from parse isolates failures and enables batch processing.

• Incremental: use OSV modified_id.csv with last_sync_timestamp; break

timestamp collisions with last_processed_id.

• Schema (normalized): TB_VULN_OSV (core), _AFFECTED_PACKAGES (PURL-unique

per vuln), _AFFECTED_RANGES (1:N, no unique — keep all ranges/events), _REFERENCES, _ALIASES, _RELATED, _REPO (+ _REPO_MAP N:M), plus TB_CRAWLER_STATUS. Raw kept in _RAW with PRE_RAW_DATA (last 10 versions) for audit.

• Repo URL extraction priority: affected[].ranges[].repo (GIT) →

database_specific.source → references[FIX/PACKAGE/WEB], parsed by regex across many hosts (GitHub, GitLab, Gitee, googlesource, kernel.org, etc.). OSS_ID mapped by joining REPO_URL against labradordb.TB_COMP_FILE_VALID_V6.SOURCE_URL (strip .git both sides).

• Concurrency safety: per-VULN_ID queues, transaction-per-vuln,

INSERT ... ON DUPLICATE KEY UPDATE (PURL) to avoid delete+insert lock contention, deadlock retry with backoff, batch-retry cap of 3.

• Commit→tag resolution (earlier filecomp OSV work): when OSV events use commit

IDs, resolve the commit via /commits/{id} then match its SHA against the tags list; rotate multiple GitHub/GitLab tokens on 401 (invalidate) / 403 / 429; compute affected tag ranges from introduced / fixed / last_affected events.

Alpine collection

• Sources: secdb.alpinelinux.org (JSON, fixed only) + crawl

security.alpinelinux.org/branch/{ver}-{repo}/ for unfixed (potentially_vulnerable, includes orphaned packages). Versions v3.2-v3.22, main+community (42 pages), edge excluded.

• Status flow: potentially_vulnerable → disappears → possibly_fixed

(7-day grace) → secdb check → fixed / not_affected / DELETE.

Debian collection

• Source: security-tracker.debian.org/tracker/data/json (huge, hundreds of

MB — consider streaming parse e.g. ijson). Status from status(resolved/open) + fixed_version.

• Version comparison MUST follow dpkg algorithm: ~ suffix = lower, + = higher

(e.g. 5.3.28+dfsg1-0.8 > 5.3.28-13.1). Use apt_pkg / debian.debian_support rather than hand-rolling — a real false-detection bug came from naive comparison.

Ubuntu collection (completed)

• Source: git.launchpad.net/ubuntu-cve-tracker (deb822-style files in

active/ and retired/); version codename→number via changelogs.ubuntu.com/meta-release.

• Git-based incremental: store last commit id+time in TB_CRAWLER_STATUS.RAW_DATA;

on each run pull + git diff to find changed/deleted files.

• Robustness engineering (durable lessons):

- Status saved only AFTER records persist (pending_status_save flag) to keep state and data consistent. - Missing-commit fallback (shallow clone / history rewrite): remote fetch → time-based search (last_commit_time, up to 1000 commits) → change-based search (up to 100 commits) → HEAD parent. Avoids unshallow (caused HTTP 503 + long runs) and avoids full re-collection. - MySQL 8.4 / Launchpad: git pull HTTP 503 + gnutls handshake fixed by Docker network MTU 1400, forcing http.version HTTP/1.1, larger postBuffer, and exponential-backoff retry / re-clone. - Status map: needed/pending/deferred/ignored/needs-triage → vulnerable (PATCHED_VERSION=0); released → fixed; not-affected/DNE → safe; delete_vuln → removed upstream.