crawler-lib-rust

# crawler-lib-rust

AI Summary

Purpose:

• Design spec + repo note (POC/BMT) for a Rust package crawler that mirrors the

Go crawler (crawler-lib-golang) data format and pipeline, but redesigns ONLY the LIST stage for crates.io.

• Code SCAFFOLDED 2026-06-02 at ~/labrador/crawler/crawler-lib-rust (Python

crawler + Rust toolchain in image). Pure-helper tests pass (10/10); all modules byte-compile.

• VERIFIED END-TO-END 2026-06-02 against the live gatheringdb

(211.115.125.165:43316, MySQL 8.4.0) + real crates.io. The 3 tables were created there. Repo config.ini keeps NO creds (placeholder restored after). - seed quick-xml: 102 versions -> 102 LIST/VERSION rows (license MIT, deps json 13, SCM, RELEASE_DATE) + PRODUCT (LATEST 0.40.1, COMP_STATUS=1). - rhwp full collection: 14 tags resolved -> 180 unique external crates -> enumerated ALL versions -> 9161 (crate,version) LIST events, all processed (8878 collected + 283 skip-existing, 0 not_found, 422 yanked flagged), 9161 VERSION rows + 180 PRODUCT rows. Top by version count: syn 350, serde 315, serde_derive 270, cc 210, libc 202, serde_json 182. ~20 min wall-clock (bulk = remote DB upserts; API/index calls cached per crate).

• DECISION: rhwp-lock collects ALL versions of each discovered external crate

(not just the lock-pinned version). The lock identifies WHICH crates rhwp uses (across all tags); the worker then collects every version of each. Matches the earlier "all versions" grain choice. RHWP_TAGS records the referencing tags.

• BUG FIXED 2026-06-02: canonical_crate_name must NOT fold _->-. crates.io

treats -/_ as equal only for publish-time UNIQUENESS, but the sparse index file and API key use the author's actual separator (serde_json index is se/rd/serde_json). Folding made all underscore crates 404 (not_found). Fix: canonical = lowercase only, separator preserved. First rhwp run lost 20 underscore crates to this; re-run after the fix collected them.

• Clean user-facing docs (no cross-language comparison) live IN THE REPO:

README.md, docs/schema.md (tables + how stored, BMT framing), docs/ONBOARDING.md. This wiki note keeps the design rationale + Go lineage for AI agents.

Key points:

• POC scope: NOT the whole crates.io registry. Start from the rhwp repo

(github.com/edwardkim/rhwp, Rust HWP/HWPX viewer) and from a user-supplied seed crate list. Collect every version + full info of the external crates those two sources surface.

• LIST has TWO input modes feeding one event-grain queue:

1. rhwp-lock: for each of rhwp's 14 tags, resolve a Cargo.lock and take the transitive external-crate closure. 2. seed: for each user-supplied crate name, register ALL its versions from the sparse index (yanked included, flagged).

• VERSION / PRODUCT tables keep the Go column format unchanged; only LIST is

Rust-specific. Identity collation FLIPS vs Go: crates.io names are case-insensitive and -/_-insensitive, so Rust identity columns are utf8mb4_0900_ai_ci with normalization (Go uses as_cs).

• Data sources follow Go's "existence vs enrichment" split: sparse index

(index.crates.io) is authoritative for existence/versions/deps/yanked; crates.io JSON API is enrichment-only.

• Rate limit: global token bucket 1 req/s on the JSON API only; sparse index

(CDN) looser + conditional GET; mandatory contact User-Agent; 429/Retry-After backoff via the Go-style PROCESSED retry ladder.

• Incremental is NOT built in this POC; §"Incremental (future)" records the plan.

Relevant when:

• Scaffolding or working on the Rust crawler.
• Reviewing the LIST schema, data-source policy, or rate-limit strategy.

Do not read full document unless:

• You need the exact table columns, status codes, source mapping, or the

resolver/worker flow.

Linked documents:

• ai/workspace/repos.md
• ai/repo-notes/crawler-lib-golang.md (the format/pipeline this mirrors)
• ai/worklog/2026/2026-W23.md

Repo Info

• Repo ID: crawler-lib-rust
• Main branch: TBD (likely master, follow Go repo convention)
• Local path: ~/labrador/crawler/crawler-lib-rust (see ai/workspace/repos.md)
• Remote: Unknown (Needs confirmation)
• Language: Python 3 crawler (mirrors Go repo) + Rust toolchain in the image

(for cargo generate-lockfile during resolution). MySQL 8 backend (gatheringdb).

POC Scope

• BMT/POC, not a full-registry crawl. Two list sources only:

1. rhwp repo — all 14 tags (v0.2.1-ext, v0.5.0, v0.6.0, v0.7.0, v0.7.2, v0.7.3, v0.7.6..v0.7.13). 2. User-supplied seed crate names.

• For every external crate surfaced, collect ALL versions + full info.
• "External crate only": the root crate rhwp and any path/git deps are

excluded (only source = registry+crates.io packages are collected).

Data Source Mapping (Go vs Rust)

Keep Go's principle: existence is decided by the authoritative index, NOT by the metadata API. A metadata 404 does not mean the crate/version is missing.

Sparse index prefix rule (names lowercased): 1-char -> 1/{name}, 2-char -> 2/{name}, 3-char -> 3/{first}/{name}, 4+ -> {first2}/{next2}/{name}.

Source-of-truth rules (mirror Go):

1. crates.io JSON API is enrichment only. Never use it to decide existence.
2. The sparse index decides crate/version existence.
3. If the JSON API is 404 but the sparse index has the version, still collect.
4. yanked versions still exist: register them in LIST with YANKED=1; collect

them (do not silently drop).

crates.io Name Identity (the flip vs Go)

• Go module paths are case-sensitive -> Go identity columns are as_cs.
• crates.io crate names are case-insensitive AND treat - == _ for

uniqueness (Foo_Bar == foo-bar). So: - Rust identity columns (LIST CRATE_NAME, VERSION/PRODUCT PRODUCT_KEY) use utf8mb4_0900_ai_ci. - Normalize the name before hashing/storing the canonical key: lowercase + map _ -> - (store canonical; keep the original display name in a separate column if needed). The dedup hash uses the canonical name.

• This is the single biggest schema difference from Go. Do NOT copy Go's

as_cs policy.

Tables (LIST = Rust-specific; VERSION/PRODUCT = Go format kept)

All created in gatheringdb. LANGUAGE='Rust', REPOSITORY='Cargo' (Needs confirmation — Go used 'Golang').

TB_COMP_LIB_RUST_LIST (NEW shape, event grain)

One row per (canonical crate name, version) event. PK is a byte hash to avoid index key-length limits and give idempotent upsert.

Indexes: CRATE_NAME, PROCESSED, TIMESTAMP, LAST_UPDATED. Table default charset utf8mb4/ai_ci; identity columns explicitly ai_ci.

TB_COMP_LIB_VERSION_RUST (gatheringdb)

Same column set as gatheringdb.TB_COMP_LIB_VERSION_GOLANG (LANGUAGE, REPOSITORY, PRODUCT_KEY, VERSION, NAME, DESCRIPTION, LICENSE, LICENSE_AI, DEPENDENCIES json, URL, SCM json, ..., RELEASE_DATE, PROCESSED). Grain (crate, version). Identity (PRODUCT_KEY, VERSION) is ai_ci. PK (LANGUAGE, REPOSITORY, PRODUCT_KEY, VERSION).

TB_COMP_LIB_PRODUCT_RUST (gatheringdb)

Same column set as gatheringdb.TB_COMP_LIB_PRODUCT_GOLANG. Grain = crate. LATEST_VERSION last-write-wins. Identity ai_ci. PK (LANGUAGE, REPOSITORY, PRODUCT_KEY).

Architecture (mirrors the Go crawler)

New repo crawler-lib-rust @ ~/labrador/crawler/crawler-lib-rust, same layout as Go (src/ai/labradorlabs/{app,crawler,db,input,util}, sql/, tests/, Dockerfile, docker-compose.yml). Image includes a Rust toolchain for the resolver.

Resolver / Indexer (replaces Go's index walker) — two modes

1. rhwp-lock mode:

- Input: rhwp repo URL + the 14 tags. - rhwp has NO committed Cargo.lock at any tag and is a single crate (no [workspace]). So per tag: checkout -> cargo generate-lockfile (or cargo metadata) -> parse the generated Cargo.lock. generate-lockfile yields the union across targets/features/optional, which is the desired "everything rhwp could pull" set. cargo's index traffic hits the CDN sparse index, NOT the rate-limited JSON API. - Keep only source = "registry+https://github.com/rust-lang/crates.io-index" packages; drop the root crate and any path/git deps. - Upsert each (name, version) into LIST with SOURCE='rhwp-lock', recording the referencing tag(s) in RHWP_TAGS.

1. seed mode:

- Input: a list of crate names (env RUST_SEED_CRATES="a,b,c" and/or a seeds.txt file — exact form is an open question below). - For each name: ONE sparse-index request -> read every version line -> upsert each (name, version) into LIST with SOURCE='seed', RESOLVE_KIND='seed', YANKED per the index. Collect ALL versions including yanked.

Both modes write the same event-grain LIST; the dedup PK sha256(canonical_name@version) makes overlapping rhwp/seed events merge idempotently into one row.

Worker (mirrors Go dataCrawlingV2)

For each claimed LIST row (GET_LOCK queue lock + per-event lock + PROCESSED ladder, same as Go):

1. If skip mode and (crate, version) already in VERSION table -> PROCESSED=1.
2. Fetch the sparse index line for the version: deps, features, yanked,

checksum (authoritative).

1. Enrich from the JSON API (rate-limited): description, repository, homepage,

documentation, license, categories, keywords, downloads.

1. Build VERSION row (Go column format; map crate deps into DEPENDENCIES

json) + PRODUCT row (LATEST_VERSION last-write).

1. Persist VERSION + PRODUCT + the resolved LIST row in one transaction.

Rate Limit Strategy

• Global token bucket 1 req/s on the JSON API only (crates.io/api/...).
• Sparse index (CDN) looser (e.g. 5-10 req/s) + conditional GET

(If-None-Match / If-Modified-Since).

• Mandatory User-Agent identifying the crawler + contact, e.g.

labrador-rust-crawler/0.1 ([email protected]). A generic/empty UA risks being blocked.

• 429 handling: honor Retry-After, exponential backoff, route to the Go-style

retry ladder (PROCESSED 30-34).

• POC volume estimate: 14 tags' union + seed list ~= a few hundred unique

crates, each with all versions. Sparse-index-dominated, so wall-clock is minutes. Even the 1 req/s JSON API over a few hundred crates is minutes.

• Multi-worker shared token bucket (DB/Redis) is a FULL-registry concern only;

POC runs a single worker.

Incremental (future — NOT built in this POC)

• rhwp-scoped: on a new rhwp tag, re-run the resolver; only new

(crate, version) events enter the queue (dedup PK). RESYNC=1 forces re-collection of a specific event (mirror Go).

• full-registry: per-crate conditional GET (ETag/Last-Modified) against the

sparse index, plus a daily db-dump.tar.gz diff. Multi-worker coordination via a shared (DB/Redis) token bucket.

Code Map (scaffolded 2026-06-02)

Repo ~/labrador/crawler/crawler-lib-rust, layout mirrors the Go crawler.

• src/ai/labradorlabs/app/main.py — entry: RustCrawler().run().
• src/ai/labradorlabs/crawler/resolver.py — fills LIST. resolve_seeds

(name -> sparse index -> all versions), resolve_rhwp + _resolve_tag (checkout tag -> cargo generate-lockfile -> parse lock -> external crates), load_seed_names (env RUST_SEED_CRATES + seeds.txt, both).

• src/ai/labradorlabs/crawler/rustCrawler.py — worker. _process (skip-exists

-> sparse index authoritative -> JSON API enrich -> write), _build_version_row, _build_product_row. Per-crate API cache (one JSON-API call per crate).

• src/ai/labradorlabs/crawler/crates_client.py — get_index_versions

(authoritative), get_crate_api (enrich). 429/Retry-After backoff.

• src/ai/labradorlabs/crawler/rate_limiter.py — token-ish limiter (1/s API).
• src/ai/labradorlabs/crawler/rust_utils.py — canonical_crate_name

(lowercase + _->-), crate_list_id (sha256), sparse_index_path, parse_cargo_lock (dependency-free), map_index_dependencies, normalize_release_date. Unit-tested: tests/test_rust_utils.py (10 pass).

• src/ai/labradorlabs/db/rustDBVo.py — upsert_list_events, claim_pending

(PROCESSED IN 0,30-34; single-worker, no GET_LOCK yet), set_processed, version_exists, write_version_and_product (VERSION+PRODUCT one txn).

• src/ai/labradorlabs/input/crawlerInfo.py — RustCrawlerInfo.LIBRARY_INFO

(endpoints, rates, UA, tables, 14 rhwp tags, seed sources).

• db/base.py, util/config.py, util/logger.py — reused from the Go repo.
• sql/ — the three DDLs. Dockerfile (ubuntu + python3 + rustup),

docker-compose.yml, seeds.txt, README.md, docs/schema.md, docs/ONBOARDING.md.

License standardization (added 2026-06-03)

Fills VERSION columns STANDARD_LICENSE, LICENSE_IDS, LICENSE_AI — ported from the Go crawler's license logic.

• Reference table gatheringdb.TB_LICENSE_V2 (1179 SPDX rows: ID, NAME_SHORT,

NAME, MAPPING regex json, URL). Loaded via rustDBVo.load_license_ref.

• crawler/license_utils.py (pure, 13 tests): parse_spdx_expression

(splits crates.io SPDX exprs — OR/AND//, parens, drops WITH <exc>), build_license_index + match_licenses (exact NAME_SHORT/NAME then MAPPING regex -> STANDARD_LICENSE=[{name,url}] json, LICENSE_IDS=compact [id,...] json), normalize_ai_license_value (Go port).

• crawler/ai_license.py AiLicenseClient — POSTs license texts to the

in-house AI API (AI_LICENSE_URL, default http://211.115.125.171/..., same endpoint as the Go repo; override via RUST_AI_LICENSE_URL). Returns the AI_LICENSE id list as json, or None if unreachable. -> LICENSE_AI.

• Worker (rustCrawler._build_version_row) now fills all three for NEW

collections, cached per license-expression string (1 AI call per distinct expr, not per version).

• Backfill for already-collected rows: crawler/licenseBackfill.py +

app/main_license_backfill.py (computes STANDARD/IDS from stored LICENSE + ref, no crates.io refetch; AI via API). RUST_LICENSE_BACKFILL_ALL=1 redoes all rows; default only fills missing.

• VERIFIED on gatheringdb (2026-06-03): backfilled 5165 rows, 5151 matched a

license. Totals over 9161 VERSION rows: LICENSE 9152, STANDARD_LICENSE/ LICENSE_IDS 5151, LICENSE_AI 4999 (AI API was reachable from python and answered most; a few timed out -> NULL). Examples: base64 MIT OR Apache-2.0 -> IDS [247,32], AI [32,247]; encoding_rs (Apache-2.0 OR MIT) AND BSD-3-Clause -> IDS [32,247,43]. Rows whose crates.io per-version license is null stay unmatched (data reality, not a bug).

Vulnerability collection (added 2026-06-02)

• Table gatheringdb.TB_COMP_LIB_VULN_CARGO_V1 (renamed from ..._VULN_RUST_V1

2026-06-03 to match the ecosystem naming *_CARGO_*) — same schema as TB_COMP_LIB_VULN_PYPI_V1 (PK (PRODUCT_KEY, VULN_ID), VULN_RANGE text, MAPPED_TYPE enum). For Cargo, MAPPED_TYPE='RE' (RustSec).

• NOTE: the shared gatheringdb runs a prod job over *_V1 vuln tables that

appears to keep CVE-format VULN_IDs only. Overnight it dropped our 6 non-CVE rows (RUSTSEC-/GHSA-only ids), leaving 25 CVE rows. So in *_V1, non-CVE advisories don't persist (matches the PyPI table being all CVE-...); the RUSTSEC/GHSA-only ones would belong in a RAW-style table.

• Source: OSV crates.io dump

(https://osv-vulnerabilities.storage.googleapis.com/crates.io/all.zip, ~2479 advisories). OSV aggregates RustSec + GHSA; VULN_ID prefers the CVE alias, else RUSTSEC, else the OSV id (GHSA).

• crawler/vuln_utils.py (pure, tested): osv_pick_vuln_id,

osv_affected_to_range (OSV SEMVER events -> interval notation (,X) / [a,b) / a] / union with |; treats introduced 0/0.0.0/ 0.0.0-0 as unbounded lower), osv_advisory_to_rows. Tests: tests/test_vuln_utils.py (13).

• crawler/vulnCollector.py — downloads/loads the OSV zip, parses, filters

(scope collected = crates in PRODUCT, default; all = whole registry), upserts via rustDBVo.upsert_vuln_rows. Entry app/main_vuln.py.

• VERIFIED on gatheringdb (scope=collected): 2479 advisories -> 31 vuln rows

across 18 of the 180 rhwp crates (e.g. smallvec 5 CVEs, tar 5, image 2; ranges like rustix CVE-2024-43806 = [0.35.11,0.35.15)|...|[0.38.0,0.38.19)).

Open Questions

• LANGUAGE/REPOSITORY RESOLVED -> rust / Cargo (in crawlerInfo).

Change there if a different convention is wanted.

• End-to-end RESOLVED (2026-06-02): seed quick-xml ran clean against the live

gatheringdb. rhwp-lock mode (cargo generate-lockfile per tag) NOT yet run end-to-end (needs the Rust toolchain / clone) — only the seed path is verified on real infra so far. config.ini DbUrl is a placeholder (no committed creds).

• Multi-worker claim locking (GET_LOCK + processing-status ladder) is NOT built;

claim_pending is single-worker. Fine for POC; needed for scale-out.

• VERSION_TABLE_DEPENDENCY / license-standardization / vuln columns are out of

BMT scope (left NULL).

• rhwp remote/clone access: public HTTPS clone assumed.

Decisions

• 2026-06-02: POC scope = rhwp 14 tags (transitive external crates via resolved

Cargo.lock) + user seed crate names; collect ALL versions of each external crate (yanked included, flagged). Confirmed with user.

• 2026-06-02: keep Go VERSION/PRODUCT column format; redesign only LIST for

crates.io. Confirmed.

• 2026-06-02: identity collation flips to ai_ci + name normalization

(lowercase, _->-) because crates.io names are case/-/_-insensitive — the inverse of Go's as_cs.

• 2026-06-02: existence from sparse index, enrichment from JSON API; rate limit

1 req/s on JSON API + contact User-Agent + 429/Retry-After backoff.

• 2026-06-02: incremental deferred; design recorded only.
• 2026-06-02: scaffolded the repo (Python, mirrors Go layout). Confirmed

decisions: seed input via BOTH env (RUST_SEED_CRATES) and file (seeds.txt); seed collects ALL versions incl. yanked (flagged); rhwp-lock resolves per tag via cargo generate-lockfile (Rust toolchain baked into the image); LANGUAGE=rust, REPOSITORY=Cargo; remote left unknown (set at push time).

• 2026-06-02: user-facing docs kept cross-language-free and in-repo

(docs/schema.md, docs/ONBOARDING.md, README.md) per user request; this wiki note retains the Go lineage as AI design memory.