Crawler / Collection DB Target (MUST READ before any collection work)

# Crawler / Collection DB Target (MUST READ before any collection work)

AI Summary

Purpose:

• Hard rule for which database all collection / crawling / missing-product /

missing-version backfill work writes to.

Key points:

• Collection (crawling) MUST target 211.115.125.165:43316 (the crawler DB).
• Do NOT write collected data to 211.115.125.164:53306. 164 is a separate

secondary DB (the data_check_script/.env shared default), not the crawl target.

• On 165 the schema is split — product/version in labradordb, license in gatheringdb.
• Every *_missing_products tool MUST use a LOCAL .env pointing to 165. If a tool

has no local .env it falls back to the shared ../data_check_script/.env (= 164) and silently writes to the wrong DB.

Relevant when:

• Running any collector: maven_missing_products, rubygems_missing_products,

composer_missing_products, or any crawler-lib backfill.

• Setting up a new collection tool / .env.

Do not read full document unless:

• You need the exact tables / the incident history.

Linked documents:

• ai/workspace/repos.md
• ai/repo-notes/crawler-lib-ruby.md, crawler-lib-java.md
• ai/worklog/2026/2026-W25.md, 2026-W24.md

Details

The rule

165 schema layout (per language collector)

• product → labradordb.TB_COMP_LIB_PRODUCT (LANGUAGE, REPOSITORY)
• version → labradordb.TB_COMP_LIB_VERSION_<LANG> (e.g. _JAVA, _RUBY, _PHP)
• license → gatheringdb.TB_LICENSE_V2 (NOT in labradordb on 165)

So a collector connects with DB_NAME=labradordb (product/version unqualified resolve there) but MUST reference the license table as gatheringdb.TB_LICENSE_V2.

Per-tool .env requirement

Each tool loads a LOCAL .env first, else the shared ../data_check_script/.env:

• maven_missing_products/.env → 165 ✓
• rubygems_missing_products/.env → 165 ✓
• composer_missing_products/.env → 165 ✓ (added 2026-06-17; previously missing → see incident)

If you add a new collector, copy a 165 .env (e.g. from rubygems_missing_products/.env). Never rely on the shared data_check_script/.env for a write target.

Incident history (why this is nailed down)

• 2026-W24 (npm missing): data_check .env (164/labradordb) was used by mistake;

target corrected to 165 (crawler config.ini).

• 2026-06-17 (composer version backfill): composer_missing_products had no local

.env, fell back to shared data_check_script/.env (164), and wrote ~37k products / 279k version rows to 164 before being caught. Stopped, added a local 165 .env, re-ran on 165. The 164 rows were left in place (to be cleaned up separately) per user.

Open Questions

• Whether to delete the mistakenly-written 164 composer rows (user: clean up later).