crawler-lib-golang

# crawler-lib-golang

AI Summary

Purpose:

• Durable note for the Go module crawler (crawler-lib-golang).
• Records the TB_COMP_LIB_GOLANG_LIST_V2 redesign: event-grain queue, declared module path resolution, and the skip-existing-version mode.

Key points:

• index.golang.org is treated as an append log of (Path, Version, Timestamp) publish events.
• Existence is decided by proxy.golang.org (@v/list -> @latest). pkg.go.dev API is metadata-only; a 404 there does not mean the module is missing.
• V2 changes the list table grain from one row per index path to one row per (INDEX_PATH, INDEX_VERSION) event.
• PRODUCT_KEY (declared module path from go.mod) is stored on the same event row. When it differs from INDEX_PATH, the event row gets PROCESSED=6, DEPRECATED=1 (no second row).
• Go module paths are case-sensitive; V2 identity columns use utf8mb4_0900_as_cs so AOC2022 != aoc2022.
• The whole V2 pipeline is gated by GO_LIST_SCHEMA=v2 (default off -> legacy V1 path runs unchanged).
• Worker fetches only the index-reported version (not the full @v/list), with @latest fallback.
• V2 incremental worker (_runV2) claims newest events first (TIMESTAMP DESC, since 2026-06-01). Resync/backfill stays TIMESTAMP ASC (oldest->newest). Queue batch size is 300 (since 2026-06-01; was 100), hardcoded in crawlerInfo.LIMIT_SELECT_QUERY (LIMIT :start, 300) and asserted by test_limit_select_query_claims_300_packages_per_batch — NOT BaseCrawler.LIMIT=1000 (a separate full-select pagination loop). 300 chosen for the main 1 + worker 3 topology: bigger than 100 to cut round-trips, but small enough to keep the per-claim GET_LOCK loop (held under the global QUEUE_CLAIM_LOCK) short and the crash blast radius (events stranded in processing/20, NOT auto-reclaimed) bounded across 3 parallel workers. (UPDATED 2026-06-08, 5670ef4: the per-claim GET_LOCK loop was REMOVED and stuck processing is now auto-reclaimed — see the queue-lock bullet below.)
• Queue-lock claim simplified + stuck reclaim (since 2026-06-08, commit 5670ef4): _claimV2Batch(where, order_by) now holds the global QUEUE_CLAIM_LOCK only for SELECT + bulk PROCESSED=20 mark — the per-event GET_LOCK loop is GONE (redundant: queue lock makes SELECT+mark atomic, so two workers never claim the same batch). This fixes the prod incident where a worker that hung/died mid-claim kept the lock and starved all others (failed to claim queue lock: golangcrawler:queue-claim spam). _reclaimStuckProcessingEvents() resets PROCESSED 20-24 older than GO_PROCESSING_RECLAIM_MINUTES (default 60, by LAST_UPDATED) back to retryable (20->0, 21->30, 22->31, 23->32, 24->33); called at _runV2 start and when a claim returns empty — so crash/hang orphans recover (this also closes the earlier "processing/20 NOT auto-reclaimed" gap). _runV2 no longer uses attempted_ids/lock_skip_offset/per-event locks; _claimEventLocks is now dead code. Tests: test_reclaim_stuck_processing_resets_to_retryable. NOTE: an already-held stuck lock must be cleared manually (KILL <conn> or restart the worker) — the patch only prevents recurrence. Set GO_PROCESSING_RECLAIM_MINUTES > max batch(300) processing time.
• dataCrawlingV2 existence pre-skip is batched (since 2026-06-01): _batchSkipExistingEvents runs ONE row-value IN SELECT (_existingVersionKeys) for the whole batch (300 events), bulk-marks already-present events processed=1 via a single insertPackageListV2, and only scrapes the missing ones. Replaces the old per-event getDBVersionExists (N SELECTs -> 1). Collation parity via _normVersionKey (exact for gatheringdb as_cs, casefold for labradordb ai_ci). The declared-path (step-3) skip stays per-event (store_path known only after go.mod fetch).
• product table grain = 1 row per module (PK (LANGUAGE, REPOSITORY, PRODUCT_KEY)), written via INSERT ... ON DUPLICATE KEY UPDATE (all cols except CREATED/TIMESTAMP/PATH overwritten = last-write-wins), so LATEST_VERSION = the version of the last-processed event for that module. Because the worker claims newest-first (DESC), dataCrawlingV2 sorts each claimed batch ascending by TIMESTAMP before processing (since 2026-06-01, commit 285034a) so the newest version of a module is written last -> LATEST_VERSION stays newest even when several new versions land in one batch. Residual edge: versions of one module split across batch boundaries (rare in incremental mode; backfill stays ASC). Verified by test_product_latest_version_is_newest_under_desc_claim.
• Go version ordering: sort_go_versions / _go_version_key (semver: major.minor.patch, release > prerelease, numeric prerelease ids < alphanumeric, pseudo-versions ordered by their fixed-width 14-digit timestamp). Build metadata (+incompatible, +meta) is stripped before parsing (since 285034a) — Go ignores it for precedence; before the fix v2.0.0+incompatible failed the regex and sorted to the bottom as invalid.
• KNOWN GAP (open, not fixed as of 2026-06-01): the version table SORT_ORDER column is meaningless under V2 event-grain ingestion — getVersionOrderRows(store_path, [version]) is called with a single version per event so every row gets SORT_ORDER=0. getVersionOrderRows does not sort; it indexes the passed list (V1 passed a sort_go_versions-ascending list, so there SORT_ORDER=0=lowest). Fixing under V2 needs a per-row sortable key column or per-module recompute (deferred).
• The case-collision fix is to write to NEW gatheringdb case-sensitive tables (gatheringdb.TB_COMP_LIB_VERSION_GOLANG, gatheringdb.TB_COMP_LIB_PRODUCT_GOLANG, both as_cs identity) instead of altering the shared labradordb tables. Gated by GO_TARGET_GATHERINGDB=1 (default off).
• LIST_V2.RESYNC column controls re-collection: default 0 = check DB and skip if exists; 1 = re-collect (skip DB check). Global override GO_RESYNC_ALL=1.
• Migration of existing data into the gatheringdb tables uses scripts/migrate_golang_to_gatheringdb.py (sharded, keyset, idempotent). Bottleneck is server disk read of wide version rows (~80 rows/s cold, ~900/s warm; product ~3000/s); parallel shards do NOT beat single-stream because storage read bandwidth saturates.
• Merged to master at c6f16a4 (2026-05-29).

Relevant when:

• Working on the Go crawler list table, indexer, or worker.
• Diagnosing "missing" Go modules that are actually alias / subpath / declared-diff / proxy-only cases.

Do not read full document unless:

• You need exact column definitions, status codes, flags, or the per-pattern handling.

Linked documents:

• ai/workspace/repos.md
• ai/worklog/2026/2026-W22.md

Repo Info

• Repo ID: crawler-lib-golang
• Local path: ~/labrador/crawler/crawler-lib-golang
• Language: Python 3 (crawler), MySQL 8 backend (labradordb)
• Tests:

- Pure helpers: python3 -m unittest tests.test_go_crawler_regressions (no DB/network). - V2 integration: tests/test_go_crawler_v2_integration.py (real MySQL + mocked network, 9 tests incl. gatheringdb routing + RESYNC + list case-sensitivity). Skips unless sqlalchemy + TEST_DB_URL are present; run inside the iotcube/golangcrawler image against a throwaway MySQL 8 container. Host python3 -m unittest discover -s tests = 57 tests OK (9 integration skipped).

Terms

• index path: the Path value from index.golang.org.
• declared module path: the value on the module line of go.mod.
• PRODUCT_KEY: the representative module path stored in DB.
• PROCESSED=6: not a failure. Means "this index path is not the representative; data was filed under the declared module path (PRODUCT_KEY)".

Source-of-Truth Rules

1. pkg.go.dev API is for documentation/metadata enrichment only. Never use it to decide existence.
2. proxy.golang.org decides module/version existence.
3. If pkg.go.dev is 404 but proxy @latest or @v/list is alive, still collect.
4. If pkg.go.dev metadata is absent, description/license/repoUrl may be empty.
5. Go module paths are case-sensitive. Do not normalize with LOWER() for identity.
6. Proxy URLs escape uppercase as ! + lowercase (e.g. eduVPN -> edu!v!p!n). See escape_go_module_proxy_path.

TB_COMP_LIB_GOLANG_LIST_V2

DDL: sql/TB_COMP_LIB_GOLANG_LIST_V2.sql in the repo.

Grain: one row per (INDEX_PATH, INDEX_VERSION) event.

Design choices:

• PK is the hash (not the raw strings) to avoid InnoDB index key-length limits and to give idempotent ON DUPLICATE KEY UPDATE.
• Identity columns are utf8mb4_0900_as_cs so case differences are distinct keys (fixes the LOWER() collision trap at schema level).
• @ separator is collision-free because Go module paths/versions cannot contain @.

PROCESSED status codes

• 0 pending, 1 complete, 5 not_found, 6 alias (filed under declared module path)
• 20-24 processing (claim ladder), 30-34 retry ladder

Running V2 (docker-compose)

docker-compose.yml services: main (indexer + worker), worker (worker only, scalable), backfill (profile index, one-shot). All set GO_LIST_SCHEMA=v2, GO_TARGET_GATHERINGDB=1, GO_SKIP_EXISTING_VERSION=1. DB conn comes from src/config.ini DbUrl (not env) — so any server with the repo + image + DB reachability works.

Deploy order (important):

1. Schema once: sql/TB_COMP_LIB_GOLANG_LIST_V2.sql (LIST_V2, includes RESYNC), sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql, sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql — all into gatheringdb. If LIST_V2 already exists without RESYNC, apply sql/ALTER_TB_COMP_LIB_GOLANG_LIST_V2_add_resync.sql.
2. Initial backfill (one-shot, before any worker): GO_INDEX_SINCE=2019-04-10... docker compose --profile index run --rm backfill (fast mode via GO_INDEX_SKIP_EXISTS_CHECK=1 baked into the service). Let it run to completion (auto-exits Exited 0 when caught up to now). Do NOT stop it midway — a partial backfill leaves a gap (main's indexer resumes from the V1 cursor ~now, so the gap is never filled). Resume after a crash with GO_INDEX_SINCE=<cursor> (cursor read from TB_CRAWLER_STATUS TYPE=GoCralwerIndexListBackfill).
3. Migrate existing labradordb data (optional, parallel-safe): scripts/migrate_golang_to_gatheringdb.py.
4. THEN workers: main server -> docker compose up -d main; each worker server -> docker compose up -d worker. ALTER must be applied before workers (claim SELECT reads RESYNC).

4-server topology (main 1 + worker 3): one main host, three worker hosts, each docker compose up -d <service>. They coordinate via the shared DB.

Workers on different servers coordinate purely through the shared DB: per-event GET_LOCK prevents double-processing, and (with GO_TARGET_GATHERINGDB=1) the skip-check + writes hit the gatheringdb as_cs tables (case-sensitive). Caveats:

• A hard worker crash leaves its event at a processing status (20-24), which the claim WHERE (processed IN (0,30-33)) does not re-claim → orphaned until manually reset.
• NEVER run the skip-mode backfill while workers run — it resets PROCESSED of done events to 0.

Feature Flags (env)

• GO_LIST_SCHEMA=v2 -> use the V2 event-grain pipeline (indexer + worker). Default/anything-else -> legacy V1.
• GO_SKIP_EXISTING_VERSION=1 -> if the version already exists, skip crawl and set PROCESSED=1 (the "check DB first" mode). The compose main/worker set this.
• GO_CRAWLER_INDEXER -> existing flag; controls whether the indexer step runs (true=indexer+worker, false=worker only).
• GO_TARGET_GATHERINGDB=1 -> V2 worker writes version/product to the gatheringdb case-sensitive tables (VERSION_TABLE_V2/PRODUCT_TABLE_V2) and runs the skip-check against them (so the check becomes case-sensitive). Default off -> legacy labradordb tables (ai_ci). Added 2026-05-29.
• GO_INDEX_SKIP_EXISTS_CHECK=1 -> backfill/indexer skips the per-event getDBPackageListV2Exists SELECT. ~68x faster initial backfill (33 -> ~2200 events/s) because the per-event remote SELECT was the bottleneck. ONLY for the initial empty-queue backfill (before any worker runs) — re-running it after the worker started would reset processed events' PROCESSED to 0 (the insert is ON DUPLICATE KEY UPDATE). Added 2026-05-29.
• GO_RESYNC_ALL=1 -> V2 worker treats every event as resync (skip DB check, re-collect all). Per-event resync is the LIST_V2.RESYNC column. Added 2026-05-29.

Collection Routine (V2 worker)

For each claimed event (INDEX_PATH, INDEX_VERSION):

1. Compute do_skip_check = skip_enabled AND not (GO_RESYNC_ALL or event.RESYNC). If the event is flagged resync, all DB-existence checks below are bypassed (force re-collect). The version table checked is the gatheringdb as_cs one when GO_TARGET_GATHERINGDB=1 (so the check is case-sensitive).
2. If do_skip_check and (INDEX_PATH, INDEX_VERSION) already exists in the version table -> set PROCESSED=1, skip fetch (cheap pre-check).
3. Fetch the index version's go.mod via proxy (getCanonicalModulePath). If it 404s, fall back to @latest.
4. If still no go.mod, restore the event to the retry ladder.
5. Parse the module line -> declared module path. store_path = declared or INDEX_PATH.
6. Classify with classify_go_index_resolution(INDEX_PATH, store_path).
7. If skip mode is on and (store_path, version) already exists -> set status and skip heavy work (second check).
8. Build the single version row + product + license under store_path.
9. Persist version + product + the resolved event row in one transaction.

- same path: PROCESSED=1, DEPRECATED=0. - different path: PROCESSED=6, DEPRECATED=1, PRODUCT_KEY=store_path.

Claim concurrency mirrors V1: GET_LOCK queue lock + per-event GET_LOCK (keyed by event ID) + processing-status ladder.

"Missing" Pattern Catalog

gatheringdb case-sensitive migration

The downstream version/product tables in labradordb are ai_ci, so case-distinct Go modules collapse (see Known Pitfalls). Rather than ALTER the huge shared tables, the fix routes V2 writes to NEW gatheringdb tables with as_cs identity:

• gatheringdb.TB_COMP_LIB_VERSION_GOLANG — same schema as the labradordb version table, PRODUCT_KEY/VERSION are as_cs. (sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql)
• gatheringdb.TB_COMP_LIB_PRODUCT_GOLANG — Go-only split of the cross-language TB_COMP_LIB_PRODUCT (which serves all languages and must stay ai_ci), PRODUCT_KEY as_cs. (sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql)
• Routing is gated by GO_TARGET_GATHERINGDB=1 (crawlerInfo keys VERSION_TABLE_V2/PRODUCT_TABLE_V2; goCrawler._v2VersionTable/_v2ProductTable; getDBVersionExists(table_name=...)).

Migration of existing rows: scripts/migrate_golang_to_gatheringdb.py.

• Server-side INSERT IGNORE ... SELECT (data stays on the server), keyset pagination on the full PK, idempotent, resumable via output/migrate_state_*.json, throttled, --shard-id/--pk-lo/--pk-hi for disjoint parallel shards, --dry-run/--reset.
• version copies the whole labradordb.TB_COMP_LIB_VERSION_GOLANG; product copies only REPOSITORY='Golang'.
• Performance (this prod, 2026-05-29): version reads ~80 rows/s cold, ~900/s warm; product ~3000/s (narrower rows). The bottleneck is reading wide JSON/text version rows — disk read bandwidth, NOT index maintenance or contention. 8 disjoint shards did NOT beat a single stream (storage bandwidth saturates with one reader). Status as of 2026-05-29: product (~1.66M) fully migrated and verified (case preserved); version migration in progress.
• Pitfall learned: never run overlapping migration INSERTs targeting the same rows — they lock-conflict and one can wedge in query end (unkillable from a non-SUPER client). Run one stream, or strictly disjoint PK ranges.

Code Map (V2 additions)

• src/ai/labradorlabs/crawler/go_utils.py: build_go_index_event_id, classify_go_index_resolution, should_skip_existing_version, is_skip_existing_version_enabled, is_go_list_schema_v2; generalized build_processing_status_update_query/params (key_column/param_prefix).
• src/ai/labradorlabs/db/labradorDBVo.py: getPackageListV2Vo.
• src/ai/labradorlabs/crawler/baseCralwer.py: getDBPackageListV2Exists, insertPackageListV2.
• src/ai/labradorlabs/crawler/goCrawler.py: _isV2ListSchema, _isSkipExistingVersionEnabled, _activeListTable, _buildV2IndexEvents (indexer; honors _isIndexSkipExistsCheck), _runV2 + dataCrawlingV2 (honors RESYNC via do_skip_check) + claim helpers (_claimV2Batch selects RESYNC, _claimEventLocks, _markEventsProcessing, _completeV2Event, _insertV2EventResult). Routing helpers _isV2TargetGatheringdb/_v2VersionTable/_v2ProductTable, resync helpers _isIndexSkipExistsCheck/_isResyncAll, and _fetchGoIndexResponse (index fetch retry, 6x5s).
• src/ai/labradorlabs/crawler/baseCralwer.py: getDBVersionExists(table_name=...) (queries the given table; gatheringdb as_cs table makes the check case-sensitive), plus getDBPackageListV2Exists, insertPackageListV2.
• src/ai/labradorlabs/input/crawlerInfo.py: PRODUCT_LIST_TABLE_V2, VERSION_TABLE_V2, PRODUCT_TABLE_V2.
• docker-compose.yml: main (indexer+worker), worker (scalable), backfill (profile index). All set GO_LIST_SCHEMA=v2 + GO_TARGET_GATHERINGDB=1 + GO_SKIP_EXISTING_VERSION=1.
• sql/TB_COMP_LIB_GOLANG_LIST_V2.sql (incl. RESYNC), sql/ALTER_TB_COMP_LIB_GOLANG_LIST_V2_add_resync.sql, sql/ALTER_TB_COMP_LIB_GOLANG_LIST_PATH_as_cs.sql (V1 list, prepared not applied).
• sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql, sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql: case-sensitive (as_cs identity) Go version/product tables for gatheringdb. Product is a Go-only split (TB_COMP_LIB_PRODUCT_GOLANG) so the shared cross-language TB_COMP_LIB_PRODUCT is left untouched. Not yet deployed; crawler not yet routed to them.
• scripts/migrate_golang_to_gatheringdb.py: one-shot batched migration (keyset + server-side INSERT IGNORE ... SELECT, idempotent, resumable via output/migrate_state_*.json, throttled). Copies labradordb.TB_COMP_LIB_VERSION_GOLANG (all) and labradordb.TB_COMP_LIB_PRODUCT (REPOSITORY='Golang' only) into the gatheringdb tables. Verified locally; not run on prod.
• tests/test_go_crawler_v2_integration.py: V2 worker integration test (claim/fetch/classify/persist one cycle + version-table case-collision reproducer + test_v2_list_table_is_case_sensitive). Bootstraps its own schema; only the 4 component network methods are mocked. The list-table test confirms at the DB level that the V2 list's as_cs identity columns + byte-hash PK keep case-distinct paths as separate rows — the inverse of the ai_ci version-table collapse.

Known Pitfalls / Open Questions

• CONFIRMED (2026-05-29): getDBVersionExists queries labradordb.TB_COMP_LIB_VERSION_GOLANG, whose PRODUCT_KEY and VERSION columns AND the PRIMARY KEY (LANGUAGE, REPOSITORY, PRODUCT_KEY, VERSION) are all utf8mb4_0900_ai_ci (case-insensitive). Verified against prod:

- 'AOC2022' = 'aoc2022' COLLATE utf8mb4_0900_ai_ci -> 1 (equal). - PRODUCT_KEY != LOWER(PRODUCT_KEY) returns 0 rows (the != operator itself can't see case), but PRODUCT_KEY != BINARY LOWER(PRODUCT_KEY) returns ~1.14M rows (out of ~19M) -> uppercase PRODUCT_KEYs are common (e.g. github.com/Aman-Codes/e2e-dashboard-backend). - Querying that real row by its lowercased path returns it (getDBVersionExists false positive proven). - Impact is NOT bounded to skip mode: because the PK is ai_ci, INSERT ... ON DUPLICATE KEY UPDATE into the version/product tables silently COLLAPSES two case-distinct Go modules (same version) into one row -> overwrite / wrong attribution, independent of GO_SKIP_EXISTING_VERSION. The V2 list table fixed case-sensitivity at the queue layer (as_cs), but the downstream version/product/license tables (ai_ci PKs) still lose case-distinct modules. - Reproduced in code: tests/test_go_crawler_v2_integration.py::test_case_insensitive_pk_collapses_distinct_modules. - FIX (2026-05-29): instead of altering the huge shared tables, route V2 writes + skip-check to NEW gatheringdb as_cs tables via GO_TARGET_GATHERINGDB=1 (see "gatheringdb case-sensitive migration"). The skip-check becomes case-sensitive automatically (the column is as_cs). Still out of scope: the license table (TB_COMP_LIB_LICENSE_GOLANG, ai_ci) and dependency table — version+product only for now. Legacy labradordb tables remain ai_ci (untouched, still used when the flag is off).

• RESOLVED (2026-05-29): V2 list table schema placement. The DDL used to create it in labradordb while crawlerInfo references it unqualified (resolves to connection default DB gatheringdb, same DB as the V1 list table). Fixed the DDL to gatheringdb.TB_COMP_LIB_GOLANG_LIST_V2 and created the table there (verified: identity columns as_cs, PK = ID hash). Live gatheringdb.TB_COMP_LIB_GOLANG_LIST (V1) is still ai_ci on PATH (~2.2M rows) — sql/ALTER_TB_COMP_LIB_GOLANG_LIST_PATH_as_cs.sql prepared but not applied.
• RESOLVED (2026-05-29): the V2 claim loop + dataCrawlingV2 now have an integration test (tests/test_go_crawler_v2_integration.py) covering same / subpath / alias_git classification, @latest fallback, retry ladder on missing go.mod, and the case-collision reproducer. 6/6 pass against MySQL 8 with only the 4 component network methods mocked.
• RESOLVED (2026-05-29): V2 resync story. Per-event re-collection via LIST_V2.RESYNC (set 1 + PROCESSED=0 to re-queue); global via GO_RESYNC_ALL=1. Default 0 = check DB and skip if exists.
• Decide whether TIMESTAMP should remain DATETIME/TIMESTAMP (microsecond) or store the raw RFC3339 nanosecond string for exact cursor fidelity. Currently mirrors V1 behavior.

Decisions

• 2026-05-29: list table redesigned to V2 event grain; migration via parallel table + GO_LIST_SCHEMA=v2 flag (not a big-bang rename) so V1 prod data/code stays intact and rollback is just turning the flag off.
• 2026-05-29: added tests/test_go_crawler_v2_integration.py (real MySQL + mocked network) instead of widening unit coverage in go_utils.py, because the value left to verify (claim/lock, transactions, collation, classification wiring) is exactly the DB-touching glue that pure helpers can't exercise. Test is self-skipping so the host-only unittest discover stays green.
• 2026-05-29: verified the VERSION_TABLE case-collision against prod and reclassified it from "needs confirmation / bounded by skip mode" to a confirmed storage-layer data-loss risk independent of skip mode (see Known Pitfalls).
• 2026-05-29: fix the case-collision by routing V2 writes to NEW gatheringdb as_cs tables (GO_TARGET_GATHERINGDB), NOT by altering the shared labradordb tables. Reason: TB_COMP_LIB_PRODUCT is cross-language (~10M rows, all ecosystems) and most are case-insensitive by design, so a blanket as_cs ALTER would be wrong/risky for non-Go. Product gets a Go-only split table; version is Go-only so same-name in gatheringdb.
• 2026-05-29: resync as a per-event LIST_V2.RESYNC column (+ global GO_RESYNC_ALL), default 0 = check-DB-and-skip. Reason: re-queue alone (PROCESSED=0) can't force re-collection in skip mode (it would skip again); RESYNC=1 persists the "re-collect this" intent in the queue.
• 2026-05-29: GO_INDEX_SKIP_EXISTS_CHECK for initial backfill — the per-event existence SELECT over the remote DB was the backfill bottleneck (~33 -> ~2200 events/s). Safe only for the empty-queue initial load; added _fetchGoIndexResponse retry so transient index.golang.org errors don't kill the long walk.
• 2026-05-29: migrate existing data with a sharded keyset INSERT IGNORE ... SELECT script rather than ALTER; confirmed empirically that parallel shards don't beat single-stream (storage read bandwidth bound), and that overlapping INSERTs cause lock wedges.
• 2026-05-29: merged the whole V2 + gatheringdb + RESYNC work to master (c6f16a4, fast-forward) via git push origin feature/...:master (no local checkout) so the running containers' mounted src/ was never disturbed.