crawler-lib-golang

# crawler-lib-golang

AI Summary

Purpose:

• Durable note for the Go module crawler (crawler-lib-golang).
• Records the TB_COMP_LIB_GOLANG_LIST_V2 redesign: event-grain queue, declared module path resolution, and the skip-existing-version mode.

Key points:

• index.golang.org is treated as an append log of (Path, Version, Timestamp) publish events.
• Existence is decided by proxy.golang.org (@v/list -> @latest). pkg.go.dev API is metadata-only; a 404 there does not mean the module is missing.
• V2 changes the list table grain from one row per index path to one row per (INDEX_PATH, INDEX_VERSION) event.
• PRODUCT_KEY (declared module path from go.mod) is stored on the same event row. When it differs from INDEX_PATH, the event row gets PROCESSED=6, DEPRECATED=1 (no second row).
• Go module paths are case-sensitive; V2 identity columns use utf8mb4_0900_as_cs so AOC2022 != aoc2022.
• The whole V2 pipeline is gated by GO_LIST_SCHEMA=v2 (default off -> legacy V1 path runs unchanged).
• Worker fetches only the index-reported version (not the full @v/list), with @latest fallback.
• V2 incremental worker (_runV2) claims newest events first (TIMESTAMP DESC, since 2026-06-01). Resync/backfill stays TIMESTAMP ASC (oldest->newest). Queue batch size is 300 (since 2026-06-01; was 100), hardcoded in crawlerInfo.LIMIT_SELECT_QUERY (LIMIT :start, 300) and asserted by test_limit_select_query_claims_300_packages_per_batch — NOT BaseCrawler.LIMIT=1000 (a separate full-select pagination loop). 300 chosen for the main 1 + worker 3 topology: bigger than 100 to cut round-trips, but small enough to keep the per-claim GET_LOCK loop (held under the global QUEUE_CLAIM_LOCK) short and the crash blast radius (events stranded in processing/20, NOT auto-reclaimed) bounded across 3 parallel workers. (UPDATED 2026-06-08, 5670ef4: the per-claim GET_LOCK loop was REMOVED and stuck processing is now auto-reclaimed — see the queue-lock bullet below.)
• V2 main interleaves indexer with worker batches + daemonizes (2026-06-18, branch feature/go-crawler-indexer-loop, NOT yet merged/deployed). _runV2(crawler_name, status_raw_data, index_enabled, index_cursor): each loop iteration, if index_enabled, runs the existing full-walk updatePackageListCrawler(cursor) first (so main re-fills LIST_V2 once per 300-event claim batch), then claims+processes; when the queue is empty and stuck-reclaim finds nothing it time.sleep(GO_IDLE_SLEEP_SECONDS) (default 60) and continues instead of returning — main is now a daemon, no longer exits-then-relies-on-docker-restart. run() skips the old one-shot pre-loop indexer walk for the V2 path (... and not self._isV2ListSchema()) and passes index_enabled+cursor into _runV2. Gating: pure workers (GO_CRAWLER_INDEXER=worker/false) never index, just claim/sleep. SAFETY: keep GO_INDEX_SKIP_EXISTS_CHECK OFF on the daemon (normal mode skips already-queued events via getDBPackageListV2Exists, goCrawler.py:1307, so repeated indexing does NOT reset PROCESSED). Tests: tests/test_go_crawler_run_loop.py (no-DB, GoCrawler.__new__+MagicMock, self-skips when runtime deps absent; run via venv with pydantic<2 or in-image). Indexer granularity stays full-walk (not single-page), so the first cycle still does the full catch-up walk before the worker starts.
• Stuck queue-lock incident replay (2026-06-17/18): symptom = all workers spam failed to claim queue lock: golangcrawler:queue-claim at a clean 10s cadence (= GET_LOCK(name, 10) timeout) because one session holds the named lock. Diagnose with SELECT IS_USED_LOCK('golangcrawler:queue-claim') (returns holder conn id; stable across polls = stuck). The MCP mysql-gatheringdb account lacks PROCESS, so PROCESSLIST/INNODB_TRX/data_locks are blocked — IS_USED_LOCK still works. wait_timeout=28800 (8h) on gatherdb01 means a half-open/dead holder pins the lock for hours; _reclaimStuckProcessingEvents recovers stranded PROCESSED=20-24 rows but NOT the named lock. Clear with KILL <conn> or by stopping the holding container. LESSON: 5670ef4 was already on origin/master (a stale local origin/master ref made it look unmerged); servers' source had the fix too — the lock got stuck because the running process was started before the fix (Python loads code at process start; pulling/rebuilding without restarting the live container does nothing). Recommended safety net: shorten the crawler session wait_timeout so a crashed holder's lock auto-frees in minutes.
• Queue-lock claim simplified + stuck reclaim (since 2026-06-08, commit 5670ef4; retry ladder updated 2026-07-02): _claimV2Batch(where, order_by) now holds the global QUEUE_CLAIM_LOCK only for SELECT + bulk PROCESSED=20 mark — the per-event GET_LOCK loop is GONE (redundant: queue lock makes SELECT+mark atomic, so two workers never claim the same batch). This fixes the prod incident where a worker that hung/died mid-claim kept the lock and starved all others (failed to claim queue lock: golangcrawler:queue-claim spam). _reclaimStuckProcessingEvents() resets stuck processing rows older than GO_PROCESSING_RECLAIM_MINUTES (default 60, by LAST_UPDATED) back to retryable (20->0, 21->30, ..., 29->38); called at _runV2 start and when a claim returns empty — so crash/hang orphans recover (this also closes the earlier "processing/20 NOT auto-reclaimed" gap). _runV2 no longer uses attempted_ids/lock_skip_offset/per-event locks; _claimEventLocks is now dead code. Tests: test_reclaim_stuck_processing_resets_to_retryable. NOTE: an already-held stuck lock must be cleared manually (KILL <conn> or restart the worker) — the patch only prevents recurrence. Set GO_PROCESSING_RECLAIM_MINUTES > max batch(300) processing time.
• V2 retry delay policy changed (2026-07-02, master f841564): retry claim now uses LAST_UPDATED gates instead of immediately re-claiming every retry rung. PROCESSED=0 is immediate; 30 waits 6h; 31 waits 12h; 32 waits 1d; 33 waits 2d; 34 waits 3d; 35 waits 5d; 36 waits 7d; 37 waits 14d; 38 waits 30d; 39 is final exhausted and is not retryable. Claim marks retry rows to processing statuses 21..29; stuck reclaim maps 20->0, 21->30, ..., 29->38. This prevents hot retry loops while preserving manual/direct backfill for terminal cases. Verified with tests.test_go_crawler_regressions and unittest discovery.
• dataCrawlingV2 existence pre-skip is batched (since 2026-06-01): _batchSkipExistingEvents runs ONE row-value IN SELECT (_existingVersionKeys) for the whole batch (300 events), bulk-marks already-present events processed=1 via a single insertPackageListV2, and only scrapes the missing ones. Replaces the old per-event getDBVersionExists (N SELECTs -> 1). Collation parity via _normVersionKey (exact for gatheringdb as_cs, casefold for labradordb ai_ci). The declared-path (step-3) skip stays per-event (store_path known only after go.mod fetch).
• product table grain = 1 row per module (PK (LANGUAGE, REPOSITORY, PRODUCT_KEY)), written via INSERT ... ON DUPLICATE KEY UPDATE (all cols except CREATED/TIMESTAMP/PATH overwritten = last-write-wins), so LATEST_VERSION = the version of the last-processed event for that module. Because the worker claims newest-first (DESC), dataCrawlingV2 sorts each claimed batch ascending by TIMESTAMP before processing (since 2026-06-01, commit 285034a) so the newest version of a module is written last -> LATEST_VERSION stays newest even when several new versions land in one batch. Residual edge: versions of one module split across batch boundaries (rare in incremental mode; backfill stays ASC). Verified by test_product_latest_version_is_newest_under_desc_claim.
• Go version ordering: sort_go_versions / _go_version_key (semver: major.minor.patch, release > prerelease, numeric prerelease ids < alphanumeric, pseudo-versions ordered by their fixed-width 14-digit timestamp). Build metadata (+incompatible, +meta) is stripped before parsing (since 285034a) — Go ignores it for precedence; before the fix v2.0.0+incompatible failed the regex and sorted to the bottom as invalid.
• KNOWN GAP (open, not fixed as of 2026-06-01): the version table SORT_ORDER column is meaningless under V2 event-grain ingestion — getVersionOrderRows(store_path, [version]) is called with a single version per event so every row gets SORT_ORDER=0. getVersionOrderRows does not sort; it indexes the passed list (V1 passed a sort_go_versions-ascending list, so there SORT_ORDER=0=lowest). Fixing under V2 needs a per-row sortable key column or per-module recompute (deferred).
• The case-collision fix is to write to NEW gatheringdb case-sensitive tables (gatheringdb.TB_COMP_LIB_VERSION_GOLANG, gatheringdb.TB_COMP_LIB_PRODUCT_GOLANG, both as_cs identity) instead of altering the shared labradordb tables. Gated by GO_TARGET_GATHERINGDB=1 (default off).
• LIST_V2.RESYNC column controls re-collection: default 0 = check DB and skip if exists; 1 = re-collect (skip DB check). Global override GO_RESYNC_ALL=1.
• Migration of existing data into the gatheringdb tables uses scripts/migrate_golang_to_gatheringdb.py (sharded, keyset, idempotent). Bottleneck is server disk read of wide version rows (~80 rows/s cold, ~900/s warm; product ~3000/s); parallel shards do NOT beat single-stream because storage read bandwidth saturates.
• Merged to master at c6f16a4 (2026-05-29).
• "60만 누락"의 정체 (2026-06-18 전수 분석): gpnf.txt(과거 누락 체크가 index Path(lower) vs LOWER(PRODUCT_KEY) 단순비교로 뽑은 not-found 스냅샷, 617,525건 중복0)를 현재 DB 와 전수 대조 → 진짜 영구 누락은 사실상 없음. 분해: 이미 해소 246,473(39.9%) = collected_exact 145,844 + case_only 15,825 + git_alias 474 + subpackage_parent_collected 27,158(상위 모듈 root 수집된 하위 package//vN, 예 .../etcd/server/v3→.../etcd) + listv2_resolved_alias 57,161(거의 전부 declared_diff, go.mod 선언경로 PRODUCT_KEY 아래 정리) + 11; backlog(미처리) PROCESSED=0 = 366,798(59.4%)(큐 대기, 영구 누락 아님); proxy 404 not_found(5) = 0; not_indexed(LIST_V2 에 INDEX_PATH 자체 없음) = 4,254(0.7%, github.com 3,780 + gitee/gitlab + metrics.pikepinetech.com/cnb.cool/github.ink 같은 소수 vanity 호스트). 근본원인 = 모듈 부재가 아니라 index path 를 PRODUCT_KEY 처럼 비교한 옛 체크 방식(declared_diff/subpath/대소문자/stale 스냅샷/큐 backlog 가 전부 not found 로 찍힘). 참고: ghproxy-9d2.pages.dev/github.hscsec.cn/github.phpd.cn 같은 GitHub mirror/proxy 호스트는 go.mod 가 정식 github.com/... 를 선언해 E2 declared_diff alias 또는 E4 pending 으로 빠짐(not_indexed 아님). 분석도구·버킷목록: ~/labrador/tool/golang_missing_products/ (analyze_gpnf.py, 2026-06-18/summary.md). 근본원인 상세 서술형 보고서: ~/labrador/tool/golang_missing_products/go-missing-index-path-report.md (2026-06-18 정량 + 2026-06-19 백필 재측정 섹션; 크롤러 repo docs/가 아니라 분석도구 디렉토리에 보관 — 크롤러 repo의 docs/go-missing-index-path-report.md는 2026-05-28 원본만 유지).
• proxy 실검증 (DB 아님): 누락 후보가 진짜 없는지 proxy.golang.org 로 직접 확인하는 게 정답(시스템엔 PROCESSED=5(not_found)를 안 써서 gone 모듈도 pending/retry 로만 남음 — DB 상태만으론 gone 판정 불가). 도구 golang_missing_products/proxy_verify.py(크롤러 go_utils 의 escape/parse/classify 재사용, @latest→@v/list→@v/{ver}.mod). 실측: not_indexed 4,254 전수 = absent_404 88%(찐 gone) / same 9.6% / declared_diff 1.4%; pending 표본 = ~98% proxy 실재(누락 아니라 수집 대기). .mod 선언명이 index 와 다른 대표 예: github.com/lgeekj/p4wnp1_aloa→github.com/mame82/P4wnP1_aloa(fork), github.com/vikramhh/etcd/server/v3→go.etcd.io/etcd/server/v3(vanity import).
• pending 백필 완료 (2026-06-18, backfill_pending.py): gpnf pending 366,798 path(=2,574,469 이벤트)를 크롤러 dataCrawlingV2 와 동일 포맷으로 수집/드레이닝. 신규 수집 1,565,267 모듈-버전, product +294,792(1.92M→2.22M). 전역 LIST_V2 pending 35.44M→32.38M(−3.06M). gpnf 재측정: pending 366,798→770, 이미 해소 39.9%→99.3%. 잔여 실제 누락 1,369(대부분 not_indexed 의 proxy-exists, LIST_V2 밖이라 pending 백필 대상 아님 — 별도 인덱싱/직접수집 필요), gone(proxy 404) ~2,871, recheck(mod_unparsed) 121. 결과 목록: 제외목록 golang_excluded.tsv(616,069, reason 포함, 누락체크 제외용) + 실제누락 2026-06-19/golang_missing_real.tsv(1,369). LATEST_VERSION 개선(사용자 요구): 백필은 product.LATEST_VERSION 을 그 PRODUCT_KEY 의 버전 테이블 전체를 sort_go_versions 정렬한 최신값으로 넣음(크롤러의 last-processed-event 방식과 다름 — KNOWN GAP 의 실질 보정). DB write 교훈: mysql.connector executemany/execute 의 클라이언트측 치환은 README 같은 임의 텍스트((/)/')에서 SQL 깨짐(1064) → cursor(prepared=True) 서버측 prepared statement 필수. + DESCRIPTION 64KB(text) 절단(strict mode 1406), epoch pseudo timestamp(v0.0.0-19700101000000)는 TIMESTAMP 최소값 미만이라 RELEASE_DATE None 처리(1292).

• 로그레벨 정책 정렬 (2026-06-23, master 푸시 7557339..27dd7d3): 누락 위험 아닌 error 로그를 전부 warning으로 강등(ruby 기준 정책). 58건 전부 강등, error 유지 0 — go엔 ruby가 error로 남긴 유실 경로(행 insert skip/피드 윈도우 포기)가 없음(insert는 re-raise 후 retry-ladder 복원, 인덱스 워크는 cursor 재개). 8파일(goCrawler/component/baseCralwer/util.licenseeUtil/db.cdbvdb/base/gatheringDBVo/labradorDBVo), 순수 .error(→.warning( 스왑(동작 불변), py_compile OK. 후속: goCrawler.py:702(per-version)/:1142(per-event)는 형제버전 성공+COMPLETE 마킹 시 단일버전 누락 가능 후보 — ruby:353 따라 warning 통일(insertErrorLog audit 남김), "1건 누락도 error" 정책이면 되돌릴 후보. W26 worklog 참조.

Relevant when:

• Working on the Go crawler list table, indexer, or worker.
• Diagnosing "missing" Go modules that are actually alias / subpath / declared-diff / proxy-only cases.

Do not read full document unless:

• You need exact column definitions, status codes, flags, or the per-pattern handling.

Linked documents:

• ai/workspace/repos.md
• ai/worklog/2026/2026-W22.md

Repo Info

• Repo ID: crawler-lib-golang
• Local path: ~/labrador/crawler/crawler-lib-golang
• Language: Python 3 (crawler), MySQL 8 backend (labradordb)
• Tests:

- Pure helpers: python3 -m unittest tests.test_go_crawler_regressions (no DB/network). - V2 integration: tests/test_go_crawler_v2_integration.py (real MySQL + mocked network, 9 tests incl. gatheringdb routing + RESYNC + list case-sensitivity). Skips unless sqlalchemy + TEST_DB_URL are present; run inside the iotcube/golangcrawler image against a throwaway MySQL 8 container. Host python3 -m unittest discover -s tests = 57 tests OK (9 integration skipped).

Terms

• index path: the Path value from index.golang.org.
• declared module path: the value on the module line of go.mod.
• PRODUCT_KEY: the representative module path stored in DB.
• PROCESSED=6: not a failure. Means "this index path is not the representative; data was filed under the declared module path (PRODUCT_KEY)".

Source-of-Truth Rules

1. pkg.go.dev API is for documentation/metadata enrichment only. Never use it to decide existence.
2. proxy.golang.org decides module/version existence.
3. If pkg.go.dev is 404 but proxy @latest or @v/list is alive, still collect.
4. If pkg.go.dev metadata is absent, description/license/repoUrl may be empty.
5. Go module paths are case-sensitive. Do not normalize with LOWER() for identity.
6. Proxy URLs escape uppercase as ! + lowercase (e.g. eduVPN -> edu!v!p!n). See escape_go_module_proxy_path.

TB_COMP_LIB_GOLANG_LIST_V2

DDL: sql/TB_COMP_LIB_GOLANG_LIST_V2.sql in the repo.

Grain: one row per (INDEX_PATH, INDEX_VERSION) event.

Design choices:

• PK is the hash (not the raw strings) to avoid InnoDB index key-length limits and to give idempotent ON DUPLICATE KEY UPDATE.
• Identity columns are utf8mb4_0900_as_cs so case differences are distinct keys (fixes the LOWER() collision trap at schema level).
• @ separator is collision-free because Go module paths/versions cannot contain @.

PROCESSED status codes

• 0 pending, 1 complete, 5 not_found, 6 alias (filed under declared module path)
• 20-29 processing (claim ladder), 30-38 delayed retry ladder, 39 retry exhausted

Running V2 (docker-compose)

docker-compose.yml services: main (indexer + worker), worker (worker only, scalable), backfill (profile index, one-shot). All set GO_LIST_SCHEMA=v2, GO_TARGET_GATHERINGDB=1, GO_SKIP_EXISTING_VERSION=1. DB conn comes from src/config.ini DbUrl (not env) — so any server with the repo + image + DB reachability works.

Deploy order (important):

1. Schema once: sql/TB_COMP_LIB_GOLANG_LIST_V2.sql (LIST_V2, includes RESYNC), sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql, sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql — all into gatheringdb. If LIST_V2 already exists without RESYNC, apply sql/ALTER_TB_COMP_LIB_GOLANG_LIST_V2_add_resync.sql.
2. Initial backfill (one-shot, before any worker): GO_INDEX_SINCE=2019-04-10... docker compose --profile index run --rm backfill (fast mode via GO_INDEX_SKIP_EXISTS_CHECK=1 baked into the service). Let it run to completion (auto-exits Exited 0 when caught up to now). Do NOT stop it midway — a partial backfill leaves a gap (main's indexer resumes from the V1 cursor ~now, so the gap is never filled). Resume after a crash with GO_INDEX_SINCE=<cursor> (cursor read from TB_CRAWLER_STATUS TYPE=GoCralwerIndexListBackfill).
3. Migrate existing labradordb data (optional, parallel-safe): scripts/migrate_golang_to_gatheringdb.py.
4. THEN workers: main server -> docker compose up -d main; each worker server -> docker compose up -d worker. ALTER must be applied before workers (claim SELECT reads RESYNC).

4-server topology (main 1 + worker 3): one main host, three worker hosts, each docker compose up -d <service>. They coordinate via the shared DB.

Workers on different servers coordinate purely through the shared DB: per-event GET_LOCK prevents double-processing, and (with GO_TARGET_GATHERINGDB=1) the skip-check + writes hit the gatheringdb as_cs tables (case-sensitive). Caveats:

• A hard worker crash leaves its event at a processing status (20-29), which the delayed retry claim WHERE does not re-claim → orphaned until stuck reclaim resets it to retryable.
• NEVER run the skip-mode backfill while workers run — it resets PROCESSED of done events to 0.

Feature Flags (env)

• GO_LIST_SCHEMA=v2 -> use the V2 event-grain pipeline (indexer + worker). Default/anything-else -> legacy V1.
• GO_SKIP_EXISTING_VERSION=1 -> if the version already exists, skip crawl and set PROCESSED=1 (the "check DB first" mode). The compose main/worker set this.
• GO_CRAWLER_INDEXER -> existing flag; controls whether the indexer step runs (true=indexer+worker, false=worker only).
• GO_TARGET_GATHERINGDB=1 -> V2 worker writes version/product to the gatheringdb case-sensitive tables (VERSION_TABLE_V2/PRODUCT_TABLE_V2) and runs the skip-check against them (so the check becomes case-sensitive). Default off -> legacy labradordb tables (ai_ci). Added 2026-05-29.
• GO_INDEX_SKIP_EXISTS_CHECK=1 -> backfill/indexer skips the per-event getDBPackageListV2Exists SELECT. ~68x faster initial backfill (33 -> ~2200 events/s) because the per-event remote SELECT was the bottleneck. ONLY for the initial empty-queue backfill (before any worker runs) — re-running it after the worker started would reset processed events' PROCESSED to 0 (the insert is ON DUPLICATE KEY UPDATE). Added 2026-05-29.
• GO_RESYNC_ALL=1 -> V2 worker treats every event as resync (skip DB check, re-collect all). Per-event resync is the LIST_V2.RESYNC column. Added 2026-05-29.
• GO_IDLE_SLEEP_SECONDS -> V2 daemon idle sleep when the queue is empty (default 60). Added 2026-06-18 (7557339-era interleave/daemonize). main no longer exits on empty queue; it sleeps this long then re-indexes/claims.
• GO_WORKER_NAME -> log-line worker tag. Every log line is now prefixed [<worker>] (resolve_worker_id: GO_WORKER_NAME -> HOSTNAME/container id -> socket.gethostname()), so scaled workers (docker compose up --scale worker=N) that share the bind-mounted golangcrawler.log are distinguishable. Container PID is always 1, so it can't tell them apart. For --scale, HOSTNAME is the (unique) container id; map id->name with docker ps. Added 2026-06-18 (logger.py, commit 7557339). Tested by tests/test_go_crawler_regressions.py::test_build_log_format_*/test_resolve_worker_id_* (host-runnable).

Collection Routine (V2 worker)

For each claimed event (INDEX_PATH, INDEX_VERSION):

1. Compute do_skip_check = skip_enabled AND not (GO_RESYNC_ALL or event.RESYNC). If the event is flagged resync, all DB-existence checks below are bypassed (force re-collect). The version table checked is the gatheringdb as_cs one when GO_TARGET_GATHERINGDB=1 (so the check is case-sensitive).
2. If do_skip_check and (INDEX_PATH, INDEX_VERSION) already exists in the version table -> set PROCESSED=1, skip fetch (cheap pre-check).
3. Fetch the index version's go.mod via proxy (getCanonicalModulePath). If it 404s, fall back to @latest.
4. If still no go.mod, restore the event to the retry ladder.
5. Parse the module line -> declared module path. store_path = declared or INDEX_PATH.
6. Classify with classify_go_index_resolution(INDEX_PATH, store_path).
7. If skip mode is on and (store_path, version) already exists -> set status and skip heavy work (second check).
8. Build the single version row + product + license under store_path.
9. Persist version + product + the resolved event row in one transaction.

- same path: PROCESSED=1, DEPRECATED=0. - different path: PROCESSED=6, DEPRECATED=1, PRODUCT_KEY=store_path.

Claim concurrency mirrors V1: GET_LOCK queue lock + per-event GET_LOCK (keyed by event ID) + processing-status ladder.

"Missing" Pattern Catalog

gatheringdb case-sensitive migration

The downstream version/product tables in labradordb are ai_ci, so case-distinct Go modules collapse (see Known Pitfalls). Rather than ALTER the huge shared tables, the fix routes V2 writes to NEW gatheringdb tables with as_cs identity:

• gatheringdb.TB_COMP_LIB_VERSION_GOLANG — same schema as the labradordb version table, PRODUCT_KEY/VERSION are as_cs. (sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql)
• gatheringdb.TB_COMP_LIB_PRODUCT_GOLANG — Go-only split of the cross-language TB_COMP_LIB_PRODUCT (which serves all languages and must stay ai_ci), PRODUCT_KEY as_cs. (sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql)
• Routing is gated by GO_TARGET_GATHERINGDB=1 (crawlerInfo keys VERSION_TABLE_V2/PRODUCT_TABLE_V2; goCrawler._v2VersionTable/_v2ProductTable; getDBVersionExists(table_name=...)).

Migration of existing rows: scripts/migrate_golang_to_gatheringdb.py.

• Server-side INSERT IGNORE ... SELECT (data stays on the server), keyset pagination on the full PK, idempotent, resumable via output/migrate_state_*.json, throttled, --shard-id/--pk-lo/--pk-hi for disjoint parallel shards, --dry-run/--reset.
• version copies the whole labradordb.TB_COMP_LIB_VERSION_GOLANG; product copies only REPOSITORY='Golang'.
• Performance (this prod, 2026-05-29): version reads ~80 rows/s cold, ~900/s warm; product ~3000/s (narrower rows). The bottleneck is reading wide JSON/text version rows — disk read bandwidth, NOT index maintenance or contention. 8 disjoint shards did NOT beat a single stream (storage bandwidth saturates with one reader). Status as of 2026-05-29: product (~1.66M) fully migrated and verified (case preserved); version migration in progress.
• Pitfall learned: never run overlapping migration INSERTs targeting the same rows — they lock-conflict and one can wedge in query end (unkillable from a non-SUPER client). Run one stream, or strictly disjoint PK ranges.

Code Map (V2 additions)

• src/ai/labradorlabs/crawler/go_utils.py: build_go_index_event_id, classify_go_index_resolution, should_skip_existing_version, is_skip_existing_version_enabled, is_go_list_schema_v2; generalized build_processing_status_update_query/params (key_column/param_prefix).
• src/ai/labradorlabs/db/labradorDBVo.py: getPackageListV2Vo.
• src/ai/labradorlabs/crawler/baseCralwer.py: getDBPackageListV2Exists, insertPackageListV2.
• src/ai/labradorlabs/crawler/goCrawler.py: _isV2ListSchema, _isSkipExistingVersionEnabled, _activeListTable, _buildV2IndexEvents (indexer; honors _isIndexSkipExistsCheck), _runV2 + dataCrawlingV2 (honors RESYNC via do_skip_check) + claim helpers (_claimV2Batch selects RESYNC, _claimEventLocks, _markEventsProcessing, _completeV2Event, _insertV2EventResult). Routing helpers _isV2TargetGatheringdb/_v2VersionTable/_v2ProductTable, resync helpers _isIndexSkipExistsCheck/_isResyncAll, and _fetchGoIndexResponse (index fetch retry, 6x5s).
• src/ai/labradorlabs/crawler/baseCralwer.py: getDBVersionExists(table_name=...) (queries the given table; gatheringdb as_cs table makes the check case-sensitive), plus getDBPackageListV2Exists, insertPackageListV2.
• src/ai/labradorlabs/input/crawlerInfo.py: PRODUCT_LIST_TABLE_V2, VERSION_TABLE_V2, PRODUCT_TABLE_V2.
• docker-compose.yml: main (indexer+worker), worker (scalable), backfill (profile index). All set GO_LIST_SCHEMA=v2 + GO_TARGET_GATHERINGDB=1 + GO_SKIP_EXISTING_VERSION=1.
• sql/TB_COMP_LIB_GOLANG_LIST_V2.sql (incl. RESYNC), sql/ALTER_TB_COMP_LIB_GOLANG_LIST_V2_add_resync.sql, sql/ALTER_TB_COMP_LIB_GOLANG_LIST_PATH_as_cs.sql (V1 list, prepared not applied).
• sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql, sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql: case-sensitive (as_cs identity) Go version/product tables for gatheringdb. Product is a Go-only split (TB_COMP_LIB_PRODUCT_GOLANG) so the shared cross-language TB_COMP_LIB_PRODUCT is left untouched. Not yet deployed; crawler not yet routed to them.
• scripts/migrate_golang_to_gatheringdb.py: one-shot batched migration (keyset + server-side INSERT IGNORE ... SELECT, idempotent, resumable via output/migrate_state_*.json, throttled). Copies labradordb.TB_COMP_LIB_VERSION_GOLANG (all) and labradordb.TB_COMP_LIB_PRODUCT (REPOSITORY='Golang' only) into the gatheringdb tables. Verified locally; not run on prod.
• tests/test_go_crawler_v2_integration.py: V2 worker integration test (claim/fetch/classify/persist one cycle + version-table case-collision reproducer + test_v2_list_table_is_case_sensitive). Bootstraps its own schema; only the 4 component network methods are mocked. The list-table test confirms at the DB level that the V2 list's as_cs identity columns + byte-hash PK keep case-distinct paths as separate rows — the inverse of the ai_ci version-table collapse.

Known Pitfalls / Open Questions

• [수정 완료 2026-07-09 2301612] 인덱서 LIST_V2 upsert 가 PROCESSED 를 덮어써 self-heal 재-walk 불가: insertPackageListV2 가 setInsertIntoDB(=ON DUPLICATE KEY UPDATE {전체컬럼})를 써서, 이미 있는 (path,version) 이벤트를 재삽입하면 PROCESSED 를 VALUES(0)로 덮어써 완료 이벤트를 pending 으로 되돌린다(repo-note "NEVER run skip-mode backfill while workers run" 경고의 정체). 그래서 과거 유실 구멍 복구용 backfillMissingPackageListPaths 재-walk 를 워커 가동 중 상시 돌릴 수 없었고, not_indexed 실누락(kiradb/alterix 등)이 남아있었다. 인덱서는 커서를 앞으로만 전진(updatePackageListCrawler)해 과거 창을 재방문 안 하고, 1a2a159(07-06)는 신규 drop 예방만 한다. 수정: 인덱서 upsert 를 중복 시 no-op(INSERT_INTO_PRESERVE_QUERY = ON DUPLICATE KEY UPDATE ID = ID)으로 바꿔 worker 소유 상태(PROCESSED/DEPRECATED/RESOLVE_KIND/PRODUCT_KEY)를 절대 안 건드림 → 재-walk self-heal 이 워커 가동 중에도 멱등·안전. INSERT IGNORE 가 아니라 no-op UPDATE 인 이유: 제약/길이 등 진짜 오류는 계속 raise 되어 _insertChunked 행별 폴백/전파(1a2a159) 유지. 테스트 tests/test_index_insert_preserve.py(3). 상태: master push + 4서버 배포·검증 완료(2026-07-09) — 전 서버 HEAD=2301612, main 라이브 로그에서 ON DUPLICATE KEY UPDATE ID = ID 사용 확인. 후속: exists-check ON(GO_INDEX_SKIP_EXISTS_CHECK 끄고) backfillMissingPackageListPaths 1회 완주로 과거 구멍 전량 복구(이제 안전). W28 worklog(2026-07-09) 참조.
• [수정 완료 2026-07-01 83882ac] 전역 queue-claim 락 제거 → FOR UPDATE SKIP LOCKED: 5670ef4(2026-06-08)가 per-event 락은 없앴지만 전역 GET_LOCK('golangcrawler:queue-claim')은 유지했는데, 이 단일 락에 모든 워커가 직렬로 줄 서고 MySQL GET_LOCK 불공정으로 한 워커가 독점 → 나머지 failed to claim queue lock starvation + 로그스팸(위 "Stuck queue-lock incident"의 non-dead 변종). _claimV2Batch를 SELECT … FOR UPDATE SKIP LOCKED + 상태매핑 UPDATE 한 트랜잭션으로 재작성해 전역 락을 완전 제거(빌더 go_utils.build_v2_claim_select_query, 테스트 GoV2ClaimSkipLockedTests 4). 동시 워커는 서로 잠근 배치를 SKIP → disjoint 병렬 claim, 중복 0, named-lock stuck 실패모드 자체 소멸. docker compose --scale worker=N 그대로(워커 설정 불필요). MySQL 8.4 필요(확인). _claimQueueLock/_releaseQueueLock은 V1 경로가 아직 써서 유지. 배포 전 이미 잡힌 stuck named-lock이 있으면 KILL <conn>/컨테이너 재시작으로 수동 해제(코드 배포=재발 방지). 상세: W27 worklog(2026-07-01).
• [수정 완료 2026-07-01 54cf534] proxy VERSION 대문자 escape 누락 → 대문자 버전 모듈 영구 정체(누락): component.py의 getGoMod/getDependency/getReleaseDateFirst가 path는 escape하지만 version은 raw로 proxy .mod/.info URL에 넣던 버그. proxy는 버전 대문자도 !escape를 요구(invalid escaped version "v0.1.0-SNAPSHOT")하므로 -SNAPSHOT/-RC1/-DeepNull/-RDK-* 등 대문자 버전은 go.mod fetch 404 → declared path 해석 실패 → LIST_V2 PROCESSED=0 영구 정체 = 조용한 누락. 규모(2026-07-01 실측): PROCESSED=0 pending 29,565,467 중 대문자 버전 45,508건 트랩. 수정(커밋·푸시 완료 master 54cf534): 3곳에 escape_go_module_proxy_path(version) 적용 + 단위테스트 4개(GoProxyVersionEscapeTests), 전체 54개 통과. 재큐 기록 주의: 당시 34를 터미널로 보고 268건을 0으로 리셋했지만, 2026-07-02 retry 정책 정정 후 터미널은 39이고 34는 지연 재시도 rung이다. go는 docker-compose로 main(1, GO_CRAWLER_INDEXER=true=index+큐)+worker(N, false=큐전용) 분리(둘 다 app/main.py, DB GET_LOCK로 조정) → 수정 반영(src bind-mount 갱신+recreate 또는 이미지 재빌드) 후 main+worker가 PROCESSED=0(45,428) 분산 드레인. 남은 일=커밋+푸시+재배포. 상세: W27 worklog(2026-07-01).
• CONFIRMED (2026-05-29): getDBVersionExists queries labradordb.TB_COMP_LIB_VERSION_GOLANG, whose PRODUCT_KEY and VERSION columns AND the PRIMARY KEY (LANGUAGE, REPOSITORY, PRODUCT_KEY, VERSION) are all utf8mb4_0900_ai_ci (case-insensitive). Verified against prod:

- 'AOC2022' = 'aoc2022' COLLATE utf8mb4_0900_ai_ci -> 1 (equal). - PRODUCT_KEY != LOWER(PRODUCT_KEY) returns 0 rows (the != operator itself can't see case), but PRODUCT_KEY != BINARY LOWER(PRODUCT_KEY) returns ~1.14M rows (out of ~19M) -> uppercase PRODUCT_KEYs are common (e.g. github.com/Aman-Codes/e2e-dashboard-backend). - Querying that real row by its lowercased path returns it (getDBVersionExists false positive proven). - Impact is NOT bounded to skip mode: because the PK is ai_ci, INSERT ... ON DUPLICATE KEY UPDATE into the version/product tables silently COLLAPSES two case-distinct Go modules (same version) into one row -> overwrite / wrong attribution, independent of GO_SKIP_EXISTING_VERSION. The V2 list table fixed case-sensitivity at the queue layer (as_cs), but the downstream version/product/license tables (ai_ci PKs) still lose case-distinct modules. - Reproduced in code: tests/test_go_crawler_v2_integration.py::test_case_insensitive_pk_collapses_distinct_modules. - FIX (2026-05-29): instead of altering the huge shared tables, route V2 writes + skip-check to NEW gatheringdb as_cs tables via GO_TARGET_GATHERINGDB=1 (see "gatheringdb case-sensitive migration"). The skip-check becomes case-sensitive automatically (the column is as_cs). Still out of scope: the license table (TB_COMP_LIB_LICENSE_GOLANG, ai_ci) and dependency table — version+product only for now. Legacy labradordb tables remain ai_ci (untouched, still used when the flag is off).

• RESOLVED (2026-05-29): V2 list table schema placement. The DDL used to create it in labradordb while crawlerInfo references it unqualified (resolves to connection default DB gatheringdb, same DB as the V1 list table). Fixed the DDL to gatheringdb.TB_COMP_LIB_GOLANG_LIST_V2 and created the table there (verified: identity columns as_cs, PK = ID hash). Live gatheringdb.TB_COMP_LIB_GOLANG_LIST (V1) is still ai_ci on PATH (~2.2M rows) — sql/ALTER_TB_COMP_LIB_GOLANG_LIST_PATH_as_cs.sql prepared but not applied.
• RESOLVED (2026-05-29): the V2 claim loop + dataCrawlingV2 now have an integration test (tests/test_go_crawler_v2_integration.py) covering same / subpath / alias_git classification, @latest fallback, retry ladder on missing go.mod, and the case-collision reproducer. 6/6 pass against MySQL 8 with only the 4 component network methods mocked.
• RESOLVED (2026-05-29): V2 resync story. Per-event re-collection via LIST_V2.RESYNC (set 1 + PROCESSED=0 to re-queue); global via GO_RESYNC_ALL=1. Default 0 = check DB and skip if exists.
• Decide whether TIMESTAMP should remain DATETIME/TIMESTAMP (microsecond) or store the raw RFC3339 nanosecond string for exact cursor fidelity. Currently mirrors V1 behavior.

Decisions

• 2026-05-29: list table redesigned to V2 event grain; migration via parallel table + GO_LIST_SCHEMA=v2 flag (not a big-bang rename) so V1 prod data/code stays intact and rollback is just turning the flag off.
• 2026-05-29: added tests/test_go_crawler_v2_integration.py (real MySQL + mocked network) instead of widening unit coverage in go_utils.py, because the value left to verify (claim/lock, transactions, collation, classification wiring) is exactly the DB-touching glue that pure helpers can't exercise. Test is self-skipping so the host-only unittest discover stays green.
• 2026-05-29: verified the VERSION_TABLE case-collision against prod and reclassified it from "needs confirmation / bounded by skip mode" to a confirmed storage-layer data-loss risk independent of skip mode (see Known Pitfalls).
• 2026-05-29: fix the case-collision by routing V2 writes to NEW gatheringdb as_cs tables (GO_TARGET_GATHERINGDB), NOT by altering the shared labradordb tables. Reason: TB_COMP_LIB_PRODUCT is cross-language (~10M rows, all ecosystems) and most are case-insensitive by design, so a blanket as_cs ALTER would be wrong/risky for non-Go. Product gets a Go-only split table; version is Go-only so same-name in gatheringdb.
• 2026-05-29: resync as a per-event LIST_V2.RESYNC column (+ global GO_RESYNC_ALL), default 0 = check-DB-and-skip. Reason: re-queue alone (PROCESSED=0) can't force re-collection in skip mode (it would skip again); RESYNC=1 persists the "re-collect this" intent in the queue.
• 2026-05-29: GO_INDEX_SKIP_EXISTS_CHECK for initial backfill — the per-event existence SELECT over the remote DB was the backfill bottleneck (~33 -> ~2200 events/s). Safe only for the empty-queue initial load; added _fetchGoIndexResponse retry so transient index.golang.org errors don't kill the long walk.
• 2026-05-29: migrate existing data with a sharded keyset INSERT IGNORE ... SELECT script rather than ALTER; confirmed empirically that parallel shards don't beat single-stream (storage read bandwidth bound), and that overlapping INSERTs cause lock wedges.
• 2026-05-29: merged the whole V2 + gatheringdb + RESYNC work to master (c6f16a4, fast-forward) via git push origin feature/...:master (no local checkout) so the running containers' mounted src/ was never disturbed.