2026-W22 Worklog · llm-wiki

# 2026-W22 Worklog

AI Summary

Purpose:

Tracks development work across repositories for this week.

Key points:

Add coding work entries here after meaningful agent sessions.
Keep entries concise.
Promote durable knowledge to ai/repo-notes/ or ai/wiki/.

Relevant when:

Reviewing this week's development work.
Creating weekly report.
Resuming context.

Do not read full document unless:

The task concerns this week.

Linked documents:

ai/worklog/index.md
ai/workspace/repos.md

Work Items

Go crawler: TB_COMP_LIB_GOLANG_LIST_V2 (event grain)

Date: 2026-05-29
Machine: company-mac
Agent: claude-code
Repo ID: crawler-lib-golang
Repo path: ~/labrador/crawler/crawler-lib-golang
Branch: unknown
Task:

- Redesign the Go index list table to event grain and fix "missing module" misclassification (alias / subpath / declared-diff / proxy-only / case-only).

Summary:

- Added TB_COMP_LIB_GOLANG_LIST_V2 DDL: one row per (INDEX_PATH, INDEX_VERSION), hash PK, 0900_as_cs identity columns, PRODUCT_KEY/RESOLVE_KIND on the same row. - Built the V2 pipeline behind GO_LIST_SCHEMA=v2 (default off, V1 untouched): indexer writes events, worker (_runV2/dataCrawlingV2) fetches only the index version with @latest fallback and resolves declared module path. - Added GO_SKIP_EXISTING_VERSION mode (skip crawl + set PROCESSED=1 when version already exists). - TDD'd pure helpers in go_utils.py; generalized build_processing_status_update_query/params for the ID key.

Files changed:

- sql/TB_COMP_LIB_GOLANG_LIST_V2.sql (new) - src/ai/labradorlabs/crawler/go_utils.py, goCrawler.py, baseCralwer.py - src/ai/labradorlabs/db/labradorDBVo.py, input/crawlerInfo.py - tests/test_go_crawler_regressions.py

Tests run:

- python3 -m unittest tests.test_go_crawler_regressions -> 46 passed (pure helpers only).

Decisions:

- Parallel V2 table + flag, not a big-bang rename, so rollback is just turning the flag off. - index.golang.org treated as a publish-event log; proxy is the existence authority; pkg.go.dev is metadata-only.

Follow-ups:

- Integration-test the V2 claim loop + dataCrawlingV2 against a real DB/network (not possible in the unit-test env). - getDBVersionExists compares against a likely case-insensitive VERSION_TABLE.PRODUCT_KEY; confirm case-only collision risk in skip mode. - Define a V2 resync story (reprocessing completed rows).

Wiki updates:

- ai/repo-notes/crawler-lib-golang.md (new), ai/workspace/repos.md, ai/repo-notes/index.md.

Go crawler V2: integration test + case-collision verification

Date: 2026-05-29
Machine: company-mac
Agent: claude-code
Repo ID: crawler-lib-golang
Repo path: ~/labrador/crawler/crawler-lib-golang
Branch: unknown
Task:

- Follow-ups from the V2 work: (1) integration-test the V2 claim loop + dataCrawlingV2; (2) confirm the VERSION_TABLE.PRODUCT_KEY case-insensitive collision against the real DB.

Summary:

- Added tests/test_go_crawler_v2_integration.py: real MySQL + mocked network (only the 4 component methods: getCanonicalModulePath, getProxyLatest, getPkgsiteModule, getReleaseDateFirst). Covers same / subpath / alias_git classification, @latest fallback, retry ladder on missing go.mod, and a case-collision reproducer. Test bootstraps its own schema and self-skips when sqlalchemy/TEST_DB_URL are absent (host unittest discover stays green: 54 tests, 6 skipped). - Ran it inside the iotcube/golangcrawler:0.1.0 image against a throwaway mysql:8.0 container (network gocrawler-test-net, native-password auth for pymysql 1.0.2): 6/6 pass. - Confirmed against prod (labradordb.TB_COMP_LIB_VERSION_GOLANG): PRODUCT_KEY/VERSION columns and the PK are all utf8mb4_0900_ai_ci. 'AOC2022' = 'aoc2022' -> equal; ~1.14M of ~19M rows have uppercase PRODUCT_KEYs; querying a real uppercase row by its lowercased path matches it. Reclassified the risk: the ai_ci PK collapses case-distinct modules on INSERT ... ON DUPLICATE KEY UPDATE -> silent overwrite, independent of skip mode.

Files changed:

- tests/test_go_crawler_v2_integration.py (new; later +test_v2_list_table_is_case_sensitive)

Tests run:

- host: python3 -m unittest discover -s tests -> 55 tests, OK (7 skipped). - docker: V2 integration suite -> 7 passed.

Note: confirmed the V2 list table is already case-sensitive by design (identity columns as_cs, PK = byte-hash sha256); the live V1 list table gatheringdb.TB_COMP_LIB_GOLANG_LIST is still ai_ci on its PATH PK (~2.2M rows) — same collision class, not changed. Added a DB-level test asserting the V2 list keeps case-distinct paths as separate rows.
Decisions:

- Integration test over more unit tests (the remaining risk is DB-touching glue). Self-skipping so host suite is unaffected.

Follow-ups:

- Fix the version/product/license case-collision: move identity columns to as_cs or a case-sensitive hash key (mirrors V2 list fix). Large-table ALTER — needs scoping. - Reconcile V2 list table schema placement: DDL creates it in labradordb, code references it unqualified (resolves to connection default DB). - Define the V2 resync story (still open).

Wiki updates:

- ai/repo-notes/crawler-lib-golang.md (Tests, Code Map, Known Pitfalls reclassified, Decisions).

Go crawler V2: deploy prep (gatheringdb table + compose)

Date: 2026-05-29
Machine: company-mac
Agent: claude-code
Repo ID: crawler-lib-golang
Task:

- Make the V2 list table deployable as-is and provide an easy compose-based run path (backfill index, then workers).

Summary:

- Fixed sql/TB_COMP_LIB_GOLANG_LIST_V2.sql to create the table in gatheringdb (was labradordb, but code references it unqualified -> default DB gatheringdb). Created the table on the live DB and verified identity columns are as_cs, PK = ID hash. - Rewrote docker-compose.yml into worker (default, scalable, GO_LIST_SCHEMA=v2 + GO_CRAWLER_INDEXER=false + GO_SKIP_EXISTING_VERSION=1), backfill (profile index, one-shot full index fill from 2019 epoch), incremental (profile incremental). Validated docker compose config. - Confirmed cross-server worker coordination: shared DB GET_LOCK (queue + per-event) prevents double-processing, getDBVersionExists skip prevents re-inserting existing versions. Noted two caveats: hard-crash orphans events at processing status 20-24 (claim WHERE only re-claims 0/30-33), and the skip is still ai_ci (case-insensitive false positives).

Files changed:

- sql/TB_COMP_LIB_GOLANG_LIST_V2.sql, docker-compose.yml

DB change:

- Created gatheringdb.TB_COMP_LIB_GOLANG_LIST_V2 (empty, additive). V1 list PATH collation NOT changed.

Wiki updates:

- ai/repo-notes/crawler-lib-golang.md (Running V2 section, schema-placement resolved).

Go crawler: case-sensitive gatheringdb version/product 테이블 + 마이그레이션 스크립트 (준비만)

Date: 2026-05-29
Machine: company-mac
Agent: claude-code
Repo ID: crawler-lib-golang
Task:

- 공용 테이블 스키마 변경 없이, Go version/product 를 case-sensitive 로 gatheringdb 에 분리. 데이터 이동 스크립트 준비(실행은 안 함).

결정(브레인스토밍):

- 방어가 아니라 분리 신설. 라우팅(크롤러가 새 테이블에 쓰게)은 다음 턴으로 미룸. 범위는 version+product 만(dependency/license 제외). 실행은 스크립트만 준비.

Summary:

- DDL 2개: sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql(원본과 동일, PRODUCT_KEY/VERSION 만 as_cs), sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql(공용 TB_COMP_LIB_PRODUCT 의 Go 분리본, PRODUCT_KEY as_cs). - 마이그레이션 스크립트 scripts/migrate_golang_to_gatheringdb.py: keyset 배치 + 서버측 INSERT IGNORE ... SELECT(데이터 client 미경유), 멱등, output/ 커서로 재개, throttle, --dry-run/--reset/--only. 복사 컬럼은 GENERATION_EXPRESSION 기준 introspect(generated VULN_COUNT 만 제외; EXTRA LIKE '%GENERATED%' 는 LAST_UPDATED 까지 잘못 빼서 안 씀). version 전량, product 는 REPOSITORY='Golang' 만. - 로컬 MySQL 8(docker)로 검증: version 6 / product 3(js 제외) 복사, 대문자 PRODUCT_KEY byte 보존, LAST_UPDATED 복사, as_cs 대상에 Foo/Bar+foo/bar 공존(=2), 멱등 재실행 무에러, dry-run 동작.

Files changed:

- sql/TB_COMP_LIB_VERSION_GOLANG_gatheringdb.sql (new), sql/TB_COMP_LIB_PRODUCT_GOLANG_gatheringdb.sql (new), scripts/migrate_golang_to_gatheringdb.py (new)

실행 안 한 것 / Follow-ups:

- prod 에 DDL 미적용, 마이그레이션 미실행(스크립트만). 실제 적재는 운용 창에서 --batch/--sleep 조절해 실행. - 크롤러 라우팅(V2 가 gatheringdb 새 테이블에 쓰게 crawlerInfo/insert 경로 변경) 미구현 — 다음 턴. - 주의: 로컬 검증 후 output/migrate_state_*.json 가 남아 있으면 실제 첫 실행에서 "already done" 으로 스킵됨 -> 삭제하거나 --reset 사용.

Go crawler V2: gatheringdb 라우팅 + RESYNC + 마이그레이션 + 배포 (master 머지)

Date: 2026-05-29
Machine: company-mac
Agent: claude-code
Repo ID: crawler-lib-golang
Repo path: ~/labrador/crawler/crawler-lib-golang
Branch: feature/go-crawler-v2-gatheringdb -> master (FF, c6f16a4)
Task:

- case-collision을 공유 테이블 ALTER 없이 해결, V2 워커를 gatheringdb case-sensitive 테이블로 라우팅, 재수집(RESYNC) 제어, 기존 데이터 이관, 배포 구성, 커밋/머지.

Summary:

- 신규 gatheringdb as_cs 테이블 DDL: TB_COMP_LIB_VERSION_GOLANG(Go 전용 동명), TB_COMP_LIB_PRODUCT_GOLANG(공용 product의 Go 분리본). prod에 생성. - 라우팅: GO_TARGET_GATHERINGDB=1이면 V2 워커가 그 테이블에 쓰고 skip-check도 그 기준(=case-sensitive). crawlerInfo V2 키 + _v2VersionTable/_v2ProductTable + getDBVersionExists(table_name). 기본 off=기존 labradordb. - RESYNC: LIST_V2.RESYNC 컬럼(기본 0=DB 확인 후 스킵, 1=재수집) + 전역 GO_RESYNC_ALL. dataCrawlingV2 do_skip_check 분기. - 마이그레이션 스크립트 scripts/migrate_golang_to_gatheringdb.py: 샤드 keyset INSERT IGNORE...SELECT, 멱등, 재개, throttle. product(166만) prod 이관 완료·검증(케이스 보존), version(~1900만) 진행 중. - 인덱스 백필: GO_INDEX_SKIP_EXISTS_CHECK(per-event SELECT 생략, 33->~2200 events/s) + _fetchGoIndexResponse retry. 2019-04부터 LIST_V2(gatheringdb) 적재 중(자동 완료 예정). - docker-compose: main/worker/backfill (4서버=main1+worker3), 모든 서비스 GO_LIST_SCHEMA=v2+GO_TARGET_GATHERINGDB=1+GO_SKIP_EXISTING_VERSION=1.

성능 발견:

- version 이관 병목 = 넓은 JSON/text 행 디스크 읽기. ~80 rows/s(cold)~900(warm), product ~3000. 8 disjoint 샤드 병렬이 단일 스트림을 못 이김(스토리지 읽기 대역폭 포화). 겹치는 INSERT는 락 wedge(query end, 비-SUPER로 kill 불가) 유발.

Files changed (커밋 c6f16a4):

- goCrawler.py, baseCralwer.py, crawlerInfo.py, go_utils.py, labradorDBVo.py, component.py - sql/TB_COMP_LIB_GOLANG_LIST_V2.sql(+RESYNC), sql/ALTER_..._add_resync.sql, sql/..._gatheringdb.sqlx2, sql/ALTER_..._PATH_as_cs.sql - scripts/migrate_golang_to_gatheringdb.py, docker-compose.yml, tests/*, .gitignore(output/ 제외)

Tests:

- host: 57 OK (9 통합 skip). docker+MySQL8: 통합 9/9 (라우팅·RESYNC·case-sensitivity 포함).

배포:

- master FF 머지(c6f16a4). 워킹트리 미변경 방식이라 실행 중 컨테이너 무영향.

Follow-ups / 운영 순서:

- 백필 완료 -> ALTER(RESYNC) 적용 -> version 이관 완료 -> main/worker 가동. 워커 가동 후 skip-백필 재실행 금지(PROCESSED 리셋). 백필 중간 정지 금지(인덱스 구멍). - license/dependency 테이블은 아직 ai_ci(범위 외). main 인덱서 커서는 V1 status(GoCralwer, ~현재)에서 재개 — V2 LIST 커서와 별개.

Wiki updates:

- ai/repo-notes/crawler-lib-golang.md 대폭 갱신(AI Summary, Flags, Running V2, gatheringdb 섹션, Code Map, Pitfalls, Decisions).

Template Entry

Date: 2026-05-28
Machine: unknown
Agent: unknown
Repo ID: unknown
Repo path: unknown
Branch: unknown
Task:

Summary:

Files changed:

Tests run:

Decisions:

Follow-ups:

Wiki updates: