LLM WikiAccess-protected knowledge portal

WIKI

2026-W24 Worklog

AI Summary Purpose Tracks development work across repositories for this week. Key points Add coding work entries here after meaningful agent sessions. Keep entries concise. Promote durable knowledge to ai/repo notes/ or ai/wiki/ . Relevant

경로ai/worklog/2026/2026-W24.md
카테고리Worklog
태그#ai-review #cicd #crawler #infra #license #monitoring #portfolio #report #security #study #w24 #worklog

# 2026-W24 Worklog

AI Summary

Purpose:

Key points:

Relevant when:

Do not read full document unless:

Linked documents:

Work Items

LLM Wiki integrated portal and graph

CI/CD AI review kit portfolio grounding

Resume/technical portfolio gap follow-up without malicious-package scope

Hermes + Discord LLM Wiki operations portfolio item

portfolio.hshouse.xyz → resume.hshouse.xyz rename and technical portfolio link sync

Agent documentation and Korean humanization policy

crawler-lib-swift·ruby: 로그 레벨 정책 통일(누락만 error)

- error 유지: baseCrawler.setInsertIgnoreDB/IntoDB insert 실패(테이블명 추가), cdbvdb.executeSql 실행 실패(swallow 구조라 여기가 행 유실의 실질 지점 — statement 머리 200자 포함), cocoapods git pull 3회 실패(Specs stale → 최근 pod 누락 가능), crawlSingleProduct 버전 실패(수동 백필 경로, 자동 재시도 없음). - warning 강등: run/crawler 외곽 catch-all 5곳(처리 실패(다음 회차 재시도)), git pull 재시도 중, pod 진입 스킵(다음 회차 재검출), version 처리 실패 2곳 (오늘 오전 error 로 만들었던 것 — swift 증분이 Specs 디렉토리 vs DB 비교라 실패 버전은 다음 회차 자동 재검출 → 정책상 warning 으로 재조정, 경로+traceback 유지), RELEASE_DATE 보강 2곳, license 매칭/항목 처리(component 201·290), cdbvdb setQuery/select 2곳(raise→상위 판단), db/base 2곳, licenseeUtil 5곳.

- 잔존하던 util/db 비누락 7곳 warning 강등: licenseeUtil 5곳(subprocess/AI/JSON/라이선스 파일 읽기), db/base.py 2곳(engine open/getDB — raise→상위 판단).

swift 는 테스트 없음 → 컴파일 검증만.

각 2곳 + ruby libraries.io api_key 1곳(데드코드) 제거 → config.ini[COMMON] 에서 읽도록 변경. config.ini 에는 빈 placeholder 만 커밋(평문 토큰 커밋은 차단됨 — 사용자 선택). swift 도 config.ini 에 커밋돼 있던 토큰 평문을 빈 값으로 교체. 세 repo 모두 토큰이 비면 무인증 clone/API 폴백으로 동작(rate limit 만 낮음). 커밋: ruby fb22fc7, dotnet fdf48d1, swift eb0f407 (master push 완료). 남은 일: (1) 노출된 기존 토큰 rotate, (2) 새 토큰을 배포 환경 config.ini 에 주입 — 주입 전까지 GitHub 라이선스 보강은 무인증(60req/h)으로 동작.

(component.py 33+/2-, dotnetCrawler.py 358±, crawlerInfo.py 4+, 신규 tests/test_dotnet_license_url_and_missing_versions.py). 미커밋.

1. licenseUrl 처리: NuGet 이 라이선스를 URL 로만 줄 때 — SPDX URL(licenses.nuget.org/<expr>, spdx.org/licenses/<id>)은 component.getSpdxNameFromLicenseUrl 로 라이선스명 환원 후 기존 매칭(IDS/AI 정상); github URL 이면 projectUrl 대신 그 repo 로 기존 GitHub license fallback 라우팅 (raw.githubusercontent.com → github.com 정규화); 그 외 일반 URL 은 LICENSE=[{"name":<url>}] 만 기록하고 LICENSE_IDS 매칭·AI 호출 스킵 (skip_license_ai). aka.ms/deprecateLicenseUrl 마커는 값 없음 취급. 2. LATEST_VERSION 무조건 최신 보정: 배치 말미 컴포넌트별 정렬 패스에서 (신규+DB+백필) 전체 버전을 sortVersions(오름차순)로 정렬해 LATEST_VERSION=sorted[-1] 보정 row 를 마지막에 upsert. 누락분 늦수집 시 과거 버전이 LATEST 를 덮던 문제 + 기존 len>1 게이트 때문에 단건 수집 시 정렬/보정이 아예 안 돌던 문제 함께 해결. 3. 누락 버전 동시 수집: _collectMissingVersions — flatcontainer (v3-flatcontainer/{id}/index.json) 전체 버전 목록과 (이번 수집분+DB) 대조, 누락 버전은 registration leaf(registration5-gz-semver2/{id}/{ver}.json)의 catalogEntry 로 catalog leaf 를 받아 본류와 동일 처리(version/license row 즉시 insert, product row 는 본류가 이미 생성 → LATEST 는 보정 패스가 담당). 버전 비교는 lower()+build metadata 제거 정규화.

추출해 본류와 백필이 공유(ctx = repo cache/누적 리스트/collect_product 플래그). 데이터 포맷(row 구조/JSON 직렬화) 불변.

각 1회 HTTP 추가. flatcontainer 는 CDN 이라 가볍지만 운영 모니터링 권장.

신규 테스트: SPDX URL 환원(LICENSE_IDS [247]), github licenseUrl 로 getRepoLicenseFromGITAPI('foo','bar'), 일반 URL → LICENSE only(IDS/AI None + AI mock 미호출), deprecated 마커 무시, flatcontainer diff 백필(1.0.0 수집 + LATEST 2.0.0), 과거버전만 수집 시 LATEST=DB 2.0.0 유지, 누락 없으면 백필 안 함. 라이브 API 형태 확인: flatcontainer versions 84개(newtonsoft.json), registration leaf catalogEntry URL ✓ (gzip 은 requests 자동 해제). 실DB 스모크 미실시.

나머지 전부 warning 강등. error 유지 2곳 = baseCralwer.setInsertIgnoreDB/ setInsertIntoDB insert 실패(여기서 swallow 되어 페이지가 processed=1 로 닫힐 수 있는 실제 유실 지점, 테이블명 컨텍스트 추가). warning 강등 ~40곳: 재시도되는 페이지/leaf 오류(processed=0 requeue), run loop, 정렬/SORT_ORDER/ LATEST 보정 실패(코어 행은 적재됨), 라이선스 매칭·GitHub API·AI·clone 보강 실패, JSON 변환 폴백(str 로 저장됨), clone 정리 실패, updatePackageListCrawler (다음 주기 last_path 부터 따라잡음), 저수준 DB(cdbvdb/base — raise 해서 상위 판단, _logProblemRow 행 상세 포함; nullify-재시도로 복구되는 케이스가 있어 warning). app/check.py(진단 CLI)는 크롤러 런타임 로그가 아니라 미변경. 검증: compileall OK, 33 테스트 전부 통과.

crawler-lib-swift: cocoapods.py:329 'version' 에러 진단 + 로그 개선 + version 누락 podspec 스킵

str(KeyError('version'))(따옴표 포함 키 이름만)을 그대로 출력한 것. 발생 지점은 getVersionVo → component.getVersion의 item_info['version'] 직접 키 접근(component.py:26) — version 키 없는 비표준 podspec JSON을 만나면 KeyError 로 그 버전이 통째로 스킵되고, 어떤 파일인지 로그에 안 남았음(파일명은 DEBUG 에만).

1. version 키 누락 podspec → warning('podspec version 키 누락(스킵): Specs/.../pod/ver/file') 후 명시적 continue — crawl·resync 두 walk 루프 + crawlSingleProduct(product_key+ver 로 표기) 세 곳. 2. 그 외 예외 catch-all(구 329행) → error('version 처리 실패: {path}/{dir5}/{dir6} - {e!r}', exc_info=True) 로 podspec 경로+traceback 포함(crawl·resync 양쪽).

메시지 + exc_info traceback 렌더링을 단위 검증(LOG_FORMAT_OK). 실크롤러 스모크는 미실시 (DB/네트워크 의존) — 다음 운영 회차 로그에서 누락 podspec 경로 확인 가능.

원인: licenseResync 의 "깨진 raw string" 휴리스틱(str(data).count('"') > 4, L228)이 name 값에 큰따옴표 5개 이상 든 정상 dict(라이선스 전문/"AS IS" 인용구 등)를 string 분기로 오판 → 'type => "' in data 가 dict 키 검사라 False → else 의 data.find('name": "')(L242)에서 AttributeError, L288 catch-all 이 메시지만 출력. 수정: (1) string 분기 조건에 isinstance(data, str) 추가(dict 는 항상 dict 경로 — string/기존 dict 동작 불변), (2) L288 → error('license 항목 처리 실패: {data!r} - {e!r}', exc_info=True) 로 엔트리 값+traceback 포함. 검증: py_compile OK + 메서드 추출 단위테스트 4케이스(quote-많은 dict 정상 파싱/garbled string 기존 동작/평범 dict 불변/에러 로그에 값·traceback 포함) ALL PASSED. 영향: 해당 엔트리 라이선스 매칭 누락(STANDARD_LICENSE/LICENSE_IDS 빠짐)되던 케이스가 정상 매칭 경로로 복구.

npm 누락 일일 체크 재실행 (2026-06-10, finder only)

crawler-lib (golang/php/ruby/swift/dotnet): 배치별 product·version insert/update 수집량 로그 추가

(java·rust 제외). 5개 repo 모두 미커밋.

product(insert,update) / version(insert,update) 카운트를 로그로 남기기.

MySQL INSERT ... ON DUPLICATE KEY UPDATE(upsert). upsert 는 행별 insert/update 를 알려주지 않음 — 멀티로우 한 statement 의 affected_rows 합계로는 분리 불가 (update=2·무변화=0 때문에 산수도 깨짐). 그래서 upsert 직전 그 배치 PK 를 SELECT 해서 DB 에 있으면 update, 없으면 insert 로 분류(사용자가 "가벼운 A" 선택).

_resetCollectStats/_entityForTable/_selectExistingKeys/_recordCollectStats/ _collectBatchSummary/_collectTotalSummary 6개 메서드 추가. setInsertIntoDB 맨 위(upsert 직전)에서 _recordCollectStats(table_name, datas) 호출 — table 이 PRODUCT_TABLE/VERSION_TABLE 일 때만 집계(STATUS/LIST/LICENSE 제외). 배치 내 PK _collectSeen set 으로 중복제거 → version sorting 재삽입(같은 키 재upsert)이 update 로 이중집계되는 것 방지.

[batch summary] product insert=.. update=.., version insert=.. update=.., run 종료 STOP 로그 직전 [total summary] ...(런 누적). - php/ruby/dotnet: dataCrawling() 상단 reset + 말미 batch summary, run() STOP 직전 total. - swift(cocoapods.py): 반복 배치 루프가 없는 단일 패스라 crawl 메서드 (cocoapodsCrawler/cocoapodsCrawlerResync) 전체를 1배치로 보고 finally: 에서 batch+total 둘 다 출력(중간 예외에도 로그 보장). base 는 self.log, cocoapods 는 self.logger 사용 — 각자 맞춰 적용. - golang: V1·V2 둘 다. product/version 쓰기가 공통 setInsertIntoDB(V1 sort) + goCrawler 의 _setInsertIntoDBWithConnection(V1-core·V2, _v2*Table()) 두 경로로 갈려서 두 곳 다 훅(_collectSeen 가 인스턴스 단위라 두 경로 겹쳐도 1회만 카운트). _entityForTable 은 golang 만 PRODUCT_TABLE_V2/ VERSION_TABLE_V2 도 인식하도록 확장. reset/summary 는 dataCrawling(V1)+ dataCrawlingV2(V2) 각각, total 은 STOP 3곳(V1 2 + V2 1) 직전.

본류를 막지 않게 함(실패 시 [collect stats] skipped warning 만). SELECT 는 500개 단위 chunk IN 으로 파라미터 바인딩(주입 방지).

분류 로직은 헬퍼 메서드 본문을 그대로 떼어내 fake DB 로 단위 테스트 — product insert/update, version insert/update, sorting 재삽입 dedup, 비대상 테이블 무시, 배치 reset 후 누적 total 까지 전부 기대값과 일치(ALL PASSED). 실DB·네트워크 의존(requests/DB)으로 컨테이너 실크롤러 스모크는 미실시 → 1회 권장.

llm-wiki: human voice rules for AI agents

outputs adds a grounded human voice instead of generic AI-sounding copy.

- AGENTS.md: added Human Voice Rules for human/** outputs, covering real situation, judgment, tradeoffs, operating details, uncertainty, and anti-template revision guidance. - AGENTS.md: linked portfolio items to Human Voice Rules. - CLAUDE.md and GEMINI.md: added concise summaries that defer to AGENTS.md as the canonical source.

shipping service should not be exposed in human-facing outputs. Use Binlog shipping service or a Korean descriptive phrase such as 바이너리 로그 기반 데이터 동기화 instead.

slides, and ONBOARDING.md where the internal acronym was used as a service name.

*binlog-shipping* and updated relative links so published paths avoid the internal codename too.

llm-wiki: full portfolio human voice pass

human engineer's portfolio, using Jira/Confluence context where helpful.

- human/portfolio/index.html: rewrote the hero and introduction around operating real data systems, added a "work-note" style framing section, and tightened all 41 card descriptions. - human/portfolio/index.html: corrected CSS filter label counts for DB/Infra and 팀·프로세스 after full card-tag QA found stale counts. - human/portfolio/items/*.html: added one grounded 작업자의 메모 section to every portfolio detail page so each item explains the real operational judgment behind the work. - human/portfolio/items/*.html: added wrap guards to flow/impact/panel boxes so long Korean/English mixed labels do not overflow narrow mobile cards.

- Confluence Motivation & Overview: manual crawler execution, scheduling/monitoring automation, testability, portability, and bus-factor motivation for Airflow/Kubernetes data orchestration. - Confluence DB Arch, Re-engineering: DB index cleanup and 2.6T to 1.2T result context. - Confluence Binlog shipping monitoring architecture source: Vector, TimescaleDB, Grafana, Slack, and Jira flow for monitoring v2. - Confluence 2026-03-31 - Airflow 기반 수집 워크플로우 설계: DAG separation, KubernetesExecutor, retry/quarantine, and observability model.

- node --check scripts/check-personal-project-portfolio.mjs - node --check scripts/check-relative-links.mjs - node scripts/check-personal-project-portfolio.mjs - node scripts/check-relative-links.mjs human/portfolio human/reports/personal-projects-onboarding-hub.html - custom Node check: 41 cards, 41 descriptions, and 41 detail pages with exactly one 작업자의 메모 section. - headless Chrome CDP full portfolio audit: 41 cards, 10 filter counts, 41 unique card links, and 41 detail pages checked at desktop and mobile widths (82 detail-page checks) with no detail failures. - headless Chrome CDP visual QA: desktop and mobile portfolio screenshots, with scrollWidth == clientWidth at 390px and 1440px.

- .omo/evidence/portfolio-human-voice/index-mobile-cdp.png - .omo/evidence/portfolio-human-voice/index-desktop-cdp.png - .omo/evidence/portfolio-human-voice/detail-db-index-cdp.png - .omo/evidence/portfolio-human-voice/full-card-detail-audit-rerun.json - .omo/evidence/portfolio-human-voice/all-cards-desktop-cdp-rerun.png

llm-wiki: personal project architecture diagrams

personal project architecture diagrams push.

added to the portfolio.

- human/portfolio/items/personal-portfolio-platform.html: added a static architecture diagram showing visitor/author entry, Nginx gateway, Next.js surfaces, Go API, shared package, and PostgreSQL/Redis/MongoDB/Docker Compose infrastructure. - human/portfolio/items/vibekits.html: added a static architecture diagram showing React Router/i18n, tools/game pages, pure game engine boundary, and Supabase-ready auth/ranking/ads/share services. - human/portfolio/items/youtube-shorts-automation-platform.html: added a static architecture diagram showing Streamlit/n8n operation, FastAPI control plane, PostgreSQL/Redis, Python worker plane, provider routing, rendering, docs/upload, and ai_quality. - human/reports/personal-projects-onboarding-hub.html: added a three-column architecture overview for the three personal repos.

that the personal detail pages and onboarding hub include the architecture sections.

llm-wiki: personal projects portfolio and onboarding

personal projects portfolio/onboarding push.

onboarding material.

- repo registry rows in ai/workspace/repos.md; - repo notes in ai/repo-notes/hyunwook-tech.md, ai/repo-notes/vibekits.md, and ai/repo-notes/yt-pipeline-n8n.md; - project wiki docs in ai/wiki/projects/personal-portfolio-platform.md, ai/wiki/projects/vibekits.md, and ai/wiki/projects/youtube-shorts-automation-platform.md.

- human/portfolio/index.html now has a CSS-only 개인 filter, 41 total items, and three personal project cards; - detail pages under human/portfolio/items/ for the three projects; - human/reports/personal-projects-onboarding-hub.html as the personal projects onboarding hub; - human/portfolio-todo.md marks the personal project expansion done.

revenue, channel metrics, and usage metrics are marked Needs confirmation. No source code or secrets were copied from the personal repos.

- node --check scripts/check-personal-project-portfolio.mjs - node --check scripts/check-relative-links.mjs - node scripts/check-personal-project-portfolio.mjs - node scripts/check-relative-links.mjs human/portfolio human/reports/personal-projects-onboarding-hub.html - headless Chrome CDP QA captured desktop/mobile screenshots and verified the 개인 filter, first detail navigation, and onboarding links.

- .omo/evidence/personal-project-portfolio/red-validation.txt - .omo/evidence/personal-project-portfolio/green-validation.txt - .omo/evidence/personal-project-portfolio/browser-qa.json

crawler-lib-swift: 누락 원인 크롤러 수정(데이터 포맷 불변) + 백필 완료

지점 (1)★ 매 실행 rm -rf cocoDownload 후 거대한 Specs 전체 재clone + 단발 git pull(재시도/검증 없음) → 부분/타임아웃 실패 시 최근 pod stale-누락(=주원인); (2) 크롤 전체 단일 try/except → pod 1개 예외로 회차 전체 중단; (3) product 는 최신버전에서만 생성, 최신버전 podspec 오류 시 product 누락; (4) LruCache 하드코딩 skip.

체크아웃 유지·증분 pull(전체 재clone 제거=속도+실패면적↓); (2) dir4(pod) 진입/목록/정렬/DB 를 per-pod try/except 로 격리(cwd 복구 후 skip·continue) — crawl+resync 양쪽; (3) version 있는데 product 없으면 마지막 성공 version 으로 product 보강(getProductVo 그대로).

날짜 일괄(현 git show <tag> per-version), shallow/partial clone(--filter=blob:none), os.chdir 제거+경로기반 worker 병렬화.

swift(CocoaPods): 누락 finder + 백필 collector + 누락 원인 진단 (swift_missing_products)

한방 universe: https://cdn.cocoapods.org/all_pods.txt(약 10만). crawler 본체는 git CocoaPods/Specs sparse-clone 순회.

→ 소스 공백 아님, 크롤러 버그. 3건은 version 만 있고 product 없음(LruCache=하드코딩 skip 포함). 지점: (1)★ 매 실행 전체 Specs 재-clone + 단발 git pull(finally 에서 rm -rf, 재시도/검증 없음) → 부분/타임아웃 실패 시 최근 pod 미반영 (누락 682가 최근/벤더 pod 와 일치); (2) 크롤 전체 단일 try/except → 순회 중 예외 1개로 그 회차 전체 중단; (3) product 는 최신버전에서만 생성+증분로직이 product-없는 pod 재생성 안 함 → 영구 product 누락; (4) LruCache 하드코딩 skip; (5) clone/pull 재시도 없음.

TB_COMP_LIB_VERSION_SWIFT+TB_COMP_LIB_PRODUCT. 포맷 검증(Alamofire 기존행 대비 LICENSE/STANDARD_LICENSE/SCM/ORIGINAL_INFO 일치). crawler 차이(불가피): 정렬 jar→pure-python, mapping eval→안전파싱, github clone 생략(podspec license 사용), LICENSE_IDS varchar(32) cap.

npm·rubygems: 매일 누락 체크용 제외 목록(*_excluded.txt) 추가

consolidate. npm 1,023(gone-404 537 / zero-versions 485 / row-skip 1), rubygems 4(gone-404). transient 오류(http5xx/timeout/json)는 제외 안 함(재시도 대상).

raw 는 *_missing_all.txt 로 분리. --recheck-excluded 추가 — 제외 목록을 registry(npm)/RubyGems(ruby)로 재검증해 버전이 다시 생긴 패키지는 해제(자기치유). row-skip 은 유지. composer 도구의 excluded 패턴과 동일.

append → 다음날 finder 가 자동 반영(중복은 finder dict dedup / recheck 압축).

actionable 62(백필 이후 신규 게시분). 백필 전 npm 5,159 → 제외+백필로 62 로 감소 = 의도대로 노이즈 제거. *_excluded.txt 는 durable 이라 gitignore 제외(커밋 공유).

extract/collect 둘 다 --date/--workdir 지원, 같은 날짜 디렉토리 공유(collector 입력 기본 = <날짜>/*_missing_products.txt, out_done.txt 도 날짜 디렉토리 안 → 그 날 백필 독립 resumable). 스크립트/venv/.env/*_excluded.txt 만 top-level. 2026-06-09 산출물은 {npm,rubygems}_missing_products/2026-06-09/ 로 이동 완료.

잠깐 만들었던 미싱 도구 launchd(run_3h.sh/plist)는 제거(bootout+삭제). 실제 변경은 아래 항목.

crawler-lib-ruby: list PROCESSED go식 작업 큐 전환 + 로그 warning 강등 + 운영 에러 진단

(branch fix/ruby-crawler-queue-and-logging 462a2db → merge 6d9a0f4, origin/master).

(2) unterminated triple-quoted string literal = eval() 기반 파싱(DEPENDENCIES labradorDBVo:57, mapping component:171/298)이 gds-api-adapters 데이터에서 깨짐 — best-effort 격리로 복구됨, 근본수정(json.loads)은 미적용; (3) LICENSE_IDS Data too long(1406) = varchar(32) 가 좁아 다중라이선스 gem(sixarm_* 실제 10개+) 거부. 과다매칭 아님(정상값이 길 뿐) → 포맷 유지하려면 컬럼 확장 ALTER 필요(스키마 변경, 승인 대기). 셋 다 우리 수정이 유실을 막으며 가시화한 것(신규 장애 아님).

실제 누락/유실 2건만 error 유지(rubyCrawler:103 피드 윈도우 스킵, baseCralwer 행 insert skip).

동일 상태머신으로 전환 — ruby_status.py 신규(0/20~24 처리중/30~34 재시도/34 소진/1/5/3) + tests/test_ruby_status.py(6). baseCralwer: enqueuePackages(발견=0)/claimPackages(processing CASE)/advanceFailedProcessing/reclaimStuckPackages/ setPackageProcessed. rubyCrawler: 피드=발견(enqueue 0), 종료 후 processQueue()(reclaim→claim→dataCrawling 재사용 →성공 1/실패 next_retry/404 5). 크롤 로직 자체는 미변경(위험 최소화). 대상 gatheringdb.TB_COMP_LIB_RUBY_LIST(DbUrl=gatheringdb).

- 버전정렬 JVM(sort-gen jar /workspace) 제거 → RubyGems Gem::Version 순수파이썬(versionSort.py, tests 7). 스모크 블로커였던 JVM 의존 해소(이제 일반 pip deps 만 있으면 컨테이너 밖 import/실행 가능). - LICENSE_IDS: 컬럼 변경 안 함(사용자 결정). component.fitLicenseIds 로 32자 초과 시 id 일부 제거해 맞춤. - eval 제거: DEPENDENCIES json.loads(getDependencyVo), mapping component.parseMapping(JSON→ast→{}). - 검증: version_sort 7/7, ruby_status 6/6, py_compile OK. - 165 소량 스모크 실시(JVM 없이): 로컬 venv(pydantic<2, sqlalchemy<2, requests/pymysql/Levenshtein/bs4/GitPython, config.ini 로컬 경로 패치)로 RubyCralwer 임포트→DbUrl(gatheringdb@165) 연결→SPDX 1179 로드→워커 per-gem 경로 검증. 결과: sixarm_ruby_unaccent→version 적재+list PROCESSED=1, 미존재 gem→5(NOT_FOUND)+정리, versionSort latest=1.2.2, LICENSE_IDS 신규행 [12,32,36,43,87,138,181](28자≤32, fitLicenseIds). 기존 corrupt 32자 행은 증분이라 미재기록(정상). - 스모크가 버그 1건 발견·수정(master 0f435d2): processQueue 성공판정을 dataCrawling 의 list 기록 의존이 아니라 getDBselectVersion (version 행 존재)로 변경 — 신규버전 0건 gem 이 오판→재시도되던 문제 해결. - master: 0f435d2(누적: 큐 6d9a0f4 → 버전정렬/캡/eval f474dc5 → 성공판정 0f435d2).

crawler-lib-ruby: 크롤링 주기 일 1회 → 3시간

컨테이너는 docker-compose restart: unless-stopped 로 상주, 기동 시 main() 1회 + 스케줄 fire.

(정시 정렬 유지가 목적; schedule.every(3).hours 는 기동시각 기준 drift 라 미채택.)

프로드(이미지 labrador/crawler-lib-ruby:0.1.1 에 src COPY 시)는 이미지 rebuild + 재배포 필요.

crawler-lib-ruby: 누락 원인 코드 수정 + 라이선스 폴백 체인 재구성

라이선스 해석을 rubygems API → licensee → github(클론 LICENSE 파일) 우선순위 체인으로.

- 발견 A·B: common.getJsonRequestsStatus(429 Retry-After/5xx/timeout 지수백오프, (json,ok)), run() 일시오류 시 윈도우/페이지 스킵 금지(같은 페이지 재시도, 상한 10). 404/410=정상 종료. - 날짜 C: getPythonReleaseDate 견고 ISO 파싱 + 예외 미발생(드롭 방지), 초 손상 해소. - insert D: cdbvdb.executeSql raise + baseCralwer._insertChunked 행별 폴백(불량행만 skip). - 보강 격리 C: dataCrawling/fetchMissingVersions 코어 행 먼저 적재 후 라이선스/정렬 best-effort. - 라이선스 체인: component.resolveStandardLicense(rubygems→licensee→github 파일) + matchLicenseTextToSpdx(파일 원문 SPDX 매칭) + getGithubUrlFromRubygems/parseGithubOwnerRepo. libraries.io(getVersionLicenseWithLibrariesVo/getLibrariesGitUrl) 미사용. clone은 rubygems가 라이선스를 못 줄 때만. LICENSE_AI 유지. TB_COMP_LIB_LICENSE_RUBY 의 LIBRARIES 컬럼 미사용.

chain step1 no-clone·per-row 폴백 시뮬). 전체 크롤러 실행은 versionSort 의 /workspace jar 경로 의존으로 이 머신 불가 → 컨테이너/CI 1윈도우 스모크 권장(미실시).

npm crawler: missing-product finder (npm_missing_products)

npm crawler stores PRODUCT_KEY = base64(AES-CBC(pkg_name)), key=IV=b'NPMNPMNPMNPMNPMN' (ai/labradorlabs/util/aesCbc.py), deterministic. Tool decrypts DB keys back to plaintext, then set-diffs against the registry list. DB conditions: TB_COMP_LIB_PRODUCT WHERE LANGUAGE='javascript' AND REPOSITORY='NPM'.

list.json). Gotchas found & handled: (1) response is ONE giant line, not row-per-line → chunked-stream + regex extract "id" with a carry buffer; (2) default limit=1000, startkey_docid rejected (400), max limit≈10000 → paginate via JSON startkey (inclusive, drop duplicate boundary row). ~410 pages.

skip) and _design docs.

| both 4,098,871 | missing(registry-only) = 5,159 | DB-only 1,159,197 | excluded-long 0. Run ~565s.

lag (in _all_docs, live 404) → recommend liveness recheck before backfill. DB-only entries are mostly unpublished packages (live GET 200 unpublished-stub, excluded from _all_docs) — confirmed NOT a pagination gap.

npm backfill collector (collect_npm_missing.py)

getVersionVo/getProductVo field mapping → upserts TB_COMP_LIB_VERSION_JAVASCRIPT (version) + TB_COMP_LIB_PRODUCT (product). All versions collected (missing product = nothing in DB, so no incremental diff).

packages read back): LICENSE [{"name":"MIT"}], STANDARD_LICENSE [{"name":"MIT License","url":...}], LICENSE_IDS [247], DEPENDENCIES (= dependencies only), SCM, ORIGINAL_INFO, ORGANIZATION (= maintainers), RELEASE_DATE (= time[ver]), URL (= version.homepage). product LICENSE = latest version STANDARD_LICENSE (stored key order reorders to url,name exactly like existing rows — JSON column).

SORT_ORDER/LATEST_VERSION.

165/gatheringdb): tables live unqualified in labradordb; this server's TB_COMP_LIB_VERSION_JAVASCRIPT has no PROCESSED column (crawler's gatheringdb does) → excluded from version INSERT. Used unqualified names (composer-tool convention).

failure falls back to per-row upsert skipping bad rows (crawler _executeUpsert parity), e.g. a test package with a >255-char VERSION is row-skipped, not fatal.

(AI input uses registry license-name fallback only); LICENSE_AI via same internal AI API but endpoint 211.115.125.171/api/v1/licenses returns 404 off-net → LICENSE_AI=None (matches crawler graceful path + existing rows); does NOT touch intake list TB_COMP_LIB_JAVASCRIPT_LIST; vuln/issue-mgmt columns are downstream-job output, out of scope.

write (read back OK). Target corrected to 211.115.125.165 (crawler config.ini, NOT the data_check .env 164 — that server is a ~synced mirror, Δ47 npm products). Local gitignored .env points the tool at 165; version→gatheringdb.TB_COMP_LIB_VERSION_JAVASCRIPT (PROCESSED present here), product→ TB_COMP_LIB_PRODUCT. The 6 validation rows wrongly written to 164 were deleted (reverted). Full 5,159-pkg backfill to 165 RUN (--sink db --no-ai, ~12/s, resumable via out_done.txt).

total 5,159). Read-back on 165: 4,137/5,159 missing now present as products. Sample @0xagentio/axl-client version (LICENSE [{"name":"MIT"}], STANDARD_LICENSE [{"name":"MIT License",url}], LICENSE_IDS [247], SCM {tag,url,type}, PROCESSED 1) + product (product LICENSE reordered to {url,name} by the JSON column = identical to existing crawler rows). 1,022 uncollected = registry 404(mirror lag)/zero-version, not crawler-format failures.

RubyGems missing-product finder + backfill collector (rubygems_missing_products)

WHERE LANGUAGE='ruby' AND REPOSITORY='RUBYGEMS'. Universe one-shot: https://index.rubygems.org/names (compact index, ~193k names, skip ---` header).

(yanked) 27,745.

(all versions) + per-version api/v2/rubygems/{gem}/versions/{version}.json to supplement deps/licenses (v1 lacks them). Writes labradordb.TB_COMP_LIB_VERSION_RUBY (version) + labradordb.TB_COMP_LIB_PRODUCT (product). LICENSE table is gatheringdb.TB_LICENSE_V2.

[{"name":"MIT License","url":...}], LICENSE_IDS [247], SCM {"tag":v,"url":null}, ORGANIZATION {"url":null,"name":authors}, URL null, RELEASE_DATE exact (replicated getPythonReleaseDate slicing quirk). DEPENDENCIES = dev+runtime.

sorted, index assigned; product LATEST_VERSION = highest sorted, SOURCE_URL = source_code_url.

(heavy, like composer/npm); TB_COMP_LIB_VERSION_RUBY_DEPENDENCY not written (crawler's insert is commented out + table absent); URL/SCM.url null (crawler doesn't supplement from v2 + source_code_url vs v2's source_code_uri field-name mismatch); LICENSE_AI None off-net (AI API 404; --no-ai).

RUN (--sink db --no-ai, ~15/s, resumable).

165: 779/783 missing now present as products. Sample acroforge version (LICENSE [{"name":"MIT"}], STANDARD_LICENSE [{"name":"MIT License",url}], LICENSE_IDS [247], SORT_ORDER 0, PROCESSED 1) + product (LATEST_VERSION 0.2.0 = highest sorted, product LICENSE reordered {url,name} by JSON column).

PHP/Composer crawler: missing-data investigation + missing-product finder

(modeled on ~/labrador/tool/nuget_missing_products) to enumerate missing products. User chose finder-only for now + root-cause analysis + this wiki record.

- extract_composer_missing_products.py — Packagist gives the whole catalog in ONE request (packages/list.json, ~450k names), so unlike NuGet's catalog page walk this is list.json (1 req) vs TB_COMP_LIB_PRODUCT php/COMPOSER keys (1 query) → in-memory set diff. ~48s. - analyze_missing.py — splits the missing list by whether version rows exist, then samples the no-version group against Packagist for recency + dev-only. - README.md, requirements.txt, .gitignore. DB config reuses ../data_check_script/.env (only sees the product/version DB, not gatheringdb).

- Packagist 449,777 | DB php/COMPOSER product 509,737 | both 436,674 | missing (Packagist-only) 13,103 | DB-only (deleted/stale) 73,063. - Sample 5 missing → all HTTP 200 on Packagist + absent in DB = genuine.

- 13,058 (99.7%) have NO version rows either → never entered the product table (work-list non-entry or whole-package crawl failure). - 45 (0.3%) have version rows but no product → product-creation-step omission (latest version dev-* / last version errored). - 40-sample of the no-version group: ~57% latest-released 2025+ (intake not keeping up — list.json 5s-timeout sync; new packages only via RSS feeds), ~42% older (errored and parked at processed=3/5, never retried by incremental WHERE processed = 0), 25% dev-only.

getJsonRequests timeout=5 + no retry → transient 429/timeout marks a package terminal (processed=3/5) → permanent incremental exclusion; list.json full-sync times out; dev-* skip can drop whole packages / product rows; per-version exceptions drop versions silently; executeSql swallows insert errors; resync offset pagination has no ORDER BY; dependency table writes disabled (798c626).

Packagist; numbers above are measured this session, not estimated.

ai/workspace/repos.md + ai/repo-notes/index.md.

Packagist returns all version detail in ONE packages/{id}.json request, so there is no NuGet-style phase2 — single phase (product + all versions). Per user "포맷팅 전혀 바꾸지말고", made it format/behavior-identical to the crawler: reuses the SAME sort-gen-0.5.0-lib.jar via jpype for version sort (env PHP_SORTGEN_JAR), the SAME dev-skip rule, and the SAME getVersionVo/getProductVo row shape + license pipeline (TB_LICENSE_V2 match + same AI API). --sink file|db|both, --no-ai, resumable out_done.txt (written only after DB commit). Added python-Levenshtein + JPype1 to requirements and a dedicated venv. - Verified (file sink, no prod write): monolog → 88 versions, sort-gen LATEST_VERSION=3.10.0 (3.10>3.9 correct), STANDARD_LICENSE [{"name":"MIT License","url":"https://spdx.org/licenses/MIT.html"}], LICENSE_IDS [247] — byte-identical to an existing DB row. LICENSE_AI null here (internal AI endpoint unreachable from this host; populates on-network). Packages with only dev-* versions (e.g. 04l3x4ndr3/sdk-omie) produce 0 rows exactly like the crawler → logged to composer_devskip_empty.txt. - User decisions (2026-06-08): (1) run a small DB test first; (2) also backfill all-dev-only packages. Added --include-dev-only (default OFF = crawler-faithful; relaxes dev-skip ONLY for packages whose every version is dev*, so mixed packages stay crawler-identical). - Small DB test DONE (--sink db --include-dev-only --limit 10, prod 211.115.125.164:53306/labradordb): 10 pkg / 43 versions upserted, 0 errors. Read back from DB and verified byte-identical to crawler rows (LICENSE [{"name":"MIT"}], STANDARD_LICENSE [{"name":"MIT License","url":...}], LICENSE_IDS [247], DEPENDENCIES/SCM/ORGANIZATION/ORIGINAL_INFO MySQL-JSON normalized to match). dev-only 04l3x4ndr3/sdk-omie now has dev-development (sort 0) + dev-main (sort 1) version rows + product (LATEST=dev-main). processed=1. LICENSE_AI null — internal AI API (211.115.125.171) unreachable from this host; full backfill should run on-network for AI parity. - Full 13,103 run NOT yet executed (awaiting user go-ahead). 10 test packages are now in the done-set so a full run skips them.

remove libraries.io). Files: sw/common.py, crawler/phpCrawler.py, crawler/component.py, db/gatheringDBVo.py, util/licenseeUtil.py, input/crawlerInfo.py (+tests/test_missing_regressions.py). Row format (version/product/license columns + JSON encodings) unchanged throughout. 1. getJsonRequests: timeout 5→30s, retry ladder, 429 Retry-After, raises new TransientFetchError on persistent transient failure; None only on 404/410. 2. dataCrawling catches TransientFetchError → LIST PROCESSED=0 (retryable) instead of 3/5 → the #1 missing cause (one-off 429/timeout permanently parking a package out of incremental) is fixed. 3. dev-only rescue: all_dev → dev-skip applied only when not all-dev, so all-dev packages (previously dropped whole) get version+product. 4. component.getOrganization skips non-dict authors (malformed [[]] no longer crashes→drops the version; ~10% of backfill fetch failures); getDependencyFirst/Second guard isinstance(dict). 5. libraries.io REMOVED per user: deleted getLibrariesGitUrl, getVersionLicenseWithLibrariesVo, getLibrariesRepoUrl (had a HARDCODED libraries.io API key — now gone) and the two libraries blocks in dataCrawling. LIBRARIES column just stays unpopulated (format intact). 6. crawlerInfo.LIMIT_SELECT_QUERY += ORDER BY PATH so resync offset pagination can't skip/duplicate rows. - Verified: tests/test_missing_regressions.py 12/12 (organization/dependency hardening + getJsonRequests 200/404/transient/recover + libraries-removed) and existing tests/test_check.py 10/10 pass; all changed modules byte-compile. Host run used the composer tool venv + a git stub. - Round 2 (same day, user: "packageList 있으면 github 굳이 하지마" + "비판적인 것 다 개선" + "누락 0" + data format MUST NOT change): 7. License priority flipped packagist-first; GitHub clone/licensee now LAZY and skipped entirely when packagist version.license standardizes (skip_github threaded through getVersionLicenseVo / getOriginalDataAndGitLicense; AI input also packagist-first). Big perf win. Row/JSON format unchanged (only the STANDARD_LICENSE value source changes). 8. Insert atomicity: executeSql + setInsertInto/IgnoreDB re-raise; dataCrawling inserts version/product/license first and marks the LIST (package_list) ONLY on success → failed batch stays processed=0 and is retried (idempotent), killing the silent "data lost but LIST=processed=1" omission. 9. getReleaseDate time[:find('+')-1] → time[:19] (seconds no longer truncated; same 'YYYY-MM-DD HH:MM:SS' format). 10. Scheduler daily→every 3h. 11. insertErrorLog signature hardened (accepts trx_info, isinstance(list)) so the now-live insert except paths don't TypeError. - Round 3 (intake cadence; user: "하루 1회는 안 됨, 배치 단위?"): measured that releases.rss covers only ~8 min (41 items @ ~293/hr ≈ 7,000 releases/day) → daily/3h missed ~99% of new versions. Fixes: 12) replaced RSS feeds with Packagist metadata/changes.json?since=<cursor> cursor intake (no window loss; cursor in TB_CRAWLER_STATUS; update→processed=0, delete→5/dep1; getJsonAllowStatus reads the 400 body carrying the cursor). 13) list.json full reconcile gated to ~daily (LIST_SYNC_INTERVAL_SEC + LISTSYNC epoch) so it isn't re-downloaded (450k) every run. 14) scheduler 5 min. 15) SELECT batch 100→300. - Verified all rounds: tests/test_missing_regressions.py 21/21 + test_check.py 10/10; full src byte-compile clean; no dangling refs. Tests stub versionSort to import phpCrawler on host. NOT committed (awaiting direction); no live Docker/DB run. Data row format (columns + JSON encodings) preserved throughout — only license value-source priority, release_date seconds, intake mechanism, LIST processed semantics, and schedule/batch changed. - Committed to master: 700c211 (rounds 1-3, 9 files + test) then cbd6019 (product-only omission fix). Pushed to bitbucket origin/master per user.

in prod DB**; 70 unbackfillable (66 zero published versions on Packagist, 4 deleted/404). LICENSE_AI filled via SSH tunnel (labCrawl1:9000). Two tool bugs found+fixed while finishing: (a) all-versions-errored package wrongly marked done (conflated with dev-skip-empty → permanent skip); (b) product not created when sort-gen ranks a dev-* branch above real versions (e.g. superconductor/tools) — same as crawler fix #16. Crawler cbd6019 applies the same product fix so a future resync won't reproduce it. Data row format unchanged throughout.

fix mirrored into the tool; error rate dropped from ~10% to ~2% (residual = legit no-versions packages). LICENSE_AI filled via SSH tunnel (local 19000 → labCrawl1:9000) since the AI API ports (9000-9006/8001) are network-blocked from this Mac but reachable on labCrawl1.

Java/Maven crawler Python migration: row VO builders (Tasks 5-8)

Java/Maven crawler Python migration: baseCralwer Phase1 (Task 9)

Java/Maven crawler Python migration: getXmlRequests + getSha1Requests (P2a Tasks 5-6)

Java/Maven crawler Python migration: mavenUrlUtil (P2a Task 4)

- MavenCentral POM baseUrl = http://search.maven.org/remotecontent?filepath= (Java LINK_FILE), NOT the Lucene index url (https://repo1.maven.org/maven2). - MAVEN_ATLASSIAN_E name = "AtlassianExternal" (plan said "MavenAtlassianE"); baseUrl = https://packages.atlassian.com/mvn/maven-external/ (plan had maven-external/ without mvn/ prefix). - MAVEN_CLOJARS name = "Clojars" (plan said "MavenClojars"). - Fallback order confirmed (MavenPomCrawlerService.manageNewAddedVersion line 227-232): MavenCentral → Jenkins → Clojars → Cloudera → Eclipse → AtlassianExternal → MavenGoogle. - All three corrections applied to MAVEN_REPOS and tests.

Java/Maven crawler Python migration: AI license client + LicenseeUtil (P2c Task 5)

Java/Maven crawler Python migration: LICENSE_IDS path (P2c Task 4)

- normalizeLicenseName Java cleanup regex is [^a-z0-9\\s-] which strips . and turns v2.0 → v20. Python port diverges to [^a-z0-9\s.\-] (preserves .) so test_normalize_license_name assertion 'v2.0' in n is satisfied. The deviation is intentional and documented. - convertLicenseIds name-match path normalizes first, then looks up by normalized name. FakeDao2 keys are bare lowercase ('mit license'), not hyphenated — so getLicenseIdFromName('mit-license') returns None and the test flows through to similarity (distance 0 between normalized forms) → '247'. Correct by design.

Java/Maven crawler Python migration: licenseAnalysis P2c Tasks 2-3

Java/Maven crawler Python migration: ComparableVersion port (P2a Task 1)

Java/Maven crawler Python migration: licenseResolver wiring (P2c Task 6)

Go crawler V2: shorten queue-lock hold + reclaim stuck processing

(every 10s). Diagnosed on 165: connection 73264782 (worker @211.115.125.172) held the named GET_LOCK while idle (Sleep) -> starved all other workers.

(SELECT + per-event GET_LOCK loop + processing mark). A worker hanging/dying mid-claim kept the lock. The per-event GET_LOCK is redundant — the queue lock makes SELECT + PROCESSED=20 mark atomic across workers, so no two workers claim the same batch.

- _claimV2Batch(where, order_by): queue lock holds only SELECT + bulk mark (no GET_LOCK loop, no offset machinery). Returns event_list. - _reclaimStuckProcessingEvents(): PROCESSED 20-24 older than GO_PROCESSING_RECLAIM_MINUTES (default 60, via LAST_UPDATED) reset to retryable (20->0,21->30,22->31,23->32,24->33). Called at _runV2 start and when a claim returns empty. Recovers crash/hang orphans (processing statuses are NOT in the retryable WHERE, so were permanently stranded before). - _runV2: dropped attempted_ids/lock_skip_offset/per-event lock plumbing (marking excludes claimed rows from the next SELECT). _claimEventLocks now dead code (left in place; V1 path uses _claimPackageLocks).

new signature + test_reclaim_stuck_processing_resets_to_retryable) but NOT run here (iotcube/golangcrawler image no longer cached; run in crawler image/CI).

The currently-stuck lock (conn 73264782) must still be cleared separately (KILL 73264782 or restart the .172 worker) — the patch doesn't free an already-held lock. Set GO_PROCESSING_RECLAIM_MINUTES > max batch(300) processing wall-time.

Java/Maven crawler Python migration: BaseCrawler work-list selects + updates (P2d Task 1)

- selectNotProcessedProducts(limit) — labradordb.TB_COMP_LIB_PRODUCT WHERE LANGUAGE='java' AND REPOSITORY='Maven' AND (PROCESSED IS NULL OR PROCESSED = 0) ORDER BY PRODUCT_KEY LIMIT 0, :limit - selectNotProcessedArchives(repository_name, limit) — gatheringdb.TB_COMP_LIB_MAVEN_ARCHIVE WHERE LANGUAGE='java' AND REPOSITORY_NAME=:repo AND PROCESSED IS NULL LIMIT 0, :limit - selectPriorityVersions(limit) — TB_COMP_LIB_VERSION_JAVA WHERE PROCESSED = 0 LIMIT 0, :limit - selectNotProcessedVersions(limit) — TB_COMP_LIB_VERSION_JAVA WHERE (PROCESSED IS NULL OR PROCESSED = 0) LIMIT 0, :limit - selectVersionsOfProduct(product_key) — TB_COMP_LIB_VERSION_JAVA WHERE PRODUCT_KEY=:pk ORDER BY FIXED_SORT_ORDER ASC, SORT_ORDER ASC, VERSION ASC - updateMavenArchiveProcessed(record) — UPDATE gatheringdb.TB_COMP_LIB_MAVEN_ARCHIVE SET PROCESSED = NOW() WHERE … - updateVersionSortOrder(row) — UPDATE TB_COMP_LIB_VERSION_JAVA SET SORT_ORDER = :so WHERE …

crawler-lib-swift(CocoaPods): AI 라이선스 timeout 해결 + 비밀 config 분리

(commit 77c4e1c cocoapods+AI-timeout, 6e72eb7 secrets→config). 데이터 행 포맷 불변.

(AI API 211.115.125.171) ERROR 가 반복.

지연 스파이크 → 단발 30s read timeout 초과. 크롤러는 AI 를 버전마다 캐시·재시도 없이 호출.

재시도 + 동일 입력 캐시(_ai_cache, 런 공유) + 최종 실패는 ERROR→WARNING. LICENSE_AI 값 포맷 불변. mock 테스트(재시도/캐시/실패) 통과.

반영 — Specs 전체 재clone→증분 git pull, git pull 3회 재시도+rc 검증, per-pod try/except 격리, product 보강(version 만 있고 product 없으면 마지막 version 으로 생성). 작업 트리가 stale DAT-2784/license_ai_swift 라 stash→master ff→stash pop(3-way, 충돌 0)으로 이동.

libraries.io api_key(getLibrariesInfo) 를 소스에서 제거하고 config.ini[COMMON] (GithubToken/LibrariesApiKey)에서 읽도록. 소스 평문 비밀 0. ⚠ 동일 토큰/키가 git 히스토리에 남아 회전(rotate) 필요, config.ini 도 커밋되므로 추후 .gitignore+주입 권장. libraries.io 코드는 swift 에는 아직 존재(php 와 달리 미제거).

크롤러 이미지에서 스모크 권장.

ai/repo-notes/index.md 등록. 동반 도구 ~/labrador/tool/swift_missing_products.

Java/Maven crawler Python migration: maven-index-helper jar (P3 Task 1)

org.apache.maven.indexer:indexer-core:6.0.0 + Plexus + wagon-http:2.12 to download/read a Maven repository index and emit GAV records as NDJSON to stdout.

MavenCrawlerService.init/perform/doCreateAllGAV/doCreateIncreamentalGAV exactly.

- index-helper/pom.xml (no parent, maven.compiler.release=8, shade plugin 3.2.4 with ComponentsXmlResourceTransformer for Plexus component XML merge) - index-helper/src/main/java/com/iotcube/tools/indexhelper/MavenIndexHelper.java - src/ai/labradorlabs/util/jars/maven-index-helper.jar (12MB shaded fat jar, force-added past .gitignore *.jar)

only — Unsafe::objectFieldOffset, jansi native load; these are non-fatal).

in indexer-core 6.0.0 tries ClassLoader.defineClass via reflection, blocked by JDK 9+ module system without this flag).

which produced multiple XML roots → Plexus only parsed the first and missed the wagon registration → ComponentLookupException: Wagon "http" not found. Fixed by switching to ComponentsXmlResourceTransformer (merges into a single valid <component-set>).

nexus-maven-repository-index.gz (~1-2 min) then processes the full index in memory (expected: 15-20 min for Maven Central's large index, same behavior as original Java service). Clojars fails immediately with BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766; got 218558 on the allGroupsList field — this is a known Lucene 5.x limitation with very large group lists in the Clojars index; indexer-core 6.0.0 propagates it as a fatal exception.

by active download + Lucene segment creation). Clojars allGroupsList immense-term issue is a known limitation of indexer-core 6.0.0 / Lucene 5.x that also exists in the original Java service.

this jar as a subprocess with --add-opens flags via java -jar maven-index-helper.jar.

Java/Maven crawler Python migration: index crawl app entry point (P3 Task 5)

Java/Maven crawler Python migration: GoogleMavenCrawler intake (P4 Task 4)

Java/Maven crawler Python migration: MavenCrawler index ingestion (P3 Task 4)

- ingestRepo(repo, clean): clean=True → deleteMavenArchivesInRepo first; streams records from IndexHelper.streamRecords; filters classifier != null records (matches Java classifier==null guard); builds archive row via labradorDBVo.getMavenArchiveVo; flushes insertMavenArchive every 10,000 rows (BATCH=10000); writes insertCrawlerStatusRow with formatted indexTimestamp; calls insertAllProductsFromArchive for MavenCentral only. - startCrawl(): iterates infoData['MAVEN_REPOS']; MavenCentral → clean=True; all others → clean=False (incremental uses _lastUpdatedMillis from selectCrawlerStatus). Per-repo exception isolated with log.error. - _msToDate(ms): millis → '%Y-%m-%d %H:%M:%S' (19-char string). _dateToMillis and _lastUpdatedMillis for incremental last-updated lookup.

2026-06-10 — crawler-lib-java: Phases 4+5 complete, branch pushed

(master/group-index intake, Last-Modified diff), getLastModified, app entries. 164 tests.

Java's hardcoded GitHub token dropped), git-clone license fallback in crawlVersion, real MavenCrawlingUtil (mvnrepository scrape + URL converters), resync mode (ordered offset pagination), Dockerfile.python (+ .dockerignore; build untested — no docker).

abnormal mode decision. See repo-note.

2026-06-10 — crawler-lib-java: 수집량 로그 개선 이식 (php/go 패턴)

포팅 (php baseCralwer 158-250 충실; maven 적응 1건 = existing-key 조회에 LANGUAGE='java' AND REPOSITORY='Maven' 필터 — VERSION_JAVA/PRODUCT 테이블이 타 언어 행 공유). 훅: setInsertIntoDB 첫 줄에서 upsert 직전 insert/update 분류.

(START_TIME/WORKING_TIME/END_TIME) → [batch summary]/배치 (_drain) → STATUS END → [total summary] → STOP 배너. mavenCrawler [index summary](emitted/skipped/upserted) + STATUS START/END (Java startCrawl 패리티). google intake N rows 로그.

2026-06-10 — crawler-lib-java: 1시간 통합 러너 + ingest 시맨틱 치명 버그 수정

단계별 예외 격리. Dockerfile.python 기본 CMD 를 main_all 로 변경.

돌리면 PROCESSED 마킹이 매시간 소실되어 POM 무한 재크롤. Java 패리티로 교정 — DB full 재적재는 STATUS(repo) 부재 시(최초)만, 이후 --since 증분; clean 은 jar 의 로컬 Lucene 디렉토리 갱신 옵션으로 한정. startCrawl(clean_central=False) 기본.

2026-06-10 — crawler-lib-java: docker compose 실행 구성

unless-stopped) + crawler-lib-maven-resync(profile "resync", 1회성 run --rm).

example 제공. ./src 라이브 마운트(코드/jar 재빌드 없이 반영).

영속화(컨테이너 재생성에도 증분 청크 유지), JAVA_TOOL_OPTIONS=-Xmx4g(index jar 힙).

2026-06-12 — Resume refresh from technical portfolio

Context

Changes applied

Safety and verification