LLM WikiAccess-protected knowledge portal

WIKI

crawler-lib-php

AI Summary Purpose Durable note for the Packagist/Composer package crawler at ~/labrador/crawler/crawler lib php . Records the 2026 06 08 missing data investigation root cause analysis only; no crawler code changed yet and the standalone mi

경로ai/repo-notes/crawler-lib-php.md
카테고리Repo Note
태그#crawler #data-pipeline #lib #license #mysql #notes #php #repo #repo-note #security

# crawler-lib-php

AI Summary

Purpose:

Key points:

Relevant when:

Do not read full document unless:

Linked documents:

Open Questions

Details

Repo Info

Pipeline

run() → intake (list.json + 2 RSS feeds, incremental only) → paginate LIST → dataCrawling() per package → insert version / product / license.

Missing-Data Root Causes (2026-06-08 analysis — NOT yet fixed)

  1. Transient HTTP failure → permanent incremental exclusion (dominant).

getJsonRequests (src/ai/labradorlabs/sw/common.py) uses timeout=5, no retry, no try/except. In dataCrawling, a non-200 (404/429/5xx) → getPackageListVo(..., 5, 1) (PROCESSED=5); an exception or bad shape → status 3 or 10. All of these are non-zero, so the package is never re-selected by incremental WHERE processed = 0. Only a full resync recovers it. This matches the ~42% "older, still missing" sample slice.

  1. list.json full-sync likely times out.

packageListCrawler fetches the multi-MB list.json with the same timeout=5 → almost always times out → caught and logged, 0 new packages added that run. New packages then enter only via the small RSS feeds, so packages outside the feed window never enter the LIST. Matches the ~57% "recently released, missing" slice.

  1. dev-* skip can drop whole packages / product rows.

In the version loop: if len(versions)>1 and version[:3]=='dev': continue. A package whose versions are all dev-* (and >1) has every version skipped → no version rows AND no product. A package whose latest sorted version (versions[-1]) is dev-* reaches continue before the if version == versions[-1] product block → version rows exist but no product (the 45-package group). Depends on the Java sort-gen ordering of dev branches.

  1. Per-version exception silently drops that version (per-version try/except) while the package can still look processed.
  1. Insert failures are swallowed. cdbvdb.executeSql catches and logs but does not re-raise; version/product/list are separate inserts, and the LIST row (processed=1) is its own insert — so a failed version batch can be lost while the package is marked done.
  1. Resync offset pagination has no ORDER BY (SELECT ... LIMIT :start, 100) → risk of skipped/duplicated rows across pages during a resync.
  1. Dependency table writes are intentionally disabled (commented out; commit 798c626). DEPENDENCIES JSON in the version row is still populated.

Crawler Fixes Applied (2026-06-08)

All changes preserve the version/product/license row format (columns + JSON encodings). They change fetch robustness, LIST PROCESSED semantics, version selection, and remove libraries.io. Verified: tests/test_missing_regressions.py 12/12 + existing tests/test_check.py 10/10; all changed modules byte-compile. Files: sw/common.py, crawler/phpCrawler.py, crawler/component.py, db/gatheringDBVo.py, util/licenseeUtil.py, input/crawlerInfo.py.

  1. getJsonRequests robustness (sw/common.py) — was timeout=5, no retry, no try/except. Now: timeout=30, retry ladder (backoff), 429 Retry-After, retries on timeout/connection/5xx. Returns None only on 404/410 (permanent); raises new TransientFetchError when transient failures persist. Fixes the multi-MB list.json always-timing-out and transient per-package failures.
  2. Transient ≠ terminal (crawler/phpCrawler.dataCrawling) — added except TransientFetchError that records the LIST row as PROCESSED=0 (retryable) instead of 3/5. This is the #1 missing cause: a one-off 429/timeout no longer parks a package out of the incremental WHERE processed = 0 set forever.
  3. dev-only rescue (phpCrawler.dataCrawling) — all_dev = all(v[:3]=='dev' for v in versions); the dev-skip (len>1 and v[:3]=='dev') is now applied only when NOT all-dev. All-dev packages (which the crawler previously dropped entirely → no version/product) are now collected. Mixed packages keep prior behavior. (Matches the backfill tool's --include-dev-only.)
  4. malformed-authors / require hardening (crawler/component.py) — getOrganization now skips non-dict author entries (e.g. authors: [[]] no longer raises AttributeError and drops the version); getDependencyFirst/Second guard isinstance(dict). This was ~10% of backfill fetch failures. Output format identical.
  5. libraries.io removed (user-requested) — deleted component.getLibrariesGitUrl, gatheringDBVo.getVersionLicenseWithLibrariesVo, licenseeUtil.getLibrariesRepoUrl (which contained a hardcoded libraries.io API key — now gone), and the two libraries.io blocks in phpCrawler.dataCrawling. License now comes from GitHub clone + packagist only. The LIBRARIES column in TB_COMP_LIB_LICENSE_PHP simply stays unpopulated (schema/format unchanged).
  6. resync pagination ORDER BY (input/crawlerInfo.py) — LIMIT_SELECT_QUERY now ... ORDER BY PATH LIMIT :start, 100. Without a stable order, offset pagination during resync could skip/duplicate rows. PATH is the LIST PK so the order uses the index.

Round 2 (2026-06-08, user: "packageList 있으면 github 굳이 하지마", "비판적인 것 다 개선", "누락 0", data format MUST NOT change)

  1. License priority flipped to packagist-first + lazy GitHub (phpCrawler.dataCrawling, gatheringDBVo.getVersionLicenseVo, component.getOriginalDataAndGitLicense) — was GitHub-first (clone+licensee always run). Now: if packagist version.license standardizes (version_dict['STANDARD_LICENSE'] not None) the package uses it and GitHub is skipped entirely (skip_github=True); the git clone is done lazily only on the first version that lacks a packagist license. getVersionLicenseVo now picks PACKAGE_REPOSITORY (packagist) first, GITHUB only as fallback. AI-license input is likewise packagist-first (clone license-file text only when packagist absent). Big perf win (no clone for the common case) and matches the user's request. Row/JSON format unchanged: version_license_dict keys and ORIGINAL_DATA={PR,GITHUB:{URL,TAG,LICENSEE_TEXT}} structure identical; only the STANDARD_LICENSE value source changes.
  2. Insert atomicity (cdbvdb.executeSql, baseCralwer.setInsertInto/IgnoreDB, phpCrawler.dataCrawling) — executeSql and the two setInsert* now re-raise instead of swallowing. dataCrawling wraps version/product/license inserts in try/except and inserts the LIST package_list (the PROCESSED markers) only after they succeed; on failure it returns without marking, so the batch stays PROCESSED=0 and is retried (idempotent upsert). Removes the silent "data lost but LIST=PROCESSED=1" omission. Trade-off: a systematic insert failure now wedges progress (loud) instead of silently dropping — correct for a no-omission goal; the row format is DB-compatible (the backfill tool upserts the same rows fine).
  3. getReleaseDate corrected (component.py) — time[:time.find('+')-1] (could drop the seconds digit) → time[:19]. Same stored format 'YYYY-MM-DD HH:MM:SS', correct value, handles +00:00/Z/no-tz.
  4. Scheduler every 3h (app/main.py) — daily 00:00 → schedule.every(3).hours, so intake/retry backlog clears faster (with fix #1 this keeps the catalog current).
  5. insertErrorLog signature hardened (baseCralwer.py) — now accepts trx_info and uses isinstance(list) (was type[list], a bug). Prevents a TypeError now that the insert except paths are live (fix #8). DB persistence to TB_CRAWLER_ERROR_LOG still intentionally off (schema unconfirmed).

Round-2 verified: tests/test_missing_regressions.py 15/15 (now also: packagist-first license skips github, getOriginalDataAndGitLicense(skip_github) doesn't call licensee, getReleaseDate keeps full seconds) + test_check.py 10/10; full src byte-compiles; no dangling refs.

Round 3 (2026-06-08, intake cadence — user: "하루 1회는 안 됨, 배치 단위?")

Measured: releases.rss covers only ~8 minutes (41 items @ ~293 releases/hr ≈ 7,000/day); packages.rss ~9h. Daily/3h runs missed ~99% of new versions — a major omission source. One incremental run already drains the entire processed=0 set, so cadence + feed-window (not batch size) drive omission.

  1. changes.json cursor intake (phpCrawler.changesCrawler, crawlerInfo.CHANGES_URL, common.getJsonAllowStatus) — replaced RSS feeds (updatePackageListCrawler/newPackageListCrawler removed) with Packagist's cursor-based metadata/changes.json?since=<cursor>: every update/delete since the cursor + a new cursor, no window loss regardless of run gaps. Cursor in TB_CRAWLER_STATUS (TYPE PhpCralwer_CHANGES). update→LIST processed=0, delete→processed=5,deprecated=1; name = action.package.rsplit('~',1)[0]. Bootstrap stores only the current cursor (history via list.json/resync). since too old → 400 → cursor reset + log to resync. Transient → cursor not advanced (next run retries same since, lossless). getJsonAllowStatus reads the 400 body that carries the cursor.
  2. list.json reconcile gated ~daily (packageListCrawler + LIST_SYNC_INTERVAL_SEC) — the 450k-name reconcile no longer runs every incremental; gated by a PhpCralwer_LISTSYNC epoch (default 24h). Stays the new-package safety net.
  3. Scheduler every 5 min (app/main.py) — daily/3h → 5 min; changes.json is cheap and list.json is gated, so no 450k re-download per run.
  4. SELECT batch 100 → 300 (crawlerInfo.LIMIT_SELECT_QUERY) — fewer round-trips (chunk size only; one run still drains all processed=0).

Round-3 verified: tests/test_missing_regressions.py 21/21 (adds changesCrawler bootstrap / update+delete dedup + ~-strip / since-too-old reset / transient-no-advance, and getJsonAllowStatus 200·400·transient) + test_check.py 10/10; full src byte-compiles. Tests stub versionSort (JVM//workspace jar) to import phpCrawler on host; changesCrawler doesn't use it.

Round 4 (2026-06-08, product-only omission)

  1. *product created from last processed version, not strictly versions[-1] (phpCrawler.dataCrawling, committed cbd6019)* — sort-gen can sort a dev-* branch ABOVE real versions (e.g. superconductor/tools: 20241105.* < dev-main). The old if version == versions[-1] then landed on the dev version, which dev-skip continues → version rows existed but NO product (the "45 version-only" omission class). Now dataCrawling tracks the last processed version and builds the product from it after the loop. getProductVo unchanged → product row format/fields identical; LATEST_VERSION = highest real version. (Same bug + fix was found in the backfill tool while completing the run.)

Round 5 (2026-07-06, zero-row 자가 백필 — hakanispirli/parasut-laravel 실측)

사건: hakanispirli/parasut-laravel 이 LIST 에는 있으나(PROCESSED=1, 6/29 16:20 KST 크롤) TB_COMP_LIB_LICENSE_PHP/TB_COMP_LIB_VERSION_PHP 에 행 0개. Packagist 의 현재 패키지 생성시각은 같은 날 23:44 KST(14:44 UTC), 전 버전 published-time 이 23:46 동일 — 즉 아침에 등록된 첫 incarnation 을 버전 게시 전에 크롤해 0개 수집→성공(1) 마킹, 이후 삭제→재등록의 changes.json update/delete 이벤트가 유실돼 영구 누락. 이벤트 유실의 정확한 지점은 확정 불가(에러로그 DB 영구화 off) — 후보: enqueue insert 조용한 실패(커서는 전진), 다운타임+커서 리셋, changesCrawler 내 예외. 7/1 ladder(9d5cf76)는 이미 1로 굳은 행을 복구하지 못한다(커밋 메시지에도 "별도 백필" 필요로 명시돼 있었음).

측정(2026-07-06): done 마킹인데 VERSION 행 0개 = 2,419건 (PROCESSED=1: 1,417건, 최신이 hakanispirli / PROCESSED=3: 1,002건, 전부 2026-04 이전). 대조군 northlab/parasut-laravel(7/5 등록)은 정상 수집 → 파이프라인 자체는 정상.

  1. zeroRowReconcile() (신규) — 하루 1회(PhpCralwer_ZEROROW epoch 게이트,

ZERO_ROW_SYNC_INTERVAL_SEC=86400): DEPRECATED=0 AND PROCESSED IN (1,3,39) 이고 VERSION 테이블에 행이 없는 LIST 를 안티조인(NOT EXISTS, IDX_PRODUCT_KEY 사용)으로 찾아 PROCESSED=0, LAST_UPDATED=NOW() 재큐잉. SELECT(무락) 후 PATH IN 500 청크 UPDATE 로 긴 갱신 락 회피. 같은 run() 의 페이지네이션이 곧바로 재수집(AI 라이선스 포함 풀 수집 — 버전-증분은 DB 에 행이 없으므로 전량 수집). 여전히 0개면 ladder 30..38→39, 404면 (5,1) → 발산 없음. 39 도 sweep 대상이라 ladder 포기 후 버전이 늦게 게시돼도 최대 하루 안에 복구. 실패 시 epoch 미갱신 → 다음 실행 재시도. → 과거 피해 2,419건이 배포 첫 실행에서 자동 백필(수동 백필 불필요). 첫 실행은 2,419×~1.3s ≈ 1시간 정도 추가 소요 예상.

  1. changes enqueue 전부 실패 시 커서 미전진 (changesCrawler) — enqueue upsert 가

전부 실패(예: DB 순단)했는데 커서만 전진하면 그 윈도우의 이벤트가 조용히 유실된다 (이번 누락의 유력 경로). insertPackageList/setInsertIntoDB/setInsertIgnoreDB/ _insertChunked 가 적재 성공 행 수(int)를 반환하도록 변경, 0 이면 커서 미저장 → 다음 실행이 같은 since 로 재시도. 부분 실패는 행별 에러 로그 후 전진(불량 행 1건이 커서를 영구히 물지 않도록).

  1. zero-version ladder 30~39 확장 (f210373) — golang 크롤러의

PACKAGE_STATUS_RETRY_1..9(30..38)/EXHAUSTED(39) 컨벤션과 정렬. 앞 5-rung 백오프(1h/5h/10h/16h/24h)는 유지(피드 앞섬은 대부분 수 시간 내 해소 + 운영 중 30..34 행 의미 보존), 뒤 4-rung 은 golang 장주기 정렬 35=2d/36=5d/37=7d/38=14d (누적 ≈24일). sweep 대상은 (1,3,LADDER_EXHAUSTED) 상수 참조. 마이그레이션: 운영 DB 에 35 이상 행 없음(31/32 각 1건) → 35 재정의(구 포기→rung 6) 무영향; 배포 전 old 코드로 35 도달한 행도 새 코드에선 재시도 계속(의도된 방향). 참고: maven(crawler-lib-java)은 90..97/98 카운터 방식이라 별개, npm(모노레포)은 30..33/34=gone — "30~39" 범위 컨벤션의 원본은 golang.

검증: tests/test_zero_row_reconcile.py 11건(sweep 게이트/대상 where/청크/epoch/실패, 커서 미전진) + test_ladder 갱신 포함 전체 suite 72건 green (composer tool venv python 으로 실행 — host 에 requests 없음). 안티조인 SQL 실 DB 검증(38s, hakanispirli 포함 확인). 커밋 d968c5c+f210373 — master 푸시 완료(5a6792d..f210373, 2026-07-06). 운영 배포 완료(2026-07-06 10:42 KST=01:42 UTC; 11:06 KST 컨테이너 소실 인시던트 후 11:13 재생성·런타임 검증 — W28 참조): 크롤러는 labCrawl1(data-crawl-01, 211.115.125.171) /product/crawler/crawler-lib/crawler-lib-php 에서 docker 컨테이너 crawler-lib-php 로 구동 (src bind-mount → 서버 git pull --ff-only + docker restart 로 반영, 이미지 재빌드 불필요. 앱 로그는 stdout 아닌 /data/logs/crawler-lib-php/phpcrawler.log). 재시작 첫 실행 실측: changes intake 3,520건 + [zero-row reconcile] 재큐잉 2,408건 → 백로그 드레인 시작.

Backfill outcome (2026-06-08)

The 13,103 missing products were backfilled into TB_COMP_LIB_PRODUCT/TB_COMP_LIB_VERSION_PHP via composer_missing_products/collect_composer_missing.py (LICENSE_AI filled through an SSH tunnel to the AI API). Final: 13,033 / 13,103 (99.5%) now have product rows; the remaining 70 are unbackfillable — 66 are registered on Packagist's list but have zero published versions, 4 are deleted (404). Two tool bugs were found and fixed during completion: (a) a package whose every version errored was wrongly marked done (conflated with dev-skip-empty) → permanent skip; (b) the same versions[-1]/dev product-omission as crawler fix #16.

Still NOT applied

Performance (2026-06-23 분석 + 버전-증분 적용)

dataCrawling 은 패키지를 순차 처리하고, 패키지마다 Packagist fetch 1회 + 버전마다 작업.

측정값:

laravel 1.8s). 순차라 풀 리싱크면 514K×~1.3s ≈ ~185h 가 fetch 대기. getJsonRequests 는 timeout=30·retries=4·backoff → 429/timeout 최악 ~90s/패키지.

avg 11.5 / max 11,092 / ≥100버전 7,232개. → 무캐시 AI 호출 총 ~5.9M회.

시간 소모 순위: ① 순차 + Packagist fetch(패키지당 1~1.8s) ② 버전당 AI 호출 무캐시(swift 크롤러엔 _ai_cache 있으나 php엔 없음) ③ AI 호출마다 요청 body(≤5000자)+응답 INFO 로깅(5.9M줄급) ④ packagist 라이선스 없을 때 full git clone ⑤ 버전마다 SPDX(~1000행) Levenshtein 매칭.

후속 권장(미적용): 패키지 병렬화(최대 wall-clock 개선), AI 입력 캐싱(B, resync 에 유효), per-version 로깅 축소(C), 라이선스 매칭 캐싱(D).

버전-증분 수집 적용 (브랜치 feature/php-version-incremental, 커밋 b3f317f)

버전만 수집**하도록 변경. 데이터 포맷 불변.

기존 버전은 AI/clone/insert 생략. getVersionVo·sort_order·last_version_dict 는 유지해 product 는 최신 버전 기준으로 항상 정확(최신이 기존 버전이어도). batch summary 에 skip 카운트.

(정정/리프레시는 resync 전체 재수집 — SPM 증분과 동일 철학).

byte-compile OK. 미푸시/미머지.

Missing-Product Finder Tool

~/labrador/tool/composer_missing_products/ (sibling of nuget_missing_products / maven_missing_products).

- Out: composer_missing_products.txt (Packagist-only = crawler omissions), composer_db_only.txt (DB-only = deleted/stale candidates).