LLM WikiAccess-protected knowledge portal

WIKI

crawler-lib-php

AI Summary Purpose Durable note for the Packagist/Composer package crawler at ~/labrador/crawler/crawler lib php . Records the 2026 06 08 missing data investigation root cause analysis only; no crawler code changed yet and the standalone mi

경로ai/repo-notes/crawler-lib-php.md
카테고리Repo Note
태그#crawler #data-pipeline #lib #license #mysql #notes #php #repo #repo-note #security

# crawler-lib-php

AI Summary

Purpose:

Key points:

Relevant when:

Do not read full document unless:

Linked documents:

Open Questions

Details

Repo Info

Pipeline

run() → intake (list.json + 2 RSS feeds, incremental only) → paginate LIST → dataCrawling() per package → insert version / product / license.

Missing-Data Root Causes (2026-06-08 analysis — NOT yet fixed)

  1. Transient HTTP failure → permanent incremental exclusion (dominant).

getJsonRequests (src/ai/labradorlabs/sw/common.py) uses timeout=5, no retry, no try/except. In dataCrawling, a non-200 (404/429/5xx) → getPackageListVo(..., 5, 1) (PROCESSED=5); an exception or bad shape → status 3 or 10. All of these are non-zero, so the package is never re-selected by incremental WHERE processed = 0. Only a full resync recovers it. This matches the ~42% "older, still missing" sample slice.

  1. list.json full-sync likely times out.

packageListCrawler fetches the multi-MB list.json with the same timeout=5 → almost always times out → caught and logged, 0 new packages added that run. New packages then enter only via the small RSS feeds, so packages outside the feed window never enter the LIST. Matches the ~57% "recently released, missing" slice.

  1. dev-* skip can drop whole packages / product rows.

In the version loop: if len(versions)>1 and version[:3]=='dev': continue. A package whose versions are all dev-* (and >1) has every version skipped → no version rows AND no product. A package whose latest sorted version (versions[-1]) is dev-* reaches continue before the if version == versions[-1] product block → version rows exist but no product (the 45-package group). Depends on the Java sort-gen ordering of dev branches.

  1. Per-version exception silently drops that version (per-version try/except) while the package can still look processed.
  1. Insert failures are swallowed. cdbvdb.executeSql catches and logs but does not re-raise; version/product/list are separate inserts, and the LIST row (processed=1) is its own insert — so a failed version batch can be lost while the package is marked done.
  1. Resync offset pagination has no ORDER BY (SELECT ... LIMIT :start, 100) → risk of skipped/duplicated rows across pages during a resync.
  1. Dependency table writes are intentionally disabled (commented out; commit 798c626). DEPENDENCIES JSON in the version row is still populated.

Crawler Fixes Applied (2026-06-08)

All changes preserve the version/product/license row format (columns + JSON encodings). They change fetch robustness, LIST PROCESSED semantics, version selection, and remove libraries.io. Verified: tests/test_missing_regressions.py 12/12 + existing tests/test_check.py 10/10; all changed modules byte-compile. Files: sw/common.py, crawler/phpCrawler.py, crawler/component.py, db/gatheringDBVo.py, util/licenseeUtil.py, input/crawlerInfo.py.

  1. getJsonRequests robustness (sw/common.py) — was timeout=5, no retry, no try/except. Now: timeout=30, retry ladder (backoff), 429 Retry-After, retries on timeout/connection/5xx. Returns None only on 404/410 (permanent); raises new TransientFetchError when transient failures persist. Fixes the multi-MB list.json always-timing-out and transient per-package failures.
  2. Transient ≠ terminal (crawler/phpCrawler.dataCrawling) — added except TransientFetchError that records the LIST row as PROCESSED=0 (retryable) instead of 3/5. This is the #1 missing cause: a one-off 429/timeout no longer parks a package out of the incremental WHERE processed = 0 set forever.
  3. dev-only rescue (phpCrawler.dataCrawling) — all_dev = all(v[:3]=='dev' for v in versions); the dev-skip (len>1 and v[:3]=='dev') is now applied only when NOT all-dev. All-dev packages (which the crawler previously dropped entirely → no version/product) are now collected. Mixed packages keep prior behavior. (Matches the backfill tool's --include-dev-only.)
  4. malformed-authors / require hardening (crawler/component.py) — getOrganization now skips non-dict author entries (e.g. authors: [[]] no longer raises AttributeError and drops the version); getDependencyFirst/Second guard isinstance(dict). This was ~10% of backfill fetch failures. Output format identical.
  5. libraries.io removed (user-requested) — deleted component.getLibrariesGitUrl, gatheringDBVo.getVersionLicenseWithLibrariesVo, licenseeUtil.getLibrariesRepoUrl (which contained a hardcoded libraries.io API key — now gone), and the two libraries.io blocks in phpCrawler.dataCrawling. License now comes from GitHub clone + packagist only. The LIBRARIES column in TB_COMP_LIB_LICENSE_PHP simply stays unpopulated (schema/format unchanged).
  6. resync pagination ORDER BY (input/crawlerInfo.py) — LIMIT_SELECT_QUERY now ... ORDER BY PATH LIMIT :start, 100. Without a stable order, offset pagination during resync could skip/duplicate rows. PATH is the LIST PK so the order uses the index.

Round 2 (2026-06-08, user: "packageList 있으면 github 굳이 하지마", "비판적인 것 다 개선", "누락 0", data format MUST NOT change)

  1. License priority flipped to packagist-first + lazy GitHub (phpCrawler.dataCrawling, gatheringDBVo.getVersionLicenseVo, component.getOriginalDataAndGitLicense) — was GitHub-first (clone+licensee always run). Now: if packagist version.license standardizes (version_dict['STANDARD_LICENSE'] not None) the package uses it and GitHub is skipped entirely (skip_github=True); the git clone is done lazily only on the first version that lacks a packagist license. getVersionLicenseVo now picks PACKAGE_REPOSITORY (packagist) first, GITHUB only as fallback. AI-license input is likewise packagist-first (clone license-file text only when packagist absent). Big perf win (no clone for the common case) and matches the user's request. Row/JSON format unchanged: version_license_dict keys and ORIGINAL_DATA={PR,GITHUB:{URL,TAG,LICENSEE_TEXT}} structure identical; only the STANDARD_LICENSE value source changes.
  2. Insert atomicity (cdbvdb.executeSql, baseCralwer.setInsertInto/IgnoreDB, phpCrawler.dataCrawling) — executeSql and the two setInsert* now re-raise instead of swallowing. dataCrawling wraps version/product/license inserts in try/except and inserts the LIST package_list (the PROCESSED markers) only after they succeed; on failure it returns without marking, so the batch stays PROCESSED=0 and is retried (idempotent upsert). Removes the silent "data lost but LIST=PROCESSED=1" omission. Trade-off: a systematic insert failure now wedges progress (loud) instead of silently dropping — correct for a no-omission goal; the row format is DB-compatible (the backfill tool upserts the same rows fine).
  3. getReleaseDate corrected (component.py) — time[:time.find('+')-1] (could drop the seconds digit) → time[:19]. Same stored format 'YYYY-MM-DD HH:MM:SS', correct value, handles +00:00/Z/no-tz.
  4. Scheduler every 3h (app/main.py) — daily 00:00 → schedule.every(3).hours, so intake/retry backlog clears faster (with fix #1 this keeps the catalog current).
  5. insertErrorLog signature hardened (baseCralwer.py) — now accepts trx_info and uses isinstance(list) (was type[list], a bug). Prevents a TypeError now that the insert except paths are live (fix #8). DB persistence to TB_CRAWLER_ERROR_LOG still intentionally off (schema unconfirmed).

Round-2 verified: tests/test_missing_regressions.py 15/15 (now also: packagist-first license skips github, getOriginalDataAndGitLicense(skip_github) doesn't call licensee, getReleaseDate keeps full seconds) + test_check.py 10/10; full src byte-compiles; no dangling refs.

Round 3 (2026-06-08, intake cadence — user: "하루 1회는 안 됨, 배치 단위?")

Measured: releases.rss covers only ~8 minutes (41 items @ ~293 releases/hr ≈ 7,000/day); packages.rss ~9h. Daily/3h runs missed ~99% of new versions — a major omission source. One incremental run already drains the entire processed=0 set, so cadence + feed-window (not batch size) drive omission.

  1. changes.json cursor intake (phpCrawler.changesCrawler, crawlerInfo.CHANGES_URL, common.getJsonAllowStatus) — replaced RSS feeds (updatePackageListCrawler/newPackageListCrawler removed) with Packagist's cursor-based metadata/changes.json?since=<cursor>: every update/delete since the cursor + a new cursor, no window loss regardless of run gaps. Cursor in TB_CRAWLER_STATUS (TYPE PhpCralwer_CHANGES). update→LIST processed=0, delete→processed=5,deprecated=1; name = action.package.rsplit('~',1)[0]. Bootstrap stores only the current cursor (history via list.json/resync). since too old → 400 → cursor reset + log to resync. Transient → cursor not advanced (next run retries same since, lossless). getJsonAllowStatus reads the 400 body that carries the cursor.
  2. list.json reconcile gated ~daily (packageListCrawler + LIST_SYNC_INTERVAL_SEC) — the 450k-name reconcile no longer runs every incremental; gated by a PhpCralwer_LISTSYNC epoch (default 24h). Stays the new-package safety net.
  3. Scheduler every 5 min (app/main.py) — daily/3h → 5 min; changes.json is cheap and list.json is gated, so no 450k re-download per run.
  4. SELECT batch 100 → 300 (crawlerInfo.LIMIT_SELECT_QUERY) — fewer round-trips (chunk size only; one run still drains all processed=0).

Round-3 verified: tests/test_missing_regressions.py 21/21 (adds changesCrawler bootstrap / update+delete dedup + ~-strip / since-too-old reset / transient-no-advance, and getJsonAllowStatus 200·400·transient) + test_check.py 10/10; full src byte-compiles. Tests stub versionSort (JVM//workspace jar) to import phpCrawler on host; changesCrawler doesn't use it.

Round 4 (2026-06-08, product-only omission)

  1. *product created from last processed version, not strictly versions[-1] (phpCrawler.dataCrawling, committed cbd6019)* — sort-gen can sort a dev-* branch ABOVE real versions (e.g. superconductor/tools: 20241105.* < dev-main). The old if version == versions[-1] then landed on the dev version, which dev-skip continues → version rows existed but NO product (the "45 version-only" omission class). Now dataCrawling tracks the last processed version and builds the product from it after the loop. getProductVo unchanged → product row format/fields identical; LATEST_VERSION = highest real version. (Same bug + fix was found in the backfill tool while completing the run.)

Backfill outcome (2026-06-08)

The 13,103 missing products were backfilled into TB_COMP_LIB_PRODUCT/TB_COMP_LIB_VERSION_PHP via composer_missing_products/collect_composer_missing.py (LICENSE_AI filled through an SSH tunnel to the AI API). Final: 13,033 / 13,103 (99.5%) now have product rows; the remaining 70 are unbackfillable — 66 are registered on Packagist's list but have zero published versions, 4 are deleted (404). Two tool bugs were found and fixed during completion: (a) a package whose every version errored was wrongly marked done (conflated with dev-skip-empty) → permanent skip; (b) the same versions[-1]/dev product-omission as crawler fix #16.

Still NOT applied

Missing-Product Finder Tool

~/labrador/tool/composer_missing_products/ (sibling of nuget_missing_products / maven_missing_products).

- Out: composer_missing_products.txt (Packagist-only = crawler omissions), composer_db_only.txt (DB-only = deleted/stale candidates).