crawler-lib-swift

# crawler-lib-swift

AI Summary

Purpose:

• Durable note for the CocoaPods (Swift/iOS ecosystem) package crawler at ~/labrador/crawler/crawler-lib-swift.
• Records the 2026-06-09 missing-data fixes, the AI-license timeout fix, and the hardcoded-secret → config.ini move.

Key points:

• Python crawler (despite the swift repo name) that collects CocoaPods packages. Main crawler logic in src/ai/labradorlabs/crawler/cocoapods.py. Java sort-gen (JVM) for version sort.
• Source of truth is the CocoaPods/Specs git repo (cloned under cocoDownload), not a JSON API. Per-pod version dirs are walked; license via clone + licensee + AI license API.
• Remote [email protected]:labradorlabs/crawler-lib-swift.git; default branch master. Feature branch DAT-2784/license_ai_swift is stale (behind master).
• 2026-06-09 fixes are on master (commits 77c4e1c cocoapods+AI-timeout, 6e72eb7 secrets→config).
• libraries.io removed (2026-06-10, commit 00ca02c, pushed): getLibrariesInfo deleted from licenseeUtil.py, fallback call removed from gatheringdb.getVersionLicenseVo, LibrariesApiKey dropped from config.ini. License now resolved from github/podspec(Specs) only; if absent, LIBRARIES column stays null. Same policy as the php crawler.
• Companion missing-products tool: ~/labrador/tool/swift_missing_products (finder + backfill, same pattern as composer_missing_products). Details Needs confirmation (built in a separate session).
• 2026-06-11 (uncommitted): version-missing podspec is now explicitly skipped with a

warning (was a bare KeyError('version') at the catch-all), and the per-version catch-all logs the podspec path. Log-level policy aligned with dotnet/ruby: only loss sites stay error (setInsertIgnoreDB/IntoDB, cdbvdb.executeSql which swallows, git pull 3-strike failure, crawlSingleProduct); retryable/enrichment logs demoted to warning. Note: swift incremental re-detects failed versions next run (Specs dir vs DB diff), so per-version failures are warnings by design.

Relevant when:

• Diagnosing missing CocoaPods packages/versions, AI-license timeouts, or working on cocoapods.py / licenseeUtil.py.

Do not read full document unless:

• You need the exact fix list, the secret-rotation follow-up, or verification details.

Linked documents:

• ai/workspace/repos.md
• ai/worklog/2026/2026-W24.md
• ai/repo-notes/crawler-lib-php.md (same fix patterns: retry / per-item isolation / product fallback / AI-license caching)

Open Questions

• Security (action needed): the GitHub PAT (ghp_…) and libraries.io API key were hardcoded in licenseeUtil.py and remain in git history; the key was later removed entirely (00ca02c) but history still holds it. Rotate both. config.ini itself is git-tracked (also holds the real DbUrl); consider .gitignore + deploy-time injection.
• No live crawler smoke test was run for the 2026-06-09 fixes (host can't run the Docker/JVM/clone path); changes were byte-compiled + unit-mock verified only. Run in the Docker image before relying on prod.
• swift_missing_products tool internals not yet documented here.

Details

Repo Info

• Repo ID: crawler-lib-swift
• Local path: ~/labrador/crawler/crawler-lib-swift
• Language: Python 3 crawler (Docker), Java sort-gen via JPype
• Ecosystem: CocoaPods (Swift/iOS)
• Branch: master (feature branch DAT-2784/license_ai_swift is stale)

Fixes Applied (2026-06-09)

Row format (version/product/license columns + JSON encodings) unchanged throughout. Verified: py_compile OK; getAiLibraryLicense retry/cache mock-tested (retry-after-failure, cache-hit, graceful-None).

cocoapods.py — missing-data + speed (commit 77c4e1c)

1. No full re-clone → incremental git pull. The Specs checkout was rm -rf'd and fully re-cloned every run; now kept and git pull'd. Full re-clone failure/timeout on the huge Specs repo dropped the most recent pods and was slow.
2. git pull retry + return-code check. Single git pull → 3 retries (15s) + rc==0 check; 3-fail → proceed on existing checkout with a warning. Single partial/failed pull silently proceeding on a stale checkout was the main cause of recent-pod omission.
3. per-pod isolation (try/except). A single pod's dir-enter / listing / version-sort(JVM) / DB error used to abort the whole crawl; now that pod is skipped and the crawl continues.
4. product fallback. If versions exist but no product was created (latest-version podspec error etc.), build the product from the last successful version (version_list[-1]). getProductVo unchanged → product row format identical.

licenseeUtil.py getAiLibraryLicense — AI license read-timeout (commit 77c4e1c)

• Symptom: repeated ERROR ... getAiLibraryLicense: Read timed out (read timeout=30) to 211.115.125.171. Root cause: the AI API is fine for simple input (~6ms) but has cold-start (~10s+) and contention (many crawlers) spikes that exceed a single 30s read timeout; the crawler called it per version, no cache, no retry.
• Fix: timeout=30 → timeout=(10,60) + 3× exponential-backoff retry; per-input cache (_ai_cache, shared across the run) so a pod's many versions with the same license call once; final failure logged WARNING not ERROR (non-fatal — only LICENSE_AI unset). LICENSE_AI value format unchanged.

Secrets → config.ini (commit 6e72eb7)

• Hardcoded ghp_… GitHub token (getGITClone + checkStatusCode) and libraries.io api_key (getLibrariesInfo) removed from source → read via Config().getConfig('COMMON', 'GithubToken' | 'LibrariesApiKey'). Values moved into config.ini[COMMON] (centralized with DbUrl). Rotation still required (see Open Questions).

Branch note

The working-tree edits were on the stale DAT-2784/license_ai_swift branch; they were moved to master via git stash → fast-forward local master to origin/master → git stash pop (3-way, no conflicts), then committed.