crawler-lib-dotnet

# crawler-lib-dotnet

AI Summary

Purpose:

• Durable note for the NuGet/.NET package crawler at ~/labrador/crawler/crawler-lib-dotnet.
• Records the 2026-06-05 missing-data investigation and the catalog page/leaf retry fixes.

Key points:

• The crawler uses NuGet V3 catalog flow: catalog index -> catalog page -> catalog leaf.
• TB_COMP_LIB_DOTNET_LIST stores catalog page URLs, not package IDs.
• dataCrawling fetches a page URL, then fetches each leaf @id; the leaf is the metadata source for id, version, dependencyGroups, license*, projectUrl, and published.
• Fixed missing-risk bugs on 2026-06-05: leading whitespace in PACKAGE_UPDATE_URL, empty LIST table not seeded, leaf/page fetch failures being able to close a page as complete, and SOURCE_URL being read from page metadata instead of leaf metadata.
• NuGet package IDs should be treated as case-insensitive identity; preserve display casing if useful, but do not treat casing-only differences as separate packages.
• Normal incremental scheduler was changed from daily midnight to every 3 hours on 2026-06-05 (src/ai/labradorlabs/app/main.py, SCHEDULE_INTERVAL_HOURS=3).
• License fallback policy was narrowed on 2026-06-05: only check/clone GitHub when the NuGet API leaf has no license value; if NuGet provides a license value, do not hit GitHub. libraries.io fallback code was removed.
• Follow-up license fallback change on 2026-06-05: when NuGet has no license and projectUrl is GitHub, the crawler now calls GitHub REST GET /repos/{owner}/{repo}/license once per catalog page/repo and shares that default-branch repo license across same-page versions from that repo. This intentionally avoids tag-specific API calls to reduce rate-limit pressure; clone/licensee remains fallback only when the repo license API returns no license data.
• Dependency rows in TB_COMP_LIB_VERSION_DOTNET_DEPENDENCY remain intentionally disabled in code since commit e0b8086; DEPENDENCIES JSON in the version row is still populated.
• 2026-06-11 (uncommitted): licenseUrl handling added — SPDX URLs

(licenses.nuget.org/<expr>, spdx.org/licenses/<id>) resolve to a license name for normal matching; GitHub licenseUrl routes the existing GitHub license fallback to that repo (instead of projectUrl); any other plain URL is stored in LICENSE only, with LICENSE_IDS matching and the AI license call skipped.

• 2026-06-11 (uncommitted): per-leaf processing extracted to

_processCatalogLeaf(versions_info, ctx); missing-version backfill _collectMissingVersions diffs flatcontainer (v3-flatcontainer/{id}/index.json) against collected+DB versions and collects gaps via registration leaf catalogEntry → catalog leaf (version + license rows only; no product rows). The end-of-batch sort pass now runs for every component (the old len>1 gate skipped single-version updates) and always upserts product LATEST_VERSION = sortVersions(all)[-1], fixing late-backfilled old versions overwriting LATEST_VERSION.

• 2026-06-11 (uncommitted): log-level policy aligned with the ruby crawler —

only confirmed data-loss sites stay error (baseCralwer.setInsertIgnoreDB/ setInsertIntoDB insert failures, which are swallowed so the page can close as processed=1; table name added to the message). Everything else demoted to warning: retryable page/leaf failures (processed=0 requeue), sorting/ LATEST_VERSION correction failures, license/GitHub/AI enrichment, JSON fallbacks, clone cleanup, updatePackageListCrawler, low-level DB (cdbvdb/base raise upward). app/check.py diagnostics CLI untouched.

Relevant when:

• Diagnosing missing NuGet packages or versions.
• Working on updatePackageListCrawler, dataCrawling, LIST processing, or NuGet metadata mapping.

Do not read full document unless:

• You need exact missing-data root causes, verification commands, or open gaps.

Linked documents:

• ai/workspace/repos.md
• ai/worklog/2026/2026-W23.md
• Microsoft NuGet catalog docs: https://learn.microsoft.com/en-us/nuget/api/catalog-resource

Open Questions

• Should the LIST worker implement a retry ladder or processing status like the Go V2 crawler instead of leaving transient failures as PROCESSED=0?
• Should dependency table insertion be re-enabled, or is version-row DEPENDENCIES JSON sufficient for downstream consumers?
• Should PackageDelete leaves be represented as deprecated/unlisted instead of ordinary version rows?
• Should GitHub clone/licensee fallback be fully removed after validating that default-branch GitHub license API coverage is good enough for .NET packages?
• No live DB/NuGet integration test was run in the 2026-06-05 fix session; only mocked unit coverage and byte-compile were run.

Details

Repo Info

• Repo ID: crawler-lib-dotnet
• Local path: ~/labrador/crawler/crawler-lib-dotnet
• Language: Python 3 crawler
• Ecosystem: NuGet (LANGUAGE='dotnet', REPOSITORY='NUGET')
• Tests:

- python3 -m unittest discover -s tests -v - python3 -m compileall src tests - git diff --check

NuGet Catalog Source-of-Truth

Use NuGet V3 catalog as an append-like event source:

1. Catalog index: https://api.nuget.org/v3/catalog0/index.json
2. Catalog page: each index items[*].@id.
3. Catalog leaf: each page items[*].@id.

Microsoft documents the hierarchy as index, page, and leaf. Index page entries do not inline leaves; page item entries contain only minimal information and leaf @id; the leaf contains package metadata such as id, version, dependencyGroups, projectUrl, license fields, and published.

NuGet package ID identity is case-insensitive. Microsoft NuGet search docs state that exact packageid matches are case-insensitive. For crawler/storage design, this means Newtonsoft.Json and newtonsoft.json should be considered the same PRODUCT_KEY identity even if the crawler preserves the original casing returned by NuGet metadata.

Missing-Data Root Causes Fixed on 2026-06-05

1. Catalog index URL had leading whitespace

File: src/ai/labradorlabs/input/crawlerInfo.py

Before:

• PACKAGE_UPDATE_URL started with a space.
• In requests, that can fail before any HTTP request is made, so incremental catalog page discovery can silently do nothing.

Fix:

• Removed the leading whitespace.
• Regression: test_package_update_url_has_no_leading_whitespace.

2. Empty LIST table could not seed itself

File: src/ai/labradorlabs/crawler/dotnetCrawler.py

Before:

• updatePackageListCrawler selected the newest LIST row first.
• If LIST was empty, it logged "No package list found in database" and returned before reading NuGet catalog index.
• A fresh or truncated LIST table would stay empty, causing the worker to have no page URLs.

Fix:

• Empty LIST now means initial seed mode: fetch catalog index and insert all page URLs.
• Regression: test_update_package_list_seeds_catalog_when_list_table_is_empty.

3. Leaf fetch failures could close a page as complete

File: src/ai/labradorlabs/crawler/dotnetCrawler.py

Before:

• If a catalog leaf @id returned None or had an unexpected JSON shape, the loop continued.
• error_flag stayed false, so the catalog page was marked PROCESSED=1.
• Result: versions on failed leaves were omitted permanently unless the page was later requeued by another catalog index update.

Fix:

• Leaf fetch/shape failures set error_flag=True.
• A page with any leaf failure remains retryable with PROCESSED=0.
• Version-level exceptions record exactly one retry status for the page.
• Regressions:

- test_leaf_fetch_failure_keeps_catalog_page_retryable - test_version_exception_records_one_retry_status_for_catalog_page

4. Page fetch failures could leave status unpersisted

Before:

• Some early continue branches appended a status but skipped the bottom-of-loop insertPackageList.
• If this happened on the last page in the batch, the status update could remain only in memory.

Fix:

• Page fetch failure or invalid page shape is immediately written as PROCESSED=0.
• Regression: test_catalog_page_fetch_failure_is_recorded_for_retry.

5. Product SOURCE_URL was read from page item metadata

Before:

• product_dict['SOURCE_URL'] used self.component.getGitUrl(item_info).
• A catalog page item only has minimal page-level metadata; projectUrl lives on the leaf.
• Result: product SOURCE_URL was commonly NULL even when NuGet leaf metadata had projectUrl.

Fix:

• Read SOURCE_URL from versions_info (the leaf).
• Regression: test_product_source_url_comes_from_catalog_leaf_metadata.

Remaining Known Gaps

• Dependency table writes are disabled:

- Code block that builds dependency_list and calls insertDependency is commented out. - This was committed as dependency-table collection being disabled (e0b8086). - Version rows still include DEPENDENCIES JSON.

• Retry semantics are still primitive:

- PROCESSED=0 keeps failures retryable but can cause repeated immediate retries if a page/leaf has a persistent bad shape or outage. - A Go-style retry ladder (30-34) or a processing status would be safer for multi-worker or persistent-failure operation.

• LIST table grain is catalog page URL, not (package, version) event grain:

- This follows the current code shape but means reprocessing a page replays all leaves in that page. - Upsert idempotency must be relied on for duplicate catalog leaf processing.

GitHub License Fallback Policy

• Primary source is still NuGet catalog leaf license metadata. When NuGet has

a license value, the crawler does not call GitHub.

• If NuGet has no license and projectUrl points to GitHub, call GitHub REST

GET /repos/{owner}/{repo}/license for the default branch. Do not make tag-specific license API calls in the main fallback path.

• Cache GitHub repo license results per catalog page/repo. This prevents

repeated API calls for one NuGet page that contains many packages from the same repository.

• Use the API response license.spdx_id/license.name to populate version

license records, decode the API content field for LICENSE_AI input, and store ORIGINAL_DATA.GITHUB.LICENSE_API.ref as None for the default-branch API result.

• This trades version-specific license precision for much lower GitHub API

volume, per the 2026-06-05 user decision.

• Existing clone/tag/licensee detection remains fallback only when the GitHub

license API returns no license data. Clone cleanup happens once per catalog page/repo, not per leaf.