Crawler Source Governance

# Crawler Source Governance

AI Summary

Purpose:

• Durable operating model for choosing and running external collection sources

without treating a technically reachable endpoint as permission to use it.

Key points:

• Source approval is a routing decision with three outcomes: use the published

interface under its conditions, use an approved alternative, or stop pending a written agreement. A crawler implementation starts only after that route is known.

• Low/medium/high are triage labels, not legal conclusions. High means the

current source path is blocked for automatic use until an agreement or alternative is confirmed; medium means runtime controls must enforce the source's conditions.

• The standard path is `inventory -> terms/license/API review -> risk route ->

runtime controls -> attribution/audit record`.

• Runtime controls are part of compliance: per-source request rate,

concurrency, retry/backoff, API-key ownership, User-Agent identity, robots handling, and a no-bypass rule.

• Prefer public APIs, public files, and public repositories over HTML scraping.

When a source disallows scraping, changing the protocol is not enough unless that protocol is an official supported interface or the source grants permission.

• Examples from the 2026-08-20 review: gemnasium-db routes to the

MIT-licensed advisories-community; Amazon/Oracle require repodata or written permission; Cloudera/Atlassian Maven require a supported interface or written agreement; npm must stay within its published crawler limits.

• A customer-operated collection-system project and LabradorLabs' own

third-party database distribution are different legal/contractual cases. Evidence for one must not be reused to claim the other is resolved.

• The current review is a technical pre-review. Final legal approval, supplier

agreements, and confirmed removals from production are Needs confirmation.

Relevant when:

• Adding or replacing an upstream source.
• Reviewing a crawler's rate-limit and retry settings.
• Writing a proposal, contract assumption, runbook, or portfolio page about

external data collection.

Do not read full document unless:

• A per-source route or the boundary between the customer project and internal

redistribution needs to be checked.

Linked documents:

• ai/sources/confluence/2026-08-20-crawler-source-legal-review.md
• ai/repo-notes/labrador-scrapers.md
• ai/wiki/projects/vulnerability-collection.md
• ai/wiki/projects/license-collection.md

Open Questions

• Which high-risk sources have written permission, and which must be replaced?
• Are published rate limits encoded in Airflow pools/client configuration and

monitored, or only documented?

• What attribution fields must be exposed in customer-facing outputs?
• What is the approved treatment of maintainer names and emails?
• Which sources are valid for the customer-operated project but not for

LabradorLabs' own redistribution?

Details

Decision route

1. Record the exact source URL, official interface, cadence, and data used.
2. Read the current terms, data license, API policy, and robots/access rules.
3. Assign a triage route:

- low: public data/OSS license; retain attribution and license obligations; - medium: allowed only with concrete runtime conditions; - high: stop and obtain written permission or select an alternative.

1. Encode the conditions in the client and scheduler, not only in a document.
2. Keep the decision, source version/date, and attribution requirements near

the crawler's durable documentation.

1. Re-review when the endpoint, terms, license, or operating model changes.

Current examples

Evidence boundary

• No claim is made that legal review is complete.
• No claim is made that a source has been removed from production merely

because removal or replacement was proposed.

• The one-page briefing intentionally excludes file/function/binary sources,

exploit/PoC data, code-level vulnerability sources, and Snyk.