# License Collection & Analysis
AI Summary
Purpose:
- Capture durable knowledge about the open-source license data pipeline: license
compatibility data, license DB-ization, and the AI-based license analysis redesign.
Key points:
- Three evolutionary stages over time:
1. (2021) License DB-ization — moved text-file-based license pattern mappings into the DB as JSON for regex-based dynamic matching; split a single unified version table into per-language tables for scalability/performance; standardized bulk writes on INSERT ... ON DUPLICATE KEY UPDATE. 2. (2023) License compatibility DB — crawl an authoritative public license compatibility matrix (OSADL) and model it as (license A, license B) → compatible Yes/No/Unknown for instant pairwise lookup, to flag legal conflicts before distribution. 3. (2025) AI-based collection/analysis V3 — collect license full_text from many sources into one unified store, then have an LLM (ChatGPT API) classify license type (permissive / protective) and key permissions, with prompt engineering to sub-classify protective into Weak / Strong; emits a single refined table.
- V3 motive: remove a complex multi-table-join read structure AND remove
dependence on an external analysis service (operational/dependency risk) by bringing analysis in-house via LLM.
Relevant when:
- Working on license data modeling, compatibility lookups, LLM-based
classification, or the per-language version-table schema.
Do not read full document unless:
- You need stage-by-stage detail.
Linked documents:
- [[vulnerability-collection.md]] (sibling collection pipelines; shares
ON DUPLICATE KEY UPDATE write standard)
- [[labrador-platform.md]] (ON DUPLICATE KEY UPDATE became the sqlmodel write
standard; crawlers orchestrated via Airflow)
- [[index.md]]
Open Questions
- The brain_hw vault "04. 라이선스 수집" folder was EMPTY at review time; this
doc is grounded in the human/portfolio items, not in raw project notes. Detailed source notes are Needs confirmation.
- Exact source list for V3
full_textcollection and the final single-table
schema are not documented here. Needs confirmation.
Details
Stage 1 (2021): License DB-ization & schema foundations
- Problem: a single unified version table for all languages caused performance
and management complexity as data grew; license info lived in text files, limiting dynamic matching/search against libraries.
- Work: split the unified version table into per-language tables (query perf +
independent management); store license pattern mappings as JSON in the DB to enable regex-based dynamic license matching; benchmark REPLACE INTO vs UPDATE vs INSERT ... ON DUPLICATE KEY UPDATE and standardize on the latter (no delete/recreate overhead) as the system-wide write query. (This same write standard later anchors the labrador-sqlmodel queue refactor.)
Stage 2 (2023): License compatibility DB
- Problem: when a product combines multiple OSS components, license-to-license
legal compatibility must be known before distribution to avoid legal risk.
- Approach: select an authoritative public compatibility matrix (OSADL),
develop a crawler to collect it, and structure it as license pairs indexed by (A, B) → compatible (Yes / No / Unknown) for instant lookup. Judgement is grounded in a trusted public source, not personal estimation.
Stage 3 (2025): AI-based collection/analysis (V3)
- Problem: prior pipeline required complex multi-table joins per query and
depended heavily on an external analysis service.
- Pipeline: Collect license
full_textfrom multiple sources → Unify into a
single full-text store → AI (LLM) classifies type and permissions → Refine into a final single table.
- LLM usage: ChatGPT API replaces the external service; classifies
permissive / protective and key permissions; prompt engineering sub-divides protective into Weak / Strong.
- Impact: simplified structure (single refined table), removed external
dependency/operational risk, deeper/more accurate analysis via AI automation.