License Collection & Analysis

# License Collection & Analysis

AI Summary

Purpose:

• Capture durable knowledge about the open-source license data pipeline: license

compatibility data, license DB-ization, and the AI-based license analysis redesign.

Key points:

• Three evolutionary stages over time:

1. (2021) License DB-ization — moved text-file-based license pattern mappings into the DB as JSON for regex-based dynamic matching; split a single unified version table into per-language tables for scalability/performance; standardized bulk writes on INSERT ... ON DUPLICATE KEY UPDATE. 2. (2023) License compatibility DB — crawl an authoritative public license compatibility matrix (OSADL) and model it as (license A, license B) → compatible Yes/No/Unknown for instant pairwise lookup, to flag legal conflicts before distribution. 3. (2025) AI-based collection/analysis V3 — collect license full_text from many sources into one unified store, then have an LLM (ChatGPT API) classify license type (permissive / protective) and key permissions, with prompt engineering to sub-classify protective into Weak / Strong; emits a single refined table.

• V3 motive: remove a complex multi-table-join read structure and replace the

prior external analysis service with an internally owned pipeline whose model step calls the ChatGPT API. Do not describe this as removing all external dependencies: the LLM API remains an explicit dependency.

Relevant when:

• Working on license data modeling, compatibility lookups, LLM-based

classification, or the per-language version-table schema.

Do not read full document unless:

• You need stage-by-stage detail.

Linked documents:

• [[vulnerability-collection.md]] (sibling collection pipelines; shares

ON DUPLICATE KEY UPDATE write standard)

• [[labrador-platform.md]] (ON DUPLICATE KEY UPDATE became the sqlmodel write

standard; crawlers orchestrated via Airflow)

• [[index.md]]

Open Questions

• The brain_hw vault "04. 라이선스 수집" folder was EMPTY at review time; this

doc is grounded in the human/portfolio items, not in raw project notes. Detailed source notes are Needs confirmation.

• Exact source list for V3 full_text collection and the final single-table

schema are not documented here. Needs confirmation.

Details

Stage 1 (2021): License DB-ization & schema foundations

• Problem: a single unified version table for all languages caused performance

and management complexity as data grew; license info lived in text files, limiting dynamic matching/search against libraries.

• Work: split the unified version table into per-language tables (query perf +

independent management); store license pattern mappings as JSON in the DB to enable regex-based dynamic license matching; benchmark REPLACE INTO vs UPDATE vs INSERT ... ON DUPLICATE KEY UPDATE and standardize on the latter (no delete/recreate overhead) as the system-wide write query. (This same write standard later anchors the labrador-sqlmodel queue refactor.)

Stage 2 (2023): License compatibility DB

• Problem: when a product combines multiple OSS components, license-to-license

legal compatibility must be known before distribution to avoid legal risk.

• Approach: select an authoritative public compatibility matrix (OSADL),

develop a crawler to collect it, and structure it as license pairs indexed by (A, B) → compatible (Yes / No / Unknown) for instant lookup. Judgement is grounded in a trusted public source, not personal estimation.

Stage 3 (2025): AI-based collection/analysis (V3)

• Problem: prior pipeline required complex multi-table joins per query and

depended heavily on an external analysis service.

• Pipeline: Collect license full_text from multiple sources → Unify into a

single full-text store → AI (LLM) classifies type and permissions → Refine into a final single table.

• LLM usage: ChatGPT API replaces the external service; classifies

permissive / protective and key permissions; prompt engineering sub-divides protective into Weak / Strong.

• Impact: simplified structure (single refined table) and removed the prior

analysis-service dependency. The replacement pipeline is internally owned, while the ChatGPT API remains an explicit model dependency. Classification quality metrics are not documented and must not be invented.