LLM WikiAccess-protected knowledge portal

WIKI

crawler-lib-java

AI Summary Purpose Durable note for the Maven/Java package crawler at ~/labrador/crawler/crawler lib java . Tracks the Python migration effort on branch python migration maven . Key points Original implementation is Java Spring/MyBatis . Th

경로ai/repo-notes/crawler-lib-java.md
카테고리Repo Note
태그#cicd #crawler #data-pipeline #infra #java #lib #license #notes #repo #repo-note #security

# crawler-lib-java

AI Summary

Purpose:

Key points:

- getVersionVo(extracted) → 24-column VERSION_JAVA row (TB_COMP_LIB_VERSION_JAVA). JSON via jsonUtil.to_json (Jackson-exact: compact, ensure_ascii=False). ALWAYS-set columns (empty→"[]" / full-null dict): LICENSE, SCM, ISSUE_MANAGEMENT, ORGANIZATION, PARENT, ORIGINAL_INFO, DEPENDENCIES, DEPENDENCY_MANAGEMENT. CONDITIONAL (None when absent): STANDARD_LICENSE, LICENSE_IDS, LICENSE_AI. - getProductVo(version_row) → 9-column PRODUCT row (TB_COMP_LIB_PRODUCT). LICENSE = version_row's LICENSE (raw POM license JSON), matching Java product.setLicense(last.getLicense()) — NOT STANDARD_LICENSE (corrected P2b Task 1). - getMavenArchiveVo(record) → 9-column MAVEN_ARCHIVE row. ORIGINAL_INFO always json.dumps (never None). - getDependencyVo(version_row) → generator of DEPENDENCY rows. ID = sha256(PRODUCT_KEY + VERSION + origin_key + version_range). Yields nothing when DEPENDENCIES is None. - getCrawlerStatusVo(name, json_raw) → STATUS row (TYPE, RAW_DATA, TIMESTAMP).

- P2d Task 1 work-list methods (all LANGUAGE='java', REPOSITORY='Maven'): - selectNotProcessedProducts(limit) → labradordb.TB_COMP_LIB_PRODUCT … ORDER BY PRODUCT_KEY LIMIT 0, :limit - selectNotProcessedArchives(repository_name, limit) → gatheringdb.TB_COMP_LIB_MAVEN_ARCHIVE … REPOSITORY_NAME=:repo AND PROCESSED IS NULL LIMIT 0, :limit - selectPriorityVersions(limit) / selectNotProcessedVersions(limit) → TB_COMP_LIB_VERSION_JAVA - selectVersionsOfProduct(product_key) → TB_COMP_LIB_VERSION_JAVA … ORDER BY FIXED_SORT_ORDER ASC, SORT_ORDER ASC, VERSION ASC - updateMavenArchiveProcessed(record) → UPDATE gatheringdb.TB_COMP_LIB_MAVEN_ARCHIVE SET PROCESSED = NOW() WHERE … - updateVersionSortOrder(row) → UPDATE TB_COMP_LIB_VERSION_JAVA SET SORT_ORDER = :so WHERE … - Test suite after P2d Task 1: 115/115 passed. Commit 44fb45a.

Relevant when:

Do not read full document unless:

Linked documents:

Open Questions

Phase 2 Decisions (2026-06-09, user-confirmed)

  1. Version sort = pure Python reproducing Java VersionSortUtil.sortVersionUsingMaven

(Apache ComparableVersion + preprocessing _→-, +→-, pr→rc; insertion-sort ascending, incl. the equal-version drop quirk). No JPype / sort-gen jar at runtime. The existing jar/Java is used only as a test oracle (differential check, skipped when no JVM).

  1. License AI API = http://211.115.125.171/api/v1/licenses (POST), body

{"texts":[{"text": <str>}, ...]}, 30s timeout, no auth. Response {"NAMES":[{NAME,CONFIDENCE}], "AI_LICENSE":[ids]}; extract AI_LICENSE/ai_license (case-insensitive) → json.dumps(list) into LICENSE_AI (None if empty). Input priority: cloned license-file text → GitHub API license → POM <licenses> name. Independent of the TB_LICENSE SPDX match that fills STANDARD_LICENSE/LICENSE_IDS. Same client as crawler-lib-dotnet/golang (licenseeUtil.getAiLibraryLicense).

  1. DEPENDENCY table NOT used. Dependencies live only in the VERSION row DEPENDENCIES

(and DEPENDENCY_MANAGEMENT) JSON columns. baseCralwer.insertDependency / VERSION_DEPENDENCY_TABLE stay dormant (not called).

Details

Branch

python-migration-maven (working branch for all Python migration work).

Run tests

cd ~/labrador/crawler/crawler-lib-java
. .venv/bin/activate
python -m pytest tests/ -v

Column order: VERSION_JAVA (TB_COMP_LIB_VERSION_JAVA)

LANGUAGE, REPOSITORY, PRODUCT_KEY, VERSION, NAME, DESCRIPTION, LICENSE, LICENSE_AI, STANDARD_LICENSE, LICENSE_IDS, URL, SCM, ISSUE_MANAGEMENT, ORGANIZATION, PARENT, ORIGINAL_INFO, SHA1_VALUE, CREATED, RELEASE_DATE, PROCESSED, SORT_ORDER, FIXED_SORT_ORDER, DEPENDENCIES, DEPENDENCY_MANAGEMENT

Column order: PRODUCT (TB_COMP_LIB_PRODUCT)

LANGUAGE, REPOSITORY, PRODUCT_KEY, LATEST_VERSION, NAME, DESCRIPTION, LICENSE, CREATED, PROCESSED

Column order: MAVEN_ARCHIVE (TB_COMP_LIB_MAVEN_ARCHIVE)

LANGUAGE, REPOSITORY_NAME, PRODUCT_KEY, VERSION, NAME, DESCRIPTION, SHA1_VALUE, ORIGINAL_INFO, LAST_MODIFIED

Column order: DEPENDENCY (TB_COMP_LIB_VERSION_JAVA_DEPENDENCY)

ID, LANGUAGE, REPOSITORY, PRODUCT_KEY, VERSION, ORIGIN_PRODUCT_KEY, ORIGIN_VERSION_RANGE, RECORD_CREATED

Column order: STATUS (TB_CRAWLER_STATUS)

TYPE, RAW_DATA, TIMESTAMP