OS Package Binary Collection — Per-Distro Reference

# OS Package Binary Collection — Per-Distro Reference

AI Summary

Purpose:

• Concise reference of the OS-package binary-collection crawlers (Ubuntu, Debian, CentOS, OpenWrt) and the package-repo-URL extraction project: repository structure, package format, extraction method, and what is stored.

Key points:

• Ubuntu/Debian: APT pool/ archives, .deb (Debian also .udeb); .deb is an ar archive of control.tar.* + data.tar.*; extract with ar + tar/xz/zstd/gzip.
• CentOS: Vault (EOL snapshots, no security updates); .rpm; extract with rpm2cpio | cpio -idm; CentOS 6–7 use os/{arch}/Packages/, CentOS 8+ split into BaseOS/ + AppStream/.
• OpenWrt: .ipk (ar archive like .deb, sometimes gzip tar); reuses the Debian/Ubuntu extraction logic; huge scale (~17.6M packages, ~1.7–3.5TB est.).
• All crawlers: HTML index parsing, DFS recursion, resumable via _state.json, binary-file filtering (ELF/.so/exec/shebang), hashes + metadata into MySQL, Docker-packaged.
• Repo-URL project: extract package name + Git repo URL — APT via Vcs-Git (sources.gz) then Homepage (packages.gz); OpenWrt via PKG_SOURCE_URL in feed Makefiles.

Relevant when:

• Working on OS-package crawlers, binary-hash collection, or repo-URL extraction; needing per-distro structural differences.

Do not read full document unless:

• You need the exact directory layout, extraction commands, or repo-URL extraction strategy for one distro.

Linked documents:

• [[../../wiki/people/career-timeline.md]]
• [[../index.md]]

Open Questions

• OpenWrt collection was implemented but real end-to-end collection test was still pending in the source notes (marked incomplete there).
• Scale figures for OpenWrt (~17.6M packages, ~1.7–3.5TB) are author estimates from late 2024.

Details

Source: author's archive notes (Obsidian vault, 600. Archives/01. OS PKG Binary 수집): Ubuntu, Debian, CentOS, OpenWrt, and the PKG_REPO_URL project. Status in source: all "완료/complete" except OpenWrt's real-collection test.

Common design across all crawlers

• Discovery: parse Apache-style HTML directory index; DFS recursive traversal.
• Resumable: state persisted in _state.json to allow restart.
• Binary filtering (to avoid hashing everything): ELF binaries, .so libraries, files under usr/bin, usr/sbin, bin, sbin, files with exec permission, script extensions (.sh, .py, .bash, .pl, .rb, ...), and files starting with a shebang (#!).
• Output: file hashes + package metadata + license info into MySQL; packaged with Docker / docker-compose.

Ubuntu (.deb)

• Source: http://archive.ubuntu.com/ubuntu/pool/main/ (also restricted, universe, multiverse).
• Layout: by source-package name, split by first letter or lib* prefix (e.g. a/abc/, libz/libzzip-0-13/). Filename: {name}_{version}_{arch}.deb.
• Extraction: .deb is an ar archive → ar x → control.tar.* (metadata) + data.tar.* (files); unified extractor handles tar/xz/zstd/gzip.

Debian (.deb, .udeb)

• Source: https://deb.debian.org/debian/pool/main/ (also non-free, contrib).
• Layout: same pool structure as Ubuntu. Filename: {name}_{version}_{arch}.deb or .udeb.
• .udeb: Debian-installer binary packages (Ubuntu mostly uses .deb).
• Extraction: identical to Ubuntu (ar → control/data tar).

CentOS (.rpm)

• Source: https://vault.centos.org/ — archived snapshots of EOL versions; no security updates.
• Layout: CentOS 6–7 use centos/{version}/os/{arch}/Packages/ (traditional); CentOS 8+ split into centos/{version}/BaseOS/{arch}/Packages/ + AppStream/{arch}/Packages/. CentOS 7 alone has 10,000+ RPMs.
• Extraction: rpm2cpio {file} | cpio -idm to expand the filesystem; metadata via rpm -qp.
• Extra: multi-source-URL support with filename-based dedup across mirrors.

OpenWrt (.ipk)

• Source: https://downloads.openwrt.org/releases/; path releases/{version}/packages/{arch}/{category}/. Categories: packages, base, luci, routing, telephony.
• Format: .ipk is usually an ar archive like .deb (debian-binary + control.tar.gz + data.tar.gz); some are gzip-compressed tar — crawler tries both. Extraction logic reused from the Debian/Ubuntu extract_and_hash.py.
• Scale (author estimate, Nov 2024): 56 versions (17.01.7 → 24.10.4), ~35–36 architectures per version, 5 categories per arch → ~17.6M packages, ~1.7–3.5TB.
• Discovery: auto-discover version → arch → category; supports selective collection; filename-based dedup.

Package-repo-URL extraction project (PKG_REPO_URL)

• Goal: extract package_name + Git repo_url for Ubuntu, Debian, OpenWrt into one DB.
• APT (Ubuntu/Debian): use Packages.gz (binary index, has Homepage) + Sources.gz (source index, has Vcs-Git). Priority: Vcs-Git first; else crawl Homepage HTML for a Git repo link (GitHub/GitLab/SourceForge patterns), with caching and request delay (0.3–0.5s). Debian additionally does dynamic architecture discovery (Ubuntu fixed to amd64/i386). Components differ: Ubuntu main/multiverse/restricted/universe; Debian main/contrib/non-free-firmware/non-free.
• OpenWrt: no Packages.gz/Sources.gz index and .ipk control files lack Homepage/Vcs-Git; instead clone the GitHub feeds (packages, luci, routing, telephony) and extract PKG_SOURCE_URL from each Makefile, then normalize (strip $(VAR) variables, codeload.github.com → github.com, drop releases/download/archive paths, git:///git@ → https://, validate repo pattern).