Forgejo Git Mirror (labGit1 / labGit2)

# Forgejo Git Mirror (labGit1 / labGit2)

AI Summary

Purpose:

• Design record for the internal-only Forgejo git mirror on two IDC servers

(labGit1 211.115.125.166, labGit2 211.115.125.174), sharded (no replication), seeded from gatheringdb.TB_COMP_FILE_VALID_V6.SOURCE_URL, with on-demand mirroring of repos not in the seed.

Key points:

• Plan A (sharding) confirmed by user 2026-08-05: each mirror repo lives on

exactly one server; no cross-server replication.

• Task 10 pilot (100 repos/shard, 2026-08-05) surfaced a "ghost repo" bug: a

failed Forgejo /repos/migrate leaves a DB record (empty:true, mirror:true, mirror_updated stuck at Go's zero-date "0001-01-01T00:00:00Z") that the seeder's/broker's "record exists → already mirrored" check couldn't tell apart from a real mirror, so retries silently skipped it forever. Fixed in fix-round-1 (commit 80ffdd7 on labgit-mirror): core.is_ghost_mirror() detects the signature; seeder and broker both delete-and-remigrate a ghost instead of marking it done. See "Task 10 fix round 1" below for the full predicate, evidence, and remaining open item (an exposed admin password on labgit2 that still needs manual rotation).

• Single entry domain; nginx on labGit1 routes per-repo with a deterministic

hash rule shared by seeder, broker, and nginx (no mapping table).

• mirror-broker service (labGit1): REST API + simple web page. `POST /ensure

{url}` creates the pull-mirror if missing; if present, triggers a per-repo mirror-sync and returns after sync ("refresh on request, then serve").

• Baseline pull-mirror interval: weekly (168h). No write-back to

TB_COMP_FILE_VALID_V6 (owned by the hash pipeline).

• Seed profile (2026-08-05): 107,198 distinct URLs, ~4.8TB total source size;

github.com 100,318 / other git hosts ~6,800 / non-git ~90 (svn 19, sourceforge 63, ftp 2, misc 6) excluded; HASH_STATUS_CD=95 (private) 2,794 excluded.

• Servers ready: Ubuntu 24.04, /data XFS 29.1T (8x8TB RAID10) empty, 128GB

RAM, EPYC 64C/128T. XFS gives ~3.1B dynamic inodes → inode exhaustion is a non-issue; the real small-file risk is git loose objects → enable Forgejo gc cron.

Relevant when:

• Building, operating, or debugging the internal git mirror; writing the

seeding/broker/nginx implementation; answering "where does repo X live".

Do not read full document unless:

• You need exact component behavior, namespace/shard rules, seeding

procedure, or open items.

Linked documents:

• [[index.md]]
• [[infra-db-monitoring.md]] (gatheringdb context)

Open Questions

• Firewall allowlist sources: exact company source IPs are not yet known.

Plan (agreed 2026-08-05): at build time, collect candidates from server SSH login history (last), the admin Mac's current public IP, and IDC-internal pipeline ranges, then show the list to the user for confirmation BEFORE applying ufw rules. Verified 2026-08-05: only port 22 currently listens externally on both servers.

• How labgit name resolution will reach consumers (internal DNS server vs

/etc/hosts on pipeline machines). Needs confirmation at build time.

• Whether original (developer-pushed) repos are in scope for this phase, or

mirror-only for now. The Confluence doc covers originals (push-mirror DR); this build phase is mirror-first. Needs confirmation.

• ~~Whether /data is mounted with noatime~~ CONFIRMED 2026-08-12 on

labgit1: rw,noatime,inode64,logbufs=8,logbsize=32k,sunit=512,swidth=512, noquota.

• ~~Server-side deploy path for the data-infra checkout~~ CONFIRMED

2026-08-12: /data/product/data-infra/labgit, copy-deployed (not a git checkout).

• Disk-failure diagnosis has no tooling. /data is a Broadcom/LSI MegaRAID

12GSAS SAS39xx hardware RAID10 (8x8TB presented as a single sdb 29.1T), so mdadm / /proc/mdstat show nothing, and NEITHER server has storcli64/perccli/megacli installed. There is currently no way to tell from the OS which slot failed. Install a controller tool or define a BMC-console procedure before a disk actually fails.

Details

Context

• Confluence design doc: "[DSGN|INFR] git 포지(Forgejo) 서버 재구성 설계"

(space EN, page 4172873742), ticket INFR-393. Hardware decisions there are already executed: OS SSD RAID1 + 8x8TB RAID10 → /data XFS 29.1T on both servers.

• User decisions in session 2026-08-05:

1. Plan A sharding, no replication (rejected full-replication option). 2. Seed = all SOURCE_URL from gatheringdb.TB_COMP_FILE_VALID_V6. 3. On-demand mirroring for repos not in the seed, via API + simple web page. 4. Access = single domain + nginx deterministic hash routing on labGit1. 5. Sync = weekly baseline + refresh-on-request-then-serve via broker. 6. No write-back of on-demand repos to TB_COMP_FILE_VALID_V6.

• User decisions in session 2026-08-05 (second round):

7. Names: entry domain labgit, per-server labgit1 / labgit2 (also the new hostnames). 8. Deployment: Docker Compose; config/code lives in bitbucket.org/labradorlabs/data-infra (repo not yet cloned on this Mac; user will clone it). Secrets (GitHub token, DB passwords) stay in server-local .env files, never committed. 9. GitHub seeding token: ONE fine-grained PAT (public repositories, read-only, no extra permissions) is sufficient — limits are per-user, so multiple tokens on one account add nothing.

GitHub rate-limit facts (checked 2026-08-05)

• Git clone/fetch does NOT consume the REST API quota (no API calls from the

git client) and has NO documented hard limit; GitHub dynamically delays (not bans) git requests under load. GitHub support guidance: authenticated git operations tolerate roughly ~5,000/h per user; anonymous git is throttled far lower (~100/h or less per IP). REST API (not used by plain git mirror): 5,000 req/h per authenticated user, 60/h unauthenticated. Sources: docs.github.com rate-limits page; github.com/orgs/community/ discussions/44515 (staff response).

• Seeding plan derived from this: pace ~1,000–1,500 clones/h total across

both servers (each server clones its own shard from its own IP), one token shared → ~3–5 days for 100k repos; real bottleneck is bandwidth (4.8TB), not the clone count.

• Weekly refresh (107k fetches/week ≈ 11/min average) is far below any

throttling zone.

Server state (verified 2026-08-05, read-only)

Components

• Forgejo #1 / #2 — one independent instance per server; PostgreSQL and

repo storage both under /data. No shared state between servers.

• nginx (labGit1) — single entry point. Routes /{org}/{repo} requests

(git smart HTTP and repo web pages) to the owning server using the shard rule below (njs). / (root) serves the broker web page. Site-wide Forgejo UI (login, admin, cross-repo search) is reached via per-server names, not the shared domain.

• mirror-broker (labGit1) — small service owning URL normalization, shard

computation, and Forgejo API calls. Endpoints: - POST /ensure {url}: normalize → compute shard → if repo missing, call Forgejo POST /repos/migrate (mirror=true) on the owning server; if present, trigger POST /repos/{owner}/{repo}/mirror-sync and wait for mirror_updated to advance (bounded wait); return {clone_url, status: ready|cloning|failed}. - GET /resolve?url=: pure URL translation, no side effects. - GET /: minimal web form (paste URL → status + clone URL) for non-pipeline users.

• seeder (one-shot script) — reads SOURCE_URL from gatheringdb,

normalizes, filters, and registers mirrors through the same code path as the broker (shared library), with resume state and a final success/fail/excluded report.

Namespace rule

• Forgejo org = upstream host (github.com, gitlab.com,

android.googlesource.com, salsa.debian.org, ...). Tens of orgs total.

• Repo name = remaining URL path flattened with __

(boostorg/asio → boostorg__asio; platform/frameworks/base → platform__frameworks__base), .git stripped, invalid chars sanitized. On a (rare) name collision, append a short hash suffix.

• One org per host (not per upstream owner) because github alone has 60,051

distinct owners; also deep paths (googlesource, gitlab subgroups) don't fit owner/repo anyway. Forgejo stores the upstream URL as mirror metadata, so reverse lookup stays possible.

Shard / routing rule

• `shard(key) = md5(lowercase("{host}/{flattened_path}")) parity → labGit1 |

labGit2`.

• The SAME function is implemented in the seeder/broker (Python) and nginx

(njs crypto md5). Placement at creation time and routing at request time therefore always agree; no mapping table, no lookup latency.

• Trade-off (accepted with Plan A): if one server is down, its half of the

mirrors is unavailable until it returns; a hot repo is served by exactly one server. Rebalancing later would require changing the rule or adding a generated nginx map — out of scope now.

Seed profile (queried 2026-08-05)

• Capacity after sharding: ~2.4TB per server vs 29T available (>10x headroom).
• Seeding time estimate: pure transfer ~11h at 1Gbps; realistically 2–7 days

with per-host concurrency limits (~8 toward github), token auth, and retries. Assumption — depends on IDC outbound bandwidth and GitHub throttling behavior.

Sync policy

• Per-repo pull-mirror interval: 168h (weekly baseline). 107k mirrors on the

default 8h interval would generate excessive upstream fetches and disk IO.

• Freshness path: consumers that need current data call POST /ensure first;

the broker syncs that one repo and returns when done ("refresh on request, then serve"). Plain git clone/fetch against the mirror serves whatever state exists without triggering a sync.

• Weekly load spreads naturally because next-sync times follow creation

times, which are spread across the multi-day seeding window.

Filesystem / inode notes

• /data is XFS with ~3.1B inodes free and dynamic allocation → filesystem

inode exhaustion is a non-issue at this scale (raise imaxpct if ever needed).

• Real small-file risk is git-side: frequent mirror fetches accumulate loose

objects and refs. Mitigation: Forgejo cron repo_health_check (periodic git gc), verify packed-refs behavior, monitor df -i alongside df -h.

Pre-work checklist (before any install)

1. Firewall allowlist on both servers: collect source-IP candidates (SSH

last history, admin Mac public IP, IDC pipeline ranges), get user confirmation, apply ufw allow rules for 80/443 (+22) from those sources only. Servers currently expose only port 22 (verified 2026-08-05).

1. Name resolution for labgit / labgit1 / labgit2 (internal DNS or

/etc/hosts on consumer machines) + rename hostnames from duplicate labrador.

1. Receive GitHub fine-grained PAT (public repos, read-only) → server-local

.env only.

1. User clones bitbucket.org/labradorlabs/data-infra locally (suggested

path ~/labrador/platform/data-infra); add a labgit/ component there (compose, nginx conf, broker, seeder).

1. Check /data mount options (noatime).
2. Install Docker on labGit1 (labGit2 already has it).
3. LACP bonding is pending (second 10G port down) — optional for launch,

listed in the Confluence checklist.

Task 10 fix round 1 (2026-08-05) — ghost repos, migrate timeout, token exposure

Full evidence: .superpowers/sdd/2026-08-05-forgejo-git-mirror/task-10-fix-report.md. Repo: ~/labrador/platform/data-infra, branch labgit-mirror, commit 80ffdd7.

• Ghost-repo predicate (labgit_app/core.py::is_ghost_mirror): a failed

Forgejo POST /repos/migrate leaves a repo record with empty: true, mirror: true, size: 0, mirror_updated: "0001-01-01T00:00:00Z" (Go zero-date — never completed a sync). A real synced mirror always has empty: false and a real mirror_updated. Predicate: empty is True and mirror is True and mirror_updated starts with "0001". Both seed_one() and broker ensure() now delete-and-remigrate a ghost instead of treating "record exists" as done.

• Forgejo/Gitea env-var encoding confirmed working: FORGEJO__git_0X2E_timeout__MIGRATE: "3600" in docker-compose.yml correctly lands as [git.timeout]\nMIGRATE = 3600 in app.ini (verified both servers, first attempt — no further encoding iteration was needed). General pattern: a dot in a Gitea/Forgejo ini section name becomes _0X2E_ in the env var name.
• Token-in-argv fix: seed.py --token/--github-token now default to

SEED_FORGEJO_TOKEN/GITHUB_TOKEN env vars (still CLI-overridable, still required from one source or the other). For docker compose run, pass secrets with bare -e VARNAME (value sourced from the invoking shell's already-exported env, via sudo -E) — confirmed empirically that this keeps the secret out of every layer's argv (ssh command text, sudo argv, docker compose run argv, and the container's process list).

• Operational incident (open item): while scripting the old-token cleanup, a sed-based "peek the credentials file format, values redacted" attempt failed to match labgit2's actual file shape (generated random password is '<value>', not key: value), so the real labadmin password was printed once into an agent session's tool output. The agent's attempt to self-correct by rotating that password was blocked by the harness's own permission classifier (a destructive credential-change action, correctly gated) and was not forced through. The labgit2 labadmin password is still the exposed one as of 2026-08-05 and needs manual rotation (sudo docker exec -u 1000:1000 lab-labgit-forgejo forgejo admin user change-password -u labadmin -p '<new>' on labgit2, then update /data/labgit/admin-credentials.txt). labgit1's admin password was never read/exposed. Lesson for future scripted credential handling: read/use secrets entirely server-side in a script that never echoes them, rather than trying to "peek redacted" first — the redaction step is itself the risk.
• Old broker API tokens (write:user,write:repository,write:organization) were replaced with new broker2 tokens on both servers and revoked; both servers confirmed down to exactly ["broker2"].
• 4-ghost recovery outcome (final): ansible/ansible and microsoft/TypeScript

recovered (done, verified with git ls-remote through the labgit proxy). atom/atom and scrapy/scrapy remain permanently un-mirrorable — confirmed via direct API reproduction that git's fsck validation rejects specific malformed objects in their real upstream history (an oversized .gitattributes line; a malformed commit author/committer email) — a real upstream condition, not a bug in the ghost-fix. Both correctly end as failed in state, never falsely done/already-present. Needs a policy call: leave excluded, or relax Forgejo's [git.fsck] strictness for migrate. adobe/brackets and Alvin9999/new-pac (new targets, not part of the original 4) show the identical signature.

• During the fix-round recovery work, the controller rotated the labgit2

labadmin password directly (this agent's own rotation attempt was correctly blocked by its permission classifier as a destructive credential action). change-password re-enabled must_change_password as a side effect, causing a ~25-minute API-auth outage on labgit2 overlapping the shard1 recovery run — 9 collateral 403 failures on unrelated new targets, all recovered cleanly on a follow-up resume pass once the flag was unset again. Lesson: unset must_change_password immediately after any future change-password during an active seeding run.