Crawler Collection-Count Log (product/version insert·update)

# Crawler Collection-Count Log (product/version insert·update)

AI Summary

Purpose:

• Define the shared per-batch "collection-count" log format emitted by the

library crawlers (crawler-lib-{golang,php,ruby,swift,dotnet}) and how the insert vs update counts are derived. One parser handles all five crawlers.

Key points:

• Two log lines, identical message format across all five crawlers:

- [batch summary] product insert=<n> update=<n>, version insert=<n> update=<n> - [total summary] product insert=<n> update=<n>, version insert=<n> update=<n>

• [batch summary] = counts for the batch that just finished. [total summary]

= cumulative over the whole run, emitted right before the run's ... STOP ... log line.

• All five crawlers write product/version with MySQL

INSERT ... ON DUPLICATE KEY UPDATE (upsert), which does NOT report per-row insert vs update. So the count is derived by SELECTing the batch's primary keys just before the upsert: key already in DB = update, key absent = insert.

• Counting lives in each crawler's shared base class (crawler/baseCralwer.py;

swift spells it baseCrawler.py), hooked at the top of setInsertIntoDB (and, in golang, also _setInsertIntoDBWithConnection). Only the PRODUCT_TABLE / VERSION_TABLE (golang also *_V2) are counted; status / list / license tables are ignored.

• Per-batch key de-dup (_collectSeen) prevents the version-sorting re-upsert

(same key written again to set SORT_ORDER) from being double-counted as an update.

Relevant when:

• Parsing/aggregating crawler logs to report how much each batch collected.
• Adding the same count log to another crawler, or changing the format.
• Debugging why insert/update numbers look off (see Open Questions / caveats).

Do not read full document unless:

• You need the exact method names, per-crawler hook placement, or the caveats.

Open Questions

• Counts are point-in-time vs the DB at upsert time. Two crawler workers writing

overlapping keys concurrently can each classify the same key as "insert" (both SELECT before either commits). Acceptable for a rough collection report, not an exact ledger.

• A version key that exists in the DB but is touched only by the version-sorting

pass (not present in the batch's main version list) is counted as an update. This is data genuinely re-touched in the batch, so it is intentionally kept, but it can make version update slightly larger than "distinct versions crawled".

• No live container smoke test was run when this was added (2026-06-10); logic

was unit-tested in isolation and all files byte-compile. A one-window container run is recommended to confirm at runtime.

Details

Source: implementation across the five repos, recorded in [[../../worklog/2026/2026-W24.md]] (2026-06-10 entry). Crawler internals: [[../../repo-notes/crawler-lib-golang.md]], crawler-lib-php, crawler-lib-ruby, crawler-lib-swift, crawler-lib-dotnet repo-notes.

Log format

Full log line (the logger format string is identical in all five crawlers):

%(asctime)s %(levelname)s %(process)s %(filename)s:%(lineno)d %(message)s

The <message> is one of:

[batch summary] product insert=12 update=3, version insert=45 update=8
[total summary] product insert=1340 update=210, version insert=5021 update=880

• product = rows written to the crawler's PRODUCT_TABLE

(golang V2: PRODUCT_TABLE_V2).

• version = rows written to the crawler's VERSION_TABLE

(golang V2: VERSION_TABLE_V2).

• insert = key not present in the table before the batch's upsert.
• update = key already present.
• Only filename:lineno differs between crawlers (different call sites); the

message itself is byte-identical, so one regex parses all five.

Suggested parse regex (message portion):

\[(batch|total) summary\] product insert=(\d+) update=(\d+), version insert=(\d+) update=(\d+)

How insert vs update is derived ("lightweight A")

Why not affected_rows: a batch is sent as ONE multi-row INSERT ... ON DUPLICATE KEY UPDATE statement. MySQL returns a single affected_rows for the whole statement (1 per insert, 2 per changed update, 0 per unchanged update), so the per-row insert/update split cannot be recovered from it. The arithmetic trick (inserts = 2N - affected) breaks as soon as any update is a no-op (unchanged row → 0).

Chosen approach: before the upsert, take the batch's primary keys and run one SELECT ... WHERE PRODUCT_KEY IN (...) (chunked at 500, parameter-bound). Keys returned = updates, keys absent = inserts.

• product key = PRODUCT_KEY (within one crawler LANGUAGE/REPOSITORY are

constant).

• version key = (PRODUCT_KEY, VERSION); the existence SELECT filters by the

batch's distinct PRODUCT_KEYs and matches (PRODUCT_KEY, VERSION) pairs in memory.

Shared base-class helper

Added to each crawler's base class:

• _resetCollectStats() — called at batch start; resets the per-batch counters

and the _collectSeen de-dup sets; initializes the cumulative counters once.

• _entityForTable(table_name) — maps a table name to 'product' / 'version'

/ None (golang also recognizes PRODUCT_TABLE_V2 / VERSION_TABLE_V2).

• _selectExistingKeys(table_name, columns, product_keys) — chunked IN lookup.
• _recordCollectStats(table_name, datas) — called just before each upsert;

classifies and accumulates into per-batch + cumulative counters. Wrapped in try/except so a stats failure logs [collect stats] skipped ... and never blocks crawling.

• _collectBatchSummary() / _collectTotalSummary() — format the message.

The hook is one line at the top of setInsertIntoDB: self._recordCollectStats(table_name, datas) (runs before the write so the SELECT sees the pre-upsert state).

Per-crawler placement

Verification (2026-06-10)

• python3 -m py_compile passes for all five repos (base + crawler files).
• Classification logic unit-tested against a fake DB: product insert/update,

version insert/update, version-sorting re-insert de-dup, non-target table ignored, and per-batch reset with cumulative total all matched expectations.

• Not yet run inside a live crawler container (needs DB/network).