# Confluence CDC Tool Research Evidence
AI Summary
Purpose:
- Preserve resume-safe evidence from the Confluence CDC research, Maxwell PoC,
production binlog-delivery documentation, and ownership record.
Key points:
- The research compared Debezium, Maxwell, and Canal across multi-instance
operation, transport flexibility, HA and scale, database support, and architectural complexity.
- Debezium was strongest for clustered HA and multi-target streaming but required
Kafka and Kafka Connect. Maxwell was lower-complexity and supported direct file output, which better matched the file-based delivery boundary.
- A hands-on Maxwell PoC on MySQL 8.4 emitted JSON for INSERT, UPDATE, DELETE, and
DDL events. The PoC also exposed that Maxwell had no official MySQL sink and would need Kafka Connect or a custom consumer to apply changes downstream.
- Customer constraints mattered: a prior MySQL replica approach was rejected
because of TCP communication, customer update schedules differed, and the product did not treat sub-hour real-time latency as the primary requirement.
- The generic real-time CDC proposal was marked deprecated and not rolled into
production. Production remained a customer-scheduled HTTPS/file delivery path.
- The quality-improvement status page assigns CDC study, design, and implementation
work to Kim Hyunwook. The production BTS history also records Hyunwook as the owner of the 1.1.0 and 2.0.0 improvements.
- Confluence supports the claims that Hyunwook compared tools, ran a Maxwell PoC,
identified operational tradeoffs, and selected a batch-oriented production boundary. It does not support a claim that Maxwell, Kafka, event-level offsets, or a DLQ were operated in production.
Relevant when:
- Writing or reviewing CDC, technology-learning, architecture-judgment, or
production-ownership claims for the Toss Payments Data Engineer application.
Do not read full document unless:
- Exact comparison criteria, PoC observations, ownership, or the boundary between
the deprecated real-time proposal and production delivery is required.
Linked documents:
../../wiki/projects/bts.md2026-07-16-cdc-resume-clarification.md../../../human/resume/toss-payments-data-engineer/resume-draft.md
Open Questions
- Confluence does not record the Maxwell failure on compressed MySQL 8.4
Transaction_payload; that claim must remain grounded in repository evidence and the separate implementation review rather than attributed to Confluence.
- The exact start date and observation period for the approximately ten production
customer deployments remain unrecorded.
Details
1. Tool comparison and customer constraints
Source:
Verified facts:
- Compared binlog, trigger, polling, application/outbox, and commercial CDC
approaches.
- Compared Debezium, Maxwell, and Canal using multi-instance support, transport
flexibility, HA/scale, supported databases, and architectural complexity.
- Recorded Debezium as the strongest clustered and multi-target option, Maxwell as
simpler but dependent on external orchestration for HA, and Canal as a server-client alternative.
- Recorded that a previous replica-style TCP connection was rejected by a customer.
- Contrasted the existing hourly HTTPS file path with a proposed Debezium/Kafka API
streaming path.
2. Initial Maxwell selection
Source:
Verified facts:
- Proposed Maxwell plus FileSink because only CDC and file output were needed.
- Explicitly preferred the lower-complexity single-process path over operating the
Debezium and Kafka stack for this requirement.
- Offset-based reprocessing and event-level history were expected benefits of the
proposal, not verified production behavior.
3. Hands-on Maxwell PoC
Source:
Verified facts:
- Ran Maxwell against a MySQL 8.4 test setup using ROW/FULL binlogs and a separate
metadata database.
- Captured JSON examples for INSERT, UPDATE, DELETE, and DDL events, including
transaction and binlog-position metadata.
- Documented that Maxwell did not provide an official MySQL sink and would require
Kafka Connect for downstream database application.
- Left bulk update/delete behavior, encryption, monitoring, and end-to-end
architecture as future test items.
4. Proposal versus production
Sources:
- [[BTS-Deprecated] 품질 개선](https://labradorlabs.atlassian.net/wiki/spaces/DT/pages/3481829378/BTS-Deprecated)
- Binlog Transfer System
Verified facts:
- The real-time CDC proposal is explicitly marked deprecated because customer and
other constraints prevented that route from proceeding.
- The proposal notes that one-hour latency itself was not a major product problem;
data consistency and recovery were the more important concerns.
- Production documentation describes customer-specific schedules, HTTPS/file
download, scrambling, separate/combined download and import modes, timeout handling, and customer error-log collection.
- Version history records Kim Hyunwook as the owner of the 1.1.0 and 2.0.0
improvements.
5. Assigned ownership
Source:
Verified facts:
- Assigns the CDC knowledge study, architecture design, and implementation work to
Kim Hyunwook.
- Marks the CDC study/improvement item complete while listing design and
implementation as the subsequent work.
Resume-safe interpretation
- Safe: compared Debezium, Maxwell, and Canal using concrete operational criteria.
- Safe: ran a Maxwell FileSink PoC and verified DML/DDL JSON event output.
- Safe: rejected a generic real-time stack after considering customer network,
cadence, sink, and operating-complexity constraints.
- Safe only with separate repository/user evidence: production Go Updater, local
go-mysql fork, ZSTD Transaction_payload handling, and approximately ten customer deployments.
- Unsafe: production Maxwell, production Kafka/Kafka Connect, production
event-level offset recovery, or production DLQ.