Remember Data Engineer Assignment — Claude Implementation

# Remember Data Engineer Assignment — Claude Implementation

AI Summary

Purpose:

Preserve implementation, review, and cross-machine resume context for the Claude

Code implementation of the Remember & Company Data Engineer take-home assignment.

Key points:

User-confirmed status on 2026-07-28: the selected DuckDB package was submitted;

the application is awaiting the next result.

Working copy lives in Google Drive under the Claude career tree, not in projects/:

career/클로드이력서/리멤버앤컴퍼니-data-engineer/과제/작업본/.

A sibling 주석본/ holds an annotated study copy for interview walkthroughs

(2026-08-11): comments only, AST-identical code, same test and run results. Never repackage from it.

Resolve that path with the ASCII marker REMEMBER_CLAUDE_WORKDIR, never by pasting

the Korean path (macOS NFC/NFD normalization differs per machine).

The submission entry point is ./up.sh: Docker is the only prerequisite; no

.env, local Python, or DuckDB CLI is required.

Stack: Python 3.11, Airflow 2.10.5, DuckDB, Docker Compose.
One year-parameterized DAG (@yearly, catchup=true, max_active_runs=1) covers

both the 2022 initial load and the 2023 sync.

Default policy follows the assignment literally (blank 2023 field overwrites with

NULL); value preservation is a documented flag, not the default.

Verified result: 30,513 initial rows; 2023 sync 2,174 inserts / 4,000 updates /

17,087 no-ops / 9,426 deactivations; master 32,687 rows with 23,261 active.

Eight post-merge reconciliation checks pass on both runs and on both policies.
Tests: local 101 passed / 1 skipped (Airflow unavailable); container 107 passed.
README.md owns the assignment's required "설계 시 고려한 중점 사항" plus the

deliverable map and the run instructions. An earlier document-boundary cleanup had moved that rationale out of README, which broke deliverable 3 of the brief.

The package ships four Korean documents: README, ARCHITECTURE, and the two reports.

과제 작업 설명.md was folded into README on the user's instruction to stop splitting the same material across files.

The first gstack technical-lead review scored 54/100. After removing review-facing

prose, splitting the merge monolith, tightening runtime boundaries, and expanding transaction tests, the final review scored 96/100 with no major blocker.

Document boundaries: README carries the deliverable map, design decisions, and

operations; data profiling owns source evidence; run-result owns execution evidence; ARCHITECTURE holds three rendered Mermaid diagrams and the key definition.

Final package 김현욱_리멤버앤컴퍼니_DE과제.zip excludes generated DBs:

52 entries, 1,670,577 bytes, SHA-256 876bef94d18fe2d2648c88248ccd06060c310389addc9651bbe8892329d3dc96.

Google Drive resets working-copy file modes to 0600. The packaging recipe now

re-applies 0755 to up.sh and 0644 elsewhere in the staging copy and asserts both, otherwise the reviewer's ./up.sh fails with permission denied.

The ZIP was extracted into a fresh Korean-named directory and ./up.sh completed

from that copy: both target runs SUCCESS, master 32,687, source/master mismatches 0, container tests 107 passed.

Relevant when:

Resuming this implementation on any machine.
Verifying or submitting the selected DuckDB package.
Rebuilding the submission zip after editing the working copy.

Do not read full document unless:

Exact commands, expected counts, or the fixed-defect list are needed.

Linked documents:

ai/repo-notes/remember-data-engineer-assignment-codex.md
ai/workspace/machines.md
ai/workspace/active-context.md
ai/worklog/2026/2026-W30.md

Open Questions

Whether the reviewer reads the is_active soft-delete flag as sound judgment or

as scope the assignment did not ask for. The assignment defines Insert and Update only; the Codex implementation leaves 2022-only rows untouched instead.

Whether a blank 2023 field is an authoritative latest value or a missing

extraction. Unresolved with the recruiter. The assignment brief invites questions by mail or chat; sending one was considered and not done because of the deadline.

Details

Resume on another Mac

ASSIGNMENT_DIR=$(dirname "$(find "$HOME/Library/CloudStorage" \
  -name 'REMEMBER_CLAUDE_WORKDIR' -print -quit)")
cd "$ASSIGNMENT_DIR"
bash resume-here.sh            # venv at ~/.venvs/remember-de-assignment, tests, pipeline
bash resume-here.sh --docker    # plus full Docker E2E
cd 작업본 && ./up.sh            # final submission path; Docker only

HANDOFF.md in that directory carries the same instructions plus the expected numbers, the zip rebuild recipe, and the three environment traps below.

The venv is deliberately created outside Google Drive. A venv inside Drive syncs thousands of files and its interpreter paths are not portable between machines anyway. The Drive folder holds source, tests, docs, and the submission zip only — 7 MB total once the regenerable DuckDB file is removed.

Directory layout

career/클로드이력서/리멤버앤컴퍼니-data-engineer/과제/
  REMEMBER_CLAUDE_WORKDIR          ASCII marker for cross-machine path resolution
  resume-here.sh                   bootstrap + verification
  HANDOFF.md                       status, expected values, zip recipe, traps
  과제-원문.pdf                     assignment brief
  김현욱_리멤버앤컴퍼니_DE과제.zip    submission package
  작업본/                           canonical working copy
  주석본/                           annotated study copy (interview prep, not shipped)

Only 작업본/ enters the zip. Everything at the parent level is working material.

Annotated study copy (added 2026-08-11)

주석본/ is a copy of 작업본/ carrying line-by-line Korean comments on the DAG and all 19 include/pipeline/ modules, made for interview walkthroughs. scripts/, sql/, the shell entry points, and tests/ are unannotated. 주석본/주석본-안내.md records the scope and the verification commands, and 주석본/면접리뷰용.md traces the ./up.sh call order in three layers (host script, DAG tasks, apply_master transaction) as Mermaid sequence diagrams, verified to render with mermaid-cli.

Code is unchanged: with SQL -- comments stripped from string literals, all 20 files have an AST identical to 작업본/. Local verification on the annotated copy (no Docker, duckdb 1.5.5 + pytest 8.3.4 in a throwaway venv): pytest tests -q gives 101 passed / 1 skipped, and scripts/run_pipeline.py --fresh --years 2022 2023 reproduces every expected value in the table below, including 8/8 reconciliation on both runs.

Repackaging must always come from 작업본/; 주석본/ never enters the zip.

Expected values (default policy)

	2022	2023
Source rows	30,513	23,261
Inserted	30,513	2,174
Updated	0	4,000
Unchanged	0	17,087
Deactivated	0	9,426
Master rows	30,513	32,687 (23,261 active)
Reconciliation	8/8	8/8

Assignment-rule check (2023 source versus master) returns 0 mismatches. Master address nulls are 2,249; with SYNC_PRESERVE_EXISTING_ON_NULL=true they are 156. If numbers differ, check for a stale SYNC_* environment variable first.

One-take submission path (verified 2026-07-27)

./up.sh destroys only prior generated state, builds the Airflow image, starts PostgreSQL and Airflow, waits for the DAG to register, waits for the 2022 and 2023 ops_sync_run rows to reach SUCCESS, prints task logs and database inspection queries, and exports the finished DuckDB to warehouse/employment_insurance.duckdb. Services remain running for Airflow UI inspection until docker compose down.

Immutable inputs are baked into the image rather than bind-mounted. This fixes an observed Docker Desktop failure where Python imports from the Google Drive File Provider path returned OSError: [Errno 35] Resource deadlock avoided. DuckDB is created in a named volume and copied to the host only after both target runs succeed. The Docker build also normalizes Korean CSV names to NFC; macOS stores them as NFD, which otherwise fails byte-exact lookup inside Linux.

The first live build also exposed that Docker preserved the Google Drive working copy's 0600 mode when copying requirements.txt, then made the layer file root-owned. COPY --chmod=0644 makes the Airflow image user able to install it.