Validation complete against both files the assessment provides. The full 40GB run is xAI's step, not the candidate's.

This page is part of the visual companion to the submitted zip (pipeline.py · requirements.txt · README.md · ARCHITECTURE.md). The take-home provides exactly two files — the small sample and one full corpus shard (part.9.parquet) — and states that tokenization, training, and evaluation run on xAI's own infrastructure after submission. Every number below is measured from real runs against those two provided files; the full-40GB wall-time figure is the README's required runtime estimate (an 18.5× extrapolation from the measured shard run), not a run that happened here.

xAI Data Engineer take-home — validated on both provided files

Scaling to 40GB

The homepage shows sample-data results — measured on the provided 100MB / 55,336-row sample. This page covers the second provided file: the full corpus shard (part.9.parquet, 2.16GB, 1,162,062 rows, the real part.N.parquet naming convention the pipeline ingests). It ran end-to-end on 2026-07-09; the first attempt failed twice, for real reasons only real-scale data could surface, and both failures became verified pipeline fixes — exactly what validating against the provided shard is for.

The capacity gate — resolved

Resolved 2026-07-09, both ways at once.

The original blocker ("only ~3.3GB free disk") was resolved externally — ~87GB was freed — and internally: the first real-shard run was OOM-killed at ~7.9GB worker RSS (a 2.16GB shard decompresses to several times its file size as one in-memory frame), which forced option (c) from the original proposal into the pipeline itself. process_file now reads shards in 100K-row batches, bounding worker memory regardless of shard size — the measured peak for the full successful run was 11.1GB RSS on a 16GB machine.

Staging plan

Stages 1–2 run (green, measured); stages 3–4 not started — no fabricated progress, no invented timing.

✓ gate resolved · ● run (measured) · ○ optional, not run · ▸ xAI's step post-submission

● Stage 1 — 1 real shard (run 2026-07-09)

part.9.parquet: 2.16GB, 1,162,062 rows, exact 4-column assessment schema. The distributions match the sample almost exactly — char_len median 2,895 vs. 2,893, language_score mean 0.930 vs. 0.930, 100% en — so the filter thresholds transfer without retuning. One real surprise: exactly 1 document in 1.16M contains literal <|endoftext|> text, which crashed tiktoken's default encoder — a contamination category with zero instances in the sample, now handled (and documented) in the pipeline.

● Stage 2 — 2.16GB end-to-end (run 2026-07-09)

Full pipeline run: 1,162,062 → 980,341 rows (84.4% retained), 17.2 min wall, 11.1GB peak RSS on the 10-core/16GB dev machine. Exact duplicates doubled vs. the sample (13.3% vs. 6.4% dropped); the "safety net" filters that never fired on the sample caught real contamination (punct 47, repetitive lines 5, digit-heavy 1, degenerate word length 2). First attempt was OOM-killed — fixed via chunked shard reads, verified byte-identical on the sample before rerunning.

○ Optional ~10GB rehearsal — not run, not required

The assessment never asks the candidate to process the full corpus (only the two provided files were ever available here). A multi-shard rehearsal would further de-risk one remaining open question: the pd.concat of all cleaned shards before global near-dedup. The per-shard side is measured (11.1GB peak for one 2.16GB shard, bounded by chunked reads); the combined-corpus side is an estimate.

▸ The full 40GB run — xAI's step, post-submission

Per the take-home's own instructions, the submitted pipeline is run, tokenized, trained, and benchmarked on xAI's infrastructure. What the assessment requires from the candidate is the runtime estimate, now grounded: the ~40GB corpus is ~18.5 shards like the provided one, projecting to roughly 45–60 minutes on the stated 64-core/256GB machine (an 18.5× extrapolation from the measured shard run, replacing the original 415× sample extrapolation) — comfortably inside the 4-hour budget.

What has to be re-verified at scale, not assumed

Filter threshold validity. Thresholds (e.g. alpha_fraction bounds, punct_fraction ceiling) were tuned to what 55,336 rows happened to show; several are generous "safety net" bounds specifically because the sample didn't exhibit the problems they guard against. At ~400× more documents it's plausible some finally get exercised for real.
Near-dedup at scale. Bucketing and union-find were left single-process because they were under 2% of cost on ~51K post-filter documents. At ~400× more surviving documents, LSH bucket/candidate-pair counts grow too — worth re-profiling at the Stage 3 scale specifically.
pd.concat memory ceiling — needs a real answer before the full run, not an assumption.
--workers sizing. Currently defaults to min(len(input_files), os.cpu_count()) — needs the real target machine's core count and real shard count to size correctly; a mismatch silently under-utilizes cores.

Gaps in pipeline.py's CLI

pipeline.py has no flag to limit which or how many shards are processed — --input unconditionally globs every .parquet file in the directory given. Staging a subset (Stages 1–3 above) therefore requires hand-copying a subset of real shards into a separate directory before running the pipeline — not a documented pipeline feature, a workaround. There is also no --dry-run/schema-check-only mode, and of PipelineConfig's 18 tunable fields, only min_language_score is exposed on the CLI — changing the alpha_fraction/punct_fraction thresholds flagged above requires editing source and redeploying.

Still not known (open inputs to the runtime estimate)

The full corpus's total shard count — the provided shard measured 2.16GB / 1,162,062 rows, implying ~18.5 shards for ~40GB, but only part.9 was provided.
The real target machine's specs — the 64-core/256GB figure is the assessment's own "Think:" guidance, and every target-machine wall-time number here is an extrapolation across hardware.
Whether bucketing/union-find stay negligible at ~18M documents (measured fine at 997K post-filter documents in the shard run).
The combined-corpus pd.concat memory ceiling — per-shard peak is measured (11.1GB), the ~19-shard concatenation before global near-dedup is not.

Resolved by the real run: the sample's thresholds DO generalize (distributions nearly identical at 21× scale, and the safety-net filters caught real, nonzero contamination); the real corpus file layout IS part.N.parquet as assumed.

What does NOT need to change

The filter logic, the exact/near-dedup algorithm, and the Pydantic/enum typed layer built this session are architecture-complete for this task — nothing here proposes rewriting the pipeline's approach, only validating it carries forward correctly at scale and adjusting execution parameters (shard count, worker count, disk/memory provisioning) based on what the staged validation above actually finds.

Sequencing summary

✓ Resolve the disk/memory capacity question — done 2026-07-09 (87GB freed + chunked reads).
✓ Get the real corpus's actual file layout — part.N.parquet at ~2.16GB/shard, measured; target-machine specs remain the assessment's stated 64-core/256GB guidance.
✓ Run Stage 1 (single real shard) — aggregate statistics match the sample's (see cards above).
✓ Run Stage 2 (2.16GB end-to-end) — 84.4% retained in 17.2 min; two real failures surfaced and fixed (OOM → chunked reads; special-token crash → tokenizer fix).
○ Optional ~10GB rehearsal — would tighten the combined-corpus memory-ceiling and union-find-at-scale answers, but is not part of the assessment's ask.
▸ The full 40GB run — performed by xAI on their infrastructure after submission; the submission carries the required runtime estimate for it.

Validation against both provided files is complete (2026-07-09). The submission — pipeline.py, requirements.txt, README.md — is ready.

agenttables