I am building talktoit to ask questions of Apple Health data, but the first obstacle was getting an export into the database. My bounded test input contained six months of data and took 1594.9 seconds to import. Before a conversational interface could be useful, that wait needed an explanation. A better answer generator would not fix a pipeline that took about 26 minutes to prepare its facts.
The first pass used fairly conventional improvements: put database writes in a transaction, replace slow timestamp parsing, combine XML passes, and tidy the regular-expression work. Import time fell to 1259.9 seconds. That was progress, but it left most of the wait intact. It also gave me a useful reason to stop assuming that database writes were the main problem.
Measure the work between the reads and writes
Profiling pointed to Python’s per-record work. The XML parser built objects, then pulled attributes back across the C/Python boundary one field at a time, under the GIL. Repeating that process around 1.35 million times dominated the run. DuckDB and disk I/O were not the bottlenecks I had expected. The next change needed to reduce that repeated object work.
- Memory-map the export so workers can scan the input bytes.
- Split work at record-safe boundaries and extract the supported records with byte-level scans.
- Write Parquet shards as an intermediate representation.
- Load and reconcile those shards in DuckDB, keeping coverage and record relationships explicit.
Parallel extraction creates correctness work of its own. A record crossing a worker boundary must not disappear or be counted twice. Child rows must still belong to the right parent records when shards meet again. Transaction boundaries matter because an interrupted load should not leave a plausible-looking partial database. The speedup is useful only alongside an account of what was imported.
The importer is part of the answer
talktoit’s local analytics and conversational layer depend on those stored facts. Local processing is the default; remote planning or narration is an explicit opt-in configuration. That choice belongs beside the ingestion design, because both affect what happens to a sensitive export. This build note uses aggregate engineering timings only. The lasting lesson for me is the one the first optimization pass exposed: sometimes the next improvement comes from changing the unit of work, rather than making each existing step slightly faster.