Data Processing

Data Processing — questions and answers

What happens if you process data in the wrong order?

Sequence matters because later steps often depend on earlier ones. If you remove duplicates before standardising formats, records that should match won't be recognised as duplicates. If you filter rows before aggregating, your totals reflect only the subset. If you apply currency conversion after summing, exchange rates compound incorrectly. Reversing steps can produce results that look plausible but are quietly wrong.

How do you verify that processing hasn't lost or altered records incorrectly?

Compare row counts and control totals before and after each step. Hash critical fields to detect unintended changes. Spot-check a random sample of outputs against inputs. For high-stakes work, process a subset with known characteristics first and confirm the results match expectations. Logging every transformation lets you trace discrepancies back to the step that introduced them.

Why do text encoding problems appear during processing even when the source file opens fine?

Applications often guess encoding when displaying files, masking issues until data moves between systems. A file saved as Windows-1252 might show correctly in Excel but produce garbled characters when imported into a UTF-8 database. Processing exposes these mismatches because it explicitly reads and writes bytes. Specifying encoding at every read and write operation prevents silent corruption.

When should you split processing into multiple passes instead of doing everything at once?

Split when intermediate outputs need manual review, when different steps require different tools, or when debugging becomes difficult. A single complex script that fails halfway through wastes time re-running early steps. Separate passes also let you checkpoint progress—useful when processing takes hours or when source data might change mid-project. The trade-off is managing intermediate files and ensuring they stay synchronised.

What makes a processing script reusable versus one that only works once?

Reusable scripts parameterise inputs, outputs and key values rather than hardcoding them. They handle variations in source data—extra columns, different delimiters, unexpected nulls—without breaking. They log what they did and flag anomalies rather than failing silently. Documentation explains assumptions: which fields must exist, what formats are expected, how errors are handled. A one-off script works for today's file; a reusable one works for next month's too.

How do you process data that's too large to fit in memory?

Stream it in chunks, processing one batch at a time and writing results incrementally. Use databases or tools designed for out-of-core processing like Dask or chunked Pandas reads. Sort or filter early to reduce volume. For aggregations, calculate running totals rather than loading everything first. Avoid operations that require seeing the entire dataset simultaneously—those force you into distributed processing frameworks or sampling approaches.