Data

Data — questions and answers

What does it mean to normalise a database, and when would you skip it?

Normalisation structures a database to reduce redundancy by splitting data into related tables. You might denormalise deliberately for read-heavy applications where query speed matters more than storage efficiency, accepting some duplication to avoid expensive joins. Data warehouses often use denormalised star schemas for exactly this reason.

How do you handle missing data without skewing the analysis?

It depends on why data is missing. If values are missing completely at random, you might drop those rows or impute using mean or median. If missingness correlates with other variables—say, high earners skip income questions—ignoring it biases results. Sometimes the best approach is analysing only complete cases and stating that limitation clearly.

What makes a dataset tidy, and why does structure matter before analysis?

Tidy data has one observation per row, one variable per column, and one value per cell. This structure lets analytical tools work predictably—filtering, grouping and joining behave as expected. Untidy data, like multiple measurements in one column or variables spread across column headers, requires reshaping before most analysis tools can process it properly.

When does a correlation actually suggest causation might exist?

Correlation alone never proves causation, but temporal precedence, dose-response relationships, and consistency across different datasets strengthen the case. Randomised controlled trials or natural experiments provide better evidence. In observational data, techniques like propensity score matching or instrumental variables help, but confounding variables can still mislead without careful controls.

Can machine learning models trained on one dataset work reliably on another?

Only if the new data comes from the same distribution. Models trained on summer sales data may fail in winter; fraud detection trained on one region may miss patterns elsewhere. Concept drift—where underlying patterns change over time—degrades performance. Regular retraining and monitoring prediction accuracy on fresh holdout data catch these problems.

What's the practical difference between batch processing and streaming data?

Batch processing handles data in scheduled chunks—nightly ETL jobs, weekly reports—which suits historical analysis and tolerates some latency. Streaming processes records as they arrive, enabling real-time dashboards or immediate alerts. Streaming requires different infrastructure like Kafka or Kinesis and complicates operations, so it only makes sense when timeliness genuinely matters.