Data Mining

Data Mining — questions and answers

What's the difference between data mining and machine learning?

Data mining focuses on discovering unknown patterns in existing data, often for immediate business insight. Machine learning builds models that can make predictions or decisions on new data. There's significant overlap—many data mining tasks use machine learning algorithms—but data mining emphasises exploration and interpretation, while machine learning emphasises generalisation and automation.

How do you avoid finding patterns that are just coincidences?

You validate findings using holdout datasets the algorithm hasn't seen, apply statistical significance tests, check whether patterns hold across different time periods or subgroups, and test whether they make domain sense. Cross-validation and setting appropriate confidence thresholds help distinguish genuine patterns from noise. Very large datasets can produce statistically significant but practically meaningless correlations.

What does overfitting mean in practice?

Overfitting occurs when a model learns the noise and peculiarities of the training data rather than underlying patterns. It performs excellently on data it's seen but poorly on new data. Signs include very high training accuracy but low test accuracy, or a model with excessive complexity relative to the dataset size. Regularisation techniques and simpler models help prevent it.

Which file formats work best for datasets with millions of rows?

Parquet and ORC offer columnar storage with compression, making them efficient for analytical queries on large datasets. HDF5 works well for scientific data with complex hierarchies. Avro handles schema evolution cleanly. CSV becomes impractical at scale due to size and parsing overhead. The choice depends on whether you need row-based or column-based access patterns.

Can you mine data that contains gaps or missing values?

Yes, but you must handle missingness appropriately. Some algorithms tolerate missing values natively; others require imputation using mean, median, or predicted values, or you can create indicator variables flagging missingness. The handling method depends on whether data is missing completely at random, at random, or not at random—each has different implications for bias.

How long does a typical data mining project take from start to finish?

Duration varies enormously based on data volume, quality, complexity, and objectives. Data preparation often consumes 60-80% of project time. A straightforward customer segmentation might take two weeks; fraud detection systems with multiple data sources and model tuning can take months. Iterative refinement based on stakeholder feedback extends timelines beyond initial technical work.