Learn / Foundations
Data preparation
Most of the work, and most of the mistakes, happen before any model is trained. How data is prepared for each kind of learning.
On this page
Intuition
A model can only learn what the data shows it. If the data that reaches the model differs from the data it will meet in production (different scale, different time period, a column that will not exist yet), the model learns the wrong thing and the test score hides it. Data preparation is the discipline of making training look like reality.
What the data looks like in each paradigm
Almost every dataset in machine learning is a table: one row per example, one column per fact about it. What changes between paradigms is what a row stands for and whether there is an answer column. Amber marks the answer, or the value the model produces.
| Paradigm | One row is | Answer column | Example columns |
|---|---|---|---|
| Regression | one day on one platform | a number: tomorrow’s transaction count | weekdaypaydayholidayvolume_last_weekvolume |
| Classification | one card payment | a category: fraud or legitimate | amount_leihourmerchant_typenew_devicekm_from_homeis_fraud |
| Clustering | one ATM over the last quarter | none in the data; the model adds a cluster number | withdrawals_daypeak_hourweekend_sharecash_outs_monthcluster |
| Dimensionality reduction | one customer with 50 spending-category columns | none in the data; the method outputs 2 new coordinates | foodfueltravelonline…pc1pc2 |
| Embeddings | one piece of text | none; the model outputs a list of 384 numbers (the embedding) | messagedim_1dim_2dim_3…dim_384 |
| Retrieval-augmented generation | one chunk of a document | none; at question time the system ranks chunks by relevance | sourcesectiontextembedding |
| Recommender systems | one user; each cell is that user’s rating of one title | the empty cells: the model predicts them | Nebula RisingVault JobParis LettersDeep OceansMidnight Score |
Supervised learning
- Labels. Decide exactly what you predict and when. “Churned within 30 days of the snapshot date” is a label; “churned” is not.
- Train / validation / test. Train fits the model, validation chooses hyper-parameters, test is opened once at the end. For anything that evolves over time, split by time.
- Leakage. Any feature that would not be known at prediction time leaks the answer. Fit scalers, encoders and imputers on the training set only; fitting them on all rows is leakage too.
- Scaling. Distance- and gradient-based models (kNN, SVM, logistic regression, neural nets) need features on comparable scales. Trees do not.
- Encoding. One-hot for low-cardinality categories; target or frequency encoding (computed inside cross-validation folds) for high cardinality.
- Imbalance. With 1% fraud, accuracy is meaningless. Use precision, recall, F1 or PR-AUC, and class weights or resampling on the training set only.
- Cross-validation. Rotate the validation fold to get a stable estimate from little data; use grouped or time-series folds when rows are related.
| rowcustomer | inputlogins_30d | inputbalance_lei | inputtenure_months | inputsalary_card | answerchurned_next_30d |
|---|---|---|---|---|---|
| C-1041 | 2 | 310 | 7 | no | yes |
| C-1042 | 26 | 18400 | 54 | yes | no |
| C-1043 | 0 | 95 | 3 | no | yes |
| C-1044 | 14 | 5200 | 22 | yes | no |
Supervised learning needs the answer column for every training row. The model learns to fill it in for new customers whose answer is not known yet.
Example rows are invented for illustration.
Unsupervised learning
- Scaling is not optional. Clustering and PCA are driven by distance and variance, so the feature with the largest numbers wins by default.
- Distance metric. Euclidean for dense numeric data, cosine for embeddings and text, Gower for mixed types.
- Outliers. A handful of extreme points can pull a centroid or define a principal component on their own. Clip, transform (log) or use a robust method.
| rowcustomer | inputavg_balance_lei | inputpayments_month | inputcard_share | inputapp_logins_month |
|---|---|---|---|---|
| C-2001 | 1200 | 38 | 0.82 | 41 |
| C-2002 | 64000 | 6 | 0.1 | 3 |
| C-2003 | 2100 | 45 | 0.91 | 52 |
| C-2004 | 58000 | 4 | 0.05 | 2 |
There is no label. Clustering will group similar rows; dimensionality reduction will summarise the columns. You judge whether the result means something.
Example rows are invented for illustration.
Embeddings and RAG
- Parsing. Get clean text out of PDFs, HTML and scans; keep headings and tables intact.
- Cleaning. Remove navigation, footers, duplicated boilerplate; normalise whitespace.
- Chunk size. Too small and an answer is split across chunks; too large and it is buried in unrelated text. The RAG lab lets you measure this.
- Metadata. Store source, section, date and access rights with every chunk so you can filter and cite.
- Deduplication. Near-identical chunks crowd out other evidence in the top k.
| rowchunk | inputsource | inputsection | inputtext | outputembedding |
|---|---|---|---|---|
| 1 | faq.md | Cards and limits | Every debit card has a daily ATM limit of 5,000 lei… | [0.02, -0.11, …] |
| 2 | faq.md | Transfers | International transfers settle within two business days… | [-0.08, 0.04, …] |
| 3 | faq.md | Branches | The Orbit City branch is open on weekdays from 09:00… | [0.05, 0.09, …] |
Documents are split into chunks, each stored with its source and an embedding. A question is embedded the same way and matched against this table.
Example rows are invented for illustration.
Recommender systems
- Interaction matrix. Rows are users, columns are items, almost every cell is empty.
- Implicit signals. Views, clicks and purchases are abundant but noisy; an absence of interaction is not a dislike.
- Negative sampling. Sample unseen items as negatives rather than treating all of them as negative.
- Time-based split. Train on the past, test on the future. A random split leaks what users will like later.
| rowuser | inputNebula Rising | inputVault Job | inputParis Letters | inputDeep Oceans | inputMidnight Score |
|---|---|---|---|---|---|
| U-01 | 5 | · | 2 | · | 4 |
| U-02 | · | 5 | · | 1 | 5 |
| U-03 | 4 | · | · | 2 | · |
| U-04 | · | 4 | 1 | · | ? |
Most cells are empty (·): people rate only a few titles. Recommending means predicting the empty cells, for example the ? for U-04.
Example rows are invented for illustration.
In Python
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
prep = ColumnTransformer([
("num", StandardScaler(), ["age", "balance", "tx_count_30d"]),
("cat", OneHotEncoder(handle_unknown="ignore"), ["segment", "channel"]),
])
# The pipeline re-fits the scaler inside every fold, so nothing leaks from validation rows.
model = Pipeline([("prep", prep), ("clf", LogisticRegression(class_weight="balanced"))])
scores = cross_val_score(model, X, y, cv=TimeSeriesSplit(5), scoring="average_precision")
Pitfalls
- Fitting preprocessing on the full dataset before splitting.
- Random splits on time-dependent data.
- Evaluating on resampled (balanced) data instead of the real class mix.
- Features that are updated after the event you are predicting.