Learn / Supervised
Classification
Predicting a category, and a probability for it. Decision boundaries, thresholds and metrics that survive imbalance.
On this page
Intuition
A classifier draws a boundary: on one side it says “A”, on the other “B”. Simple models draw straight boundaries; flexible ones draw curves, islands and checkerboards. Good classifiers also say how sure they are, because the decision (block the payment, call the customer) depends on the cost of being wrong.
What the data looks like
Each row is one example, the inputs describe it, and the answer is a category.
| rowpayment | inputamount_lei | inputhour | inputmerchant_type | inputnew_device | inputkm_from_home | answeris_fraud |
|---|---|---|---|---|---|---|
| P-88310 | 249 | 14 | grocery | no | 2 | no |
| P-88311 | 9800 | 3 | electronics | yes | 1430 | yes |
| P-88312 | 75 | 9 | fuel | no | 11 | no |
| P-88313 | 4200 | 23 | jewellery | yes | 860 | yes |
The answer is a category, so this is classification. In the playground each dot is one row: two inputs (x, y) and the class as its colour.
Example rows are invented for illustration.
How the data is prepared
- Stratify the split so every set keeps the same class mix.
- Scale features for kNN, SVM, logistic regression and neural nets.
- Imbalance: keep the test set at the real class ratio; rebalance only the training set, or use class weights.
- Leakage: the most common cause of a “95% accurate” model that fails in production. See data preparation.
Algorithms
Logistic regression
A linear score squashed through a sigmoid into a probability.
- Cost
- O(epochs·n·d)
- Strength
- Calibrated probabilities, interpretable, hard to beat as a baseline.
- Weakness
- Linear decision boundary.
- Used for
- credit scoring, churn baseline
k-nearest neighbours
Vote among the k closest labelled points.
- Cost
- O(n) per query
- Strength
- Simple and flexible boundaries.
- Weakness
- Needs feature scaling; slow on large data.
- Used for
- similar-customer lookup, anomaly triage
Naive Bayes
Assume features are independent given the class and apply Bayes’ rule.
- Cost
- O(n·d)
- Strength
- Tiny, fast, works with little data.
- Weakness
- The independence assumption is usually false.
- Used for
- spam filtering, text routing
Decision tree
Ask a sequence of yes/no questions that make the groups as pure as possible.
- Cost
- O(n·d·log n)
- Strength
- Readable rules, no scaling needed.
- Weakness
- Overfits when grown deep.
- Used for
- eligibility rules, explainable decisions
Random forest
Average many deep trees, each trained on a bootstrap sample with random feature choices.
- Cost
- O(trees·n·log n)
- Strength
- Robust, little tuning, good out of the box.
- Weakness
- Larger and slower than a single tree; less interpretable.
- Used for
- fraud detection, churn prediction
Support vector machine
Find the boundary with the widest margin; kernels bend it.
- Cost
- O(n²) - O(n³) exact
- Strength
- Effective in high dimensions; strong with the right kernel.
- Weakness
- Scores are not probabilities; kernel SVMs scale poorly.
- Used for
- text classification, small tabular sets
Neural net (MLP)
Learn the features and the boundary together.
- Cost
- O(epochs·n·weights)
- Strength
- Learns arbitrary boundaries.
- Weakness
- Data-hungry; easy to overfit.
- Used for
- onboarding conversion, behaviour models
The maths
Logistic regression turns a linear score into a probability with the sigmoid, and is trained by minimising log-loss:
A decision tree instead picks the split that most reduces impurity, for example Gini . A support vector machine maximises the margin by minimising the hinge loss plus .
From the confusion matrix:
Real-world examples
- Churn prediction: which mobile app users are about to leave, early enough to act.
- Fraud detection: rare positives and expensive misses, so recall at a fixed false-positive budget is the metric.
- Onboarding conversion: who will finish signing up, and where a nudge helps.
- Credit scoring and spam filtering: the textbook cases.
Playground
A single tree, a random forest and logistic regression on moons: the tree’s boundary is a jagged staircase, the forest averages thirty of them into something smooth, and logistic regression can only draw one straight line. Switch the dataset to spirals to see which models can follow it at all.
Tap or click the plot to add a class A point. Tick "Add clicks as class B" for class B.
Press Train to compare.
Press Train to compare.
Train a model to see its loss curve.
| Step - | train ━ | test ┅ |
|---|
class A class B · shading = the model's prediction, stronger where it is surer · ● train ○ test
Press Train. The verdict appears here.
In Python
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import average_precision_score, classification_report
from sklearn.model_selection import train_test_split
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
clf = RandomForestClassifier(n_estimators=300, class_weight="balanced", n_jobs=-1).fit(X_tr, y_tr)
proba = clf.predict_proba(X_te)[:, 1]
print("PR-AUC", average_precision_score(y_te, proba))
print(classification_report(y_te, proba > 0.3)) # pick the threshold from costs, not 0.5 by habit
Pitfalls
- Accuracy on imbalanced data. 99% accuracy can mean “never predicts fraud”.
- The 0.5 threshold. It is a default, not a decision. Choose it from the cost of false positives and false negatives.
- Uncalibrated scores. A forest’s 0.8 is not an 80% chance; calibrate if you use probabilities as numbers.
- A boundary that is too clever. If kNN with k = 1 scores 100% on training data, that is memorisation.