Feature Engineering for Tabular ML in 2026: What Still Beats a Transformer

The 2026 Tabular Reality Check

The pitch is seductive: throw a transformer at your tabular data, skip the feature engineering, and let attention do the work. A few years into the tabular foundation-model era, most production teams that tried this have quietly gone back to gradient boosting with hand-built features. Not because transformers are bad — because on the tabular problems most US enterprises actually have, they are expensive, harder to maintain, and rarely more accurate.

Here is the thesis up front: for the majority of tabular ML problems in 2026, well-engineered features paired with gradient boosting match or beat transformer-based models — and they do it at a fraction of the cost, latency, and operational risk. "Beats" here means four things at once: accuracy, inference cost, latency, and maintainability. A transformer that wins by a fraction of a point of AUC but triples your serving bill and resists debugging is not a win. (The numbers in this article are illustrative of typical enterprise trade-offs, not measured results from a single study.)

This article covers why the gap persists, the feature engineering that closes it, and a workflow that keeps your team from chasing the wrong model. One caveat before we start: this is the evidence-weighted default, not a law of nature. The frontier is moving — TabPFN v2, TabuLa, and successors are real — so re-benchmark against the current state of the art once a year rather than treating this as settled. (Related: when to migrate legacy ML pipelines to modern architectures.)

Why Transformers Still Lose on Tabular Data

Transformers were built for sequences and images, where position and locality carry meaning. Tabular data has no such structure. Column order is arbitrary — shuffle the features and a transformer's input changes while the underlying problem does not. That mismatch forces the model to learn structure that a well-designed feature gives it for free.

There is a second, subtler mismatch. Neural networks are approximately invariant to rotation of the input space, and gradient-boosted trees are not. On tabular problems the axes matter: "age" and "income" are not interchangeable directions, and a decision boundary aligned to a single feature is common. Trees exploit axis-aligned structure directly; a transformer has to spend capacity learning that the axes are special.

What the benchmark record actually shows

The honest summary: gradient boosting remains highly competitive on tabular benchmarks, and the transformer wins are narrower than the headlines suggest.

The most-cited evidence is Grinsztajn et al. (2022), "Why do tree-based models still outperform deep learning on typical tabular data?" (NeurIPS Datasets & Benchmarks). Their finding is specific: on medium-sized datasets (roughly 10k rows), tree-based models outperformed neural networks, largely because trees handle uninformative features gracefully and because neural nets' rotation invariance is a liability when feature axes carry meaning. That is not a claim that trees win at every scale — it is a claim about the regime most enterprise tabular data lives in.

Later work on tabular foundation models (TabPFN-style approaches) shows genuine gains — but typically on small datasets (hundreds to low thousands of rows) and specific regimes, not universally. Anyone claiming transformers categorically win on tabular data is overreading a contested literature. (See our breakdown of the tabular foundation-model benchmarks.)

Where transformers do win (and why)

Transformers earn their keep in specific conditions:

Large-scale data — hundreds of thousands to millions of rows, where the model has enough signal to learn interactions itself.
High-cardinality categoricals with rich semantics — text-like IDs, product descriptions, or embeddings where pretrained representations help.
Multimodal tables — rows that mix numbers with free text or images.
Pretrained tabular foundation models that already fit your domain and let you skip training from scratch — especially on small datasets where TabPFN-class models shine.
Interaction-heavy problems where you cannot enumerate the crossings by hand and have the data volume to let attention find them.

Outside those zones, the economics rarely favor the transformer.

"The question is not whether transformers can model tabular data. It is whether they beat a tuned gradient-boosted model with good features — after you account for training cost, latency, and the engineer-hours to maintain them."

The Feature Engineering That Still Beats a Transformer

This is where the leverage lives. The features below encode domain knowledge a transformer must otherwise infer from raw columns — and gradient boosting exploits them with remarkable efficiency. (Deep dive: our feature engineering checklist for production models.)

Interaction and crossing features

Domain-driven combinations — price_per_sqft, days_since_last_login × plan_tier, ratio and difference features — give tree models the splits they need in one step. A transformer can learn interactions in principle, but it needs data volume and compute to do so. Explicit crossings are cheap signal; let the model spend its capacity elsewhere.

Target and frequency encoding done right

High-cardinality categoricals (customer IDs, ZIP codes, SKUs) break one-hot encoding. Frequency encoding captures rarity directly and is safe. Target encoding is powerful but dangerous — it leaks the target. Two fixes are required, not one:

Out-of-fold encoding. Compute the encoding on training folds only, then apply to the held-out fold. Critically, the encoding must be recomputed inside the cross-validation loop — computing it once before splitting still leaks.
Smoothing. Raw category means overfit rare categories. Use an empirical-Bayes / m-estimate blend: encoded = (count × category_mean + m × global_mean) / (count + m), with m tuned (typically 10–100). Categories with few observations shrink toward the global mean instead of memorizing noise.

Together these keep the signal while preventing the model from memorizing the label through the encoding itself. (More: how to encode high-cardinality categoricals safely.)

Time-aware features for tabular ML

Rolling means, recency-weighted counts, lag features, and time-since-event columns capture temporal dynamics that a transformer will not extract from raw timestamp columns. A 30-day rolling average of a customer's spend is signal; a raw epoch integer is mostly noise to a tree. Raw timestamps do carry information, but trees split poorly on them — decompose the timestamp into the features that matter: day-of-week and month as cyclical encodings (sine/cosine), holiday-proximity flags, and business-day offsets. These aggregates and decompositions are where tabular feature engineering most reliably outperforms generic architectures.

Handling leakage as a first-class concern

Leakage is the silent killer of tabular models, and it is the single most common reason a "great" offline score collapses in production. Common sources:

Target-derived features computed across the full dataset before splitting.
Time-travel — using future data to predict the past.
Train/test contamination from shared preprocessing steps (scalers, imputers, encoders fit on all data).
Group leakage — the same customer or entity appearing in both train and test.

Prevention is structural: wrap all feature logic in a pipeline, enforce out-of-fold encoding inside the CV loop, use GroupKFold when entities repeat, and always use temporal splits for time-series data. Treat leakage checks as a gate, not an afterthought. (See our leakage audit template.)

Gradient boosting + engineered features: the baseline that won't die

XGBoost, LightGBM, and CatBoost remain the models to beat. They handle mixed types natively, tolerate uninformative features, train in minutes, and serve at low latency. CatBoost in particular handles categoricals well out of the box, with ordered target statistics that reduce leakage without manual out-of-fold work.

The tuning levers that matter most, in order of impact:

Depth and min_child_weight / min_data_in_leaf — control overfitting on the small-to-medium datasets where trees already win.
Learning rate and number of rounds, tuned together with early stopping on a proper validation split.
Regularization (lambda, alpha, subsample, colsample) — cheap insurance against noisy features.
Categorical handling — native categorical support (CatBoost, LightGBM) usually beats manual encoding for medium cardinality.

Resist the urge to tune hyperparameters before you have good features. On tabular problems, feature quality dominates hyperparameter quality by a wide margin.

A Workflow That Keeps You Off the Wrong Model

The thesis promised a workflow. Here it is — a decision sequence that costs hours, not weeks, and prevents the two expensive mistakes: shipping a transformer you didn't need, and dismissing one you did.

Establish the honest baseline first. Gradient boosting with sensible defaults and your engineered features, evaluated on a correctly constructed validation split (temporal for time-series, grouped for repeated entities). This is the number every other model must beat.
Diagnose the gap. If the baseline is close to the business target, stop. If it is far off, ask why: is it underfitting (need more capacity or better features), overfitting (need regularization or more data), or a data problem (leakage, label noise, missing signal)? A transformer will not fix a data problem.
Check whether you are in a transformer-favorable regime. Large data, high-cardinality semantic categoricals, multimodal rows, or a pretrained foundation model that fits your domain. If none apply, the transformer is unlikely to pay for itself.
If you do try a transformer, benchmark on total cost of ownership, not just AUC. Include training time, serving latency, GPU cost, and the engineer-hours to debug and maintain it. A 0.3-point AUC gain that doubles inference cost and makes failures opaque is a loss.
Re-benchmark annually. The tabular foundation-model frontier moves fast. What lost in 2025 may win in 2026 — but only if you check.

The point of the workflow is not to avoid transformers forever. It is to make the decision on evidence and economics rather than on the pitch.

Bottom Line

For the tabular problems most enterprises actually have — medium-sized datasets, mixed types, high-cardinality categoricals, temporal dynamics — engineered features plus gradient boosting remain the model to beat in 2026. The leverage is in the features: explicit interactions, correctly smoothed and out-of-fold target encodings, decomposed time features, and leakage controls treated as a gate. Reach for a transformer when the data volume, semantics, or multimodality justify it — and measure the decision on total cost, not just accuracy.