Tabular Foundation Models Redefine AI Landscape, Outperforming Traditional GBDTs in Zero-Shot Predictions

A seismic shift is underway in the realm of machine learning for tabular data, long considered the bedrock of enterprise analytics and a stronghold for gradient-boosted decision trees (GBDTs). A new class of models, referred to as "tabular LLMs" or tabular foundation models, is rapidly emerging, demonstrating superior performance on community benchmarks in zero-shot prediction tasks, effectively inverting the decade-long dominance of highly tuned GBDT configurations. Independent audits confirm that these novel architectures are not only more accurate but also significantly more efficient in their "training" phase, which fundamentally transforms into an inference process.

The pivotal revelation comes from the TabArena leaderboard, a dynamic benchmark for tabular machine learning models. Every single-model entry currently positioned above the best-tuned and ensembled GBDT configuration is now a tabular foundation model. The only other contenders surpassing traditional trees are complex, multi-hour ensemble pipelines like AutoGluon. This marks a profound departure from the established paradigm where meticulously optimized GBDTs—such as XGBoost, LightGBM, and CatBoost—were the de facto solution for most structured data problems.

The Rise of a New Paradigm: Zero-Shot Tabular AI

For years, gradient-boosted decision trees have been the undisputed champions of tabular data, celebrated for their robust performance, interpretability, and ability to handle diverse feature types. Data scientists routinely engaged in extensive hyperparameter tuning and ensemble methods to squeeze every last drop of performance from these models. However, this process often demanded significant computational resources and expert knowledge, particularly for optimal configuration.

The advent of foundation models, initially popularized by large language models (LLMs) in natural language processing, has now extended its influence to tabular data. These "tabular LLMs," while not modeling language, adopt the transformer architecture and the principle of in-context learning. Instead of requiring extensive gradient-based training on a new dataset, they predict missing values or labels in a spreadsheet zero-shot. Users simply provide a few labeled examples as context, similar to how one might prompt a text LLM with worked examples, and the model generates predictions for unseen rows in a single forward pass. This paradigm shift effectively transforms the "training" phase into an inference operation, drastically reducing deployment time and complexity.

Verifying the Breakthrough: An Independent Audit

Given the historical tendency for leaderboards to sometimes present an incomplete picture, an independent audit was undertaken to rigorously verify these claims. The author recomputed TabArena’s scoring from its published artifacts and then re-ran TabICLv2, identified as the strongest model with unrestricted open weights, from scratch on controlled hardware. This audit focused on the benchmark’s official Lite protocol, covering 51 datasets with one train/test split per dataset.

The results were compelling. The independent run of TabICLv2 achieved an Elo rating of 1559, remarkably close to the official 1575, falling well within the expected bootstrap intervals of ±60–86. Per-task metric values were identical to four decimal places for 16 of 51 datasets, with a median relative difference of 0.08% and a maximum difference of 3.5%—a variation typically attributed to GPU non-determinism rather than methodological discrepancies. The entire verification sweep, including environment setup, completed in a mere 2.1-hour GPU session, costing just over $2. This robust validation confirms the integrity of the TabArena leaderboard and the performance claims of these new tabular foundation models.

Anatomy of a Tabular Foundation Model: TabICLv2 in Detail

TabICLv2, developed by Inria’s SODA team, exemplifies the architecture and operational principles of these new models. It is a single, pretrained transformer designed to predict on any table zero-shot, eliminating the need for hyperparameter search or gradient training on new data. Its operational workflow mirrors a learned k-nearest-neighbors approach, where the model attends over provided labeled rows to infer labels for unseen data, but crucially, the "similarity" and combination logic are learned during pretraining, not fixed.

The architecture of TabICLv2 comprises three stacked transformers, each addressing a distinct aspect of tabular data processing:

1. First Transformer (Column-wise Embedding): This stage processes each column independently. It acts as a set transformer, learning the distribution of values within each feature and converting raw cell values into context-aware embeddings. For instance, the number "450" in a ‘price’ column would be embedded differently from "450" in a ‘postcode’ column, reflecting their distinct semantic roles.
2. Second Transformer (Row-wise Aggregation): After column-wise embedding, each row consists of a "bag" of feature embeddings. This transformer uses learned query tokens (similar to [CLS] tokens in NLP) to attend over these embeddings, collapsing the entire row into a single, fixed-length vector. This vector encapsulates the essential information of the entire example, regardless of the original number of columns.
3. Third Transformer (In-Context Learning and Prediction): This is where the zero-shot magic happens. Unlabeled test-row vectors attend to all labeled train-row vectors, identifying the most similar training examples and reading off their labels. The attention is unidirectional (test to train), ensuring that test data does not influence the understanding of training labels. This mechanism effectively performs nearest-neighbors reasoning, but with learned similarity and combination rules.

TabICLv2 outputs predictions via two specialized heads: a hierarchical classifier for diverse classification tasks and a regression head that emits 999 quantiles. This quantile output provides a full predictive distribution, offering not just a point estimate but also a reliable confidence interval, a significant advantage over single-point predictions. These quantiles are trained using pinball loss, which asymmetrically penalizes overshoots and undershoots, naturally nudging each output towards its target percentile.

Notably, TabICLv2 is compact, with approximately 28 million parameters—a fraction of the billions found in typical text LLMs. Its pretraining relies purely on synthetic data generated from random structural causal models. This crucial design choice guarantees that no real-world benchmark dataset could have "leaked" into its pretraining, ensuring pristine data provenance and uncontaminated performance evaluation. The model’s open weights (BSD-3 license) and scikit-learn compatible interface further underscore its reproducibility and accessibility.

The TabArena Leaderboard: A New Hierarchy Emerges

The TabArena benchmark, featuring 51 datasets varying in size (748 to 150,000 rows) and complexity (5 to 1,777 features), employs an Elo rating system to compare models, anchoring a default RandomForest at 1000. A 400-point Elo difference signifies a 10:1 win odds. The latest snapshot (2026-07-15) unequivocally showcases the dominance of tabular foundation models.

TabICLv2, with its 158 Elo lead over the best tuned-and-ensembled LightGBM (1432 Elo), stands out as the strongest open-source, zero-shot solution. Other foundation models, such as the gated TabPFN line, perform even higher. This significant margin was unexpected, particularly considering the extensive tuning budgets afforded to the GBDTs on the board.

Beyond accuracy, serving cost and efficiency present another compelling argument. TabICLv2’s median prediction time is 0.38 seconds per 1,000 rows, faster than tuned-and-ensembled LightGBM (2.64 s/1K rows). Its "fit" time (context storage, no gradient training) is roughly 100 times shorter than GBDT protocols. While foundation models typically leverage GPUs and GBDTs run on CPUs, the overall cost-efficiency is striking; the entire 51-dataset TabICLv2 sweep on an on-demand A10G GPU cost a mere $2.

At the pinnacle of the leaderboard sits Google Research’s TabFM, boasting a remarkable 98 Elo lead over the next entry (an AutoGluon pipeline) and 123 Elo over the next single model. TabFM, also pretrained on synthetic data, appears to avoid benchmark contamination. However, its lack of a public technical paper, published parameter count, and a non-commercial license raise concerns about transparency and full auditability, preventing independent verification of its underlying mechanisms.

The "Tabular LLM" Misnomer: Text-Based vs. Table-Native Approaches

The phrase "tabular LLM" can be misleading, as it sometimes refers to a distinct line of research where general text-based LLMs are prompted with tables serialized into text format. To clarify this distinction, a direct comparison was made using Claude Opus 4.8. The model was given datasets as text, with 64 labeled example rows and batches of test rows for prediction, requesting structured probabilistic outputs.

The results were decisive: on 49 scorable datasets, the text LLM beat TabICLv2 on only 6, and the best tuned-and-ensembled GBDT on the same 6. Its median error was 57% higher than TabICLv2’s. The text LLM performed marginally better on datasets where column names carried significant real-world semantic meaning (e.g., churn, credit, blood-donation), which its language prior could exploit. Conversely, it performed poorly on purely pattern-based data like app-permission vectors or sensor data. This empirically confirms that table-native foundation models, specifically designed to understand tabular structures, vastly outperform generic text LLMs applied to serialized tabular data, offering superior accuracy and orders of magnitude lower inference costs (approximately $1.50 per thousand predictions for the prompted LLM versus a fraction of a GPU-second for TabICLv2).

The Enduring Strengths of Traditional GBDTs

Despite the clear lead of foundation models, traditional GBDTs are not entirely obsolete. The TabArena analysis revealed specific "regimes" where trees still hold an advantage. While TabICLv2 wins on 40 of 51 datasets against the best-tuned GBDTs, the 11 losses cluster in two critical areas:

1. High Dimensionality: For tables with over 100 features, GBDTs consistently outperform foundation models. Of the six datasets exceeding this feature count (e.g., Bioresponse with 1,776 features, hiva_agnostic with 1,617), GBDTs won five. Below 100 features, TabICLv2’s win rate soared to 86-88%. This suggests that current tabular foundation models may struggle with the extreme sparsity and complex feature interactions inherent in very wide datasets.
2. Scale with High-Cardinality Categoricals: GBDTs retain an edge on very large datasets (above 20,000 rows) combined with numerous high-cardinality categorical features. The largest single tree win (Amazon_employee_access) is a classic example of this data shape. TabICLv2’s win rate dropped from 89% for tables under 3,000 rows to 64% for those above 20,000.

These findings suggest that while foundation models offer a general-purpose solution for a broad range of tabular tasks, tuned GBDTs remain the preferred choice for specific, challenging data characteristics.

The Critical Question of Data Provenance: Contamination Concerns

The integrity of benchmark results hinges on the provenance of pretraining data. In the context of foundation models, particularly those leveraging real-world data, the risk of "benchmark contamination" – where benchmark datasets inadvertently leak into the pretraining corpus – is a serious concern.

TabICLv2 and the original TabPFN models, being pretrained exclusively on synthetic data, are entirely immune to this issue. Their performance is guaranteed to be based on learned general principles, not memorized benchmark specifics. However, newer models are increasingly incorporating real-world data to boost performance. RealTabPFN-2.5, for instance, continues pretraining on "a curated corpus of 43 real-world tabular datasets," employing extensive deduplication pipelines. TabDPT goes further, pretraining on 123 OpenML datasets, a source from which many TabArena datasets originate.

While rigorous deduplication efforts are described, the practical challenge of independently verifying such claims remains. The incentive gradient points towards using more real data for higher Elo scores, placing the burden of proof for "clean" data acquisition and processing on the model developers. This creates an epistemic challenge, similar to self-declared leakage flags in other benchmarks, where trust becomes a factor. For organizations demanding the highest level of evidentiary cleanliness, synthetic-only pretrained models like TabICLv2 offer an unassailable position. For those seeking marginal gains, a more pragmatic approach might involve accepting the inherent trust requirement or conducting rigorous held-out evaluations.

Hybrid Approaches: The Promise of Model Routing

The complementarity between different model types suggests potential benefits from hybrid approaches. TabArena’s availability of per-split validation errors allowed for an exploration of "oracle" and "deployable" model routers. An oracle router, which hypothetically selects the best model per dataset using test error (a "cheating" ceiling), demonstrated significant headroom. An oracle picking between TabICLv2 and tuned LightGBM achieved an Elo of 1674, a +103 gain over TabICLv2 alone, despite LightGBM being 160 Elo lower overall. An FM+FM oracle reached an even higher 1707 Elo.

More practically, a deployable router, which selects models based on validation error (information available at deployment), proved effective between foundation models. Picking between TabICLv2 and TabPFN-2.6 by validation error yielded an Elo of 1645, surpassing both individual models and recovering roughly half of the oracle’s potential. This confirms that intelligent model selection can enhance performance using real-world data.

However, the same validation-based router backfired when applied across the FM/GBDT divide, resulting in an Elo of 1534—37 Elo below simply always using TabICLv2. This suggests that validation errors from different model families (e.g., GBDTs with bagged internal validation versus FMs) may not be directly comparable, leading to suboptimal routing decisions. Effectively leveraging the complementary strengths of FMs and GBDTs, particularly for wide tables, will require more sophisticated selection signals than raw validation error.

The Future of Tabular AI: A New Era of Efficiency and Performance

The emergence of tabular foundation models marks a pivotal moment in the evolution of AI for structured data. The verified performance of open-source models like TabICLv2, pretrained solely on synthetic data, demonstrates a compelling combination of accuracy, efficiency, and transparency. For many tabular tasks involving moderate row counts and up to approximately 100 features, TabICLv2 zero-shot presents the strongest readily available solution.

While traditional GBDTs will retain their niche in specific challenging regimes—particularly for extremely wide tables or those with complex, high-cardinality categorical features—the overall trend points towards a future dominated by foundation models. The shift from resource-intensive gradient-based training to rapid, in-context inference dramatically accelerates the deployment lifecycle of tabular AI solutions.

Future research will likely focus on further optimizing these models for challenging data types, developing more robust routing mechanisms for hybrid model systems, and rigorously addressing the data provenance concerns associated with using real-world pretraining data. The landscape of tabular AI is no longer static; it is rapidly transforming, driven by the innovation and efficiency of foundation models.