Realistic Evaluation of TabPFN v2 in Open Environments (2025)

2.1 Open Environments Challenges

Most tabular machine learning models are typically trained and tested in closed environments where critical learning factors remain stable. However, various real-world tasks operate in open environments where dynamic changes occur in key factors, posing challenges to model generalization[42].Zhou [61] categorizes four core challenges in open environments: Emerging New Classes, Decremental/Incremental Features, Changing Data Distributions, and Varied Learning Objectives.

These challenges are pivotal in open environments machine learning. Emerging new classes, involving unseen classes during testing, have been addressed in natural language processing[13] and computer vision[14, 15]. Decremental/Incremental features, caused by changes in feature sets, lead to mismatched training-testing spaces. TabFSBench[10] evaluates model performance under such variations, andHou etal. [31] enhances performance by restoring ephemeral features. Changing data distributions, where test data violate the i.i.d.assumption, have led to benchmark datasets such as Tableshift[18], and methods such as domain adaptation[60] and domain generalization[59]. Varied learning objectives, which prioritize adaptive optimization beyond accuracy, include multi-objective learning[62, 64] and self-evolving training[37]. However, research on these challenges remains fragmented, lacking a unified framework to evaluate models on all four challenges.

2.2 Tabular Data in Machine Learning

Tabular data, with structured and heterogeneous features, is used in healthcare, finance, and recommendation systems[5, 32, 48]. Unlike images and texts, it has high dimensionality, heterogeneity, and complex dependencies, posing challenges for machine learning models[16]. Current approaches are mainly tree-based models (e.g., XGBoost[11], LightGBM[4], CatBoost[44]) and deep learning models. Tree-based models handle irregular patterns and uninformative features well[20], while deep learning models like DCN V2[53], FT-Transformer[19], and NODE[43] aim to capture complex feature interactions for better performance[19, 43].

In the realm of tabular machine learning tasks, tree-based models have traditionally held a dominant position over deep learning models[20, 39]. The emergence of the novel deep learning model TabPFN v2[26] has surpassed tree-based models. TabPFN v2 has demonstrated superior performance compared to tree-based models across multiple benchmarks. However, the majority of these benchmarks are confined to closed environments. Consequently, the comparative performance of TabPFN v2 and tree-based models in open environments remains underexplored.

2.3 Research on TabPFN

TabPFN[25], short for Tabular Prior-Fitted Network, is a model pre-trained on large-scale synthetic datasets, enabling efficient zero-shot learning. It can efficiently perform classification and regression tasks without the need for hyperparameter tuning. Compared to existing models, TabPFN shows significant advantages on small- to medium-scale datasets with low computational cost, making it an efficient solution for tabular tasks. Recent research, however, reveals limitations in TabPFN’s performance on high-dimensional, large-scale, or multi-class tasks[57, 35].

Various optimization strategies have been proposed to enhance TabPFN’s adaptability to current limitations and more complex scenarios, including local context construction via retrieval-based methods[33], model fine-tuning[50, 55], and pretraining dataset expansion[6]. Moreover, TabPFN’s strong performance has prompted its application to challenges such as distribution shift adaptation[22], time series forecasting[29], and various domains including healthcare[41, 51], ecology[21], and cybersecurity[45]. However, these studies primarily focus on closed environments or target only a single challenge in open settings, lacking a comprehensive evaluation of TabPFN under diverse open environments scenarios.

3.1 TabPFN

Developed byHollmann etal. [25], TabPFN reimagines classification through an innovative adaptation of a Transformer-based architecture. At its core, the method reformulates the classification task as a sequence processing problem with the following key components.

TabPFN standardizes each data point $(x_{i},y_{i})$ to $(\tilde{x}_{i},\tilde{y}_{i})$ in the $k$-dimensional space via linear projections with zero-padding ensuring uniform dimensionality. A context matrix $\mathcal{A}$ is constructed by concatenating $N$ training samples and a test sample $x^{*}$, where $\mathcal{A}=\left[\tilde{x}_{i}\oplus\tilde{y}_{i}\right]_{i=1}^{N}\parallel$ and $\oplus$ denotes vector concatenation. This formulation treats each data point as a token in a sequence, enabling flexible handling of variable dataset sizes. The context matrix is then processed through Transformer layers and an MLP head, which converts the test sample’s output token into class probabilities.

3.2 TabPFN v2

Building upon TabPFN, TabPFN v2[26] introduces architectural innovations that redefine feature processing in tabular data analysis. The proposed method encompasses a feature space transformation where each raw feature is projected into a $k$-dimensional latent space and subjected to controlled perturbation, creating unique positional identifiers[57, 19]. The computational framework processes a three-dimensional tensor structure using dual attention mechanisms: cross-sample attention for dataset-level patterns and intra-feature attention for feature relationships. Pre-trained weights derived from synthetic data generated by structural causal models which facilitate zero-shot transfer, thereby addressing the challenges of tabular data diversity.

Current research[26, 57] has extensively evaluated TabPFN v2’s performance in closed environments, but largely overlooked its adaptability to open environments, leaving a critical gap. To fully realize its potential and practical value, we conduct comprehensive evaluations of TabPFN v2 under various open environments challenges.

4.1 Emerging New Classes

In closed environments machine learning tasks, it is commonly assumed that the class of any test sample must belong to the class set seen during training. However, this assumption does not always hold in open environments. We formally define this challenge by partitioning the class set $L$ into $L^{\text{train}}$ and $L^{\text{test}}$, corresponding to the training and testing phases, respectively. In closed environments, the class set remains consistent between the training and testing phases, i.e., $L^{\text{train}}=L^{\text{test}}$. In contrast, in open environments, test samples may belong to novel classes $l$ that are not present during training, i.e., $\exists~{}l\in L^{\text{test}}$ such that $l\notin L^{\text{train}}$. In such cases, the model must be capable of identifying and handling these new classes.

4.2 Decremental/Incremental Features

Decremental and incremental features represent open environments challenges characterized by partial removal or augmentation of the input feature set, known as feature shift. Let $C$ denote the full feature set, partitioned into $C^{\text{train}}$ and $C^{\text{test}}$ for training and testing, respectively. In closed environments, $C^{\text{train}}=C^{\text{test}}$, whereas in open environments, $C^{\text{train}}$ remains fixed but $C^{\text{test}}$ may differ. Specifically, when $C^{\text{test}}\subsetneqq C^{\text{train}}$, imputation of shifted features in $C^{\text{test}}$ is necessary to maintain input dimension consistency and enable accurate model prediction (Decremental Features). Conversely, when $C^{\text{train}}\subsetneqq C^{\text{test}}$, the model typically truncates the newly added features in $C^{\text{test}}$, retaining only those corresponding to $C^{\text{train}}$, thus ensuring input dimension consistency between the training and testing phases (Incremental Features).

4.3 Changing Data Distributions

Closed environments machine learning research generally assumes that all data in both the training and testing phases are independent samples from the identical distribution. Unfortunately, this assertion does not always hold true in open environments. Changing data distributions has two scenarios. Covariate Shift[49] occurs when the input distribution $p(x)$ changes between training and testing phases, while the conditional probability $p(y|x)$ remains constant. Concept Shift[17] involves changes in the conditional probability $p(y|x)$ with a stable input distribution $p(x)$.

4.4 Varied Learning Objectives

The performance of the machine learning model $f$ can be measured by a learning objective $M_{f}$, such as accuracy, F1 score, or ROC-AUC. Learning towards different objectives may lead to a model with different strengths. A model that is optimal on one measure is not necessarily optimal on others. Machine learning research in closed environments generally assumes that the $M_{f}$ used to evaluate model performance is fixed and known in advance. However, this assumption may not always hold in open environments. When facing this challenge, the model should perform well across various learning objectives without requiring data to be recollected and a completely new model to be trained.

4.5 Evaluation Framework for open environments Challenges

Existing benchmarks for evaluating model performance in open environments typically focus on a single task, such as distribution shift[18] or feature shift[10]. However, they lack a unified and comprehensive assessment across multiple open environments challenges. Hence, we propose a modular and extensible evaluation framework that assesses both model performance and robustness across diverse real-world scenarios. The framework formalizes four representative open environments challenges: Emerging New Classes, Decremental/Incremental Features, Changing Data Distributions, and Varied Learning Objectives. It builds testing protocols by leveraging existing benchmarks, including WhyShift[34] and TableShift[18] for distribution shifts, and TabFSBench[10] for feature shifts. It supports comprehensive evaluation of tabular models and enables exporting datasets under different open environments scenarios with just a few lines of Python code. Details are in AppendixC.

5.1 Emerging New Classes

Current tabular models (e.g., TabPFN v2) are fundamentally constrained by fixed input-output dimensions, limiting new class incorporation. To evaluate their adaptability, we design a novel class detection task from SMOOD[23]. Based on multi-class datasets (AppendixE.1), we implement a leave-one-class-out protocol: for $k$-class problems, we perform $k$ runs, each excluding one class during training and treating it as novel. We evaluate models on the metrics of Area Under the Precision-Recall curve (AUPR) and ROC-AUC. Results represent averages across all runs. The detailed computation procedure is described in SectionE.2.

TabPFN v2 has the potential to detect new classes.As illustrated in Table1, TabPFN v2 consistently achieves better AUPR and ROC-AUC in the task of new class detection across all four datasets when compared to other models. This empirical evidence indicates that TabPFN v2 possesses a robust capability for identifying new classes. Results from other metrics of predicted probability are in AppendixE.

5.2 Decremental/Incremental Features

To conduct a comprehensive evaluation of decremental/incremental features, we adopt TabFSBench[10], a benchmark specifically designed for this challenge. It includes twelve datasets covering eight classification tasks and four regression tasks across various domains, dataset sizes, and feature types. Descriptions and results are provided in AppendixF.

TabPFN v2 exhibits heightened vulnerability to decremental features.To assess TabPFN v2’s adaptability to decremental features, we conduct random-shift experiments in TabFSBench and use the performance gap as a metric. The performance gap, explained in AppendixF, measures the impact of feature shifts by comparing model performance metrics between original and shifted features. As shown in Tables2, TabPFN v2’s performance gap widens significantly with increasing feature shifts, indicating weaker adaptability and higher sensitivity to feature space changes. In contrast, MLP and CatBoost show greater robustness against decremental features, possibly due to their inherent anti-shift properties, which TabPFN v2 may lack.

TabPFN v2 can not address new added features in the testing phase.When the dimensionality of input features dynamically increases, TabPFN v2 cannot process the additional features and can only truncate them to retain those present during training. This is because its internal parameters and feature representations are based on a fixed feature dimensionality during training. Consequently, TabPFN v2 cannot leverage the information from new features during testing. However, this limitation won’t degrade TabPFN v2’s performance.

5.3 Changing Data Distributions

We evaluate TabPFN v2 under scenarios of changing data distributions, using metrics including Accuracy, Balanced Accuracy, F1-score, and ROC-AUC. Detailed results are given in AppendixG. The evaluation is conducted on nine fully numerical datasets drawn from the WhyShift[34] and TableShift[18] benchmarks, which contain three types of data distribution scenarios. To accommodate memory constraints and the current limitations of TabPFN v2 in handling very large datasets, we apply stratified subsampling (up to 50,000 instances) while preserving the original train/test splits. Detailed dataset statistics are provided in AppendixG.1.

TabPFN v2 reveals limited robustness when concepts shift.We present a comparative analysis of accuracy between XGBoost and TabPFN v2 under two distinct data distribution shifts: Concept Shift and Covariate Shift. XGBoost is the best model in the changing data distributions task. Figure3 shows that both models exhibit enhanced accuracy when shifting from Concept to Covariate Shift. Meanwhile, XGBoost maintains consistent superiority over TabPFN v2 across both shift types. However, in Covariate Shift scenarios, TabPFN v2 demonstrates more improved performance compared to XGBoost, reducing the performance difference. Results on the other three metrics are given in AppendixG.3. These results suggest that TabPFN v2 demonstrates promising discriminative capacity on covariate-shift datasets instead of concept-shift datasets.

5.4 Varied Learning Objectives

We conduct an exhaustive comparative analysis across four primary classification learning objectives: Accuracy, ROC-AUC, F1-score, and Balanced Accuracy. The analysis is performed on i.i.d. datasets used in changing data distribution.

TabPFN v2 has statistically significant bias toward majority classes.Figure3 reveals that the performance of TabPFN v2 degrades significantly on class-imbalance-sensitive metrics (F1-score and Balanced Accuracy), suggesting inherent limitations in handling minority classes. Specifically, Balanced Accuracy, a metric designed to address class imbalance by computing the arithmetic mean of per-class accuracies, shows that TabPFN v2 struggles to effectively adapt to varying sample sizes across different classes. Similarly, F1-score, as the harmonic mean of precision and recall, further confirms the model’s suboptimal predictive capability for minority classes. Hence, TabPFN v2 is suitable to handle datasets with balanced classes.

TabPFN v2 fails to maintain competitive performance across various learning objectives.As illustrated in Figure3, TabPFN v2 demonstrates competitive performance with respect to accuracy and ROC-AUC, achieving comparable results to other models in these specific evaluation criteria. However, a comparative analysis reveals statistically significant performance deficiencies in both F1 Score and Balanced Accuracy when contrasted with tree-based models and RealMLP. These empirical observations highlight an important limitation of TabPFN v2: while exhibiting superior performance for particular learning objectives, the model fails to maintain consistent efficacy across all evaluated performance metrics.

5.5 Holistic Assessment

We conduct a comprehensive assessment to evaluate the robustness of TabPFN v2 relative to compared models in open environments, employing a performance ranking analysis across four above challenges in open environments.

TabPFN v2’s robustness is inherently data-dependent.Through a thorough and comprehensive evaluation of the performance of TabPFN v2 across four distinct open environments challenges, our analysis has revealed that TabPFN v2 consistently demonstrates superior efficacy, particularly when applied to small-scale datasets. This observed performance characteristic is in precise alignment with the fundamental design objective of TabPFN v2, which is specifically optimized for small-scale datasets. The empirical results obtained substantiate the theoretical premise underlying TabPFN v2’s development, thereby confirming the particular suitability of TabPFN v2 for applications where the volume of training data is inherently limited.

Tree-based models remain the optimal approach for general tabular tasks in open environments.As shown in Table3, tree-based models, particularly CatBoost and RandomForest, consistently outperform TabPFN v2. CatBoost achieves the best overall ranking, excelling in both changing data distributions and varied learning objectives, demonstrating stronger adaptability in open environments. In contrast, while TabPFN v2 remains competitive in closed environments, its performance declines relative to tree-based methods in open environments. These results suggest that tree-based models are better suited for open environments tasks requiring robustness.

5.6 Recommendations

During the experimental investigation, we observe that the majority of existing high-performance models predominantly demonstrate their superior performance in closed environments. However, these models tend to fall short in adapting to the open environments challenges that are more frequently encountered in real-world scenarios. To further enhance the performance of models in open environments and to provide guidance for the development of subsequent research, the following recommendations are proposed:

Develop benchmarks targeting unexplored open environments tabular challenges. Existing benchmarks are primarily designed around distribution shifts and feature shifts, lacking variations in open environments tabular challenges such as new classes and changes in learning objectives. Considering that the construction of benchmarks can effectively improve the model performance evaluation and methodological improvements on corresponding tasks. Therefore, it is urgent to develop benchmarks based on various open environments tabular challenges.

Evaluate models on various open environments metrics. Current research typically relies on OOD Accuracy, Performance Gap, or Balanced Accuracy to assess the robustness of a model. However, these metrics are mostly applicable to tasks involving distribution shifts or feature shifts and do not cover diverse open environments challenges. Therefore, additional general open environments metrics should be introduced in model evaluation, such as Open-World Tracking Accuracy[38] and Mean Average Precision[47].

Take model robustness as a critical metric when comparing model quality. Current research often judges the quality of a model solely based on its performance in closed environments, without considering the robustness of the model in open environments as an important evaluation criterion. While robustness is a crucial indicator for determining whether a model has practical value. Therefore, robustness should be regarded as a critical metric when comparing model quality, and the quality of a model should be comprehensively assessed based on both closed environments performance and open environments robustness.

Design universal modules to enhance the robustness of diverse existing models. From the aforementioned experiments, we learn that although some models perform well in certain open environments challenges, these models rely on their specific modules and lack universality. They cannot be transferred to other models to further improve robustness. Moreover, these models do not achieve excellent performance in all open environments challenges. Therefore, future research should focus on designing highly universal and transferable modules to enhance the overall performance of models in handling open environments tasks.

A.1 TabPFN

Developed by[25], TabPFN reimagines classification through an innovative adaptation of a Transformer-based architecture. At its core, the method reformulates the classification task as a sequence processing problem with the following key components:

A.2 TabPFN v2

Building upon TabPFN, TabPFN v2[26] introduces three key architectural innovations that redefine feature processing in tabular data analysis:

B.1 Emerging New Classes

In closed environments, it is commonly assumed that the label of any testing sample must come from the label set used during training. However, this assumption is not always valid in open environments. For instance, in a forest disease monitoring system that relies on a machine learning model trained with signals from sensors deployed in the forest, it is impractical to enumerate all possible classes in advance, as some forest diseases may be entirely novel, such as those caused by invasive insect pests that have never been encountered in the region before.

B.2 Decremental/Incremental Features

Decremental/Incremental Features are another open environments challenge, wherein the feature set previously utilized as inputs is either partially removed or expanded by new features, also known as feature shift. Given a forest disease monitoring system that relies on a machine learning model trained with signals from sensors deployed in the forest, certain existing sensors may cease to function, leading to a reduction of the feature set (Decremental Features). Meanwhile, additional sensors may be deployed to monitor, resulting in an expansion of the feature set (Incremental Features).

B.3 Changing Data Distributions

Machine learning research in closed environments generally assumes that all data in both the training and testing phases are independent samples from the identical distribution. Unfortunately, this assertion does not always hold true in open environments. In the forest disease monitoring system, the model may be built in summer based on sensor signals specific to that season, but it is expected to perform well across all seasons.

B.4 Varied Learning Objectives

The performance of the machine learning model $f$ can be measured by a learning objective $M_{f}$, such as accuracy, F1 score, or ROC-AUC. Learning towards different objectives may lead to a model with different strengths. Being optimal on one measure does not mean that the model will also be optimal on other measures. Machine learning research in closed environments generally assumes that the $M_{f}$ used to measure model performance is invariant and known in advance. However, this assertion may not invariably be valid in open environments. In the forest disease monitoring system, the sensor dispatch task may prioritize different objectives over time. Initially, various sensors are dispatched to pursue high monitoring accuracy; later, after a relatively high accuracy has been achieved, different sensors may be used to ensure that the system operates with minimal energy consumption. When facing this challenge, the model should be able to perform well on various learning objectives without requiring the data to be recollected and a completely new model to be trained.

D.1 Traning settings

Deep learning models are trained on an NVIDIA 4090 GPU. Tree-based models are trained on an AMD Ryzen 5 7500F 6-Core Processor. All experimental results are reported as the average of three different random seeds to ensure statistical reliability.

D.2 Models

In this subsection, we provide detailed descriptions of all the models used in our paper.

D.3 Hyperparameter Tuning

In this subsection, we provide hyperparameter grids of tree-based and deep learning models in Table4,5.

For tree-based models, we employ GridSearchCV from the scikit-learn library to conduct an exhaustive hyperparameter search. This approach systematically explores a predefined parameter grid through 5-fold cross-validation to ensure the reproducibility of results. The search process is optimized for computational efficiency by enabling parallel processing.

Regarding deep learning models, we implement an adaptive hyperparameter optimization strategy based on the Optuna framework[1], following methodologies established in prior studies[36]. The optimization protocol maintains a constant batch size of 1024 and performs 100 independent trials using training-validation splits to prevent potential data leakage from the test set.

E.1 Dataset

E.2 Results

To evaluate a model’s ability to detect novel classes, we adopt a leave-one-class-out protocol. For each class label, we exclude its samples from training and treat them as novel (label 1) during testing. An equal number of samples from the remaining classes are randomly sampled as known (label 0). After training the model on the reduced dataset, we compute confidence scores for test samples using the maximum predicted probability. Samples with low confidence (within a fixed interval $[\theta_{\min},\theta_{\max}]$) are predicted as novel. We assess performance using ROC-AUC and AUPR, measuring the model’s ability to separate known and novel instances based on confidence.

To further assess whether models exhibit appropriate uncertainty when encountering novel classes, we hold out one or more classes during training and evaluate the predicted probabilities on test samples from these unseen classes. A prediction is deemed uncertain if its maximum confidence falls within a predefined low-confidence interval $[a,b]$. We report the proportion of novel samples falling into this interval as an indicator of the model’s ability to recognize unfamiliar inputs. This metric complements ROC-AUC and AUPR by directly measuring how often the model expresses uncertainty when presented with out-of-distribution classes.

Table6, Table7, and Table8 show the results for uncertainty intervals $[0.4,0.6]$, $[0.45,0.55]$, and $[0.49,0.51]$, respectively.

F.1 Dataset

We refer to TabFSBench[10] for details of the evaluated datasets.

F.2 Performance Gap

F.3 Results

To systematically assess TabPFN v2 in the presence of decremental features, we design random-shift experiments in TabFSBench. We use accuracy and ROC-AUC for classification tasks and RMSE for regression tasks. Table2 provides detailed model performance.

G.1 Dataset

We evaluate our models on two established benchmarks for distribution shift in tabular data: TableShift[18] and WhyShift[34].

From the TableShift benchmark, we select three datasets—College Scorecard, Hospital Readmission, and Diabetes. To ensure scalability and consistency across experiments, we apply stratified subsampling to limit each dataset to 50,000 total instances while preserving the original train/test split ratio. Detailed statistics are provided in Table9.

From the WhyShift benchmark, we adopt six pre-defined settings provided by the original paper, covering a variety of real-world covariate and concept shift scenarios. For datasets containing more than 50,000 instances, we similarly apply stratified subsampling to retain a total of 50,000 samples. The configuration and statistics for all selected WhyShift settings are summarized in Table10.

G.2 Domain Shift Metrics

Domain shift can be categorized into covariate shift, concept shift, and label shift. We adopt the metrics proposed in TableShift[18] to quantify the degree of these three types of domain shift.

Measuring Concept Shift with Frechet Dataset Distance (FDD):

Inspired by the widely used Frechet Inception Distance (FID) in machine learning[24], FDD utilizes intermediate representations of a classifier to quantify distributional discrepancies. It calculates the Frechet distance (also known as the Wasserstein-2 distance) between two distributions to assess the extent of concept shift.

The computation of this metric proceeds as follows: First, a classifier (we use MLPs) is trained on the source domain using the best hyperparameters obtained through hyperparameter search. Then, for each input $x\in\mathcal{D}$, we compute the activation values at each layer of the model, obtaining the activation vector $\hat{x}:=f_{\theta}[i](x)$, where $i$ denotes the $i$-th layer of the model. Finally, the Frechet Dataset Distance is calculated to measure the divergence between the two distributions.

$$\textrm{DFD}(\mathcal{D}^{\text{train}},\mathcal{D}^{\text{test}})=||\mu_{$$

(3)

$\mu_{D}$ represents the set of feature vectors extracted from domain $D$. Based on $\mu_{D}$, we construct the covariance matrix $\Sigma_{D}$. In the discussion below, we refer to the resulting measure as $\Delta_{y|x}$. A lower FDD score indicates a smaller distance between any $x_{i}$ in the training domain $D_{\text{train}}$ and $x_{j}$ in the test domain $D_{\text{test}}$.

G.3 Results

We evaluate TabPFN v2 alongside several mainstream tabular models under changing data distributions scenarios, using metrics including Accuracy, Balanced Accuracy, F1-score, and ROC-AUC. The evaluation is conducted on nine fully numerical datasets drawn from the WhyShift[34] and TableShift[18] benchmarks, which contain three types of data distribution scenarios. To accommodate memory constraints and the current limitations of TabPFN in handling very large datasets, we apply stratified subsampling (up to 50,000 instances) while preserving the original train/test splits. Figure4, 5,6and7 demonstrates results on changing data distributions. Although there has been recent work attempting to apply TabPFN to scenarios involving temporal distribution shift[22], their implementation is not publicly available. Hence, we do not include this method in our evaluation.