The respiratory is partially inspired by benchopt, but aimed for a more R-user-friendly, easily-hacking, more statistically benchmarking.
This repository supports an automated GitHub Action bot that listens for issues with the title
@new-csv DataName DataURL Idx_of_YThe bot will create a new commit and open a pull request to add new data for benchmarking automatically. For example, Issue #39 triggers Pull Request #38, which adds the abalone dataset for benchmarking.
@new-csv abalone https://raw.githubusercontent.com/jbrownlee/Datasets/refs/heads/master/abalone.csv 9The response variable is the 9-th column, so Idx_of_Y = 9. (Currently, assume no header column names.)
The source files are stored in R/, which contains three files
datasets.R: each dataset (either simulated or real dataset) is defined as a function
For the simulation data, we consider
- different covariance structure of the design matrix
- Independent
- AR(1)
- AR(1)+
- Factor
- different data generating model
- Friedman
- Checkerboard
- Linear
- Max
For the real datasets, we consider
| Data | Description | n | p | URL |
|---|---|---|---|---|
| BostonHousing | Housing values and other information about Boston census tracts. | 506 | 13 | 🔗 |
| CaliforniaHousing | Aggregated housing data from each of 20,640 neighborhoods (1990 census block groups) in California. | 20640 | 8 | 🔗 |
| CASP | Physicochemical Properties of Protein Tertiary Structure | 45730 | 9 | 🔗 |
| Energy | Appliances Energy Prediction | 19735 | 27 | 🔗 |
| AirQuality | Air Quality | 9357 | 12 | 🔗 |
| BiasCorrection | Bias correction of numerical prediction model temperature forecast | 7590 | 21 | 🔗 |
| ElectricalStability | Electrical Grid Stability Simulated Data | 10000 | 12 | 🔗 |
| GasTurbine | Gas Turbine CO and NOx Emission Data Set | 36733 | 10 | 🔗 |
| ResidentialBuilding | Residential Building | 372 | 107 | 🔗 |
| LungCancerGenomic | Lung cancer genomic data from the Chemores Cohort Study | 123 | 945 | 🔗 |
| StructureActivity | Qualitative Structure Activity Relationships (triazines) | 186 | 60 | 🔗 |
| BloodBrain | Blood Brain Barrier Data data("BloodBrain", package = "caret") |
208 | 134 | 🔗 |
| GSE65904 | Whole-genome expression analysis of melanoma tumor biopsies from a population-based cohort. | 210 | 47323 | 🔗 |
methods.R: each method is defined as a function and with tuning parameters
Here are the method we considered:
| Method | Software |
|---|---|
| Bayesian Additive Regression Trees (BART) |  |
| Accelerated BART (XBART) |  |
| Random Forests |  |
| XGBoost |  |
| Multivariate Adaptive Regression Splines (MARS) |  |
evaluate.R: evaluate each method (with particular parameter) on each dataset, and report the CV error and running time (the criteria can be customized)
To interactively display the results, we adopt the Shiny App, and the related files are in the folder benchmark-tree-regressions/
Note that a typical shiny app is server-based. With the help of shinylive, we deploy the shiny website on the GitHub pages https://hohoweiya.xyz/benchmark.tree.regressions/.
The repository used renv to manage the compatible R packages. It is also better to use the same R version 4.1.2 in case.
Then one can install the dependencies of R package via
> renv::restore() # Restores packages from the lockfile (if using renv)Next, you can check the scripts run-local.R and run-action.R.
Due to the limited computational resources, run-action.R take shorter time to finish.
Specifically, the current local results are run on the Yale Mccelary HPC.
$ R
> renv::restore() # restore packages from the lockfile
> system.time({source("run-local.R")})
user system elapsed
207084.40 56119.33 107226.75 Automated Way: open an issue with title @new-csv DataName URL Idx_of_Y
Step by Step: Add a real data
- add data info into
lst_real_dataindatasets.R, and write out a meta data
real.data.meta = df_data_meta()
saveRDS(real.data.meta, file = "benchmark-tree-regressions/real-data-meta.rds")-
run
print_to_readme()to update the table in thisREADME.mdfile -
add the preparation step for the data (starting from downloading the data) as a function
real_XXXindatasets.R -
update the list of real data
benchmark-tree-regressions/choices.real.data.R
The current respiratory is mainly benchmarking various tree-based regressions, but the structure is much more flexible for other tasks. If you find this structure useful, please feel free to use it as a template for your benchmarking analysis via the link 🔗.