Skip to content

Getting Started

The AutoML Benchmark is a tool for benchmarking AutoML frameworks on tabular data. It automates the installation of AutoML frameworks, passing it data, and evaluating their predictions. Our paper describes the design and showcases results from an evaluation using the benchmark. This guide goes over the minimum steps needed to evaluate an AutoML framework on a toy dataset.

Full instructions can be found in the API Documentation.

Installation

These instructions assume that Python 3.9 (or higher) and git are installed, and are available under the alias python and git, respectively. We recommend Pyenv for managing multiple Python installations, if applicable. We support Ubuntu 22.04, but many linux and MacOS versions likely work (for MacOS, it may be necessary to have brew installed).

First, clone the repository:

1
2
git clone https://github.com/openml/automlbenchmark.git --branch stable --depth 1
cd automlbenchmark

Create a virtual environments to install the dependencies in:

Linux

1
2
python -m venv venv
source venv/bin/activate

MacOS

1
2
python -m venv venv
source venv/bin/activate

Windows

1
2
python -m venv ./venv
venv/Scripts/activate

Then install the dependencies:

1
2
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
Note for Windows users

The automated installation of AutoML frameworks is done using shell script, which doesn't work on Windows. We recommend you use Docker to run the examples below. First, install and run docker. Then, whenever there is a python runbenchmark.py ... command in the tutorial, add -m docker to it (python runbenchmark.py ... -m docker).

Problem with the installation?

On some platforms, we need to ensure that requirements are installed sequentially. Use xargs -L 1 python -m pip install < requirements.txt to do so. If problems persist, open an issue with the error and information about your environment (OS, Python version, pip version).

Running the Benchmark

To run a benchmark call the runbenchmark.py script specifying the framework to evaluate.

See the API Documentation. for more information on the parameters available.