Example: AutoML on a specific task and fold

The defaults are very useful for performing a quick test, as the datasets are small and cover different task types (binary classification, multiclass classification, and regression). We also have a "validation" benchmark suite for more elaborate testing that also includes missing data, categorical data, wide data, and more. The benchmark defines 9 tasks, and evaluating two folds with a 10-minute time constraint would take roughly 3 hours (=9 tasks * 2 folds * 10 minutes, plus overhead). Let's instead use the --task and --fold parameters to run only a specific task and fold in the benchmark when evaluating the flaml AutoML framework:

python runbenchmark.py flaml validation test -t eucalyptus -f 0

This should take about 10 minutes plus the time it takes to install flaml. Results should look roughly like this:

Processing results for flaml.validation.test.local.20230711T122823
Summing up scores for current run:
               id       task  fold framework constraint    result      metric  duration       seed
openml.org/t/2079 eucalyptus     0     flaml       test -0.702976 neg_logloss     611.0 1385946458

Similarly to the test run, you will find additional files in the results directory.