study
OpenMLBenchmarkSuite
¶
Bases: BaseStudy
An OpenMLBenchmarkSuite represents the OpenML concept of a suite (a collection of tasks).
It contains the following information: name, id, description, creation date, creator id and the task ids.
According to this list of task ids, the suite object receives a list of OpenML object ids (datasets).
Parameters:
Name | Type | Description | Default |
---|---|---|---|
suite_id
|
int
|
the study id |
required |
alias
|
str(optional)
|
a string ID, unique on server (url-friendly) |
required |
main_entity_type
|
str
|
the entity type (e.g., task, run) that is core in this study. only entities of this type can be added explicitly |
required |
name
|
str
|
the name of the study (meta-info) |
required |
description
|
str
|
brief description (meta-info) |
required |
status
|
str
|
Whether the study is in preparation, active or deactivated |
required |
creation_date
|
str
|
date of creation (meta-info) |
required |
creator
|
int
|
openml user id of the owner / creator |
required |
tags
|
list(dict)
|
The list of tags shows which tags are associated with the study. Each tag is a dict of (tag) name, window_start and write_access. |
required |
data
|
list
|
a list of data ids associated with this study |
required |
tasks
|
list
|
a list of task ids associated with this study |
required |
Source code in openml/study/study.py
OpenMLStudy
¶
Bases: BaseStudy
An OpenMLStudy represents the OpenML concept of a study (a collection of runs).
It contains the following information: name, id, description, creation date, creator id and a list of run ids.
According to this list of run ids, the study object receives a list of OpenML object ids (datasets, flows, tasks and setups).
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study_id
|
int
|
the study id |
required |
alias
|
str(optional)
|
a string ID, unique on server (url-friendly) |
required |
benchmark_suite
|
int(optional)
|
the benchmark suite (another study) upon which this study is ran. can only be active if main entity type is runs. |
required |
name
|
str
|
the name of the study (meta-info) |
required |
description
|
str
|
brief description (meta-info) |
required |
status
|
str
|
Whether the study is in preparation, active or deactivated |
required |
creation_date
|
str
|
date of creation (meta-info) |
required |
creator
|
int
|
openml user id of the owner / creator |
required |
tags
|
list(dict)
|
The list of tags shows which tags are associated with the study. Each tag is a dict of (tag) name, window_start and write_access. |
required |
data
|
list
|
a list of data ids associated with this study |
required |
tasks
|
list
|
a list of task ids associated with this study |
required |
flows
|
list
|
a list of flow ids associated with this study |
required |
runs
|
list
|
a list of run ids associated with this study |
required |
setups
|
list
|
a list of setup ids associated with this study |
required |
Source code in openml/study/study.py
185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 |
|
attach_to_study(study_id, run_ids)
¶
Attaches a set of runs to a study.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study_id
|
int
|
OpenML id of the study |
required |
run_ids
|
list(int)
|
List of entities to link to the collection |
required |
Returns:
Type | Description |
---|---|
int
|
new size of the study (in terms of explicitly linked entities) |
Source code in openml/study/functions.py
attach_to_suite(suite_id, task_ids)
¶
Attaches a set of tasks to a benchmarking suite.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
suite_id
|
int
|
OpenML id of the study |
required |
task_ids
|
list(int)
|
List of entities to link to the collection |
required |
Returns:
Type | Description |
---|---|
int
|
new size of the suite (in terms of explicitly linked entities) |
Source code in openml/study/functions.py
create_benchmark_suite(name, description, task_ids, alias=None)
¶
Creates an OpenML benchmark suite (collection of entity types, where the tasks are the linked entity)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
name
|
str
|
the name of the study (meta-info) |
required |
description
|
str
|
brief description (meta-info) |
required |
task_ids
|
list
|
a list of task ids associated with this study
more can be added later with |
required |
alias
|
str(optional)
|
a string ID, unique on server (url-friendly) |
None
|
Returns:
Type | Description |
---|---|
OpenMLStudy
|
A local OpenML study object (call publish method to upload to server) |
Source code in openml/study/functions.py
create_study(name, description, run_ids=None, alias=None, benchmark_suite=None)
¶
Creates an OpenML study (collection of data, tasks, flows, setups and run), where the runs are the main entity (collection consists of runs and all entities (flows, tasks, etc) that are related to these runs)
Parameters:
Name | Type | Description | Default |
---|---|---|---|
benchmark_suite
|
int(optional)
|
the benchmark suite (another study) upon which this study is ran. |
None
|
name
|
str
|
the name of the study (meta-info) |
required |
description
|
str
|
brief description (meta-info) |
required |
run_ids
|
list
|
a list of run ids associated with this study,
these can also be added later with |
None
|
alias
|
str(optional)
|
a string ID, unique on server (url-friendly) |
None
|
benchmark_suite
|
int | None
|
the ID of the suite for which this study contains run results |
None
|
Returns:
Type | Description |
---|---|
OpenMLStudy
|
A local OpenML study object (call publish method to upload to server) |
Source code in openml/study/functions.py
delete_study(study_id)
¶
Deletes a study from the OpenML server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study_id
|
int
|
OpenML id of the study |
required |
Returns:
Type | Description |
---|---|
bool
|
True iff the deletion was successful. False otherwise |
Source code in openml/study/functions.py
delete_suite(suite_id)
¶
Deletes a study from the OpenML server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
suite_id
|
int
|
OpenML id of the study |
required |
Returns:
Type | Description |
---|---|
bool
|
True iff the deletion was successful. False otherwise |
Source code in openml/study/functions.py
detach_from_study(study_id, run_ids)
¶
Detaches a set of run ids from a study.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study_id
|
int
|
OpenML id of the study |
required |
run_ids
|
list(int)
|
List of entities to unlink from the collection |
required |
Returns:
Type | Description |
---|---|
int
|
new size of the study (in terms of explicitly linked entities) |
Source code in openml/study/functions.py
detach_from_suite(suite_id, task_ids)
¶
Detaches a set of task ids from a suite.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
suite_id
|
int
|
OpenML id of the study |
required |
task_ids
|
list(int)
|
List of entities to unlink from the collection |
required |
Returns:
Type | Description |
---|---|
int
|
|
new size of the study (in terms of explicitly linked entities)
|
|
Source code in openml/study/functions.py
get_study(study_id, arg_for_backwards_compat=None)
¶
Retrieves all relevant information of an OpenML study from the server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study
|
study id (numeric or alias) |
required | |
arg_for_backwards_compat
|
str
|
The example given in https://arxiv.org/pdf/1708.03731.pdf uses an older version of the API which required specifying the type of study, i.e. tasks. We changed the implementation of studies since then and split them up into suites (collections of tasks) and studies (collections of runs) so this argument is no longer needed. |
None
|
Returns:
Type | Description |
---|---|
OpenMLStudy
|
The OpenML study object |
Source code in openml/study/functions.py
get_suite(suite_id)
¶
Retrieves all relevant information of an OpenML benchmarking suite from the server.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study
|
study id (numeric or alias) |
required |
Returns:
Type | Description |
---|---|
OpenMLSuite
|
The OpenML suite object |
Source code in openml/study/functions.py
list_studies(offset=None, size=None, status=None, uploader=None, benchmark_suite=None, output_format='dict')
¶
Return a list of all studies which are on OpenML.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
offset
|
int
|
The number of studies to skip, starting from the first. |
None
|
size
|
int
|
The maximum number of studies to show. |
None
|
status
|
str
|
Should be {active, in_preparation, deactivated, all}. By default active studies are returned. |
None
|
uploader
|
list(int)
|
Result filter. Will only return studies created by these users. |
None
|
benchmark_suite
|
int
|
|
None
|
output_format
|
Literal['dict', 'dataframe']
|
The parameter decides the format of the output. - If 'dict' the output is a dict of dict - If 'dataframe' the output is a pandas DataFrame |
'dict'
|
Returns:
Name | Type | Description |
---|---|---|
datasets |
dict of dicts, or dataframe
|
|
Source code in openml/study/functions.py
557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 |
|
list_suites(offset=None, size=None, status=None, uploader=None, output_format='dict')
¶
Return a list of all suites which are on OpenML.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
offset
|
int
|
The number of suites to skip, starting from the first. |
None
|
size
|
int
|
The maximum number of suites to show. |
None
|
status
|
str
|
Should be {active, in_preparation, deactivated, all}. By default active suites are returned. |
None
|
uploader
|
list(int)
|
Result filter. Will only return suites created by these users. |
None
|
output_format
|
Literal['dict', 'dataframe']
|
The parameter decides the format of the output. - If 'dict' the output is a dict of dict - If 'dataframe' the output is a pandas DataFrame |
'dict'
|
Returns:
Name | Type | Description |
---|---|---|
datasets |
dict of dicts, or dataframe
|
|
Source code in openml/study/functions.py
update_study_status(study_id, status)
¶
Updates the status of a study to either 'active' or 'deactivated'.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
study_id
|
int
|
The data id of the dataset |
required |
status
|
(str)
|
'active' or 'deactivated' |
required |
Source code in openml/study/functions.py
update_suite_status(suite_id, status)
¶
Updates the status of a study to either 'active' or 'deactivated'.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
suite_id
|
int
|
The data id of the dataset |
required |
status
|
(str)
|
'active' or 'deactivated' |
required |