core.dataset.DingoDataset
core.dataset.DingoDataset(
file_name=None,
dictionary=None,
data_keys=None,
leave_on_disk_keys=None,
)This is a generic dataset class with save / load methods.
A common use case is to inherit multiply from DingoDataset and torch.utils.data.Dataset, in which case the subclass picks up these I/O methods, and DingoDataset is acting as a Mixin class.
Alternatively, if the torch Dataset is not needed, then DingoDataset can be subclassed directly.
For constructing, provide either file_name, or dictionary containing data and settings entries, or neither.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| file_name | str | HDF5 file containing a dataset | None |
| dictionary | dict | Contains settings and data entries. The data keys should be the same as save_keys | None |
| data_keys | list | Variables that should be saved / loaded. This allows for class to store additional variables beyond those that are saved. Typically, this list would be provided by any subclass. | None |
| leave_on_disk_keys | Optional[list] | Keys for which the values are not loaded into RAM when initializing the dataset. This reduces the memory footprint during training. Instead, the values are loaded from the HDF5 file during training. | None |
Attributes
| Name | Description |
|---|---|
| dataset_type | |
| settings |
Methods
| Name | Description |
|---|---|
| from_dictionary | |
| from_file | |
| to_dictionary | |
| to_file |
from_dictionary
core.dataset.DingoDataset.from_dictionary(dictionary)from_file
core.dataset.DingoDataset.from_file(file_name)to_dictionary
core.dataset.DingoDataset.to_dictionary()to_file
core.dataset.DingoDataset.to_file(file_name, mode='w')