malva.datastore module

class malva.datastore.ChunkedStore

Bases: object

attrs

close()

contains(key)

Check if dataset exists.

copy_dataset(source_name, dest_name)

Copy a dataset within the store

create_dataset(name, shape, dtype='float64', compression_enabled=False, **kwargs)

Create a new dataset with the specified shape and type, with optional Elias compression.

Parameters:

namestr

Name of the dataset

shapetuple

Shape of the dataset (e.g., (100, 200) for 2D array)

dtypestr or numpy.dtype

Data type of the dataset

**kwargsdict

Additional arguments (for compatibility)

Returns:

: PageAlignedArray

The newly created dataset

flush()

Flush all changes to disk and save metadata

get_chunk_indices(dataset_name)

Generate chunk indices for efficient iteration over a dataset

Parameters:

dataset_namestr

Name of the dataset

Yields:

tuple

Slice objects for each dimension

items()

Return dataset items.

keys()

Return dataset keys.

resize_dataset(name, new_shape)

Resize a dataset to a new shape

values()

Return dataset values.

class malva.datastore.EliasPageCache

Bases: object

class malva.datastore.PageAlignedArray

Bases: object

flush()

Ensure all changes are written to disk

get_shape()

Public method to get the shape

shape

Public property to access array shape

class malva.datastore.PageCache

Bases: object

malva.datastore.convert_h5_to_chunked(h5_filename, output_base, chunk_size=10000000)

Convert HDF5 file to chunked store format with memory-efficient processing.