Datasets and Files¶

Each qDrive dataset is uniquely identified by a UUID (Universally Unique Identifier). A dataset has metadata associated with it that helps you sort and filter your data — some user-defined (attributes, tags, ranking) and some system-defined (creation time, UUID).

A dataset can hold multiple files where the actual data is stored. Files can be of any format; the DataQruiser app can render netCDF4 files, JSON, code/text files, images (tiff, png, jpeg, svg), PDF, Markdown and HTML and unsupported files with the system viewer.

Creating datasets¶

An empty dataset can be created with:

from qdrive import dataset

# Minimal example — a dataset with just a name
ds_1 = dataset.create('my_dataset_name')

# Complete example — a dataset with extended metadata
ds_2 = dataset.create(
    'Qubit 2 T2*',
    description='T2* measurement of qubit 2, RF power is also applied to qubit 1',
    scope_name='2Q SC processor A14',
    tags=['calibration'],
    attributes={
        'set_up': 'Fridge B256',
        'sample_id': 'Q7-R3',
    },
    alt_uid='exp20240115-124501',
)

Every dataset is automatically assigned a UUID, accessible after creation:

print(f"Dataset UUID: {ds_2.uuid}")

Parameters:

name (required): a descriptive name for your dataset.
description (optional): a longer description of the dataset contents.
scope_name (optional): the scope where the dataset will be stored.
tags (optional): a list of tags (e.g. calibration, tuning).
attributes (optional): a dictionary of key-value pairs providing structured metadata.
alt_uid (optional): an alternative identifier that can be used instead of the UUID to access the dataset. Usually not needed.

Note

If no scope_name is provided, the dataset is created in the default scope (see Scopes).
scope_name accepts a scope name (string), a scope object returned by get_scopes(), or a scope UUID (e.g. uuid.UUID('12345678-1234-5678-1234-567812345678')).

Tip

Use descriptive names and consistent attributes across related datasets to make them easier to find and filter later. Attributes and tags are fully searchable in both the Python API and the DataQruiser app.

Files within a dataset¶

The dataset object manages files within a dataset. This section covers adding files, inspecting and selecting file versions, and using file-type-specific methods to access data.

Adding files¶

You can add files to a dataset by assigning them directly:

from qdrive import dataset
from pathlib import Path
import numpy as np
import xarray as xr

new_dataset = dataset.create('my_dataset_name')

# add a file from a path, or an xarray object
new_dataset.add_file('C:/location/of/file.extension')
new_dataset.add_xarray("analysis.hdf5", xr.Dataset({'a': (['x'], np.arange(10))}))

# alternative way to add a file

# from a path
new_dataset['my_file.extension'] = Path('C:/location/of/file.extension')

# from a Python object (list, dict, numpy array, xarray)
new_dataset['my_array.npz'] = np.linspace(0, 10, 100)
new_dataset['my_json.json'] = {'a': 1, 'b': [1, 2, 3]}
new_dataset['my_xarray.hdf5'] = xr.Dataset({'a': (['x'], np.arange(10))})

# add the current script file
new_dataset['my_script.py'] = __file__

Note

Assigning a file with the same key multiple times creates a new version of that file.

Adding thumbnails¶

A thumbnail is the image shown next to a dataset in the dataset list of the DataQruiser app, which makes a long list of measurements far easier to scan. Add one from a file, a matplotlib figure, or an image you already have in memory:

import matplotlib.pyplot as plt

ds = dataset.create('my_dataset_name')

# from a file
ds.add_thumbnail('C:/location/of/overview.png')

# directly from the figure you just plotted
fig, ax = plt.subplots()
ax.plot(x_data, y_data)
ds.add_thumbnail(fig)

A dataset can have several thumbnails, numbered with id. Thumbnail 0 — the default — is the one the dataset is shown with:

ds.add_thumbnail(fig_overview)          # id=0, the image shown in the dataset list
ds.add_thumbnail(fig_fit, id=1)         # additional thumbnails
ds.add_thumbnail(fig_raw, id=2)

Inspecting and selecting file versions¶

Each file can have multiple versions, inspected with:

print(my_dataset['my_file.extension'].version_info)

which displays something like:

File object information
=======================
Name : my_file
Selected File version : 1720711563075
File versions (3) :
      1720711517406 (created on 11/07/2024 17:25:17)
      1720711551649 (created on 11/07/2024 17:25:51)
    * 1720711563075 (created on 11/07/2024 17:26:03)

By default, accessing a file returns its latest version (marked with *). To access a specific version by its ID:

my_dataset['my_file.extension'].version_id(1720711517406)

Tip

You can also access file versions by their position in the version history:

my_dataset['my_file.extension'].version(0)  # first version
my_dataset['my_file.extension'].version(1)  # second version
my_dataset['my_file.extension'].version(2)  # third version

File-type-specific methods¶

Numerical data

For numerical data we recommend .hdf5 files in the NETCDF4 format. While you can assign an HDF5 file directly, a more convenient approach is to work with xarray datasets — when you assign an xarray dataset, it is automatically converted to an HDF5 file and uploaded.

Tip

xarray lets you label NumPy arrays and define relationships between dimensions. Assigning data this way also enables automatic plotting in the DataQruiser app.

import xarray as xr
import numpy as np

x_data = np.linspace(0, 30, 100)
y1_data = np.sin(x_data)
y2_data = np.cos(x_data)

xr_ds = xr.Dataset(
    {
        "y1": (["x"], y1_data, {"units": "mV"}),
        "y2": (["x"], y2_data, {"units": "mV"}),
    },
    coords={"x": ("x", x_data, {"units": "s"})},
)

# Optionally link y1 and y2 for joint plotting (temporary workaround):
xr_ds["y1"].attrs.update({"__join_plot": "y2"})
xr_ds["y2"].attrs.update({"__join_plot": "y1"})

HDF5 files saved in NETCDF4 format can be accessed in several ways:

from qdrive import dataset
dsq = dataset("my_dataset_uuid_or_alt_id")

xarray_ds   = dsq["measurement.hdf5"].xarray    # load as an xarray (recommended)
pandas_ds   = dsq["measurement.hdf5"].pandas    # load as a pandas DataFrame
hdf5_handle = dsq["measurement.hdf5"].hdf5       # load as an h5py File

Adding metadata

Metadata can be added directly to a dataset or stored within files. A common approach is to save metadata as JSON, which is accessed as a dictionary in Python:

dsq['my_json_data.json'] = {'item1': "value1", 'item2': "value2"}
# Keys can be added dynamically, though assigning all keys at once is faster.
dsq['my_json_data.json']['item3'] = "value3"

Lists can also be assigned for metadata storage.

Note

Metadata directly associated with numerical data can be embedded within an xarray dataset, which makes it visible in the DataQruiser app:

import xarray as xr
import numpy as np

xr_ds = xr.Dataset(
    {"y": (["x"], np.linspace(0, 100), {"units": "mV"})},
    coords={"x": ("x", np.linspace(0, 100), {"units": "s"})},
)
xr_ds.attrs['my_fit_params'] = {'param1': 1, 'param2': 2, 'param3': 3}
dsq['my_xarray.hdf5'] = xr_ds

Adding source files

To automatically upload a Python script to a dataset each time it runs, add this to your script:

from qdrive import dataset
from pathlib import Path

dsq = dataset.create('my_dataset')   # or load an existing one: dataset('uuid')
dsq['my_script.py'] = Path(__file__)

In a Jupyter notebook, upload the notebook file by running, in a cell:

dsq['my_notebook.ipynb'] = Path('my_notebook_name.ipynb')

Note

Save your file before uploading to ensure all changes are included.

Retrieving datasets¶

Retrieve a dataset by its UUID (which you can copy with one click from the DataQruiser app):

from qdrive import dataset
dsq = dataset('d30eec8071014f99b09cc3dfce60187d')  # UUID or alternative identifier
print(dsq)

Printing the dataset shows its contents:

Contents of dataset :: single shot - sensor tuning
==================================================

uuid :: 59c40af3-cef3-49aa-8747-64707a9b080a
Alternative identifier :: 1695914164228175126
Scope :: my_scope
Ranking :: 0
Attributes ::
        set-up : my_setup
        sample : my_sample
Files ::
name              type                  selected version number (version_id)      Maximal version number
----------------  --------------------  --------------------------------------  ------------------------
measurement       FileType.HDF5_NETCDF  0 (1719573749579)                                              0
snapshot          FileType.JSON         0 (1719573749579)                                              0
analysis          FileType.HDF5_NETCDF  1 (1719573831702)                                              1
fit_params        FileType.JSON         1 (1719573831822)                                              1

uuid: the universal unique identifier of the dataset.
Alternative identifier: the identifier assigned by your data-acquisition software (e.g. the Core Tools uid or QCoDeS GUID).
Scope: the scope this dataset belongs to.
Ranking: an integer indicating how much you like the dataset, useful for filtering in the DataQruiser app.
Attributes: searchable key-value fields.
Files: the files in the dataset, each with its own version history.

Modifying dataset metadata¶

After creating a dataset, you can modify its metadata:

ds = dataset("59c40af3-cef3-49aa-8747-64707a9b080a")

# Update the description
ds.description = 'T2* measurement of qubit 2 with RF power applied to qubit 1'

# Replace all tags
ds.tags = ['calibration']
# Add a tag without replacing the others
ds.tags.append('tuning')

# Replace all attributes
ds.attributes = {'set_up': 'Fridge B256', 'sample_id': 'Q7-R3'}
# Add or update a single attribute
ds.attributes['sample_id'] = 'Q7-R4'

# Set the ranking (useful for filtering)
ds.ranking = 1  # 1 = like, 0 = neutral, -1 = dislike/hidden

Note

Changes are made locally first and then synchronized with the server (this can take a few seconds to appear on other devices).

Searching and filtering datasets¶

from qdrive.dataset.search import search_datasets

search_result = search_datasets(search_query='my_coolest_dataset', attributes={'sample': 'sample_1'}, ranking=1)

# iterate over the results
for ds in search_result:
    print(ds)

# get the first dataset
ds = search_result.first

search_datasets accepts:

search_query: a string matched against the dataset name, UUID, or alternative identifier.
attributes: attributes to filter on, e.g. {'set_up': 'my_setup'}. A list of values matches any of them, e.g. {'set_up': ['my_setup1', 'my_setup2']}.
ranking: the minimum ranking to include (default 0; hidden datasets have ranking -1).
start_date / end_date: only include datasets collected after / before this date (e.g. datetime.datetime(2024, 12, 1)).
scopes: a list of scopes to filter on (name, UUID, or scope object — see Scopes).

Data conversion policies¶

qDrive converts Python objects assigned to datasets as follows:

xarray.Dataset / xarray.DataArray: converted to a NetCDF4 file. dict attributes are serialized to JSON; an error occurs if attributes contain types incompatible with JSON.
h5py.File: stored directly as an HDF5 file.
Python str, int, float, bool, tuple, dict, list: converted to JSON and stored as a .json file.
NumPy arrays: stored as .npz files (also supports list[np.ndarray] and dict[str, np.ndarray]).
Other Python objects: no predefined conversion.

The sync agent also performs specific conversions:

QCoDeS: converted with qcodes.dataset.to_xarray_dataset(), then to NetCDF4.
Core Tools: converted with core_tools.data.ds.ds2xarray.ds2xarray(), then to NetCDF4.
FolderBase: .zarr files are not uploaded by default; use a converter to convert them to a ZIP or NetCDF4 file.

Note

All NetCDF4 conversions use engine='h5netcdf' and invalid_netcdf=True.

Create a dataset from a folder¶

Use upload_folder to upload a directory (recursively) as a single dataset. All regular files are included except manifest files; optional Zarr conversion is supported.

from qdrive.utility.uploads import upload_folder
from qdrive.scopes import get_scope

upload_scope = get_scope("my_scope")
upload_folder("/my/location", scope=upload_scope.uuid)

If the dataset name is not provided, it defaults to the folder name.

All upload_folder parameters

folder_path: path to the directory to upload.
scope: scope name or UUID. Defaults to the current default scope.
dataset_name: dataset name. Defaults to the basename of folder_path.
dataset_description: optional description.
dataset_tags: optional list of tags.
dataset_attributes: optional key/value attributes.
dataset_alt_uid: optional alternative identifier.
dataset_collected: the datetime the dataset was collected. Defaults to the earliest file creation time in folder_path.
direct_upload: if True, upload directly to the server; if False, register locally and let the sync agent upload.
convert_zarr_to_hdf5: if True, each .zarr directory is converted to a single .hdf5 (NetCDF via xarray) and the original .zarr is skipped.
allow_scope_override: a manifest stored alongside the folder remembers which scope it was uploaded to. Re-uploading the same folder to a different scope normally raises an error. Set this to True to deliberately change the scope.

A complete example

import datetime

from qdrive.utility.uploads import upload_folder
from qdrive.scopes import get_scope

scope = get_scope("my_scope")

upload_folder(
    "/data/experiment_42",
    scope=scope.uuid,
    dataset_name="experiment_42",
    dataset_description="Room-temperature sweep",
    dataset_tags=["rt", "sweep"],
    dataset_attributes={"operator": "alice", "run": "42"},
    dataset_alt_uid="exp-42",
    dataset_collected=datetime.datetime.now(),
    direct_upload=True,
    convert_zarr_to_hdf5=True,
)

Copying a dataset to another scope¶

Use copy_dataset to duplicate a dataset from one scope to another. The destination dataset receives a new UUID; the alt_uid is preserved (or set to the source UUID if absent), so re-running the copy is idempotent. Metadata (name, description, tags, collected time, creator, ranking, attributes) and all files with their versions are copied.

from qdrive.utility.copy import copy_dataset
from qdrive.scopes import get_scope

source_scope = get_scope("scope_1")
target_scope = get_scope("scope_2")

copy_dataset("90c7fc9474754691a989f061bf5a6752", source_scope.uuid, target_scope.uuid)

Parameters:

dataset_src_uuid: UUID of the dataset to copy.
scope_src_uuid: UUID of the scope the dataset is currently in.
scope_dst_uuid: UUID of the scope to copy it to.

Datasets in the DataQruiser app¶

In addition to browsing and visualizing your data, you can create datasets, modify their metadata, and upload files directly within the DataQruiser app.