FolderBase¶

The FolderBase connector synchronizes datasets from folders you organize yourself: you mark a folder as a dataset by adding a small info file to it. It continuously watches a directory, detects these datasets, and uploads them — the most flexible connector, and the right choice when your data doesn't come from one of the supported acquisition systems.


Package	`etiket_sync_agent_folderbase` (installed by default)
Identifier	`etiket_sync_agent_folderbase`
Type	File-based
Scope	Required default scope
Live sync	No

Configuration¶

Parameter	Type	Description
`root_directory`	path	The folder to watch for datasets.
`is_server_folder`	bool	Whether the folder lives on a shared/network location.

Note

When your data is stored on a network drive, set is_server_folder to True.

Create a sync source¶

from qdrive.scopes import get_scopes
from etiket_sdk.sync import SyncSources

print(get_scopes())  # find the scope UUID you want to sync to

source = SyncSources.create(
    name="my_folderbase_source",
    connector_identifier="etiket_sync_agent_folderbase",
    config_data={
        "root_directory": "~/Downloads/MyData",
        "is_server_folder": False,
    },
    default_scope="6c319227-f6f8-4bc6-b7f9-ed278a8fb040",
)

You can also add this source from the DataQruiser app.

How datasets are detected¶

FolderBase identifies a dataset by the presence of a _QH_dataset_info.yaml file in a folder. Every other file in that folder and its subfolders is treated as a data file and added to the dataset. Dataset folders are detected at any depth under root_directory, so you can organize your tree however you like:

main_folder/
├── 20240101/
│   └── 20240101-211245-experiment_1/
│       ├── _QH_dataset_info.yaml      ← marks this folder as a dataset
│       ├── data.json
│       └── data.hdf5
├── 20240102/
│   └── 20240102-220655-experiment_2/
│       ├── _QH_dataset_info.yaml
│       ├── data.hdf5
│       └── analysis/
│           └── results.hdf5
└── some_other_folder/
    ├── _QH_dataset_info.yaml
    └── data.json

When a file is added to one of these folders, or a new dataset folder is created, the sync agent picks it up automatically.

The `_QH_dataset_info.yaml` file¶

This file holds the metadata used to create the dataset. The minimal version only needs a version:

version: 0.1

It supports the following fields:

Field	Required	Description
`version`	yes	File-format version. The current version is `0.1`.
`dataset_name`	no	Dataset name. Defaults to the name of the folder.
`collected`	no	Data-collection timestamp in ISO format `YYYY-MM-DDTHH:MM:SS`. Defaults to the creation time of the first file.
`description`	no	A description of the dataset.
`attributes`	no	A mapping of attributes; values must be strings or numbers.
`tags`	no	A list of tags.
`skip`	no	Glob patterns for files/folders to ignore (e.g. `['.json', 'raw_data/']`).
`thumbnails`	no	Images shown next to the dataset in the dataset list, most important first (e.g. `['overview.png', 'plots/*.png']`) — see Thumbnails.
`converters`	no	File converters to run during sync — see Converters.

A fuller example:

version: 0.1
dataset_name: my_dataset_name
description: "Description of the experiment I want to do."
attributes:
  initials: QH
  set_up: XLD001
  sample: my_sample
tags: ['rabi', 'test']
skip: ['*.json', 'raw_data/*']
thumbnails: ['overview.png', 'plots/*.png']

Use spaces, not tabs

The YAML file must use spaces for indentation. Tabs cause parsing errors and the dataset will fail to synchronize.

Creating the info file programmatically¶

When you run measurements, the easiest approach is to write _QH_dataset_info.yaml from your acquisition script. qDrive provides a helper:

from qdrive.dataset import generate_dataset_info

generate_dataset_info(
    "/path/to/dataset",
    dataset_name="my_dataset_name",
    description="Description of the experiment I want to do.",
    attributes={"sample": "my_sample"},
    tags=["rabi", "test"],
    skip=["*.json", "raw_data/*"],
    thumbnails=["overview.png", "plots/*.png"],
)

All arguments except the path are optional.

Thumbnails¶

The images listed under thumbnails are shown next to the dataset in the dataset list of the DataQruiser app, which makes a folder of measurements far easier to scan than a list of names.

version: 0.1
thumbnails:
  - 'overview.png'      # the image the dataset is shown with
  - 'plots/*.png'       # the rest, in alphabetical order

The order of the list is the order of the thumbnails, so put the plot that best represents the dataset first. Entries are either a literal path or a glob pattern, both relative to the dataset folder:

Patterns use the same syntax as skip and only ever match images, so thumbnails: ['*'] means "every image in this dataset" and a pattern that also catches a .json file simply ignores it.
Literal paths are treated as a deliberate choice: a typo, a missing file, or a file that is not an image is reported as a synchronization error.
Files excluded by skip are not uploaded at all, so they cannot be thumbnails. Listing the same file under both is reported as an error.
At most 10 thumbnails are added per dataset.

Converters¶

FolderBase can run converters that transform files during synchronization — for example turning a .zarr directory into a .zip, or a CSV into HDF5 so it plots automatically in qHarbor. A converter is referenced from a dataset's _QH_dataset_info.yaml by module and class:

version: 0.1
converters:
  zarr_to_zip_converter:
    module: my_converters.zarr_to_zip
    class: ZarrToZipConverter

See File converters for how to write, test, package, and install your own.