File converters

A converter transforms a file into another format during synchronization — for example turning a .zarr directory into a .zip, or a CSV into HDF5 so it can be plotted automatically in dataQruiser. Converters are used by the FolderBase connector, and can also be used from your own custom connectors.

Available converters

A set of ready-to-use converters is provided in the etiket_sync_agent_qh_converters package:

Converts	Module	Class	Converter name
CSV → HDF5	etiket_sync_agent_qh_converters	CSVToHDF5Converter	csv_to_hdf5_converter
Zarr → ZIP	etiket_sync_agent_qh_converters	ZarrToZipConverter	zarr_to_zip_converter
Zarr → HDF5	etiket_sync_agent_qh_converters	ZarrToNETCDF4Converter	zarr_to_hdf5_converter

The three columns map directly onto the _QH_dataset_info.yaml fields:

• Module — the import path to load the class from. The package re-exports its converter classes from its top level, so you use the package name directly (no submodule path needed).
• Class — the converter class.
• Converter name — the YAML key. It must follow the <input>_to_<output>_converter pattern; the agent then applies the converter to files whose extension matches the class's input_type.

To use one, install the package that provides it (here through the SDK; you can also do this from the DataQruiser app):

from etiket_sdk.sync import Converters

Converters.install_from_pypi("etiket_sync_agent_qh_converters")
print(Converters.list())

Then reference it from a dataset's _QH_dataset_info.yaml by module and class. The converter name follows the pattern <input>_to_<output>_converter:

version: 0.1
converters:
  zarr_to_zip_converter:
    module: etiket_sync_agent_qh_converters
    class: ZarrToZipConverter

The agent runs the converter during sync on any matching file in the dataset. The converter package must be installed in the environment where the sync agent runs.

Creating a new converter

If you need a conversion that isn't listed above, you can write your own. A converter is a small Python package — registered the same way as a connector.

Package structure

etiket-sync-agent-zarr-to-zip-converters/
├── pyproject.toml                  # package config + entry point
└── etiket_sync_agent_zarr_to_zip_converters/
    ├── __init__.py                 # exports the converter class
    └── zarr_to_zip_converter.py    # the FileConverter subclass

You don't have to create these by hand — the scaffold generator below produces exactly this structure.

Generating the package

The etiket_sync_agent package (pip install etiket_sync_agent) ships a scaffold generator that creates the package for you — exactly like the one for connectors:

generate_converter_scaffold zarr-to-zip \
    --dependency zarr

Options: --version (initial package version), --dependency (add a dependency, repeatable), and --entry-name (the entry-point name; defaults to the sanitized converter name).

Technical detail: how converters are discovered

Converters are registered the same way as connectors: each package declares an entry point, in the etiket_sync_agent.converters group (the scaffold writes this for you):

[project.entry-points."etiket_sync_agent.converters"]
zarr_to_zip = "etiket_sync_agent_zarr_to_zip_converters:ZarrToZipConverter"

When the package is installed into the sync agent's environment, the agent discovers it automatically — no central registration needed.

The converter class

Fill in the generated class: subclass FileConverter, declare input_type and output_type, and implement convert(), which receives two arguments and returns the path to the result:

• input_path — the file (or folder) to convert.
• output_dir — a pathlib.Path to a scratch directory provided by the sync agent. Write your output (and any intermediate files) here, and return the path to the result.

output_dir is temporary

The sync agent creates output_dir fresh before each convert() call and deletes it automatically once the converted file has been uploaded. So:

• Always write your output inside output_dir (don't write next to input_path or in the current working directory).
• Don't put anything there you need to keep — it won't survive the call.
• You don't need to create or clean up the directory yourself; just use it.

import shutil, pathlib
from etiket_sync_agent.converters.converter_abstract import FileConverter

class ZarrToZipConverter(FileConverter):
    input_type = "zarr"     # input file extension
    output_type = "zip"     # output file extension

    def convert(
        self, input_path: pathlib.Path, output_dir: pathlib.Path
    ) -> pathlib.Path:
        shutil.make_archive(
            base_name=str(output_dir / input_path.name),
            format="zip",
            root_dir=str(input_path),
        )
        return output_dir / f"{input_path.name}.zip"

You can test it locally before wiring it into a sync source:

import tempfile, pathlib

with tempfile.TemporaryDirectory() as tmp:
    converter = ZarrToZipConverter()
    output = converter.convert(pathlib.Path("/path/to/test.zarr"), pathlib.Path(tmp))
    print(output)   # inspect the result before tmp is removed

Bundling multiple converters

The scaffold creates a package with a single converter, but one package can provide several — the built-in etiket_sync_agent_qh_converters ships three. To add more, repeat three steps per converter:

1. Add another FileConverter subclass (in its own module, e.g. csv_to_hdf5.py).
2. Re-export it from the package's __init__.py.
3. Add one line to the entry-point table in pyproject.toml.

The entry-point table simply lists one converter per line:

[project.entry-points."etiket_sync_agent.converters"]
csv_to_hdf5 = "etiket_sync_agent_qh_converters:CSVToHDF5Converter"
zarr_to_zip = "etiket_sync_agent_qh_converters:ZarrToZipConverter"
zarr_to_netcdf4 = "etiket_sync_agent_qh_converters:ZarrToNETCDF4Converter"

Each line is entry_name = "<package>:<ConverterClass>"; the entry name only has to be unique within the package. The <package>:<Class> reference resolves because every converter class is re-exported from __init__.py:

# etiket_sync_agent_qh_converters/__init__.py
from etiket_sync_agent_qh_converters.csv_to_hdf5 import CSVToHDF5Converter
from etiket_sync_agent_qh_converters.zarr_to_zip import ZarrToZipConverter
from etiket_sync_agent_qh_converters.zarr_to_netcdf4 import ZarrToNETCDF4Converter

Installing your converter

While developing, install your package into the sync agent's environment in editable mode through the SDK:

from etiket_sdk.sync import Converters

Converters.install_from_local(
    "/path/to/etiket-sync-agent-zarr-to-zip-converters", editable=True
)
print(Converters.list())

Once published, others install it with Converters.install_from_pypi("etiket_sync_agent_zarr_to_zip_converters"). Restart the sync agent if it was already running, then reference the converter from your _QH_dataset_info.yaml as shown above.