xcp_d.config module

A Python module to maintain unique, run-wide XCP-D settings.

This module implements the memory structures to keep a consistent, singleton config. Settings are passed across processes via filesystem, and a copy of the settings for each run and subject is left under <output_dir>/sub-<participant_id>/log/<run_unique_id>/xcp_d.toml. Settings are stored using ToML. The module has a to_filename() function to allow writing out the settings to hard disk in ToML format, which looks like:

Example file representation of XCP-D settings.

[environment]
cpu_count = 1
exec_env = "posix"
free_mem = 2.2
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.5.0"
templateflow_version = "0.4.2"
version = "0.6.0"

[execution]
bids_description_hash = "123456"
bids_filters = {}
boilerplate_only = false
debug = []
fmri_dir = "ds000005/"
fs_license_file = "/opt/freesurfer/license.txt"
log_dir = "/opt/xcp_d"
log_level = 40
low_mem = false
md_only_boilerplate = false
notrack = false
output_dir = "ds000005/derivatives/xcp_d"
reports_only = false
run_uuid = "20240205-123456"
templateflow_home = "~/.cache/templateflow"
work_dir = "work/"
write_graph = false
atlases = ["Gordon"]

[workflow]
file_format = "nifti"
dummy_scans = 0
input_type = "fmriprep"
despike = false
smoothing = 6
combine_runs = false
motion_filter_order = 4
head_radius = 50
fd_thresh = 0.3
min_time = 100
bandpass_filter = true
high_pass = 0.01
low_pass = 0.1
bpf_order = 2
min_coverage = 0.5
correlation_lengths = []
process_surfaces = false
abcc_qc = false

[nipype]
crashfile_format = "txt"
get_linked_libs = false
memory_gb = 48
nprocs = 2
omp_nthreads = 2
plugin = "MultiProc"
resource_monitor = false
stop_on_first_crash = false

[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

This config file is used to pass the settings across processes, using the load() function.

Configuration sections

class xcp_d.config.environment

Read-only options regarding the platform and environment.

Crawls runtime descriptive settings (e.g., default FreeSurfer license, execution environment, nipype and XCP-D versions, etc.). The environment section is not loaded in from file, only written out when settings are exported. This config section is useful when reporting issues, and these variables are tracked whenever the user does not opt-out using the --notrack argument.

cpu_count = 2

Number of available CPUs.

exec_docker_version = None

Version of Docker Engine.

exec_env = 'posix'

A string representing the execution platform.

free_mem = 5.9

Free memory at start.

nipype_version = '1.11.0'

Nipype’s current version.

overcommit_limit = '50%'

Linux’s kernel virtual memory overcommit limits.

overcommit_policy = 'heuristic'

Linux’s kernel virtual memory overcommit policy.

templateflow_version = '25.1.2'

The TemplateFlow client version installed.

version = '26.0.1'

XCP-D’s version.

class xcp_d.config.execution

Configure run-level settings.

aggr_ses_reports = None

Maximum number of sessions aggregated in one subject’s visual report.

atlases = []

Selection of atlases to apply to the data.

bids_database_dir = None

Path to the directory containing SQLite database indices for the input BIDS dataset.

bids_description_hash = None

Checksum (SHA256) of the dataset_description.json of the BIDS dataset.

bids_filters = None

A dictionary of BIDS selection filters.

boilerplate_only = None

Only generate a boilerplate.

confounds_config = None

Nuisance regressors to include in the postprocessing.

dataset_links = {}

A dictionary of dataset links to be used to track Sources in sidecars.

datasets = {}

Path(s) to search for other datasets (either derivatives or atlases).

debug = []

Debug mode(s).

fmri_dir = None

An existing path to the preprocessing derivatives dataset, which must be BIDS-compliant.

fs_license_file = None

An existing file containing a FreeSurfer license.

classmethod init()

Create a new BIDS Layout accessible with layout.

layout = None

A BIDSLayout object, see init().

log_dir = None

The path to a directory that contains execution logs.

log_level = 25

Output verbosity.

low_mem = None

Utilize uncompressed NIfTIs and other tricks to minimize memory allocation.

md_only_boilerplate = None

Do not convert boilerplate from MarkDown to LaTex and HTML.

notrack = None

Do not collect telemetry information for XCP-D.

output_dir = None

Folder where derivatives will be stored.

output_layout = None

Output layout for the derivatives.

parameters_hash = None

Unique identifier for this set of configurable parameters.

participant_label = None

List of participant identifiers that are to be preprocessed.

processing_list = []

List of (subject_id, anat_session_id, [func_session_id, …]) to be postprocessed.

report_output_level = None

Directory level at which the html reports should be written.

reports_only = None

Only build the reports, based on the reportlets found in a cached working directory.

run_uuid = '20260331-185743_ae87c778-c7fb-4d0b-9ce8-3d83f4cb0974'

Unique identifier of this particular run.

session_id = None

Select a particular session from all available in the dataset.

task_id = None

Select a particular task from all available in the dataset.

templateflow_home = PosixPath('/home/docs/.cache/templateflow')

The root folder of the TemplateFlow client.

work_dir = PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/xcp-d/checkouts/26.0.1/docs/work')

Path to a working directory where intermediate results will be available.

write_graph = None

Write out the computational graph corresponding to the planned preprocessing.

class xcp_d.config.workflow

Configure the particular execution graph of this workflow.

abcc_qc = None

Run DCAN QC.

band_stop_max = None

High cutoff frequency for the band-stop filter.

band_stop_min = None

Low cutoff frequency for the band-stop filter.

bandpass_filter = None

Apply a band-pass filter to the data.

bpf_order = None

Order of the band-pass filter.

combine_runs = None

Combine runs of the same task.

correlation_lengths = None

Produce correlation matrices limited to each requested amount of time. If this list includes ‘all’ then correlations from the full (censored) time series will be produced.

despike = None

Despike the BOLD data before postprocessing.

dummy_scans = None

Number of label-control volume pairs to delete before CBF computation.

fd_thresh = None

Framewise displacement threshold for censoring.

file_format = None

The file format to process. May be “auto”, “cifti”, or “nifti”.

head_radius = None

Radius of the head in mm.

high_pass = None

Lower bound of the band-pass filter.

input_type = None

Postprocessing pipeline type.

linc_qc = None

Run LINC QC.

low_pass = None

Upper bound of the band-pass filter.

min_coverage = None

Coverage threshold to apply to parcels in each atlas.

min_time = None

Post-scrubbing threshold to apply to individual runs in the dataset.

mode = None

Study- or pipeline-specific mode used to set other parameters.

motion_filter_order = None

Order of the filter to apply to the motion regressors.

motion_filter_type = None

Type of filter to apply to the motion regressors.

output_interpolated = None

Output interpolated data, not censored data.

output_run_wise_correlations = None

Output run-wise correlations.

process_surfaces = None

Warp fsnative-space surfaces to the MNI space.

skip_outputs = []

List of outputs to skip during postprocessing (e.g., ‘alff’, ‘reho’, ‘parcellation’, ‘connectivity’).

smoothing = None

Full-width at half-maximum (FWHM) of the smoothing kernel.

class xcp_d.config.nipype

Nipype settings.

crashfile_format = 'txt'

The file format for crashfiles, either text or pickle.

get_linked_libs = False

Run NiPype’s tool to enlist linked libraries for every interface.

classmethod get_plugin()

Format a dictionary for Nipype consumption.

classmethod init()

Set NiPype configurations.

memory_gb = None

Estimation in GB of the RAM this workflow can allocate at any given time.

nprocs = 2

Number of processes (compute tasks) that can be run in parallel (multiprocessing only).

omp_nthreads = None

Number of CPUs a single process can access for multithreaded execution.

plugin = 'MultiProc'

NiPype’s execution plugin.

plugin_args = {'maxtasksperchild': 1, 'raise_insufficient': False}

Settings for NiPype’s execution plugin.

resource_monitor = False

Enable resource monitor.

stop_on_first_crash = True

Whether the workflow should stop or continue after the first error.

Usage

A config file is used to pass settings and collect information as the execution graph is built across processes.

from xcp_d import config
config_file = config.execution.work_dir / '.xcp_d.toml'
config.to_filename(config_file)
# Call build_workflow(config_file, retval) in a subprocess
with Manager() as mgr:
    from xcp_d.cli.workflow import build_workflow
    retval = mgr.dict()
    p = Process(target=build_workflow, args=(str(config_file), retval))
    p.start()
    p.join()
config.load(config_file)
# Access configs from any code section as:
value = config.section.setting

Logging

class xcp_d.config.loggers

Keep loggers easily accessible (see init()).

cli = <Logger cli (WARNING)>

Command-line interface logging.

default = <RootLogger root (WARNING)>

The root logger.

classmethod init()

Set the log level, initialize all loggers into loggers.

• Add new logger levels (25: IMPORTANT, and 15: VERBOSE).

• Add a new sub-logger (cli).

• Logger configuration.

interface = <Logger nipype.interface (INFO)>

NiPype’s interface logger.

utils = <Logger nipype.utils (INFO)>

NiPype’s utils logger.

workflow = <Logger nipype.workflow (INFO)>

NiPype’s workflow logger.

Other responsibilities

The config is responsible for other conveniency actions.

• Switching Python’s multiprocessing to forkserver mode.

• Set up a filter for warnings as early as possible.

• Automated I/O magic operations. Some conversions need to happen in the store/load processes (e.g., from/to Path <-> str, BIDSLayout, etc.)

xcp_d.config.dismiss_hash(entities: list | None = None)

Set entities to dismiss in a DerivativesDataSink.

xcp_d.config.dumps()

Format config into toml.

class xcp_d.config.environment

Bases: _Config

Read-only options regarding the platform and environment.

Crawls runtime descriptive settings (e.g., default FreeSurfer license, execution environment, nipype and XCP-D versions, etc.). The environment section is not loaded in from file, only written out when settings are exported. This config section is useful when reporting issues, and these variables are tracked whenever the user does not opt-out using the --notrack argument.

cpu_count = 2

Number of available CPUs.

exec_docker_version = None

Version of Docker Engine.

exec_env = 'posix'

A string representing the execution platform.

free_mem = 5.9

Free memory at start.

nipype_version = '1.11.0'

Nipype’s current version.

overcommit_limit = '50%'

Linux’s kernel virtual memory overcommit limits.

overcommit_policy = 'heuristic'

Linux’s kernel virtual memory overcommit policy.

templateflow_version = '25.1.2'

The TemplateFlow client version installed.

version = '26.0.1'

XCP-D’s version.

class xcp_d.config.execution

Bases: _Config

Configure run-level settings.

aggr_ses_reports = None

Maximum number of sessions aggregated in one subject’s visual report.

atlases = []

Selection of atlases to apply to the data.

bids_database_dir = None

Path to the directory containing SQLite database indices for the input BIDS dataset.

bids_description_hash = None

Checksum (SHA256) of the dataset_description.json of the BIDS dataset.

bids_filters = None

A dictionary of BIDS selection filters.

boilerplate_only = None

Only generate a boilerplate.

confounds_config = None

Nuisance regressors to include in the postprocessing.

dataset_links = {}

A dictionary of dataset links to be used to track Sources in sidecars.

datasets = {}

Path(s) to search for other datasets (either derivatives or atlases).

debug = []

Debug mode(s).

fmri_dir = None

An existing path to the preprocessing derivatives dataset, which must be BIDS-compliant.

fs_license_file = None

An existing file containing a FreeSurfer license.

classmethod init()

Create a new BIDS Layout accessible with layout.

layout = None

A BIDSLayout object, see init().

log_dir = None

The path to a directory that contains execution logs.

log_level = 25

Output verbosity.

low_mem = None

Utilize uncompressed NIfTIs and other tricks to minimize memory allocation.

md_only_boilerplate = None

Do not convert boilerplate from MarkDown to LaTex and HTML.

notrack = None

Do not collect telemetry information for XCP-D.

output_dir = None

Folder where derivatives will be stored.

output_layout = None

Output layout for the derivatives.

parameters_hash = None

Unique identifier for this set of configurable parameters.

participant_label = None

List of participant identifiers that are to be preprocessed.

processing_list = []

List of (subject_id, anat_session_id, [func_session_id, …]) to be postprocessed.

report_output_level = None

Directory level at which the html reports should be written.

reports_only = None

Only build the reports, based on the reportlets found in a cached working directory.

run_uuid = '20260331-185743_ae87c778-c7fb-4d0b-9ce8-3d83f4cb0974'

Unique identifier of this particular run.

session_id = None

Select a particular session from all available in the dataset.

task_id = None

Select a particular task from all available in the dataset.

templateflow_home = PosixPath('/home/docs/.cache/templateflow')

The root folder of the TemplateFlow client.

work_dir = PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/xcp-d/checkouts/26.0.1/docs/work')

Path to a working directory where intermediate results will be available.

write_graph = None

Write out the computational graph corresponding to the planned preprocessing.

xcp_d.config.from_dict(settings, init=True, ignore=None)

Read settings from a flat dictionary.

Parameters:

• setting (dict) – Settings to apply to any configuration

• init (bool or Container) – Initialize all, none, or a subset of configurations.

• ignore (Container) – Collection of keys in setting to ignore

xcp_d.config.get(flat=False)

Get config as a dict.

xcp_d.config.hash_config(conf: dict[str, Any], *, fields_required: dict[str, list[str]] = {'execution': ['bids_filters', 'confounds_config', 'atlases', 'output_layout'], 'workflow': ['mode', 'file_format', 'dummy_scans', 'input_type', 'despike', 'smoothing', 'output_interpolated', 'combine_runs', 'motion_filter_type', 'band_stop_min', 'band_stop_max', 'motion_filter_order', 'head_radius', 'fd_thresh', 'min_time', 'bandpass_filter', 'high_pass', 'low_pass', 'bpf_order', 'min_coverage', 'correlation_lengths', 'process_surfaces', 'output_run_wise_correlations', 'abcc_qc', 'linc_qc']}, version: str | None = None, digest_size: int = 4) → str

Generate a unique BLAKE2b hash of configuration attributes. By default, uses a preselected list of workflow-altering parameters.

This will also grab the configuration hash from the fmri_dir’s dataset_description.json if it exists. If it does, it will be prefixed to the hash.

Parameters:

• conf (dict) – Configuration dictionary.

• fields_required (dict) – Dictionary of required fields for each level.

• version (str) – Version of the configuration.

• digest_size (int) – Size of the digest in bytes.

Returns:

A unique BLAKE2b hash of the configuration.

Return type:

str

xcp_d.config.load(filename, skip=None, init=True)

Load settings from file.

Parameters:

• filename (os.PathLike) – TOML file containing XCP-D configuration.

• skip (dict or None) – Sets of values to ignore during load, keyed by section name

• init (bool or Container) – Initialize all, none, or a subset of configurations.

class xcp_d.config.loggers

Bases: object

Keep loggers easily accessible (see init()).

cli = <Logger cli (WARNING)>

Command-line interface logging.

default = <RootLogger root (WARNING)>

The root logger.

classmethod init()

Set the log level, initialize all loggers into loggers.

• Add new logger levels (25: IMPORTANT, and 15: VERBOSE).

• Add a new sub-logger (cli).

• Logger configuration.

interface = <Logger nipype.interface (INFO)>

NiPype’s interface logger.

utils = <Logger nipype.utils (INFO)>

NiPype’s utils logger.

workflow = <Logger nipype.workflow (INFO)>

NiPype’s workflow logger.

class xcp_d.config.nipype

Bases: _Config

Nipype settings.

crashfile_format = 'txt'

The file format for crashfiles, either text or pickle.

get_linked_libs = False

Run NiPype’s tool to enlist linked libraries for every interface.

classmethod get_plugin()

Format a dictionary for Nipype consumption.

classmethod init()

Set NiPype configurations.

memory_gb = None

Estimation in GB of the RAM this workflow can allocate at any given time.

nprocs = 2

Number of processes (compute tasks) that can be run in parallel (multiprocessing only).

omp_nthreads = None

Number of CPUs a single process can access for multithreaded execution.

plugin = 'MultiProc'

NiPype’s execution plugin.

plugin_args = {'maxtasksperchild': 1, 'raise_insufficient': False}

Settings for NiPype’s execution plugin.

resource_monitor = False

Enable resource monitor.

stop_on_first_crash = True

Whether the workflow should stop or continue after the first error.

class xcp_d.config.seeds

Bases: _Config

Initialize the PRNG and track random seed assignments.

classmethod init()

Initialize a seeds object.

master = None

Master random seed to initialize the Pseudorandom Number Generator (PRNG).

xcp_d.config.to_filename(filename)

Write settings to file.

class xcp_d.config.workflow

Bases: _Config

Configure the particular execution graph of this workflow.

abcc_qc = None

Run DCAN QC.

band_stop_max = None

High cutoff frequency for the band-stop filter.

band_stop_min = None

Low cutoff frequency for the band-stop filter.

bandpass_filter = None

Apply a band-pass filter to the data.

bpf_order = None

Order of the band-pass filter.

combine_runs = None

Combine runs of the same task.

correlation_lengths = None

Produce correlation matrices limited to each requested amount of time. If this list includes ‘all’ then correlations from the full (censored) time series will be produced.

despike = None

Despike the BOLD data before postprocessing.

dummy_scans = None

Number of label-control volume pairs to delete before CBF computation.

fd_thresh = None

Framewise displacement threshold for censoring.

file_format = None

The file format to process. May be “auto”, “cifti”, or “nifti”.

head_radius = None

Radius of the head in mm.

high_pass = None

Lower bound of the band-pass filter.

classmethod init()

input_type = None

Postprocessing pipeline type.

linc_qc = None

Run LINC QC.

low_pass = None

Upper bound of the band-pass filter.

min_coverage = None

Coverage threshold to apply to parcels in each atlas.

min_time = None

Post-scrubbing threshold to apply to individual runs in the dataset.

mode = None

Study- or pipeline-specific mode used to set other parameters.

motion_filter_order = None

Order of the filter to apply to the motion regressors.

motion_filter_type = None

Type of filter to apply to the motion regressors.

output_interpolated = None

Output interpolated data, not censored data.

output_run_wise_correlations = None

Output run-wise correlations.

process_surfaces = None

Warp fsnative-space surfaces to the MNI space.

skip_outputs = []

List of outputs to skip during postprocessing (e.g., ‘alff’, ‘reho’, ‘parcellation’, ‘connectivity’).

smoothing = None

Full-width at half-maximum (FWHM) of the smoothing kernel.