columnflow.tasks.external

Tasks dealing with external data.

Classes:

GetDatasetLFNs(*args, **kwargs)	Task to get list of logical file names (LFNs).
GetDatasetLFNsWrapper(*args, **kwargs)
BundleExternalFiles(*args, **kwargs)	Task to collect external files.

class GetDatasetLFNs(*args, **kwargs)

Bases: DatasetTask, TransferLocalFile

Task to get list of logical file names (LFNs).

Attributes:

version
sandbox	Defines sandbox for this task.

Methods:

resolve_param_values(params)	Resolve parameter values params from command line and propagate them to this set of parameters.
single_output()	Creates a remote target file for the final .json file containing the list of LFNs.
run()	Run function for this task.
get_dataset_lfns_dasgoclient(dataset_inst, ...)	Get the LNF information with the dasgoclient.
iter_nano_files(task[, remote_fs, ...])	Generator function that reduces the boilerplate code for looping over files referred to by lfn_indices given the lfns obtained by this task which needs to be complete for this function to succeed.

version = None

classmethod resolve_param_values(params)

Resolve parameter values params from command line and propagate them to this set of parameters.

Parameters:

params (DotDict) – Parameters provided at command line level.

Return type:

DotDict

Returns:

Updated list of parameter values.

property sandbox: str

Defines sandbox for this task.

Returns:

Path to shell script that sets up the requested sandbox.

single_output()

Creates a remote target file for the final .json file containing the list of LFNs.

Return type:

FileSystemFileTarget

Returns:

Law remote target with the initialized output name

run()

Run function for this task.

Raises:

ValueError – If number of loaded LFNs does not correspond to number of LFNs specified in this dataset_info_inst.

get_dataset_lfns_dasgoclient(dataset_inst, shift_inst, dataset_key)

Get the LNF information with the dasgoclient.

Parameters:

• dataset_inst (Dataset) – Current dataset instance, currently not used.

• shift_inst (Shift) – Current shift instance, currently not used.

• dataset_key (str) – DAS key identifier for the current dataset.

Raises:

Exception – If query with dasgoclient fails.

Return type:

list[str]

Returns:

The list of LFNs corresponding to the dataset with the identifier dataset_key.

iter_nano_files(task, remote_fs=None, lfn_indices=None, eager_lookup=1)

Generator function that reduces the boilerplate code for looping over files referred to by lfn_indices given the lfns obtained by this task which needs to be complete for this function to succeed.

When lfn_indices are not given, task must be a branch of a DatasetTask workflow whose branch value is used instead.

Parameters:

• task (AnalysisTask | DatasetTask) – Current task that needs to access the nanoAOD files

• remote_fs (str | Sequence[str] | None, default: None) – Name of the remote file system where the LFNs are located, defaults to None

• lfn_indices (list[int] | None, default: None) – List of indices of LFNs that are processed by this task instance, defaults to None

• eager_lookup (bool | int, default: 1) – Look at the next remote fs in remote_fs if stat takes too long, defaults to 1

Raises:

• TypeError – If task is not of type BaseWorkflow or not a task analyzing a single branch in the task tree

• Exception – If current task is not complete as indicated with self.complete()

• ValueError – If no remote fs is provided at call and none can be found in either the config instance or the law config.

• Exception – If a given LFN cannot be found at any remote file system

Yield:

a file target that points to a LFN

Return type:

None

class GetDatasetLFNsWrapper(*args, **kwargs)

Bases: AnalysisTask, WrapperTask

Methods:

requires()

The Tasks that this Task depends on.

Attributes:

version

requires()

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires

version = None

class BundleExternalFiles(*args, **kwargs)

Bases: ConfigTask, TransferLocalFile

Task to collect external files.

This task is intended to download source files for other tasks, such as files containing corrections for objects, the “golden” json files, source files for the calculation of pileup weights, and others.

All information about the relevant external files is extracted from the given config_inst, which must contain the keyword external_files in the auxiliary information. This can look like this:

# cfg is the current config instance
cfg.x.external_files = DotDict.wrap({
# The following assumes that the zip files are reachable under the
# url ``SOURCE_URL``
# jet energy correction
"jet_jerc": (f"{SOURCE_URL}/POG/JME/{year}{corr_postfix}_UL/jet_jerc.json.gz", "v1"),

# tau energy correction and scale factors
"tau_sf": (f"{SOURCE_URL}/POG/TAU/{year}{corr_postfix}_UL/tau.json.gz", "v1"),

# electron scale factors
"electron_sf": (f"{SOURCE_URL}/POG/EGM/{year}{corr_postfix}_UL/electron.json.gz", "v1"),

The entries in this DotDict can either be simply the path to the source files or can be a tuple of the format (path/or/url/to/source/file, VERSION) to introduce a versioning mechanism for external files.

Attributes:

version
files_hash	Create a hash based on all external files.
file_names	Create a unique basename for each external file.

Methods:

create_unique_basename(path)	Create a unique basename.
run()	The task run method, to be overridden in a subclass.

version = None

classmethod create_unique_basename(path)

Create a unique basename.

Parameters:

path (tuple[str] | str) – path to create a unique basename for

Return type:

str

Returns:

Unique basename

property files_hash: str

Create a hash based on all external files.

Returns:

Hash based on the flattened list of external files in the current config instance.

property file_names: DotDict

Create a unique basename for each external file.

Returns:

DotDict of same shape as external_files DotDict with unique basenames.

run()

The task run method, to be overridden in a subclass.

See Task.run