columnflow.util

Collection of general helpers and utilities.

Summary

UNSET	Placeholder for an unset value.
env_is_remote	bool(x) -> bool
env_is_dev	bool(x) -> bool
primes	List of the first 100 primes.
maybe_import(name[, package, force])	Calls importlib.import_module internally and returns the module if it exists, or otherwise a MockModule instance with the same name.
import_plt()	Lazily imports and configures matplotlib pyplot.
import_ROOT()	Lazily imports and configures ROOT.
import_file(path[, attr])	Loads the content of a python file located at path and returns its package content as a dictionary.
create_random_name()	Returns a random string based on UUID v4.
expand_path(*path)	Takes path fragments, joins them and recursively expands all contained environment variables.
real_path(*path)	Takes path fragments and returns the real, absolute location with all variables expanded.
ensure_dir(path)	Ensures that a directory at path (and its subdirectories) exists and returns the full, expanded path.
wget(src, dst[, force])	Downloads a file from a remote src to a local destination dst, creating intermediate directories when needed.
call_thread(func[, args, kwargs, timeout])	Execute a function func in a thread and aborts the call when timeout is reached.
call_proc(func[, args, kwargs, timeout])	Execute a function func in a process and aborts the call when timeout is reached.
ensure_proxy(fn, opts, task, *args, **kwargs)	Law task decorator that checks whether either a voms or arc proxy is existing before calling the decorated method.
dev_sandbox(sandbox[, add, remove])	Takes a sandbox key sandbox and adds or removes the substring "_dev" right before the file extension (if any), depending on whether the current environment is used for development (see env_is_dev) and the add and remove flags.
safe_div(a, b)	Returns a divided by b if b is not zero, and zero otherwise.
try_float(f)	Tests whether a value f can be converted to a float.
is_pattern(s)	Returns True if a string s contains pattern characters such as "*" or "?", and False otherwise.
is_regex(s)	Returns True if a string s is a regular expression starting with "^" and ending with "$", and False otherwise.
pattern_matcher(pattern[, mode])	Takes a string pattern which might be an actual pattern for fnmatching, a regular expressions or just a plain string and returns a function that can be used to test of a string matches that pattern.
dict_add_strict(d, key, value)	Adds key-value pair to dictionary, but only if it does not change an existing value; Raises KeyError otherwise.
get_source_code(obj[, indent])	Returns the source code of any object obj as a string.
DotDict	Subclass of OrderedDict that provides read and write access to items via attributes by implementing __getattr__ and __setattr__.
MockModule(name)	Mockup object that resembles a module with arbitrarily deep structure such that, e.g.,
FunctionArgs(*args, **kwargs)	Light-weight utility class that wraps all passed args and kwargs and allows to invoke different functions with them.
ClassPropertyDescriptor(fget[, fset])	Generic descriptor class that is used by classproperty().
classproperty(func)	Propety decorator for class-level methods.
DerivableMeta(cls_name, bases, cls_dict)	Meta class for Derivable objects providing class-level features such as improved tracing and lookup of subclasses, and single-line subclassing for partial-like overwriting of class-level attributes.
Derivable()	Derivable base class with features provided by the meta DerivableMeta.

Attributes

UNSET

util.UNSET = <object object>

env_is_remote

util.env_is_remote = False

env_is_dev

util.env_is_dev = False

primes

util.primes = [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 541]

Functions

maybe_import

maybe_import(name, package=None, force=False)

Calls importlib.import_module internally and returns the module if it exists, or otherwise a MockModule instance with the same name. When force is True and the import fails, an ImportError is raised.

Return type:

ModuleType | MockModule

import_plt

import_plt()

Lazily imports and configures matplotlib pyplot.

Return type:

ModuleType

import_ROOT

import_ROOT()

Lazily imports and configures ROOT.

Return type:

ModuleType

import_file

import_file(path, attr=None)

Loads the content of a python file located at path and returns its package content as a dictionary. When attr is set, only the attribute with that name is returned.

The file is not required to be importable as its content is loaded directly into the interpreter. While this approach is not necessarily clean, it can be useful in places where custom code must be loaded.

create_random_name

create_random_name()

Returns a random string based on UUID v4.

Return type:

str

expand_path

expand_path(*path)

Takes path fragments, joins them and recursively expands all contained environment variables.

Return type:

str

real_path

real_path(*path)

Takes path fragments and returns the real, absolute location with all variables expanded.

Return type:

str

ensure_dir

ensure_dir(path)

Ensures that a directory at path (and its subdirectories) exists and returns the full, expanded path.

Return type:

str

wget

wget(src, dst, force=False)

Downloads a file from a remote src to a local destination dst, creating intermediate directories when needed. When dst refers to an existing file, an exception is raised unless force is True.

The full, normalized destination path is returned.

Return type:

str

call_thread

call_thread(func, args=(), kwargs=None, timeout=None)

Execute a function func in a thread and aborts the call when timeout is reached. args and kwargs are forwarded to the function.

The return value is a 3-tuple (finsihed_in_time, func(), err).

Return type:

tuple[bool, Any, str | None]

call_proc

call_proc(func, args=(), kwargs=None, timeout=None)

Execute a function func in a process and aborts the call when timeout is reached. args and kwargs are forwarded to the function.

The return value is a 3-tuple (finsihed_in_time, func(), err).

Return type:

tuple[bool, Any, str | None]

ensure_proxy

ensure_proxy(fn, opts, task, *args, **kwargs)

Law task decorator that checks whether either a voms or arc proxy is existing before calling the decorated method.

Return type:

tuple[Callable, Callable, Callable]

dev_sandbox

dev_sandbox(sandbox, add=True, remove=True)

Takes a sandbox key sandbox and adds or removes the substring “_dev” right before the file extension (if any), depending on whether the current environment is used for development (see env_is_dev) and the add and remove flags.

If sandbox does not contain the “_dev” postfix and both env_is_dev and add are True, the postfix is appended.

If sandbox does (!) contain the “_dev” postfix, env_is_dev is False and remove is True, the postfix is removed.

In any other case, sandbox is returned unchanged.

Examples:

# if env_is_dev and /path/to/script_dev.sh exists
dev_sandbox("bash::/path/to/script.sh")
# -> "bash::/path/to/script_dev.sh"

# otherwise
dev_sandbox("bash::/path/to/script.sh")
# -> "bash::/path/to/script.sh"

Return type:

str

safe_div

safe_div(a, b)

Returns a divided by b if b is not zero, and zero otherwise.

Return type:

float

try_float

try_float(f)

Tests whether a value f can be converted to a float.

Return type:

bool

is_pattern

is_pattern(s)

Returns True if a string s contains pattern characters such as “*” or “?”, and False otherwise.

Return type:

bool

is_regex

is_regex(s)

Returns True if a string s is a regular expression starting with “^” and ending with “$”, and False otherwise.

Return type:

bool

pattern_matcher

pattern_matcher(pattern, mode=<built-in function any>)

Takes a string pattern which might be an actual pattern for fnmatching, a regular expressions or just a plain string and returns a function that can be used to test of a string matches that pattern.

When pattern is a sequence, all its patterns are compared the same way and the result is the combination given a mode which typically should be any or all.

Example:

matcher = pattern_matcher("foo*")
matcher("foo123")  # -> True
matcher("bar123")  # -> False

matcher = pattern_matcher(r"^foo\d+.*$")
matcher("foox")  # -> False
matcher("foo1")  # -> True

matcher = pattern_matcher(("foo*", "*bar"), mode=any)
matcher("foo123")  # -> True
matcher("123bar")  # -> True

matcher = pattern_matcher(("foo*", "*bar"), mode=all)
matcher("foo123")     # -> False
matcher("123bar")     # -> False
matcher("foo123bar")  # -> True

Return type:

Callable[[str], bool]

dict_add_strict

dict_add_strict(d, key, value)

Adds key-value pair to dictionary, but only if it does not change an existing value; Raises KeyError otherwise.

Return type:

None

get_source_code

get_source_code(obj, indent=None)

Returns the source code of any object obj as a string. When indent is not None, the code indentation is first removed and then re-applied with indent if it is a string, or by that many spaces in case it is an integer.

Return type:

str

classproperty

classproperty(func)

Propety decorator for class-level methods.

Return type:

ClassPropertyDescriptor

Classes

DotDict

class DotDict

Bases: OrderedDict

Subclass of OrderedDict that provides read and write access to items via attributes by implementing __getattr__ and __setattr__. In case a item is accessed via attribute and it does not exist, an AttriuteError is raised rather than a KeyError. Example:

d = DotDict()
d["foo"] = 1

print(d["foo"])
# => 1

print(d.foo)
# => 1

print(d["bar"])
# => KeyError

print(d.bar)
# => AttributeError

d.bar = 123
print(d.bar)
# => 123

# use wrap() to convert a nested dict
d = DotDict({"foo": {"bar": 1}})
print(d.foo.bar)
# => 1

__setattr__(attr, value)

Implement setattr(self, name, value).

Return type:

None

classmethod wrap(*args, **kwargs)

Takes a dictionary d and recursively replaces it and all other nested dictionary types with DotDict’s for deep attribute-style access.

Return type:

DotDict

MockModule

class MockModule(name)

Bases: object

Mockup object that resembles a module with arbitrarily deep structure such that, e.g.,

coffea = MockModule("coffea")
print(coffea.nanoevents.NanoEventsArray)
# -> "<MockupModule 'coffea' at 0x981jald1>"

will always succeed at declaration, but most likely fail at execution time. In fact, each attribute access will return the mock object again. This might only be useful in places where a module is potentially not existing (e.g. due to sandboxing) but one wants to import it either way a) to perform only one top-level import as opposed to imports in all functions of a package, or b) to provide type hints for documentation purposes.

_name

type: str

The name of the mock module.

FunctionArgs

class FunctionArgs(*args, **kwargs)

Bases: object

Light-weight utility class that wraps all passed args and kwargs and allows to invoke different functions with them.

ClassPropertyDescriptor

class ClassPropertyDescriptor(fget, fset=None)

Bases: object

Generic descriptor class that is used by classproperty().

__init__(fget, fset=None)

__weakref__

list of weak references to the object (if defined)

DerivableMeta

class DerivableMeta(cls_name: str, bases: tuple, cls_dict: dict)

Bases: ABCMeta

Meta class for Derivable objects providing class-level features such as improved tracing and lookup of subclasses, and single-line subclassing for partial-like overwriting of class-level attributes.

static __new__(metacls, cls_name, bases, cls_dict)

Class creation.

Return type:

DerivableMeta

has_cls(cls_name, deep=True)

Returns True if this class has a subclass named cls_name and False otherwise. When deep is True, the lookup is recursive through all levels of subclasses.

Return type:

bool

get_cls(cls_name, deep=True, silent=False)

Returns a previously created subclass named cls_name.

When deep is True, the lookup is recursive through all levels of subclasses. When no such subclass was found an exception is raised, unless silent is True in which case None is returned.

Parameters:

• cls_name (str) – Name of the subclass to load

• deep (bool, default: True) – Search for the subclass cls_name throughout the whole inheritance tree of this class (True) or just in the direct inheritance line (False)

• silent (bool, default: False) – If True, raise an error if no subclass cls_name was found, otherwise return None

Raises:

• ValueError – If deep is False and the name cls_name is not found in the direct line of subclasses of this class

• ValueError – If deep is True and the name cls_name is not found at any level of the inheritance tree starting at this class

Return type:

DerivableMeta | None

Returns:

The requested subclass

derive(cls_name, bases=(), cls_dict=None, module=None)

Creates a subclass named cls_name inheriting from this class an additional, optional bases.

cls_dict will be attached as class-level attributes.

Parameters:

• cls_name (str) – Name of the newly-derived class

• bases (tuple, default: ()) – Additional bases to derive new class from

• cls_dict (Optional[dict], default: None) – Dictionary to forward to init function of derived class

• module (Optional[str], default: None) – extract module name from this module

Return type:

DerivableMeta

Returns:

Newly derived class instance

derived_by(other)

Returns if a class other is either this or derived from this class, and False otherwise.

Return type:

bool

Derivable

class Derivable

Bases: object

Derivable base class with features provided by the meta DerivableMeta.

classattribute cls_name

type: str read-only

A shorthand to access the name of the class.

__weakref__

list of weak references to the object (if defined)