production
#
Tasks related to producing new columns.
Classes:
|
|
alias of |
|
|
- class ProduceColumns(*args, **kwargs)[source]#
Bases:
ProducerMixin
,SelectorStepsMixin
,CalibratorsMixin
,ChunkedIOMixin
,MergeReducedEventsUser
,LocalWorkflow
,RemoteWorkflow
Attributes:
Methods:
Hook to add workflow requirements.
requires
()The Tasks that this Task depends on.
output
()The output that this Task produces.
run
()The task run method, to be overridden in a subclass.
- sandbox = 'bash::$CF_BASE/sandboxes/venv_columnar.sh'#
- reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'MergeReducedEvents': <class 'columnflow.tasks.reduction.MergeReducedEvents'>, 'MergeReductionStats': <class 'columnflow.tasks.reduction.MergeReductionStats'>}#
- register_producer_shifts = True#
- missing_column_alias_strategy = 'original'#
- workflow_requires()[source]#
Hook to add workflow requirements. This method is expected to return a dictionary. When this method is called from a branch task, an exception is raised.
- requires()[source]#
The Tasks that this Task depends on.
A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.
See Task.requires
- output()[source]#
The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
- check_finite_output = <luigi.parameter.BoolParameter object>#
- check_overlapping_inputs = <luigi.parameter.BoolParameter object>#
- exclude_index = False#
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'check_finite_output', 'check_overlapping_inputs', 'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- ProduceColumnsWrapperBase#
alias of
Wrapper
Attributes:configs
datasets
exclude_index
exclude_params_index
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
shifts
skip_configs
skip_datasets
skip_shifts
Methods:
requires
()Collect requirements defined by the underlying
require_cls
of theWrapperTask
depending on optional additional parameters.update_wrapper_params
(params)
- class ProduceColumnsWrapper(*args, **kwargs)[source]#
Bases:
Wrapper
,ProducersMixin
Attributes:
- exclude_index = False#
- exclude_params_index = {}#
- exclude_params_repr = {}#
- exclude_params_repr_empty = {}#
- exclude_params_req = {}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'log_file', 'sandbox'}#