cutflow
#
Tasks to be implemented: MergeSelectionMasks, PlotCutflow
Classes:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- class CreateCutflowHistograms(*args, **kwargs)[source]#
Bases:
VariablesMixin
,SelectorStepsMixin
,CalibratorsMixin
,ChunkedIOMixin
,DatasetTask
,LocalWorkflow
,RemoteWorkflow
Attributes:
Methods:
Define the branch map for when this task is used as a workflow.
Hook to add workflow requirements.
requires
()The Tasks that this Task depends on.
output
()The output that this Task produces.
run
()The task run method, to be overridden in a subclass.
- sandbox = 'bash::$CF_BASE/sandboxes/venv_columnar.sh'#
- selector_steps_order_sensitive = True#
- initial_step = 'Initial'#
- default_variables = ('event', 'cf_*')#
- reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'MergeSelectionMasks': <class 'columnflow.tasks.selection.MergeSelectionMasks'>}#
- missing_column_alias_strategy = 'original'#
- create_branch_map()[source]#
Define the branch map for when this task is used as a workflow. By default, use the merging information provided by
file_merging_factor
to return a dictionary which maps branches to one or more input file indices. E.g. 1 -> [3, 4, 5] would mean that branch 1 is simultaneously handling input file indices 3, 4 and 5.
- workflow_requires()[source]#
Hook to add workflow requirements. This method is expected to return a dictionary. When this method is called from a branch task, an exception is raised.
- requires()[source]#
The Tasks that this Task depends on.
A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.
See Task.requires
- output()[source]#
The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
- exclude_index = False#
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'check_finite_output', 'check_overlapping_inputs', 'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- class CreateCutflowHistogramsWrapper(*args, **kwargs)#
Bases:
AnalysisTask
,WrapperTask
Attributes:
Methods:
requires
()Collect requirements defined by the underlying
require_cls
of theWrapperTask
depending on optional additional parameters.update_wrapper_params
(params)- configs = <law.parameter.CSVParameter object>#
- datasets = <law.parameter.CSVParameter object>#
- exclude_index = False#
- exclude_params_index = {}#
- exclude_params_repr = {}#
- exclude_params_repr_empty = {}#
- exclude_params_req = {}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'log_file', 'sandbox'}#
- requires() Requirements #
Collect requirements defined by the underlying
require_cls
of theWrapperTask
depending on optional additional parameters.- Return type:
- Returns:
Requirements for the
WrapperTask
instance.
- shifts = <law.parameter.CSVParameter object>#
- skip_configs = <law.parameter.CSVParameter object>#
- skip_datasets = <law.parameter.CSVParameter object>#
- skip_shifts = <law.parameter.CSVParameter object>#
- update_wrapper_params(params)#
- class PlotCutflowBase(*args, **kwargs)[source]#
Bases:
SelectorStepsMixin
,CategoriesMixin
,CalibratorsMixin
,PlotBase
,ShiftTask
,LocalWorkflow
,RemoteWorkflow
Attributes:
Methods:
Create parts to create the output path to store intermediary results for the current
Task
.- sandbox = 'bash::$CF_BASE/sandboxes/venv_columnar.sh'#
- exclude_index = True#
- selector_steps_order_sensitive = True#
- reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'CreateCutflowHistograms': <class 'columnflow.tasks.cutflow.CreateCutflowHistograms'>}#
- store_parts()[source]#
Create parts to create the output path to store intermediary results for the current
Task
.Calls
store_parts()
of thesuper
class and inserts {“selector”: “__steps__LIST_OF_STEPS”}, whereLIST_OF_STEPS
is the sorted list of selector steps. For more information, see e.g.store_parts()
.- Returns:
Updated parts to create output path to store intermediary results.
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- class PlotCutflow(*args, **kwargs)[source]#
Bases:
PlotCutflowBase
,PlotBase1D
,ProcessPlotSettingMixin
,LocalWorkflow
,RemoteWorkflow
Attributes:
Methods:
Abstract method that must be overwritten by inheriting tasks to define the branch map.
Hook to add workflow requirements.
requires
()The Tasks that this Task depends on.
output
()The output that this Task produces.
run
()The task run method, to be overridden in a subclass.
- plot_function = <luigi.parameter.Parameter object>#
- variable = <luigi.parameter.Parameter object>#
- reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'CreateCutflowHistograms': <class 'columnflow.tasks.cutflow.CreateCutflowHistograms'>}#
- create_branch_map()[source]#
Abstract method that must be overwritten by inheriting tasks to define the branch map.
- workflow_requires()[source]#
Hook to add workflow requirements. This method is expected to return a dictionary. When this method is called from a branch task, an exception is raised.
- requires()[source]#
The Tasks that this Task depends on.
A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.
See Task.requires
- output()[source]#
The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
- exclude_index = False#
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- class PlotCutflowWrapper(*args, **kwargs)#
Bases:
AnalysisTask
,WrapperTask
Attributes:
Methods:
requires
()Collect requirements defined by the underlying
require_cls
of theWrapperTask
depending on optional additional parameters.update_wrapper_params
(params)- configs = <law.parameter.CSVParameter object>#
- exclude_index = False#
- exclude_params_index = {}#
- exclude_params_repr = {}#
- exclude_params_repr_empty = {}#
- exclude_params_req = {}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'log_file', 'sandbox'}#
- requires() Requirements #
Collect requirements defined by the underlying
require_cls
of theWrapperTask
depending on optional additional parameters.- Return type:
- Returns:
Requirements for the
WrapperTask
instance.
- shifts = <law.parameter.CSVParameter object>#
- skip_configs = <law.parameter.CSVParameter object>#
- skip_shifts = <law.parameter.CSVParameter object>#
- update_wrapper_params(params)#
- class PlotCutflowVariablesBase(*args, **kwargs)[source]#
Bases:
VariablePlotSettingMixin
,ProcessPlotSettingMixin
,PlotCutflowBase
,LocalWorkflow
,RemoteWorkflow
Attributes:
Methods:
Abstract method that must be overwritten by inheriting tasks to define the branch map.
Hook to add workflow requirements.
requires
()The Tasks that this Task depends on.
output
()The output that this Task produces.
run_postprocess
(hists, category_inst, ...)run
()The task run method, to be overridden in a subclass.
- only_final_step = <luigi.parameter.BoolParameter object>#
- initial_step = 'Initial'#
- default_variables = ('cf_*',)#
- exclude_index = True#
- reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'CreateCutflowHistograms': <class 'columnflow.tasks.cutflow.CreateCutflowHistograms'>}#
- create_branch_map()[source]#
Abstract method that must be overwritten by inheriting tasks to define the branch map.
- workflow_requires()[source]#
Hook to add workflow requirements. This method is expected to return a dictionary. When this method is called from a branch task, an exception is raised.
- requires()[source]#
The Tasks that this Task depends on.
A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.
See Task.requires
- abstract output()[source]#
The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- class PlotCutflowVariables1D(*args, **kwargs)[source]#
Bases:
PlotCutflowVariablesBase
,PlotBase1D
Attributes:
Methods:
output
()The output that this Task produces.
run_postprocess
(hists, category_inst, ...)- plot_function = <luigi.parameter.Parameter object>#
- plot_function_processes = 'columnflow.plotting.plot_functions_1d.plot_variable_per_process'#
- plot_function_steps = 'columnflow.plotting.plot_functions_1d.plot_variable_variants'#
- per_plot = <luigi.parameter.ChoiceParameter object>#
- output()[source]#
The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
- exclude_index = False#
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- class PlotCutflowVariables2D(*args, **kwargs)[source]#
Bases:
PlotCutflowVariablesBase
,PlotBase2D
Attributes:
Methods:
output
()The output that this Task produces.
run_postprocess
(hists, category_inst, ...)- plot_function = <luigi.parameter.Parameter object>#
- output()[source]#
The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
- exclude_index = False#
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#
- class PlotCutflowVariablesPerProcess2D(*args, **kwargs)[source]#
Bases:
WrapperTask
,PlotCutflowVariables2D
Attributes:
Methods:
requires
()The Tasks that this Task depends on.
- workflow = 'local'#
- reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'CreateCutflowHistograms': <class 'columnflow.tasks.cutflow.CreateCutflowHistograms'>, 'PlotCutflowVariables2D': <class 'columnflow.tasks.cutflow.PlotCutflowVariables2D'>}#
- requires()[source]#
The Tasks that this Task depends on.
A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.
See Task.requires
- exclude_index = False#
- exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}#
- exclude_params_htcondor_workflow = {}#
- exclude_params_index = {'effective_workflow', 'local_shift'}#
- exclude_params_remote_workflow = {'local_shift'}#
- exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}#
- exclude_params_repr_empty = {'selector_steps'}#
- exclude_params_req = {'effective_workflow', 'local_shift'}#
- exclude_params_req_get = {}#
- exclude_params_req_set = {}#
- exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}#
- exclude_params_slurm_workflow = {}#
- exclude_params_workflow = {'branch'}#