plotting

Tasks to plot different types of histograms.

Classes:

PlotVariablesBase(*args, **kwargs)
PlotVariablesBaseSingleShift(*args, **kwargs)
PlotVariables1D(*args, **kwargs)
PlotVariables2D(*args, **kwargs)
PlotVariablesPerProcess2D(*args, **kwargs)
PlotVariablesBaseMultiShifts(*args, **kwargs)
PlotShiftedVariables1D(*args, **kwargs)
PlotShiftedVariablesPerProcess1D(*args, **kwargs)

class PlotVariablesBase(*args, **kwargs)

Bases: VariablePlotSettingMixin, ProcessPlotSettingMixin, CategoriesMixin, MLModelsMixin, WeightProducerMixin, ProducersMixin, SelectorStepsMixin, CalibratorsMixin, LocalWorkflow, RemoteWorkflow

Attributes:

sandbox	sandbox to use for this task.
exclude_index
reqs	Set upstream requirements, in this case MergeHistograms
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

Methods:

store_parts()	Create parts to create the output path to store intermediary results for the current Task.
create_branch_map()	Abstract method that must be overwritten by inheriting tasks to define the branch map.
workflow_requires()	Hook to add workflow requirements.
get_plot_shifts()
run()	The task run method, to be overridden in a subclass.

sandbox = 'bash::$CF_BASE/sandboxes/venv_columnar.sh'

sandbox to use for this task. Defaults to default_columnar_sandbox from analysis config.

exclude_index = True

reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'MergeHistograms': <class 'columnflow.tasks.histograms.MergeHistograms'>}

Set upstream requirements, in this case MergeHistograms

store_parts()

Create parts to create the output path to store intermediary results for the current Task.

Calls store_parts() of the super class and inserts {“producers”: “prod__{HASH}”} before keyword version. Here, HASH is the joint string of the first five producer names + a hash created with law.util.create_hash() based on the list of producers, starting at its 5th element (i.e. self.producers[5:]) For more information, see e.g. store_parts().

Returns:

Updated parts to create output path to store intermediary results.

create_branch_map()

Abstract method that must be overwritten by inheriting tasks to define the branch map.

workflow_requires()

Hook to add workflow requirements. This method is expected to return a dictionary. When this method is called from a branch task, an exception is raised.

abstract get_plot_shifts()

run()

The task run method, to be overridden in a subclass.

See Task.run

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow'}

exclude_params_remote_workflow = {}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotVariablesBaseSingleShift(*args, **kwargs)

Bases: PlotVariablesBase, ShiftTask

Attributes:

exclude_index
reqs	Set upstream requirements, in this case MergeHistograms
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

Methods:

create_branch_map()	Abstract method that must be overwritten by inheriting tasks to define the branch map.
requires()	The Tasks that this Task depends on.
output()	The output that this Task produces.
get_plot_shifts()

exclude_index = True

reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'MergeHistograms': <class 'columnflow.tasks.histograms.MergeHistograms'>}

Set upstream requirements, in this case MergeHistograms

create_branch_map()

Abstract method that must be overwritten by inheriting tasks to define the branch map.

requires()

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires

output()

The output that this Task produces.

The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single Target or a list of Target instances.

Implementation note

If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.

See Task.output

get_plot_shifts()

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow', 'local_shift'}

exclude_params_remote_workflow = {'local_shift'}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow', 'local_shift'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotVariables1D(*args, **kwargs)

Bases: PlotVariablesBaseSingleShift, PlotBase1D

Attributes:

plot_function
exclude_index
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

plot_function = <luigi.parameter.Parameter object>

exclude_index = False

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow', 'local_shift'}

exclude_params_remote_workflow = {'local_shift'}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow', 'local_shift'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotVariables2D(*args, **kwargs)

Bases: PlotVariablesBaseSingleShift, PlotBase2D

Attributes:

plot_function
exclude_index
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

plot_function = <luigi.parameter.Parameter object>

exclude_index = False

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow', 'local_shift'}

exclude_params_remote_workflow = {'local_shift'}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow', 'local_shift'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotVariablesPerProcess2D(*args, **kwargs)

Bases: WrapperTask, PlotVariables2D

Attributes:

workflow
exclude_index
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

Methods:

requires()

The Tasks that this Task depends on.

workflow = 'local'

requires()

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires

exclude_index = False

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow', 'local_shift'}

exclude_params_remote_workflow = {'local_shift'}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow', 'local_shift'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'local_shift', 'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotVariablesBaseMultiShifts(*args, **kwargs)

Bases: PlotVariablesBase, ShiftSourcesMixin

Attributes:

legend_title	Description: sets the title of the legend; when empty and only one process is present in the plot, the process_inst label is used; empty default
exclude_index
reqs	Set upstream requirements, in this case MergeHistograms
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

Methods:

create_branch_map()	Abstract method that must be overwritten by inheriting tasks to define the branch map.
requires()	The Tasks that this Task depends on.
output()	The output that this Task produces.
get_plot_shifts()
get_plot_parameters()

legend_title = <luigi.parameter.Parameter object>

Description: sets the title of the legend; when empty and only one process is present in the plot, the process_inst label is used; empty default

exclude_index = True

reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'MergeHistograms': <class 'columnflow.tasks.histograms.MergeHistograms'>, 'MergeShiftedHistograms': <class 'columnflow.tasks.histograms.MergeShiftedHistograms'>}

Set upstream requirements, in this case MergeHistograms

create_branch_map()

Abstract method that must be overwritten by inheriting tasks to define the branch map.

requires()

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires

output()

The output that this Task produces.

The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single Target or a list of Target instances.

Implementation note

If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.

See Task.output

get_plot_shifts()

get_plot_parameters()

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow'}

exclude_params_remote_workflow = {}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotShiftedVariables1D(*args, **kwargs)

Bases: PlotBase1D, PlotVariablesBaseMultiShifts

Attributes:

plot_function
exclude_index
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow

plot_function = <luigi.parameter.Parameter object>

exclude_index = False

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow'}

exclude_params_remote_workflow = {}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

class PlotShiftedVariablesPerProcess1D(*args, **kwargs)

Bases: WrapperTask, PlotShiftedVariables1D

Attributes:

exclude_index
exclude_params_branch
exclude_params_htcondor_workflow
exclude_params_index
exclude_params_remote_workflow
exclude_params_repr
exclude_params_repr_empty
exclude_params_req
exclude_params_req_get
exclude_params_req_set
exclude_params_sandbox
exclude_params_slurm_workflow
exclude_params_workflow
workflow
reqs	Set upstream requirements, in this case MergeHistograms

Methods:

requires()

The Tasks that this Task depends on.

exclude_index = False

exclude_params_branch = {'acceptance', 'branches', 'cancel_jobs', 'cleanup_jobs', 'htcondor_cpus', 'htcondor_flavor', 'htcondor_gpus', 'htcondor_logs', 'htcondor_memory', 'htcondor_pool', 'htcondor_scheduler', 'htcondor_share_software', 'ignore_submission', 'job_workers', 'max_runtime', 'no_poll', 'parallel_jobs', 'pilot', 'poll_fails', 'poll_interval', 'retries', 'shuffle_jobs', 'slurm_flavor', 'slurm_partition', 'submission_threads', 'tasks_per_job', 'tolerance', 'transfer_logs', 'walltime'}

exclude_params_htcondor_workflow = {}

exclude_params_index = {'effective_workflow'}

exclude_params_remote_workflow = {}

exclude_params_repr = {'cancel_jobs', 'cleanup_jobs', 'workflow'}

exclude_params_repr_empty = {'ml_models', 'selector_steps'}

exclude_params_req = {'effective_workflow'}

exclude_params_req_get = {}

exclude_params_req_set = {}

exclude_params_sandbox = {'log_file', 'sandbox'}

exclude_params_slurm_workflow = {}

exclude_params_workflow = {'branch'}

workflow = 'local'

reqs = {'BuildBashSandbox': <class 'columnflow.tasks.framework.remote.BuildBashSandbox'>, 'BundleBashSandbox': <class 'columnflow.tasks.framework.remote.BundleBashSandbox'>, 'BundleCMSSWSandbox': <class 'columnflow.tasks.framework.remote.BundleCMSSWSandbox'>, 'BundleRepo': <class 'columnflow.tasks.framework.remote.BundleRepo'>, 'BundleSoftware': <class 'columnflow.tasks.framework.remote.BundleSoftware'>, 'MergeHistograms': <class 'columnflow.tasks.histograms.MergeHistograms'>, 'MergeShiftedHistograms': <class 'columnflow.tasks.histograms.MergeShiftedHistograms'>, 'PlotShiftedVariables1D': <class 'columnflow.tasks.plotting.PlotShiftedVariables1D'>}

Set upstream requirements, in this case MergeHistograms

requires()

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires