Code Documentation

Created on Thu Feb 10 12:45:21 2022

@author: khs3z

class flim.plugin.ABC

Helper class that provides a standard way to create an ABC using inheritance.

class flim.plugin.AbstractPlugin(name: str = 'flim.plugin', input: Dict[str, Any] = <built-in function input>, input_select: str = None, **kwargs)

Abstract class used to template plugins for data manipulation, analysis, plotting.

configure(input: dict = <built-in function input>, input_select: Optional[str] = None, **kwargs)

Updates the configuration with the passed arguments.

The configuration is updated not replaced, i.e values of matching keys are overwritten, values of non-matching keys remain unaltered.

Parameters

kwargs – the new key:value pairs.

abstract execute()

Executes the analysis using the analyzer’s set input and configuration.

Returns

Dictionary of pandas.DataFrame and/or matplotlib.pyplot.Figure objects.

Keys represent window titles, values represent the DataFrame or Fugure objects.

Return type

dict

get_config_name() → str

Returns a string copy of plugin’s name (self.name) in which whitespaces and non alphanumeric characters have been removed. It’s used to defines the plugin’s identifier in the config file.

Returns

cleaned up plugin name

Return type

str

get_default_parameters() → dict

Provides the plugin’s default parameters.

Returns

default parameters

Return type

dict

get_description() → str

Returns a description for this analyzing module. This may include instructions on how to use the parameters.

Returns

the description.

Return type

str

get_icon() → Any

Returns icon for this analysis.

Returns

The bitmap of the icon.

Return type

wx.Bitmap

get_mapped_parameters() → List[Dict[str, Any]]

Provides a list of the plugins current parameters. Each list item defines a parameters for the smallest independent work unit. The list can be mapped for parallel flow execution.

Returns

list of current parameters

Return type

list[dict]

get_parameters() → dict

Defines the plugins current parameters.

Returns

current parameters

Return type

dict

abstract get_required_categories() → List[str]

Returns the category column names that are required in the input to be analyzed.

Category columns use ‘category’ as dtype in Pandas DataFrame.

Returns

list of column names.

Return type

list(str)

abstract get_required_features() → List[str]

Returns the non-category column names that are required in the input to be analyzed.

Non-category columns are all those columns that do not use ‘category’ as dtype in Pandas dataframe.

Returns

list of column names.

Return type

list(str)

input_definition() → List[Type]

Provides type definition of plugin’s required input.

Returns

list of object types

Return type

list[type]

output_definition() → Dict[str, Type]

Provides type definition of plugin’s execute method.

Returns

keys describe output object labels; values represent corresponding

object types

Return type

dict[type]

run(input={}, input_select=None, **kwargs)

The run() method is called (with arguments, if appropriate) to run a task.

Note: The implemented run method cannot have *args in its signature. In addition, the following keywords are reserved: upstream_tasks, task_args and mapped.

If a task has arguments in its run() method, these can be bound either by using the functional API and _calling_ the task instance, or by using self.bind directly.

In addition to running arbitrary functions, tasks can interact with Prefect in a few ways: <ul><li> Return an optional result. When this function runs successfully,

the task is considered successful and the result (if any) can be made available to downstream tasks. </li>

<li> Raise an error. Errors are interpreted as failure. </li> <li> Raise a [signal](../engine/signals.html). Signals can include FAIL, SUCCESS,

RETRY, SKIP, etc. and indicate that the task should be put in the indicated state.

<ul> <li> FAIL will lead to retries if appropriate </li> <li> SUCCESS will cause the task to be marked successful </li> <li> RETRY will cause the task to be marked for retry, even if max_retries

has been exceeded </li>

<li> SKIP will skip the task and possibly propogate the skip state through the

flow, depending on whether downstream tasks have skip_on_upstream_skip=True.

</li></ul>

</li></ul>

run_configuration_dialog(parent: Any, data_choices: Dict[str, Any] = {}) → Dict[str, Any]

Executes the plugin’s configuration dialog.

The dialog is initialized with values of the analyzer’s Config object.

Parameters

  • parent – parent GUI element

  • data_choices (dict) – available data tables to choose from. Keys correspond to table names; values correspond to DataFrame objects.

Returns

The key:value pairs of specified config parameters.

Return type

dict

set_input(input: dict, input_select: list = []) → dict
validate()

Validates the parameters currently set for the plugin.

Returns

True if set parameters satisfy plugin’s requirements

Return type

boolean

class flim.plugin.DataBucket(name='Data Bucket', **kwargs)
execute()

Executes the analysis using the analyzer’s set input and configuration.

Returns

Dictionary of pandas.DataFrame and/or matplotlib.pyplot.Figure objects.

Keys represent window titles, values represent the DataFrame or Fugure objects.

Return type

dict

output_definition()

Provides type definition of plugin’s execute method.

Returns

keys describe output object labels; values represent corresponding

object types

Return type

dict[type]

run(input={}, input_select=None, name=None, **kwargs)

The run() method is called (with arguments, if appropriate) to run a task.

Note: The implemented run method cannot have *args in its signature. In addition, the following keywords are reserved: upstream_tasks, task_args and mapped.

If a task has arguments in its run() method, these can be bound either by using the functional API and _calling_ the task instance, or by using self.bind directly.

In addition to running arbitrary functions, tasks can interact with Prefect in a few ways: <ul><li> Return an optional result. When this function runs successfully,

the task is considered successful and the result (if any) can be made available to downstream tasks. </li>

<li> Raise an error. Errors are interpreted as failure. </li> <li> Raise a [signal](../engine/signals.html). Signals can include FAIL, SUCCESS,

RETRY, SKIP, etc. and indicate that the task should be put in the indicated state.

<ul> <li> FAIL will lead to retries if appropriate </li> <li> SUCCESS will cause the task to be marked successful </li> <li> RETRY will cause the task to be marked for retry, even if max_retries

has been exceeded </li>

<li> SKIP will skip the task and possibly propogate the skip state through the

flow, depending on whether downstream tasks have skip_on_upstream_skip=True.

</li></ul>

</li></ul>

class flim.plugin.Task(name: str = None, slug: str = None, tags: Iterable[str] = None, max_retries: int = None, retry_delay: datetime.timedelta = None, retry_on: Union[Type[Exception], Iterable[Type[Exception]]] = None, timeout: Union[int, datetime.timedelta] = None, trigger: Callable[[Dict[Edge, State]], bool] = None, skip_on_upstream_skip: bool = True, cache_for: datetime.timedelta = None, cache_validator: Callable = None, cache_key: str = None, checkpoint: bool = None, state_handlers: List[Callable] = None, on_failure: Callable = None, log_stdout: bool = False, result: Result = None, target: Union[str, Callable] = None, task_run_name: Union[str, Callable] = None, nout: int = None)

The Task class which is used as the full representation of a unit of work.

This Task class can be used directly as a first class object where it must be inherited from by a class that implements the run method. For a more functional way of generating Tasks, see [the task decorator](../utilities/tasks.html).

Inheritance example: ```python class AddTask(Task):

def run(self, x, y):

return x + y

```

Note: The implemented run method cannot have *args in its signature. In addition, the following keywords are reserved: upstream_tasks, task_args and mapped.

An instance of a Task can be used functionally to generate other task instances with the same attributes but with different values bound to their run methods.

Example: ```python class AddTask(Task):

def run(self, x, y):

return x + y

a = AddTask()

with Flow(“My Flow”) as f:

t1 = a(1, 2) # t1 != a t2 = a(5, 7) # t2 != a

```

To bind values to a Task’s run method imperatively (and without making a copy), see Task.bind.

Parameters

  • name (-) – The name of this task

  • slug (-) – The slug for this task. Slugs provide a stable ID for tasks so that the Prefect API can identify task run states. If a slug is not provided, one will be generated automatically once the task is added to a Flow.

  • tags (-) – A list of tags for this task

  • max_retries (-) – The maximum amount of times this task can be retried

  • retry_delay (-) – The amount of time to wait until task is retried

  • retry_on (-) – Exception types that will allow retry behavior to occur. If not set, all exceptions will allow retries. If set, retries will only occur if the exception is a subtype of the exception types provided.

  • timeout (-) – The amount of time (in seconds) to wait while running this task before a timeout occurs; note that sub-second resolution is not supported, even when passing in a timedelta.

  • trigger (-) – a function that determines whether the task should run, based on the states of any upstream tasks.

  • skip_on_upstream_skip (-) – if True, if any immediately upstream tasks are skipped, this task will automatically be skipped as well, regardless of trigger. By default, this prevents tasks from attempting to use either state or data from tasks that didn’t run. If False, the task’s trigger will be called as normal, with skips considered successes. Defaults to True.

  • cache_for (-) – The amount of time to maintain a cache of the outputs of this task. Useful for situations where the containing Flow will be rerun multiple times, but this task doesn’t need to be.

  • cache_validator (-) – Validator that will determine whether the cache for this task is still valid (only required if cache_for is provided; defaults to prefect.engine.cache_validators.duration_only)

  • cache_key (-) – if provided, a cache_key serves as a unique identifier for this Task’s cache, and can be shared across both Tasks _and_ Flows; if not provided, the Task’s _name_ will be used if running locally, or the Task’s database ID if running in Cloud

  • checkpoint (-) – if this Task is successful, whether to store its result using the configured result available during the run; Also note that checkpointing will only occur locally if prefect.config.flows.checkpointing is set to True

  • result (-) – the result instance used to retrieve and store task results during execution

  • target (-) – location to check for task Result. If a result exists at that location then the task run will enter a cached state. target strings can be templated formatting strings which will be formatted at runtime with values from prefect.context. If a callable function is provided, it should have signature callable(**kwargs) -> str and at write time all formatting kwargs will be passed and a fully formatted location is expected as the return value. The callable can be used for string formatting logic that .format(**kwargs) doesn’t support.

  • state_handlers (-) –

    A list of state change handlers that will be called whenever the task changes state, providing an opportunity to inspect or modify the new state. The handler will be passed the task instance, the old (prior) state, and the new (current) state, with the following signature:

    state_handler(task: Task, old_state: State, new_state: State) -> Optional[State]

    If multiple functions are passed, then the new_state argument will be the result of the previous handler.

  • on_failure (-) – A function with signature fn(task: Task, state: State) -> None that will be called anytime this Task enters a failure state

  • log_stdout (-) – Toggle whether or not to send stdout messages to the Prefect logger. Defaults to False.

  • task_run_name (-) – a name to set for this task at runtime. task_run_name strings can be templated formatting strings which will be formatted at runtime with values from task arguments, prefect.context, and flow parameters (in the case of a name conflict between these, earlier values take precedence). If a callable function is provided, it should have signature callable(**kwargs) -> str and at write time all formatting kwargs will be passed and a fully formatted location is expected as the return value. The callable can be used for string formatting logic that .format(**kwargs) doesn’t support. Note: this only works for tasks running against a backend API.

  • nout (-) – for tasks that return multiple results, the number of outputs to expect. If not provided, will be inferred from the task return annotation, if possible. Note that nout=1 implies the task returns a tuple of one value (leave as None for non-tuple return types).

Raises

  • - TypeError – if tags is of type str

  • - TypeError – if timeout is not of type int

  • - TypeError – if positional-only parameters are present in task’s signature

bind(*args: Any, mapped: bool = False, upstream_tasks: Iterable[Any] = None, flow: Flow = None, **kwargs: Any) → Task

Binding a task to (keyword) arguments creates a _keyed_ edge in the active Flow that will pass data from the arguments (whether Tasks or constants) to the Task’s run method under the appropriate key. Once a Task is bound in this manner, the same task instance cannot be bound a second time in the same Flow.

To bind arguments to a _copy_ of this Task instance, see __call__. Additionally, non-keyed edges can be created by passing any upstream dependencies through upstream_tasks.

Parameters

  • *args (-) –

    arguments to bind to the current Task’s run method

  • mapped (-) – Whether the results of these tasks should be mapped over with the specified keyword arguments; defaults to False. If True, any arguments contained within a prefect.utilities.edges.unmapped container will _not_ be mapped over.

  • upstream_tasks (-) – a list of upstream dependencies for the current task.

  • flow (-) – The flow to set dependencies on, defaults to the current flow in context if no flow is specified

  • **kwargs (-) –

    keyword arguments to bind to the current Task’s run method

Returns

the current Task instance

Return type

  • Task

copy(**task_args: Any) → prefect.core.task.Task

Creates and returns a copy of the current Task.

Parameters

**task_args (-) –

a dictionary of task attribute keyword arguments, these attributes will be set on the new copy

Raises

- AttributeError – if any passed task_args are not attributes of the original

Returns

a copy of the current Task, with any attributes updated from task_args

Return type

  • Task

inputs() → Dict[str, Dict]

Describe the inputs for this task. The result is a dictionary that maps each input to a type, required, and default. All values are inferred from the run() signature; this method can be overloaded for more precise control.

Returns

  • dict

is_equal(other: object) → prefect.core.task.Task

Produces a Task that evaluates self == other

This can’t be implemented as the __eq__() magic method because of Task comparisons.

Parameters

other (-) – the other operand of the operator. It will be converted to a Task if it isn’t one already.

Returns

  • Task

is_not_equal(other: object) → prefect.core.task.Task

Produces a Task that evaluates self != other

This can’t be implemented as the __neq__() magic method because of Task comparisons.

Parameters

other (-) – the other operand of the operator. It will be converted to a Task if it isn’t one already.

Returns

  • Task

map(*args: Any, upstream_tasks: Iterable[Any] = None, flow: Flow = None, task_args: dict = None, **kwargs: Any) → Task

Map the Task elementwise across one or more Tasks. Arguments that should _not_ be mapped over should be placed in the prefect.utilities.edges.unmapped container.

For example:

` task.map(x=X, y=unmapped(Y)) `

will map over the values of X, but not over the values of Y

Parameters

  • *args (-) –

    arguments to map over, which will elementwise be bound to the Task’s run method

  • upstream_tasks (-) – a list of upstream dependencies to map over

  • flow (-) – The flow to set dependencies on, defaults to the current flow in context if no flow is specified

  • task_args (-) – a dictionary of task attribute keyword arguments, these attributes will be set on the new copy

  • **kwargs (-) –

    keyword arguments to map over, which will elementwise be bound to the Task’s run method

Raises

- AttributeError – if any passed task_args are not attributes of the original

Returns

a new Task instance

Return type

  • Task

not_() → prefect.core.task.Task

Produces a Task that evaluates not self

Returns

  • Task

or_(other: object) → prefect.core.task.Task

Produces a Task that evaluates self or other

Parameters

other (-) – the other operand of the operator. It will be converted to a Task if it isn’t one already.

Returns

  • Task

outputs() → Any

Get the output types for this task.

Returns

  • Any

pipe(_prefect_task: Union[Type[prefect.utilities.edges.EdgeAnnotation], prefect.core.task.Task], **kwargs: Any) → prefect.core.task.Task

“Pipes” the result of this task through another task. some_task().pipe(other_task) is equivalent to other_task(some_task()), but can result in more readable code when used in a long chain of task calls.

Parameters

  • _prefect_task (-) – The task to execute after this task.

  • **kwargs (-) –

    Additional keyword arguments to include as task arguments.

Returns

A new task with the new arguments bound to it.

Return type

  • Task

run() → None

The run() method is called (with arguments, if appropriate) to run a task.

Note: The implemented run method cannot have *args in its signature. In addition, the following keywords are reserved: upstream_tasks, task_args and mapped.

If a task has arguments in its run() method, these can be bound either by using the functional API and _calling_ the task instance, or by using self.bind directly.

In addition to running arbitrary functions, tasks can interact with Prefect in a few ways: <ul><li> Return an optional result. When this function runs successfully,

the task is considered successful and the result (if any) can be made available to downstream tasks. </li>

<li> Raise an error. Errors are interpreted as failure. </li> <li> Raise a [signal](../engine/signals.html). Signals can include FAIL, SUCCESS,

RETRY, SKIP, etc. and indicate that the task should be put in the indicated state.

<ul> <li> FAIL will lead to retries if appropriate </li> <li> SUCCESS will cause the task to be marked successful </li> <li> RETRY will cause the task to be marked for retry, even if max_retries

has been exceeded </li>

<li> SKIP will skip the task and possibly propogate the skip state through the

flow, depending on whether downstream tasks have skip_on_upstream_skip=True.

</li></ul>

</li></ul>

serialize() → Dict[str, Any]

Creates a serialized representation of this task

Returns

  • dict representing this task

set_dependencies(flow: Flow = None, upstream_tasks: Iterable[object] = None, downstream_tasks: Iterable[object] = None, keyword_tasks: Mapping[str, object] = None, mapped: bool = False, validate: bool = None) → Task

Set dependencies for a flow either specified or in the current context using this task

Parameters

  • flow (-) – The flow to set dependencies on, defaults to the current

  • specified (flow in context if no flow is) –

  • upstream_tasks (-) – A list of upstream tasks for this task

  • downstream_tasks (-) – A list of downtream tasks for this task

  • keyword_tasks (-) – The results of these tasks will be provided

  • arguments. (to this task under the specified keyword) –

  • mapped (-) – Whether the results of the _upstream_ tasks should be mapped over with the specified keyword arguments

  • validate (-) – Whether or not to check the validity of the flow. If not provided, defaults to the value of eager_edge_validation in your Prefect configuration file.

Returns

  • self

Raises

- ValueError – if no flow is specified and no flow can be found in the current context

set_downstream(task: Task, flow: Flow = None, key: str = None, mapped: bool = False) → Task

Sets the provided task as a downstream dependency of this task.

Parameters

  • task (-) – A task that will be set as a downstream dependency of this task.

  • flow (-) – The flow to set dependencies on, defaults to the current flow in context if no flow is specified

  • key (-) – The key to be set for the new edge; the result of this task will be passed to the downstream task’s run() method under this keyword argument.

  • mapped (-) – Whether this dependency is mapped; defaults to False

Returns

  • self

Raises

- ValueError – if no flow is specified and no flow can be found in the current context

set_upstream(task: object, flow: Flow = None, key: str = None, mapped: bool = False) → Task

Sets the provided task as an upstream dependency of this task.

Parameters

  • task (-) – A task or object that will be converted to a task that will be set as a upstream dependency of this task.

  • flow (-) – The flow to set dependencies on, defaults to the current flow in context if no flow is specified

  • key (-) – The key to be set for the new edge; the result of the upstream task will be passed to this task’s run() method under this keyword argument.

  • mapped (-) – Whether this dependency is mapped; defaults to False

Returns

  • self

Raises

- ValueError – if no flow is specified and no flow can be found in the current context

flim.plugin.abstractmethod(funcobj)

A decorator indicating abstract methods.

Requires that the metaclass is ABCMeta or derived from it. A class that has a metaclass derived from ABCMeta cannot be instantiated unless all of its abstract methods are overridden. The abstract methods can be called using any of the normal ‘super’ call mechanisms.

Usage:

class C(metaclass=ABCMeta):

@abstractmethod def my_abstract_method(self, …):

…

flim.plugin.create_instance(clazz)

Creates analyzer instance.

Parameters

clazz (class) – analyzer class to instantiate.

Returns

analyzer object (AbstractPlugin subclass)

flim.plugin.discover_plugin_classes(dirs=['data', 'analysis', 'workflow'])

Creates labels and class objects for all Plugin subclasses in this package.

Returns

key-value pairs of analyzer labels and associated classes.

Return type

dict

flim.plugin.get_plugin_class(key, tosearch={'Analysis': {'Autoencoder: Augment Data': <class 'flim.analysis.aeaugment.AEAugment'>, 'Autoencoder: Run': <class 'flim.analysis.aerun.RunAE'>, 'Autoencoder: Train': <class 'flim.analysis.aetraining.AETraining'>, 'Categorize Data': <class 'flim.analysis.categorizer.Categorizer'>, 'K-Means': <class 'flim.analysis.kmeans.KMeansClustering'>, 'KS-Statistics': <class 'flim.analysis.ksstats.KSStats'>, 'PCA': <class 'flim.analysis.pca.PCAnalysis'>, 'Random Forest': <class 'flim.analysis.randomforest.RandomForest'>, 'Relative Change': <class 'flim.analysis.relativechange.RelativeChange'>, 'Series Analysis': <class 'flim.analysis.seriesanalyzer.SeriesAnalyzer'>, 'Summarize': <class 'flim.analysis.summarystats.SummaryStats'>}, 'Data': {'Concat Data': <class 'flim.data.concatdata.Concatenator'>, 'Filter': <class 'flim.data.filterdata.Filter'>, 'Merge': <class 'flim.data.mergedata.Merger'>, 'Order Categories': <class 'flim.data.categories.CategoryOrder'>, 'Pivot': <class 'flim.data.pivotdata.Pivot'>, 'Sort': <class 'flim.data.sortdata.Sort'>, 'Unpivot': <class 'flim.data.unpivotdata.UnPivot'>}, 'Plot': {'Bar Plot': <class 'flim.analysis.barplots.BarPlot'>, 'Box Plot': <class 'flim.analysis.boxplots.BoxPlot'>, 'FLIRR Plot': <class 'flim.analysis.flirr.FLIRRPlot'>, 'Frequency Histogram': <class 'flim.analysis.freqhisto.FreqHisto'>, 'Heatmap': <class 'flim.analysis.heatmap.Heatmap'>, 'KDE': <class 'flim.analysis.kde.KDE'>, 'Line Plot': <class 'flim.analysis.lineplots.LinePlot'>, 'Pair Plot': <class 'flim.analysis.pairplots.PairPlot'>, 'Scatter Plot': <class 'flim.analysis.scatterplots.ScatterPlot'>, 'Swarm Plot': <class 'flim.analysis.swarmplot.SwarmPlot'>, 'Violin Plot': <class 'flim.analysis.violinplot.ViolinPlot'>}, 'Workflow': {'Basic FLIRR Analysis': <class 'flim.workflow.basicflow.BasicFLIRRWorkFlow'>, 'FLIM Data Augmentation Tuning': <class 'flim.workflow.aetune.AEWorkflow'>, 'FLIM Feature Analysis': <class 'flim.workflow.aefeature.AEFeatureWorkflow'>}})
flim.plugin.init_plugins()

Initializes an instance for each individual Plugin subclass in this package.

Returns

analyzer object instances.

Return type

list

flim.plugin.init_plugins_configs()

Creates a single configuration with default settings for a given group of Plugin objects.

Parameters

list – plugin objects.

Returns

configuration parameters.

Return type

dict

flim.plugin.plugin(plugintype)

Register an instantiated plugin to the PLUGINS dict.

flim.plugin.task(fn: Optional[Callable] = None, **task_init_kwargs: Any) → Union[prefect.tasks.core.function.FunctionTask, Callable[[Callable], prefect.tasks.core.function.FunctionTask]]

A decorator for creating Tasks from functions.

Parameters

  • fn (-) – the decorated function

  • **task_init_kwargs (-) –

    keyword arguments that will be passed to the Task constructor on initialization.

Returns

A instance of a FunctionTask

Return type

  • FunctionTask

Raises

- ValueError – if the provided function violates signature requirements for Task run methods

Usage:

``` @task(name=’hello’, max_retries=3, retry_delay=1) def hello(name):

print(‘hello, {}’.format(name))

with Flow(“My Flow”) as flow:

t1 = hello(‘foo’) t2 = hello(‘bar’)

```

The decorator is best suited to Prefect’s functional API, but can also be used with the imperative API.

``` @task def fn_without_args():

return 1

@task def fn_with_args(x):

return x

# both tasks work inside a functional flow context with Flow(“My Flow”):

fn_without_args() fn_with_args(1)

```