opera.pge.base package

Submodules

opera.pge.base.base_pge module

base_pge.py

Module defining the Base PGE interfaces from which all other PGEs are derived.

class opera.pge.base.base_pge.PgeExecutor(pge_name, runconfig_path, **kwargs)

Bases: PreProcessorMixin, PostProcessorMixin

Main class for execution of a PGE, including the SAS layer.

The PgeExecutor class is primarily responsible for defining the interface for PGE execution and managing the actual execution of the SAS executable within a subprocess. PGE’s also define pre- and post-processing stages, which are invoked by PgeExecutor, but whose implementations are defined by use of Mixin classes.

The use of Mixin classes allows for flexibility of PGE design, where inheritors of PgeExecutor can compose a custom PGE by providing overloaded implementations of the Mixin classes to tailor the behavior of the pre- and post-processing phases, where necessary, while still inheriting any common functionality from this class.

LEVEL = 'L0'

Processing Level for Base PGE Products (dummy value)

NAME = 'BasePge'

Short name for the Base PGE

PGE_VERSION = '4.0.0-er.3.0'

Version of the PGE (tracks repo version by default)

PROJECT = 'OPERA'

Name of the project associated to this PGE

SAS_VERSION = '0.1'

Version of the SAS wrapped by this PGE (dummy value)

run(**kwargs)

Main entry point for PGE execution.

The pre-processor stage is run to initialize the PGE, followed by SAS execution, then completed with the post-processing steps to complete the job.

run_sas_executable(**kwargs)

Kicks off a SAS executable as defined by the RunConfig provided to the PGE.

Execution time for the SAS is collected and logged by this method.

Parameters:

**kwargs (dict) – Any keyword arguments needed for SAS execution.

class opera.pge.base.base_pge.PostProcessorMixin

Bases: object

Mixin class which is responsible for handling all post-processing steps for the PGE. The post-processing phase is defined as all steps necessary after SAS execution has completed.

This class is intended for use as a Mixin for use with the PgeExecutor class and its inheritors, and as such, this class assumes access to the instance attributes defined by PgeExecutor.

Inheritors of PostProcessorMixin may provide overloaded implementations for any of the exiting pre-processing steps, and even provide additional steps as necessary.

run_postprocessor(**kwargs)

Executes the post-processing steps for PGE job completion.

Inheritors of this Mixin may override this function to tailor the order of post-processing steps.

Parameters:

**kwargs (dict) – Any keyword arguments needed by the post-processor

class opera.pge.base.base_pge.PreProcessorMixin

Bases: object

Mixin class which is responsible for handling all pre-processing steps for the PGE. The pre-processing phase is defined as all steps necessary prior to SAS execution.

This class is intended for use as a Mixin for use with the PgeExecutor class and its inheritors, and as such, this class assumes access to the instance attributes defined by PgeExecutor.

Inheritors of PreProcessorMixin may provide overloaded implementations for any of the exiting pre-processing steps, and even provide additional steps as necessary.

run_preprocessor(**kwargs)

Executes the pre-processing steps for PGE initialization.

Inheritors of this Mixin may override this function to tailor the order of pre-processing steps.

Parameters:

**kwargs (dict) – Any keyword arguments needed by the pre-processor

opera.pge.base.runconfig module

runconfig.py

Module for parsing and validating run configuration files for OPERA PGEs.

This module is adapted for OPERA from the NISAR PGE R2.0.0 util/l1_l2_runconfig.py Original Author: David White Adapted By: Scott Collins

opera.pge.base.runconfig.BASE_PGE_SCHEMA = '/var/lib/jenkins/workspace/se-pipeline_release_3.0.0-rc.4.0/src/opera/pge/base/schema/base_pge_schema.yaml'

Path to the Yamale schema applicable to the PGE portion of each RunConfig

class opera.pge.base.runconfig.RunConfig(filename)

Bases: object

Class used to parse and validate run configuration files for OPERA PGEs.

RunConfig files are written in YAML, and contain distinct configuration sections for both the PGE and SAS executables. Schema-based validation is performed via the Yamale library (https://github.com/23andMe/Yamale).

_filename

Name of the file parsed to create the RunConfig

Type:

str

_run_config

Parsed contents of the provided RunConfig file

Type:

dict

_pge_config

Short-cut to the PGE-specific section of the parsed RunConfig

Type:

dict

_sas_config

Short-cut to the SAS-specific section of the parsed RunConfig

Type:

dict

property algorithm_parameters_file_config_path: str

Returns the path to the algorithm parameters run configuration file

property algorithm_parameters_schema_path: str

Returns the path to the optional algorithm parameter Schema file for DSWX-S1

property ancillary_file_map: dict

Returns the Ancillary File Map from the Dynamic Ancillary Files Group

asdict() → dict

Returns the entire parsed RunConfig in its dictionary representation

property composite_release_id: str

Returns the Composite Release ID (CRID) from the Primary Executable Category

property data_validity_start_date: str

Returns the DataValidityStartDate value for the Primary Executable

property debug_switch: bool

enabled/disabled.

Type:

Returns a boolean indicating the debugging state

property error_code_base: int

Returns the Error Code Base for a particular Primary Executable

property execute_via_shell: bool

enabled/disabled

Type:

Returns a boolean indicating the state of ExecuteViaShell

property filename: str

Returns the of the file parsed to create the RunConfig

get_ancillary_filenames()

Returns a list of all ancillary filenames listed in the DynamicAncillaryFilesGroup section of the run config file. The returned list only has the filenames, not the types of the files.

Returns:

ancillary_filenames – List of all ancillary filenames listed in the RunConfig under the DynamicAncillaryFilesGroup section.

Return type:

list of str

get_input_filenames()

Iterates over the input_file list from the runconfig and returns a list of all files.

Files in the list are immediately included in the returned list.

Directories in the list will be examined and any files found will be added to the list.

Returns:

input_file – The expanded list of input files determined from the RunConfig setting. The list is sorted prior to being returned.

Return type:

list of str

Raises:

OSError – If anything that is not a file or directory is encountered while traversing the set of input file locations.

get_output_product_filenames()

Returns a sorted list of all product file paths currently written to the output location specified by the RunConfig.

Any hidden files (starting with “.”) are ignored, as well as any files within the designated “scratch” path (if it happens to be defined within the output product directory).

property input_files: list

Returns the path from the Input Files Group

property iso_template_path: str

Returns the ISO Template Path for a Primary Executable

property name: str

Returns the name of the RunConfig file

property output_product_path: str

Returns the Output Product Path from the Product Path Group

property pge_name: str

Returns the PGE Name from the PGE Name Group

property product_identifier: str

Returns the Product Identifier from the Primary Executable Category

property product_type: str

Returns the product type as defined in the SAS portion of the RunConfig

property product_version: str

Returns the Product Version from the Primary Executable Category

property qa_enabled: bool

enabled/disabled

Type:

Returns a boolean indicating the state of QAExecutable

property qa_program_options: str

Return program options (arguments) for an executable command

property qa_program_path: str

Return the path to a QA Executable

property sas_config: dict

Returns the short-cut to the SAS-specific section of the parsed RunConfig

property sas_program_options: str

Returns the Program Options (arguments) to a Primary Executable

property sas_program_path: str

Returns the Program Path from a Primary Executable Category

property sas_schema_path: str

Returns the path to the Schema file for a Primary Executable

property scratch_path: str

Returns the Scratch Path from the Product Path Group

validate(pge_schema_file='/var/lib/jenkins/workspace/se-pipeline_release_3.0.0-rc.4.0/src/opera/pge/base/schema/base_pge_schema.yaml', strict_mode=True)

Validates the RunConfig using a combination of the base PGE schema, and the specific SAS schema defined by the RunConfig itself.

Parameters:

• pge_schema_file (str, optional) – The Yamale schema file for the PGE portion of the RunConfig. Defaults to the base PGE schema. Inheritors of RunConfig that overload this method may use this argument to provide their own tailored schema.

• strict_mode (bool, optional) – Toggle for Yamale validation strict mode. When enabled, unexpected elements not defined by the schema will raise validation errors.

Raises:

• RuntimeError – If the SAS schema defined by the parsed RunConfig cannot be located.

• YamaleError – If the RunConfig does not validate against the combined PGE/SAS schema.

Module contents