This task utilizes the Cumulus Message Adapter to interpret and construct incoming and outgoing messages.
Links to the npm package, task input, output and configuration schema definitions and more can be found on the auto-generated Cumulus Tasks page.
The purpose of this task is to do the following with the incoming PDR object:
Stage it to an internal S3 bucket
Parse the PDR
Archive the PDR and remove the staged file if successful
Outputs a payload object containing metadata about the parsed PDR (e.g. total size of all files, files counts, etc) and a granules object
The constructed granules object is created using PDR metadata to determine values like data type and version, collection definitions to determine a file storage location based on the extracted data type and version number.
Granule file types are converted from the PDR spec types to CNM types according to the following translation table:
HDF: 'data', HDF-EOS: 'data', SCIENCE: 'data', BROWSE: 'browse', METADATA: 'metadata', BROWSE_METADATA: 'metadata', QA_METADATA: 'metadata', PRODHIST: 'qa', QA: 'metadata', TGZ: 'data', LINKAGE: 'data'
Files missing file types will have none assigned, files with invalid types will result in a PDR parse failure.
This task expects an incoming input that contains name and path information about the PDR to be parsed. For the specifics, see the Cumulus Tasks page entry for the schema.
This task does expect values to be set in the
workflow_config CMA parameters for the workflows. A schema exists that defines the requirements for the task.
For the most recent config.json schema, please see the Cumulus Tasks page entry for the schema.
Below are expanded descriptions of selected config keys:
A Cumulus provider object. Used to define connection information for retrieving the PDR.
Defines the bucket where the 'pdrs' folder for parsed PDRs will be stored.
A Cumulus collection object. Used to define granule file groupings and granule metadata for discovered files.
This task outputs a single payload output object containing metadata about the parsed PDR (e.g. filesCount, totalSize, etc), a pdr object with information for later steps and a the generated array of granule objects.
See the SIPS workflow cookbook for an example of this task in a workflow