For a workflow to run, it needs to be associated with a rule (see rule configuration). The rule configuration determines how and when a workflow execution is triggered. Rules can be triggered one time, on a schedule, or by new data written to a kinesis stream.
There are three lambda functions in the API package responsible for scheduling and starting workflows:
message consumer, and
SF starter. Each Cumulus instance comes with a Start SF SQS queue.
SF scheduler lambda puts a message onto the
Start SF queue. This message is picked up the
Start SF lambda and an execution is started with the body of the message as the input.
When a one time rule is created, the
schedule SF lambda is triggered. Rules that are not one time are associated with a CloudWatch event which will manage the trigger of the lambdas that trigger the workflows.
For a scheduled rule, the Cloudwatch event is triggered on the given schedule which calls directly to the
schedule SF lambda.
For a kinesis rule, when data is added to the kinesis stream, the Cloudwatch event is triggered, which calls the
message consumer lambda. The
message consumer lambda parses the kinesis message and finds all of the rules associated with that message. For each rule (which corresponds to one workflow), the
schedule SF lambda is triggered to queue a message to start the workflow.
For an sns rule, when a message is published to the SNS topic, the
message consumer receives the SNS message (JSON expected), parses it into an object, starts a new execution of the workflow associated with the rule and passes the object in the
payload field of the Cumulus message.