Cumulus API

The Cumulus API allows developers to interact with the Cumulus Framework, such as monitoring status or creating, editing, and deleting records. This is the same API that powers the Cumulus dashboard.

By utilizing this API, a developer can integrate with the Cumulus framework in any language or environment; although interacting with Cumulus through the Cumulus dashboard may be appropriate for many end users, for some use cases it's best to have the flexibility of a web-accessible API.

The API accepts and responds with JSON payloads at various HTTPS endpoints.

In order to use these endpoints, you must include authentication information in your HTTPS request; authentication is explained in the following section.

The following table lists the query string parameters that can be used with most of the Cumulus API endpoints. {fieldName} is a stand-in for any of the fields in the record, and for nested objects dot notation can be used; for example, valid fieldNames include: pdrName, status, and recipe.processStep.description.

query string parameter	description
`limit={number}`	number of records to be returned by the API call; default is `1`, maximum is `100`
`page={number}`	page number, 1-indexed; default is `1`
`searchContext={value}`	searchContext value returned by a previous query, must be included on subsequent queries to maintain the context. Allows listing past 10,000 results. Incompatible with the `from` and `to` parameters which will both override searchContext behavior and make the query subject to the 10,000 result cap again.
`sort_by={fieldName}`	which field to sort by; default is `timestamp`
`order={asc\|desc}`	whether to sort in `asc` or `desc` order
`sort_key[]={-fieldName1}&sort_key[]={fieldName2}`	One or more sort keys can be specified using the sort_key[] parameter. The order used impacts searching. Fields can be prepended with a `-` to sort in descending order or a `+` to sort in ascending. Ascending order is the default. The + must be escaped with %2B
`prefix={value}`	`startsWith` search of the `granuleId`, `status`, `pdrName`, `name`, `error`, `id` and `msg` fields
`infix={value}`	`includes` search of the `granuleId`, `status`, `pdrName`, `name`, `error`, `id` and `msg` fields
`fields={fieldName1, fieldName2}`	which fields to return, separated by a comma
`{fieldName}={value}`	exact value match for the given field
`{fieldName}__from={number}`	for numeric fields, field value must be greater than the given number
`{fieldName}__to={number}`	for numeric fields, field value must be less than the given number
`{fieldName}__not={value}`	field does not match the given value
`{fieldName}__in={value1, value2}`	field matches one of the values in the comma-separated list
`{fieldName}__exists={true\|false}`	field exists or doesn't exist in the record
`q="fieldName:[1 TO 2] AND fieldName2:[3 TO 4]"`	arbitrary Apache Lucene query syntax, not needed for most uses of the API; if the `q` parameter is used, all other query parameters will be ignored, besides `limit`, `page`, and `fields`

Versioning

The Cumulus API is versioned and the current version is v2. Retrieve the latest API version from Cumulus.

To use any version, include the version number in the path before the query endpoint. {API_URL}/{version}/{query}

GET

/version

Example Request

$ curl https://example.com/version

Example Response

{
    "response_version": "v2",
    "api_version": "15.0.0"
}

Token

Returns a bearer token using oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).

GET

/token

Query Parameters

query string parameter	description
`state={string}`	The URI to redirect to after if oAuth was successful

Example request

$ curl https://example.com/token

Example response

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}

Refresh token

Refreshes a bearer token received from oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).

POST

/refresh

Request body

parameter	type	required	description
`token`	string	`true`	The JWT received from the `/token` endpoint to refresh

Example request

$ curl --request POST https://example.com/refresh --header 'Content-Type: application/json' --data '{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}'

Example response

{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDcxMjk4ODEzIn0.vO6RlSRo47kkH15_muoUYNvv74fRzFxs7FlmVaarHlc"}

Delete token

Delete the record for an access token received from oAuth with Earthdata Login service.

token is the JWT containing access token information to delete

DELETE

/tokenDelete/{token}

Example request

$ curl --request DELETE https://example.com/tokenDelete/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY

Example response

{"message": "Token record was deleted"}

Authorization header

When a request is made to the Cumulus API, it must contain a bearer token generated by the API.

The token is generated after the user login with Earthdata Login service.

The token is included in requests using the Authorization header.

If no token is provided, the Cumulus API server will respond with an error, requesting credentials. If an incorrect token is provided, the server will respond with a separate error noting this.

Authentication example

#! /bin/sh

ORIGIN=$(dirname $CUMULUS_BASEURL)
LOGIN_URL="$CUMULUS_BASEURL/token"

# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)

# Request the Earthdata url with client id and redirect uri to use with Cumulus
AUTHORIZE_URL=$(curl -s -i ${LOGIN_URL} | grep location | sed -e "s/^location: //");

# Request an authorization grant code
TOKEN_URL=$(curl -s -i -X POST \
  -F "credentials=${AUTH}" \
  -H "Origin: ${ORIGIN}" \
  ${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")

# Request the token through the CUMULUS API url that's returned from Earthdata
# Response is a JSON object of the form { token: String }
# This uses the cli tool jq to parse the JSON and get the token string
# More info on jq: https://stedolan.github.io/jq/
TOKEN=$(curl -s ${TOKEN_URL%$'\r'} | jq -r '.message.token')

echo $TOKEN

List providers

List providers in the Cumulus system.

GET

/providers

Example request

$ curl https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "daac-cumulus",
        "table": "provider",
        "limit": 1,
        "page": 1,
        "count": 2
    },
    "results": [
        {
            "id": "HTTP_MODIS",
            "globalConnectionLimit": 10,
            "maxDownloadTime": 300,
            "protocol": "http",
            "host": "https://data.modis.gov/",
            "timestamp": 1508861082226
        }
    ]
}

Retrieve provider

Retrieve a single provider.

GET

/providers/{id}

Example request

$ curl https://example.com/providers/HTTP_MODIS --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "createdAt": 1508861081785,
    "id": "HTTP_MODIS",
    "host": "https://data.modis.gov/",
    "globalConnectionLimit": 10,
    "maxDownloadTime": 300,
    "updatedAt": 1508861081785,
    "protocol": "http"
}

Create provider

Create a provider. For more information on creating providers and the contents of a request see the Cumulus setup documentation.

Overview of the schema fields:

Field	Value	Description
`id`	`string`	provider id/name
`protocol`	`"s3"`\|`"http"`\|`"https"`\|`"ftp"`	file transfer (sync) protocol
`host`	`string`	provider host endpoint
`port`	`number`	provider host port
`globalConnectionLimit`	`number`	limit to number of concurrent connections. This is the maximum number of connections Cumulus compatible ingest lambdas are expected to make to a provider. Defaults to unlimited
`maxDownloadTime`	`number`	Maximum download time in seconds for all granule files on a sync granule task. The timeout is used together with globalConnectionLimit to limit concurrent downloads.
`username`	`string`	provider connection username
`password`	`string`	provider connection password
`privateKey`	`string`	filename assumed to be in s3://bucketInternal/stackName/crypto
`cmKeyId`	`string`	AWS KMS Customer Master Key arn or alias
`allowedRedirects`	`Array<string>`	Only hosts in this list will have the provider username/password forwarded for authentication. Entries should be specified as host.com or host.com:7000 if redirect port is different than the provider port.
`certificateUri`	`string`	Optional SSL Certificate S3 URI for custom or self-signed SSL (TLS) certificate

POST

/providers

Example request

$ curl --request POST https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
    "host": "https://www.example.gov",
    "id": "MY_DAAC_SATELLITE",
    "protocol": "http",
    "globalConnectionLimit": 10,
    "maxDownloadTime": 300
}'

Example response

{
    "message": "Record saved",
    "record": {
        "createdAt": 1491941727851,
        "host": "https://www.example.gov",
        "id": "MY_DAAC_SATELLITE",
        "protocol": "http",
        "globalConnectionLimit": 10,
        "maxDownloadTime": 300,
        "timestamp": 1513956151186
    }
}

Update/replace provider

Update/replace an existing provider. Expects payload to specify the entire provider object, and will completely replace the existing provider with the specified payload. For a field reference see "Create provider".

Returns status 200 on successful replacement, 400 if the id property in the payload does not match the corresponding value in the resource URI, or 404 if there is no provider with the specified ID.

PUT

/providers/{id}

Example request

$ curl --request PUT https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "id": "MY_DAAC_SATELLITE",
  "host": "https://www.example.co.uk",
  "globalConnectionLimit": 10,
  "maxDownloadTime": 300,
  "protocol": "http"
}'

Example successful response

{
  "createdAt": 1491941727851,
  "id": "MY_DAAC_SATELLITE",
  "host": "https://www.example.co.uk",
  "globalConnectionLimit": 10,
  "maxDownloadTime": 300,
  "updatedAt": 1513956150733,
  "protocol": "http",
  "timestamp": 1513956555713
}

Delete provider

Delete a provider from Cumulus. The related PDRs and granules remain in the Cumulus and CMR systems.

DELETE

/providers/{id}

Example request

$ curl --request DELETE https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List collections

List collections in the Cumulus system.

If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink

If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.

GET

/collections

Example request

$ curl https://example.com/collections --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "collection",
        "limit": 1,
        "page": 1,
        "count": 3
    },
    "results": [
        {
            "name": "MOD11A1",
            "version": "006",
            "dataType": "MOD11A1",
            "process": "modis",
            "provider_path": "/",
            "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
            "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "files": [
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
                },
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
                }
            ],
            "timestamp": 1513020427284,
            "createdAt": 1510761441174,
            "updatedAt": 1513020427162,
            "edpa": true,
            "some_other_field": "field",
            "duplicateHandling": "skip",
        }
    ]
}

List collections with active granules

List collections in the Cumulus system that have active associated granules.

If time parameters are specified, the query will return collections that have granules that have been updated in that time frame.

If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink

If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.

GET

/collections/active

Example request

$ curl https://example.com/collections/active?includeStats=true --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "collection",
        "limit": 1,
        "page": 1,
        "count": 3
    },
    "results": [
        {
            "name": "MOD11A1",
            "version": "006",
            "dataType": "MOD11A1",
            "process": "modis",
            "provider_path": "/",
            "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
            "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "files": [
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
                    "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
                },
                {
                    "bucket": "private",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
                },
                {
                    "bucket": "protected",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
                },
                {
                    "bucket": "public",
                    "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
                    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
                }
            ],
            "timestamp": 1513020427284,
            "createdAt": 1510761441174,
            "updatedAt": 1513020427162,
            "edpa": true,
            "some_other_field": "field",
            "duplicateHandling": "skip",
            "stats": {
                "running": 0,
                "completed": 6,
                "failed": 1,
                "total": 7
            }
        }
    ]
}

Retrieve collection

Retrieve a single collection.

GET

/collections/{name}/{version}

Example request

$ curl https://example.com/collections/MOD11A1/006 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "process": "modis",
    "granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
    "version": "006",
    "dataType": "MOD11A1",
    "some_other_field": "field",
    "createdAt": 1510761441174,
    "edpa": true,
    "name": "MOD11A1",
    "duplicateHandling": "skip",
    "provider_path": "/",
    "files": [
        {
            "bucket": "protected",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
        },
        {
            "bucket": "private",
            "sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
            "regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
        },
        {
            "bucket": "private",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$"
        },
        {
            "bucket": "protected",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$"
        },
        {
            "bucket": "public",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$"
        },
        {
            "bucket": "public",
            "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg",
            "regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$"
        }
    ],
    "updatedAt": 1513020427162,
    "granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
    "sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
}

Create collection

Create a collection. For more information on creating collections and the contents of a request see the Cumulus setup documentation. A collection generally includes, but is not limited to, the fields listed below.

Overview of the schema fields:

Field	Value	Description
`name`	`string`	collection name
`version`	`string`	collection version
`dataType`	`string`	matches collection with PDR datatype
`duplicateHandling`	`"replace"`\|`"version"`\|`"skip"`\|`"error"`	duplicate handling protocol
`process`	`string`	process choice step variable
`provider_path`	`string`	path of remote files to sync
`granuleId`	`string (regex)`	regex to match granule IDs
`granuleIdExtraction`	`string (regex)`	regex to extract ID from files
`sampleFileName`	`string`	sample filename for granule ID
`url_path`	`string`	s3 prefix template
`files`	`array`	array of file specification objects
`-- file.bucket`	`string`	file destination bucket
`-- file.regex`	`string (regex)`	regex to match file names
`-- file.sampleFileName`	`string`	sample filename
`-- file.url_path`	`string`	s3 prefix template
`-- file.fileType`	`string`	granule file fileType mapping

POST

/collections

Example request

$ curl --request POST https://example.com/collections --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "process": "modis",
  "duplicateHandling": "replace",
  "provider_path": "cumulus-test-data/pdrs",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ]
}'

Example response

{
    "message": "Record saved",
    "record": {
        "name": "MOD09GQ",
        "version": "006",
        "dataType": "MOD09GQ",
        "duplicateHandling": "replace",
        "process": "modis",
        "provider_path": "cumulus-test-data/pdrs",
        "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
        "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
        "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
        "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
        "files": [
            {
                "bucket": "protected",
                "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
                "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
                "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
            }
        ],
        "createdAt": 1491946535919
    }
}

Update/replace collection

Update/replace an existing collection. Expects payload to specify the entire collection object, and will completely replace the existing collection with the specified payload. For a field reference see "Create collection".

Returns status 200 on successful replacement, 400 if the name or version property in the payload does not match the corresponding value in the resource URI, or 404 if there is no collection with the specified name and version.

PUT

/collections/{name}/{version}

Example request

$ curl --request PUT https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "duplicateHandling": "error",
  "newNeededField": "myCustomFieldValue",
  "process": "modis",
  "provider_path": "new_path/test-data",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ]
}'

Example successful response

{
  "name": "MOD09GQ",
  "version": "006",
  "dataType": "MOD09GQ",
  "duplicateHandling": "error",
  "newNeededField": "myCustomFieldValue",
  "process": "modis",
  "provider_path": "new_path/test-data",
  "granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
  "granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
  "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
  "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
  "files": [
    {
      "bucket": "protected",
      "regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
      "sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
      "url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
    }
  ],
  "createdAt": 1491946535919,
  "updatedAt": 1514304825894
}

Delete collection

Delete a collection from Cumulus, but not from CMR. All related granules in Cumulus must have already been deleted from Cumulus.

DELETE

/collections/{name}/{version}

Example request

$ curl --request DELETE https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List granules

List granules in the Cumulus system. If the query includes a value of true for getRecoveryStatus, recoveryStatus will be included in each granule's return value when applicable.

GET

/granules

Example request

$ curl https://example.com/granules --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "granule",
        "limit": 1,
        "page": 1,
        "count": 8,
        "searchContext": "%5B%221577836800000%22%5D",
    },
    "results": [
        {
            "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "completed",
            "provider": "LP_TS2_DataPool",
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
            "files": [
                {
                    "bucket": "cumulus-devseed-protected",
                    "checksum": 964704694,
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 1447347,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "checksum": 121318124,
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
                    "fileSize": 22559,
                    "fileType": "metadata",
                    "checksumType": "CKSUM",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "checksum": 2188150664,
                    "key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 18118,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
                    "fileType": "browse",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
                },
                {
                    "bucket": "cumulus-devseed-protected",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
                    "fileType": "metadata",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
                    "fileType": "browse",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
                }
            ],
            "error": null,
            "createdAt": 1513020455831,
            "timestamp": 1513020462156,
            "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
            "duration": 6.325,
            "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS"
        }
    ]
}

List granules with searchContext

List granules in the Cumulus system with searchContext from a previous query. Must include all of previous query's sort, order, or sort_key query string parameters. Must not include from and to query string parameters.

GET

/granules

Example request

$ curl https://example.com/granules?searchContext=%5B%221577836800000%22%5D --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "granule",
        "limit": 1,
        "page": 1,
        "count": 8,
        "searchContext": "%5B%221606780899999%22%5D",
    },
    "results": [
        {
            "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "completed",
            "provider": "LP_TS2_DataPool",
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
            "files": [
                {
                    "bucket": "cumulus-devseed-protected",
                    "checksum": 964704694,
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 1447347,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "checksum": 121318124,
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
                    "fileSize": 22559,
                    "fileType": "metadata",
                    "checksumType": "CKSUM",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
                },
                {
                    "bucket": "cumulus-devseed-private",
                    "checksum": 2188150664,
                    "key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
                    "fileSize": 18118,
                    "fileType": "data",
                    "checksumType": "CKSUM",
                    "fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
                    "fileType": "browse",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
                },
                {
                    "bucket": "cumulus-devseed-protected",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
                    "fileType": "metadata",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
                },
                {
                    "bucket": "cumulus-devseed-public",
                    "key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
                    "fileType": "browse",
                    "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
                }
            ],
            "error": null,
            "createdAt": 1513020455831,
            "timestamp": 1513020462156,
            "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
            "duration": 6.325,
            "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS"
        }
    ]
}

Retrieve granule

Retrieve a single granule. Two routes are currently available. The preferred query includes both a Collection ID and a Granule ID to identify a unique granule.

Please note -- Querying by the Granule ID alone (e.g. GET /granules/{granuleId}) is supported but may be deprecated in the future.

If the query includes a value of true for getRecoveryStatus, the returned granule will include recoveryStatus when applicable.

GET

/granules/{collectionId}/{granuleId}

Example request

$ curl https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
    "collectionId": "MOD11A1___006",
    "status": "completed",
    "provider": "LP_TS2_DataPool",
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
    "files": [
        {
            "bucket": "cumulus-devseed-protected",
            "checksum": 964704694,
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileSize": 1447347,
            "checksumType": "CKSUM",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-private",
            "checksum": 121318124,
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
            "fileSize": 22559,
            "checksumType": "CKSUM",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
        },
        {
            "bucket": "cumulus-devseed-private",
            "checksum": 2188150664,
            "key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileSize": 18118,
            "checksumType": "CKSUM",
            "fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-protected",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
        },
        {
            "bucket": "cumulus-devseed-public",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
        },
        {
            "bucket": "cumulus-devseed-protected",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
        },
        {
            "bucket": "cumulus-devseed-public",
            "key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
            "fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
        }
    ],
    "error": null,
    "createdAt": 1513020455831,
    "timestamp": 1513020462156,
    "published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
    "duration": 6.325,
    "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
    "_id": "MOD11A1.A2017137.h20v17.006.2017138085755"
}

Create Granule

Create a granule. A granule can have the following fields.

Field	Type	Required	Description
`beginningDateTime`	`string`	`no`	The time when the granule's temporal coverage begins
`cmrLink`	`string`	`no`	link to CMR
`collectionId`	`string`	`yes`	Collection associated with the granule
`createdAt`	`integer`	`no`	Time granule record was created (now)
`duration`	`number`	`no`	Ingest duration milliseconds
`endingDateTime`	`string`	`no`	The time when the granule's temporal coverage ends
`error`	`object`	`no`	The error details for this granule
`execution`	`string`	`no`	Step Function Execution URL
`files`	`array`	`no`	Files associated with the granule
`granuleId`	`string`	`yes`	Granule ID
`lastUpdateDateTime`	`string`	`no`	The date/time that data provider last updated the granule info on data provider's database
`pdrName`	`string`	`no`	PDR associated with the granule
`processingEndDateTime`	`string`	`no`	Time processing of granule ends. Usually a StepFunction's stop time
`processingStartDateTime`	`string`	`no`	Time processing of granule began. Usually a StepFunction's start time
`productVolume`	`number`	`no`	Sum of the granule's file's sizes in bytes
`productionDateTime`	`string`	`no`	The date and time a specific granule was produced by a PGE
`provider`	`string`	`no`	Granule's provider
`published`	`boolean`	`no`	If granule is published to CMR
`queryFields`	`object`	`no`	Arbitrary query fields assigned to the granule
`status`	`string`	`yes`	Ingest status of the granule one of ['running', 'completed', 'failed']
`timeToArchive`	`number`	`no`	Time to post to CMR in seconds
`timeToPreprocess`	`number`	`no`	Time to sync granule in seconds
`timestamp`	`integer`	`no`	Timestamp for granule record (now)
`updatedAt`	`integer`	`no`	Update Time for granule (now)

POST

/granules

Example request

$ curl --request POST https://example.com/granules \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --data '{
    "granuleId": "granuleId.A20200113.006.1005",
  "collectionId": "alpha___006",
  "status": "running",
  "beginningDateTime": "1995-01-01T00:00:00.000Z",
  "cmrLink": "https://mmt.uat.earthdata.nasa.gov/collections/C1237256734-CUMULUS",
  "createdAt": 1631547286190,
  "duration": 60000,
  "endingDateTime": "2021-09-13T00:00:00.000Z",
  "error": {},
  "execution": "https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestStepFunction2:cff1266e-ef36-664f-ff00-3a4d26bd1735",
  "files": [
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.hdf",
      "fileName": "granuleId.A20200113.006.1005.hdf"
    }
  ],
  "lastUpdateDateTime":"2021-09-12T15:10:01.000Z",
  "pdrName": "aPdrName",
  "processingEndDateTime": "2020-10-13T23:59:59.999Z",
  "processingStartDateTime": "2020-10-13T00:00:00.000Z",
  "productVolume": 59632353,
  "productionDateTime": "2020-10-14T15:40:05.546Z",
  "provider": "s3-local",
  "published": true,
  "queryFields": {"custom": "values can go here"},
  "timeToArchive": 5001,
  "timeToPreprocess": 3240,
  "timestamp":1631547675248,
  "updatedAt":1631547675248
  }'

Example response

{
    "message": "Successfully wrote granule with Granule Id: granuleId.A20200113.006.1005"
}

Replace granule

Replace an existing granule. Expects payload to contain the modified parts of the granule and the existing granule values will be overwritten by the modified portions. Any unspecified values will be removed, and appropriate fields will be replaced with defaults. Executions associated will not be modified if not specified. The same fields are available as are for creating a granule..

Returns status 200 on successful update, 201 on new granule creation, 404 if the granuleId can not be found in the database, or 400 when the granuleId in the payload does not match the corresponding value in the resource URI.

Please note -- In versions of CUMULUS prior to release v16, PUT endpoint was identical to PATCH requests.

PUT

/granules/{collectionId}/{granuleId}

Example request

$ curl --request PUT https://example.com/granules/COLLECTION___VERSION/granuleId.A19990103.006.1000 \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2'\  --data '{
  "granuleId": "granuleId.A20200113.006.1005",
  "files": [
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.hdf",
      "fileName": "granuleId.A20200113.006.1005.hdf"
    },
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.jpg",
      "fileName": "granuleId.A20200113.006.1005.jpg"
    }
  ],
  "duration": 1000,
  "status": "completed"
  }'

Example response

{
    "message": "Successfully updated granule with Granule Id: granuleId.A20200113.006.1005, CollectionId: COLLECTION___VERSION"
}

Update granule

Update an existing granule. Expects payload to contain the modified parts of the granule and the existing granule values will be overwritten by the modified portions. Unspecified keys will be retained. Keys set to null will be removed. Executions will not be disassociated from the granule via null deletion. The same fields are available as are for creating a granule..

Please note -- Querying by the Granule ID alone (e.g. PATCH /granules/{granuleId}) is supported but may be deprecated in the future.

PATCH

/granules/{collectionId}/{granuleId}

Example request

$ curl --request PATCH https://example.com/granules/granuleId.A19990103.006.1000 \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2'\
  --data '{
  "granuleId": "granuleId.A20200113.006.1005",
  "files": [
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.hdf",
      "fileName": "granuleId.A20200113.006.1005.hdf"
    },
    {
      "bucket": "stack-protected",
      "key": "granuleId.A20200113.006.1005.jpg",
      "fileName": "granuleId.A20200113.006.1005.jpg"
    }
  ],
  "duration": 1000,
  "status": "completed"
  }'

Example response

{
    "message": "Successfully updated granule with Granule Id: granuleId.A20200113.006.1005"
}

Associate execution

Associate an execution with a granule. Returns status 200 on successful association, 404 if the granule or execution can not be found in the database, or 400 when the granuleId in the payload does not match the corresponding value in the resource URI.

The request should have the following fields.

Field	Type	Description
`collectionId`	`string`	Collection associated with the granule (e.g. `<name>___<version>` where `<name>` is the collection name and `<version>` is the collection version)
`granuleId`	`string`	Granule ID
`executionArn`	`string`	Execution arn

POST

/granules/{granuleId}/executions

Example request

$ curl --request POST https://example.com/granules/granuleId.A19990103.006.1000/executions \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2'\
  --data '{
  "granuleId": "granuleId.A19990103.006.1000",
  "collectionId": "MOD09GQ___006",
  "executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
  }'

Example response

{
    "message": "Successfully associated execution arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 with granule granuleId granuleId.A19990103.006.1000 collectionId MOD09GQ___006"
}

Reingest granule

Reingest a granule. This causes the granule to re-download to Cumulus from source, and begin processing from scratch. Reingesting a granule will overwrite existing granule files.

You trigger the reingest by posting with the data's action set to reingest.

A reingest request may also include one optional parameters, either executionArn or workflowName. Use of one of these will cause the reingest occur, but before running the ingest, a different original payload is found from existing executions and used in place of the most recent execution. This has the result of allowing you to reingest a granule with any of it's previous inputs. If executionArn is specified, the originalMessage is pulled directly from that execution. When workflowName is specified, the database is search to find all of the executions with that workflowName that were run on the granuleid, and the most recent originalMessage is pulled and used for the reingest. Remember only to supply either executionArn or workflowName, if both are present, workflowName is ignored and executionArn is used to determining the input message to the reingest.

A warning message is included in the response if the collection's duplicateHandling is not set to 'replace'.

PATCH

/granules/{granuleId}

Example request

$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755
       --header 'Authorization: Bearer ReplaceWithTheToken'
       --header 'Content-Type: application/json'
       --header 'Cumulus-API-Version: 2'\
       --data '{"action": "reingest",
               ["executionArn": "arn:aws:states:us-east-1:123456789012:execution:stack-lambdaName:9da47a3b-4d85-4599-ae78-dbec2e042520"],
               ["workflowName": "TheWorkflowName"] }'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "action": "reingest",
    "status": "SUCCESS",
    "warning": "The granule files may be overwritten"
}

Apply workflow to granule

Apply the named workflow to the granule. Workflow input will be built from template and provided entire Cumulus granule record as payload.

PATCH

/granules/{granuleid}

Example request

$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'   --header 'Cumulus-API-Version: 2'\ --data '{ "action": "applyWorkflow", "workflow": "inPlaceWorkflow" }'

Example response

{
    "granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
    "action": "applyWorkflow inPlaceWorkflow",
    "status": "SUCCESS"
}

Move a granule

Move a granule from one location on S3 to another. Individual files are moved to specific locations by using a regex that matches their filenames.

PATCH

/granules/{granuleId}

Example request

$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'   --header 'Cumulus-API-Version: 2'\
 --data '{ "action": "move", "destinations": [{ "regex": ".*.hdf$", "bucket": "s3-bucket", "filepath": "new/filepath/" }]}'

Example response

{
  "granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
  "action": "move",
  "status": "SUCCESS"
}

Remove granule from CMR

Remove a Cumulus granule from CMR.

PATCH

/granules/{granuleId}

Example request

$ curl --request PATCH https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750   --header 'Cumulus-API-Version: 2'\
 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --header 'Cumulus-API-Version: 2'\ --data '{"action": "removeFromCmr"}'

Example response

{
    "granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
    "action": "removeFromCmr",
    "status": "SUCCESS"
}

Delete granule

Delete a granule from Cumulus. It must already be removed from CMR.

Please note -- Querying by the Granule ID alone (e.g. DELETE /granules/{granuleId}) is supported but may be deprecated in the future.

DELETE

/granules/{collectionId}/{granuleId}

Example request

$ curl --request DELETE https://example.com/granules/1A0000-2016121001_002_001 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "detail": "Record deleted"
}

Bulk Operations

Apply a workflow to the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field	Required	Value	Description
`workflow`	`Y`	`string`	Workflow to be applied to all granules
`granules`	`yes` - if `query` not present	`Array<object>`	List of Granules to process e.g. { granuleId, collectionId }. Required if there is no Elasticsearch query provided
`query`	`yes` - if `granules` not present	`Object`	Query to Elasticsearch to determine which Granules to go through given workflow. Required if no Granules are given.
`index`	`yes` - if `query` is present	`string`	Elasticsearch index to search with the given query
`knexDebug`	`N`	`bool`	Sets knex PostgreSQL connection pool/query debug output. Defaults to false
`queueUrl`	`N`	`string`	URL of SQS queue to use for scheduling granule workflows (e.g. `https://sqs.us-east-1.amazonaws.com/12345/queue-name`)

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST

/granules/bulk

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "workflowName": "HelloWorldWorkflow",
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granules:

curl -X POST
  https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [ { "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD11A1___006" } ], "workflowName": "HelloWorldWorkflow"}'

Example response

{
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e"
}

Bulk Delete

Bulk delete the provided granules. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field	Required	Value	Description
`granules`	`yes` - if `query` not present	`Array<object>`	List of Granules to process e.g. { granuleId, collectionId }. Required if there is no Elasticsearch query provided
`index`	`yes` - if `query` is present	`string`	Elasticsearch index to search with the given query
`query`	`yes` - if `granules` not present	`Object`	Query to Elasticsearch to determine which Granules to delete. Required if no Granules are given
`concurrency`	`N`	`integer`	Sets the granule concurrency for the bulk deletion operation. Defaults to `10`
`forceRemoveFromCmr`	`N`	`bool`	Whether to remove published granules from CMR before deletion. You must set this value to `true` to do bulk deletion of published granules, otherwise deleting them will fail.
`knexDebug`	`N`	`bool`	Sets knex PostgreSQL connection pool/query debug output. Defaults to false
`maxDbConnections`	`N`	`integer`	Sets the maximum database connections to allocate for the operation. Defaults to `concurrency` value

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST

/granules/bulkDelete

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "forceRemoveFromCmr": true,
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granules:

curl -X POST
  https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' --data '{"granules": [ { "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD11A1___006" } ], "forceRemoveFromCmr": true}'

Example response

{
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e"
}

Bulk Reingest

Reingest the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field	Required	Value	Description
`granules`	`yes` - if `query` not present	`Array<object>`	List of Granules to process e.g. { granuleId, collectionId }. Required if there is no Elasticsearch query provided
`index`	`yes` - if `query` is present	`string`	Elasticsearch index to search with the given query
`query`	`yes` - if `ids` not present	`Object`	Query to Elasticsearch to determine which Granules to be reingested. Required if no Granules are given.
`knexDebug`	`N`	`bool`	Sets knex postgreSQL connection pool/query debug output. Defaults to false
`workflowName`	`no`	`string`	optional workflow name that allows different workflow and initial input to be used during reingest. See below.

An optional data parameter of workflowName is also available to allow you to override the input message to the reingest. If workflowName is specified, the original message is pulled directly by finding the most recent execution of the workflowName associated with the granuleId.

Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.

POST

/granules/bulkReingest

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/granules/bulkReingest --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"status": "FAILED"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example request with given Granules and optional workflow parameter:

curl -X POST
  https://example.com/granules/bulkReingest
  --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'
  --data '{
        "granules": [ { "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD11A1___006" } ],
        "workflow": "workflowName",
        }'

Example response

{
    "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e"
}

List PDRs

List PDRs in the Cumulus system.

GET

/pdrs

Example request

$ curl https://example.com/pdrs --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "pdr",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
            "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
            "collectionId": "MOD11A1___006",
            "status": "failed",
            "provider": "LP_TS2_DataPool",
            "progress": 0,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:ACCOUNT:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
            "PANSent": false,
            "PANmessage": "N/A",
            "stats": {
                "processing": 0,
                "completed": 0,
                "failed": 0,
                "total": 0
            },
            "createdAt": 1514305411204,
            "timestamp": 1514305424036,
            "duration": 12.832
        }
    ]
}

Retrieve PDR

Retrieve a single PDR.

GET

/pdrs/{pdrName}

Example request

$ curl https://example.com/pdrs/7970bff5-128a-489f-b43c-de4ad7834ce5.PDR --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
    "collectionId": "MOD11A1___006",
    "status": "failed",
    "provider": "LP_TS2_DataPool",
    "progress": 0,
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:ACCOUNT:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
    "PANSent": false,
    "PANmessage": "N/A",
    "stats": {
        "processing": 0,
        "completed": 0,
        "failed": 0,
        "total": 0
    },
    "createdAt": 1514305411204,
    "timestamp": 1514305424036,
    "duration": 12.832,
    "_id": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR"
}

Delete PDR

Delete a PDR from Cumulus. Its granules will remain, and the PDR may be re-discovered and re-ingested/re-processed from scratch in the future.

DELETE

/pdrs/{pdrName}

Example request

$ curl --request DELETE https://example.com/pdrs/good_25grans.PDR --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

List rules

List rules in the Cumulus system.

GET

/rules

Example request

$ curl https://example.com/rules --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "rule",
        "limit": 1,
        "page": 1,
        "count": 7
    },
    "results": [
        {
            "name": "repeat",
            "workflow": "DiscoverPdrs",
            "provider": "local",
            "collection": {
                "name": "AST_L1A",
                "version": "003"
            },
            "rule": {
                "type": "scheduled",
                "value": "rate(5 minutes)"
            },
            "timestamp": 1511232462534,
            "state": "DISABLED"
        }
    ]
}

Retrieve rule

Retrieve a single rule.

GET

/rules/{name}

Example request

$ curl https://example.com/rules/repeat --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "workflow": "DiscoverPdrs",
    "collection": {
        "name": "AST_L1A",
        "version": "003"
    },
    "updatedAt": 1511232462507,
    "createdAt": 1510903518741,
    "provider": "local",
    "name": "repeat",
    "rule": {
        "type": "scheduled",
        "value": "rate(5 minutes)"
    },
    "state": "DISABLED"
}

Create rule

Create a rule. For more information on creating rules and the contents of a request see the Cumulus setup documentation.

Overview of the schema fields:

Field	Value	Required	Description
`name`	`string`	`yes`	rule name (letters, numbers, underscores only)
`state`	`"DISABLED"`\|`"ENABLED"`	`yes`	rule state (default: ENABLED)
`workflow`	`string`	`yes`	name of workflow started by the rule
`rule`	`Object`	`yes`	rule object
`-- rule.type`	`"onetime"`\|`"scheduled"`\|`"kinesis"`\| `"sns"`\|`"sqs"`	`yes`	rule trigger type
`-- rule.value`	`onetime`: N/A `scheduled`: cron-type or rate expression `kinesis`: Kinesis stream ARN `sns`: SNS topic ARN `sqs`: SQS queue URL	`no`	required value differs by type
`-- rule.arn`	`string`	`no`	`kinesis` scheduled event arn
`-- rule.logEventArn`	`string`	`no`	`kinesis` scheduled log event arn
`collection`	`Object`	`no`	collection record provided to workflow
`-- collection.name`	`string`	`yes`	collection name
`-- collection.version`	`string`	`yes`	collection version
`executionNamePrefix`	`string`	`no`	Execution Name Prefix
`meta`	`Object`	`no`	contents to add to workflow input's `meta` field
`payload`	`string`	`no`	input payload to be used in onetime and scheduled rules
`provider`	`string`	`no`	provider record provided to workflow
`queueUrl`	`string`	`no`	queue URL for Scheduled Executions
`tags`	`array`	`no`	tags (for search)

POST

/rules

Example request

$ curl --request POST https://example.com/rules --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "workflow": "DiscoverPdrs",
    "collection": {
        "name": "AST_L1A",
        "version": "003"
    },
    "provider": "local",
    "name": "repeat_test",
    "rule": {
        "type": "scheduled",
        "value": "rate(5 minutes)"
    },
    "meta": { "publish": false },
    "state": "DISABLED"
}'

Example response

{
    "message": "Record saved",
    "record": {
        "workflow": "DiscoverPdrs",
        "collection": {
            "name": "AST_L1A",
            "version": "003"
        },
        "createdAt": 1510903518741,
        "provider": "local",
        "name": "repeat_test",
        "rule": {
            "type": "scheduled",
            "value": "rate(5 minutes)"
        },
        "meta": { "publish": false },
        "state": "DISABLED"
    }
}

Replace rule

Replace an existing rule. Expects payload to specify the entire rule object, and will completely replace the existing rule with the specified payload. For a field reference see "Create rule".

Returns status 200 on successful replacement, 400 if the name property in the payload does not match the corresponding value in the resource URI, or 404 if there is no rule with the specified name.

Special case:

Field	Value	Description
`action`	`"rerun"`	rerun rule (`onetime` rule only), and rule record is not replaced

PUT

/rules/{name}

Example request

$ curl --request PUT https://example.com/rules/repeat_test \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2' \
  --data '{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED"
}'

Example successful response

{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "updatedAt": 1521755265130,
  "createdAt": 1510903518741,
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED"
}

Rerun request (`onetime`-rule special case)

$ curl --request PUT https://example.com/rules/my_onetime_rule \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2' \
  --data '{
  "name": "repeat_test",
  "workflow": "DiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "state": "ENABLED",
  "action": "rerun"
}'

Update rule

Update an existing rule. Expects payload to contain the modified and/or additional fields of the rule and the existing rule values will be overwritten by the modified portions. Unspecified keys will be retained. Keys set to null will be removed. For a field reference see "Create rule".

Returns status 200 on successful update, 400 if the name property in the payload does not match the corresponding value in the resource URI, or 404 if there is no rule with the specified name.

Special case:

Field	Value	Description
`action`	`"rerun"`	rerun rule (`onetime` rule only), and rule record is not updated

PATCH

/rules/{name}

Example request

$ curl --request PATCH https://example.com/rules/repeat_test \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2' \
  --data '{
  "name": "repeat_test",
  "workflow": "NewDiscoverPdrs",
  "meta": {
    "additionalKey": "additionalKeyValue"
  },
  "state": "ENABLED"
}'

Example successful response

{
  "name": "repeat_test",
  "workflow": "NewDiscoverPdrs",
  "collection": {
    "name": "AST_L1A",
    "version": "003"
  },
  "updatedAt": 1521755265130,
  "createdAt": 1510903518741,
  "provider": "local",
  "rule": {
    "type": "scheduled",
    "value": "rate(5 minutes)"
  },
  "meta": {
    "publish": false,
    "additionalKey": "additionalKeyValue"
  },
  "state": "ENABLED"
}

Rerun request (`onetime`-rule special case)

$ curl --request PATCH https://example.com/rules/my_onetime_rule \
  --header 'Authorization: Bearer ReplaceWithToken' \
  --header 'Content-Type: application/json' \
  --header 'Cumulus-API-Version: 2' \
  --data '{"action": "rerun"}'

Delete rule

Delete a rule from Cumulus.

DELETE

/rules/{name}

Example request

$ curl --request DELETE https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

Summary

Retrieve a summary of statistics around the granules in the system. The collections returned are the number of distinct collections for the granules active during the given time period, defaulted to the last day if none is specified.

GET

/stats

Example Request

$ curl https://example.com/stats --header 'Authorization: Bearer ReplaceWithTheToken'

Example success response

{
    "errors": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 2,
        "aggregation": "count",
        "unit": "error"
    },
    "collections": {
        "dateFrom": "1970-01-01T12:00:00+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 3,
        "aggregation": "count",
        "unit": "collection"
    },
    "processingTime": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": null,
        "aggregation": "average",
        "unit": "second"
    },
    "granules": {
        "dateFrom": "1970-01-18T12:36:59+00:00",
        "dateTo": "2017-12-26T04:38:15+00:00",
        "value": 8,
        "aggregation": "count",
        "unit": "granule"
    }
}

Count

Count the value frequencies for a given field, for a given type of record in Cumulus. Requires the following query parameters, and may include the regular filter parameters:

query string parameter	description
`type={providers\|collections\|granules\|pdrs\|logs}`	type of Cumulus record to query
`field={fieldName}`	which field to count frequencies for; no default

GET

/stats/aggregate

Example request

curl 'https://example.com/stats/aggregate?field=status&type=pdrs' --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "meta": {
    "name": "cumulus-api",
    "count": 52,
    "field": "status.keyword"
  },
  "count": [
    {
      "key": "failed",
      "count": 43
    },
    {
      "key": "completed",
      "count": 5
    },
    {
      "key": "parsed",
      "count": 3
    }
  ]
}

List logs

Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.

List processing logs from the Cumulus engine. A log's level field should be specified by their string value and be one of:

fatal (60)
error (50)
warn (40)
info (30)
debug (20)
trace (10)

GET

/logs

Example request

$ curl https://example.com/logs?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "logs",
        "limit": 5,
        "page": 1,
        "count": 750
    },
    "results": [
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "https://ba008ffc.ngrok.io/ not found",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "HostNotFound: https://ba008ffc.ngrok.io/ not found\n    at discover.discover.then.catch.e (/var/task/index.js:9006:22)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "https://ba008ffc.ngrok.io/ not found",
            "v": 1,
            "timestamp": 1513881407867
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "file": "discover-pdrs/index.js",
            "message": "Received a 404 error from undefined. Check your endpoint!",
            "details": {
                "host": "ba008ffc.ngrok.io",
                "path": "/",
                "port": "",
                "protocol": "https",
                "uriPath": "/",
                "url": "https://ba008ffc.ngrok.io/",
                "depth": 1,
                "fetched": true,
                "status": "notfound",
                "stateData": {
                    "requestLatency": 66,
                    "requestTime": 66,
                    "contentLength": 34,
                    "contentType": "text/plain",
                    "code": 404,
                    "headers": {
                        "content-length": "34",
                        "connection": "close",
                        "content-type": "text/plain"
                    }
                },
                "id": 0
            },
            "v": 1,
            "timestamp": 1513881407867
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Cannot read property 'provider_path' of undefined",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "TypeError: Cannot read property 'provider_path' of undefined\n    at HttpDiscover.Discover (/var/task/index.js:10098:33)\n    at HttpDiscover._class (/var/task/index.js:96368:255)\n    at new HttpDiscover (/var/task/index.js:10342:241)\n    at handler (/var/task/index.js:8968:23)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "v": 1,
            "timestamp": 1513881031694
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Provider info not provided",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "ProviderNotFound: Provider info not provided\n    at handler (/var/task/index.js:8962:20)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "Provider info not provided",
            "v": 1,
            "timestamp": 1513879654040
        },
        {
            "pid": 1,
            "hostname": "ip-10-29-4-186",
            "name": "cumulus",
            "level": 50,
            "msg": "Provider info not provided",
            "file": "discover-pdrs/index.js",
            "type": "Error",
            "stack": "ProviderNotFound: Provider info not provided\n    at handler (/var/task/index.js:8962:20)\n    at Promise (/var/task/cumulus-sled/index.js:82:5)\n    at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n    at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n    at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
            "message": "Provider info not provided",
            "v": 1,
            "timestamp": 1513879435010
        }
    ]
}

Example request for errors

$ curl https://example.com/logs?level=error --header 'Authorization: Bearer ReplaceWithTheToken'

Example response for errors

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "logs",
        "limit": 1,
        "page": 1,
        "count": 181
    },
    "results": [
        {
            "level": 50,
            "msg": "Unexpected HTTP status code: 404",
            "pid": 1,
            "hostname": "ip-10-26-125-174",
            "name": "cumulus",
            "file": "sync-granule/index.js",
            "type": "Error",
            "stack": "Error: Unexpected HTTP status code: 404\n    at ClientRequest.<anonymous> (/var/task/sync-granule/index.js:125136:29)\n    at ClientRequest.g (events.js:292:16)\n    at emitOne (events.js:96:13)\n    at ClientRequest.emit (events.js:188:7)\n    at HTTPParser.parserOnIncomingClient (_http_client.js:473:21)\n    at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)\n    at TLSSocket.socketOnData (_http_client.js:362:20)\n    at emitOne (events.js:96:13)\n    at TLSSocket.emit (events.js:188:7)\n    at readableAddChunk (_stream_readable.js:176:18)",
            "code": 404,
            "v": 1,
            "timestamp": 1515446810226
        }
    ]
}

Retrieve Logs

Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.

Retrieve all logs from a specific execution.

GET

/logs/{executionName}

Example request

$ curl https://example.com/logs/93e5e049-89aa-47e5-900d-0eec87327260?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "test-src-integration",
        "table": "logs",
        "limit": 5,
        "page": 1,
        "count": 10
    },
    "results": [
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:57.752Z",
            "sender": "test-src-integration-PostToCmr",
            "version": "$LATEST",
            "message": "Published MOD09GQ.A8009365.7JQhni.006.6594428626875 to the CMR. conceptId: G1223210348-CUMULUS",
            "RequestId": "05712fb1-5593-4640-b5a3-a99db41d8003"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:42.054Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.913Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "Finishing downloading s3://cumulus-test-sandbox-internal/cumulus-test-data/pdrs/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.888Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875_ndvi.jpg",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        },
        {
            "level": 30,
            "executions": "93e5e049-89aa-47e5-900d-0eec87327260",
            "timestamp": "2018-08-15T19:23:41.886Z",
            "sender": "test-src-integration-SyncGranuleNoVpc",
            "version": "$LATEST",
            "message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf.met",
            "RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
        }
    ]
}

Get Granule CSV file

Get a CSV file of all the granule in the Cumulus database.

GET

/granule-csv

Example request

$ curl https://example.com/granule-csv --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published"
"MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false
"MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true

List executions

List executions.

GET

/executions

Example request

$ curl https://example.com/executions?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "meta": {
    "name": "cumulus-api",
    "stack": "test-src-integration",
    "table": "execution",
    "limit": 3,
    "page": 1,
    "count": 447
  },
  "results": [
    {
      "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "duration": 1.693,
      "createdAt": 1534443719571,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
      "error": {},
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "running",
      "timestamp": 1534443721559,
      "updatedAt": 1534443721265
    },
    {
      "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
      "duration": 1.718,
      "createdAt": 1534443706867,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
      "name": "bee18782-e987-4bd4-b726-41669e489e2f",
      "error": {},
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "running",
      "timestamp": 1534443709043,
      "updatedAt": 1534443708585
    },
    {
      "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "duration": 29.397,
      "createdAt": 1534443668185,
      "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
      "error": {
        "Cause": "None",
        "Error": "Unknown Error"
      },
      "type": "IngestGranule",
      "collectionId": "MOD09GQ___006",
      "tasks": {},
      "status": "completed",
      "timestamp": 1534443698322,
      "updatedAt": 1534443697582
    }
  ]
}

Retrieve Execution

Retrieve details for a specific execution.

GET

/executions/{executionArn}

Example request

$ curl https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "updatedAt": 1534441782302,
    "status": "completed",
    "timestamp": 1534441782302,
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "createdAt": 1534441752026,
    "duration": 30.276,
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "collectionId": "MOD09GQ___006",
    "error":
    {
        "Error": "Unknown Error",
        "Cause": "None"
    },
    "type": "IngestGranule",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" }
}

Retrieve Execution Status

Retrieve details and status of a specific execution. This also returns execution history and details of the state machine when the execution exists in Step Function API.

GET

/executions/status/{executionArn}

Example request

$ curl https://example.com/executions/status/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl6sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "presignedS3Url": "https://example.amazonaws.com/example",
  "data": "Execution status data, see example below"
}

{
  "presignedS3Url": "https://example.amazonaws.com/example",
  "data": "Error: Execution Status for execution123 exceeded maximum allowed payload size"
}

Execution status

{
    "execution":
    {
        "executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2",
        "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "name": "d0a6584b-bea6-476e-a745-c1feb2ad00b2",
        "status": "SUCCEEDED",
        "startDate": "2018-08-16T18:39:32.209Z",
        "stopDate": "2018-08-16T18:39:59.089Z",
        "input":
        {
            "foo": "input"
        },
        "output":
        {
            "bar": "output"
        }
    },
    "executionHistory":
    {
        "events":
        [
            {
                "taskName": "foo bar 1"
            },
            {
                "taskName": "foo bar 2"
            }
        ]
    },
    "stateMachine":
    {
        "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "name": "TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
        "status": "ACTIVE",
        "definition":
        {
            "Comment": "Ingest Granule",
            "StartAt": "First State",
            "States":
            {
                "First State": {}
            }
        },
        "roleArn": "arn:aws:iam::123456789012:role/test-src-integration-steprole",
        "creationDate": "2018-06-11T16:08:17.533Z"
    }
}

Search Executions By Granules

Return executions associated with specific granules. Granules can be sent as a list of granule objects containing granuleIds and collectionIds or as an Elasticsearch query to the Metrics' Elasticsearch.

Overview of the request fields:

Field	Required	Value	Description
`granules`	`yes` - if `query` not present	`Array<Object>`	List of granules to process. Each granule must contain a granuleId and collectionId
-- `granule.granuleId`	`yes` - if using `granules`	`string`	GranuleId for a granule
-- `granule.collectionId`	`yes` - if using `granules`	`string`	CollectionId for a granule
`query`	`yes` - if `granules` not present	`Object`	Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given.
`index`	`yes` - if `query` is present	`string`	Elasticsearch index to search with the given query

POST

/executions/search-by-granules

Example request with given Granules:

curl -X POST
  https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"collectionId": "MOD09GQ__006"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example response

{
   "meta": {
       "count": 3
    },
    "results": [
        {
            "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
            "duration": 1.693,
            "createdAt": 1534443719571,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
            "name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
            "error": {},
            "type": "IngestGranule",
            "collectionId": "MOD09GQ___006",
            "tasks": {},
            "status": "running",
            "timestamp": 1534443721559,
            "updatedAt": 1534443721265
        },
        {
            "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
            "duration": 1.718,
            "createdAt": 1534443706867,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
            "name": "bee18782-e987-4bd4-b726-41669e489e2f",
            "error": {},
            "type": "IngestGranule",
            "collectionId": "MOD09GQ___006",
            "tasks": {},
            "status": "running",
            "timestamp": 1534443709043,
            "updatedAt": 1534443708585
        },
        {
            "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
            "duration": 29.397,
            "createdAt": 1534443668185,
            "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
            "name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
            "error": {
            "Cause": "None",
            "Error": "Unknown Error"
            },
            "type": "IngestGranule",
            "collectionId": "MOD09GQ___006",
            "tasks": {},
            "status": "completed",
            "timestamp": 1534443698322,
            "updatedAt": 1534443697582
        }
    ]
}

Workflows By Granules

Return the workflows that have run on specific granules. If multiple granules are specified, it will return the intersection of workflows that have run on ALL the granules. That is, only a workflow that has been run on every inputted granule will be returned

Overview of the request fields:

Field	Required	Value	Description
`granules`	`yes` - if `query` not present	`Array<Object>`	List of granules to process. Each granule must contain a granuleId and collectionId
-- `granule.granuleId`	`yes` - if using `granules`	`string`	GranuleId for a granule
-- `granule.collectionId`	`yes` - if using `granules`	`string`	CollectionId for a granule
`query`	`yes` - if `granules` not present	`Object`	Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given.
`index`	`yes` - if `query` is present	`string`	Elasticsearch index to search with the given query

POST

/executions/workflows-by-granules

Example request with given Granules:

curl -X POST
  https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'

Example request with Elasticsearch query generated by Kibana:

$ curl --request POST \
    https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "index": "index-in-es",
        "query": {
            "size": 500,
            "query": {
                "filter": [
                    {
                        "bool": {
                            "filter": [
                                {
                                    "bool": {
                                        "should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
                                        "minimum_should_match": 1
                                    }
                                },
                                {
                                    "bool": {
                                        "should": [{"match": {"collectionId": "MOD09GQ__006"}}],
                                        "minimum_should_match": 1
                                    }
                                }
                            ]
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    }'

Example response

[
    "DiscoverGranules",
    "IngestGranules"
]

Create execution

Create an execution. An execution generally includes the fields listed below.

Overview of the schema fields:

Field	Type	Required	Description
`arn`	`string`	`yes`	execution arn (this field is unique)
`asyncOperationId`	`string`	`no`	`id` of the associated async operation
`collectionId`	`string`	`no`	collectionId of the associated collection (e.g. `<name>___<version>` where `<name>` is the collection name and `<version>` is the collection version)
`cumulusVersion`	`string`	`no`	cumulus version for the execution (e.g. `9.0.0`)
`duration`	`number`	`no`	duration of the execution
`error`	`object`	`no`	the error details in case of a failed execution
`execution`	`string`	`no`	the execution page url on AWS console
`finalPayload`	`object`	`no`	the final payload of this execution
`name`	`string`	`yes`	execution name
`originalPayload`	`object`	`no`	the original payload for this execution
`parentArn`	`string`	`no`	arn of the parent execution
`status`	`string`	`yes`	the execution status, possible values: `running`, `completed`, `failed`, `unknown`
`tasks`	`object`	`no`	the tasks for this execution
`type`	`string`	`no`	the workflow name, e.g. IngestGranule
`timestamp`	`number`	`no`	the timestamp for this execution

POST

/executions

Example request

$ curl --request POST https://example.com/executions --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
    "collectionId": "MOD09GQ___006",
    "cumulusVersion": "1.2.3",
    "error": {},
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
    "status": "running",
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "type": "IngestGranule",
    "timestamp": 1513020462156
}'

Example response

{
    "message": "Successfully wrote execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}

Update or replace execution

Update/replace an existing execution. Expects payload to specify the entire execution object, and will completely replace the existing execution with the specified payload. For a field reference see "Create execution".

Returns status 200 on successful replacement, 400 if the arn property in the payload does not match the corresponding value in the resource URI, or 404 if there is no execution with the specified arn.

PUT

/executions/{executionArn}

Example request

$ curl --request PUT https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
    "arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
    "collectionId": "MOD09GQ___006",
    "cumulusVersion": "1.2.3",
    "duration": 30.276,
    "error": {},
    "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
    "finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" },
    "name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
    "originalPayload": { "somePayloadKey": "object containing payload at execution start" },
    "parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
    "status": "completed",
    "tasks":
    {
        "MoveGranuleStep":
        {
            "name": "test-src-integration-MoveGranules",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
            "version": "$LATEST"
        },
        "ProcessingStep":
        {
            "name": "test-src-integration-FakeProcessing",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
            "version": "$LATEST"
        },
        "StopStatus":
        {
            "name": "test-src-integration-SfSnsReport",
            "arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
            "version": "$LATEST"
        }
    },
    "type": "IngestGranule",
    "timestamp": 1513020462156
}'

Example response

{
    "message": "Successfully updated execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}

Delete execution

Delete an execution from Cumulus.

DELETE

/executions/{executionArn}

Example request

$ curl --request DELETE https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Record deleted"
}

Bulk Delete Executions by `collectionId`

This request, when given an existing collectionId will trigger a bulk action that will take the following actions:

Remove all records from elastic search matching the collectionId (in a single query with a limit specified by esBatchSize in the payload)
Remove all records from the postgreSQL database matching the collectionId (in a single query with a limit specified by dbBatchSize in the payload)
- Records will be removed in the order postgreSQL returns them. Where deletion would cause a violation with a record's parent_cumulus_id field constraints, the child record will have that field set to null.
Failure on any deletion will result in operation failure - if that occurs, the issue will need to be corrected and this operation can be re-run.
This operation runs asynchronously, and will return an asynchronous operation ID

$ curl --request POST \
    https://example.com/executions/bulk-delete-by-collection --header 'Authorization: Bearer ReplaceWithTheToken' \
  --header 'Content-Type: application/json' \
  --data '{
        "collectionId": "COLLECTION_NAME___COLLECTION_VERSION",
        "esBatchSize": 100000,
        "dbBatchSize": 50000
    }'

POST

/executions/bulk-delete-by-collection

Example request

Example response

{
    "id": "12345678-2222-3333-4444-1234567890ab"                        
}

List workflows

List workflows in the Cumulus system.

GET

/workflows

Example request

$ curl https://example.com/workflows --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

[
    {
        "name": "TestLambdaVersionWorkflow",
        "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/TestLambdaVersionWorkflow.json",
        "definition": {
            "Comment": "Tests Lambda update after redeploy",
            "StartAt": "StartStatus",
            "States": {
                "StartStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Next": "WaitForDeployment"
                },
                "WaitForDeployment": {
                    "Type": "Task",
                    "Resource": "${WaitForDeploymentLambdaAliasOutput}",
                    "Next": "VersionUpTest"
                },
                "VersionUpTest": {
                    "Type": "Task",
                    "Resource": "${VersionUpTestLambdaAliasOutput}",
                    "Next": "StopStatus"
                },
                "StopStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Catch": [
                        {
                            "ErrorEquals": [
                                "States.ALL"
                            ],
                            "Next": "WorkflowFailed"
                        }
                    ],
                    "End": true
                },
                "WorkflowFailed": {
                    "Type": "Fail",
                    "Cause": "Workflow failed"
                }
            }
        }
    },
    {
        "name": "HelloWorldWorkflow",
        "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
        "definition": {
            "Comment": "Returns Hello World",
            "StartAt": "StartStatus",
            "States": {
                "StartStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Next": "HelloWorld"
                },
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": "${HelloWorldLambdaAliasOutput}",
                    "Next": "StopStatus"
                },
                "StopStatus": {
                    "Type": "Task",
                    "Resource": "${SfSnsReportLambdaAliasOutput}",
                    "Catch": [
                        {
                            "ErrorEquals": [
                                "States.ALL"
                            ],
                            "Next": "WorkflowFailed"
                        }
                    ],
                    "End": true
                },
                "WorkflowFailed": {
                    "Type": "Fail",
                    "Cause": "Workflow failed"
                }
            }
        }
    }
]

Retrieve workflow

Retrieve a single workflow.

GET

/workflow/{name}

Example request

$ curl https://example.com/workflows/HelloWorldWorkflow --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "name": "HelloWorldWorkflow",
    "template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
    "definition": {
        "Comment": "Returns Hello World",
        "StartAt": "StartStatus",
        "States": {
            "StartStatus": {
                "Type": "Task",
                "Resource": "${SfSnsReportLambdaAliasOutput}",
                "Next": "HelloWorld"
            },
            "HelloWorld": {
                "Type": "Task",
                "Resource": "${HelloWorldLambdaAliasOutput}",
                "Next": "StopStatus"
            },
            "StopStatus": {
                "Type": "Task",
                "Resource": "${SfSnsReportLambdaAliasOutput}",
                "Catch": [
                    {
                        "ErrorEquals": [
                            "States.ALL"
                        ],
                        "Next": "WorkflowFailed"
                    }
                ],
                "End": true
            },
            "WorkflowFailed": {
                "Type": "Fail",
                "Cause": "Workflow failed"
            }
        }
    }
}

List async operations

Async operations are long-running requests serviced by the Cumulus API.

These tend to be bulk operations. Examples include bulk granule operations and replaying ingest notifications.

This endpoint lists async operations in the Cumulus system.

GET

/asyncOperations

Example request

$ curl https://example.com/asyncOperations --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "asyncOperation",
        "limit": 1,
        "page": 1,
        "count": 8
    },
    "results": [
        {
          "output": "{\"deletedGranules\":[\"granule-id-f02a53418f\"]}",
          "createdAt": 1591384094512,
          "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
          "description": "Bulk granule deletion",
          "operationType": "Bulk Granule Delete",
          "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
          "status": "SUCCEEDED",
          "updatedAt": 1591384094512,
          "timestamp": 1591384095235
        }
    ]
}

Retrieve async operation

Retrieve information about a single async operation. Useful for determining the status of an async operation.

GET

/asyncOperations/{id}

Example request

$ curl https://example.com/asyncOperations/0eb8e809-8790-5409-1239-bcd9e8d28b8e --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
  "updatedAt": 1574730504762,
  "status": "RUNNING",
  "taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
  "description": "Bulk granule deletion",
  "operationType": "Bulk Granule Delete"
}

Ingest Replays

The Cumulus API supports requests to replay ingest notifications. The schema below describes the expected fields in a replay request

Field	Type	Required	Description
`type`	string	required	Currently only accepts `kinesis`.
`kinesisStream`	string	for type `kinesis`	Any valid kinesis stream name (not ARN)
`kinesisStreamCreationTimestamp`	*	optional	Any input valid for a JS Date constructor. For reasons to use this field see AWS documentation on StreamCreationTimestamp.
`endTimestamp`	*	optional	Any input valid for a JS Date constructor. Messages newer than this timestamp will be skipped.
`startTimestamp`	*	optional	Any input valid for a JS Date constructor. Messages will be fetched from the Kinesis stream starting at this timestamp. Ignored if it is further in the past than the stream's retention period.

Example Request

$ curl -X POST https://example.com/replays --header 'Authorization: Bearer ReplaceWithTheToken' --data '{ "type: "kinesis", "kinesisStream": "my-stream", "endTimestamp": 1567890123456, "startTimestamp": "2019-08-31T22:22:03.456Z"}'

Example Response

{
  "asyncOperationId": "208a463a-e096-4dd9-b006-e09345319ae6"
}

Retrieve schema

Retrieve the data schema for a particular type of Cumulus record.

This schema describes the expected format of a record's JSON object when retrieving from Cumulus, as well as a summary of what each field may contain. The schema response can also be used to determine which fields are required when creating a new record using the API.

Supported type values are provider, collection, granule, and pdr.

GET

/schemas/{type}

Example request

$ curl https://example.com/schemas/provider --header 'Authorization: Bearer ReplceWithTheToken'

Example response

{
  "$schema": "http://json-schema.org/draft-04/schema#",
  "title": "Provider Object",
  "description": "Keep the information about each ingest endpoint",
  "type": "object",
  "properties": {
    "name": {
      "title": "Title",
      "description": "A title for the provider record",
      "type": "string",
      "pattern": "^([\\w\\d_\\-]*)$"
    },
    "providerName": {
      "title": "Provider, e.g. MODAPS",
      "description": "Name of the SIP",
      "type": "string"
    },
    "protocol": {
      "title": "Protocol",
      "type": "string",
      "enum": [
        "http",
        "ftp"
      ],
      "default": "http"
    },
    "host": {
      "title": "Host",
      "type": "string"
    },
    "path": {
      "title": "Path to the PDR/files folder",
      "type": "string"
    },
    "config": {
      "title": "Configuration",
      "type": "object",
      "properties": {
        "username": {
          "type": "string"
        },
        "password": {
          "type": "string"
        },
        "port": {
          "type": "string"
        }
      }
    },
    "status": {
      "title": "Status",
      "type": "string",
      "enum": [
        "ingesting",
        "stopped",
        "failed"
      ],
      "default": "stopped",
      "readonly": true
    },
    "isActive": {
      "title": "Is Active?",
      "type": "boolean",
      "default": false,
      "readonly": true
    },
    "regex": {
      "type": "object",
      "patternProperties": {
        "^([\\S]*)$": {
          "type": "string"
        }
      },
      "readonly": true
    },
    "lastTimeIngestedAt": {
      "title": "Last Time Ingest from the Provider",
      "type": "number",
      "readonly": true
    },
    "createdAt": {
      "type": "number",
      "readonly": true
    },
    "updatedAt": {
      "type": "number",
      "readonly": true
    }
  },
  "required": [
    "name",
    "providerName",
    "protocol",
    "host",
    "path",
    "isActive",
    "status",
    "createdAt",
    "updatedAt"
  ]
}

List reconciliation reports

List reconciliation reports in the Cumulus system.

GET

/reconciliationReports

Example request

$ curl https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
    "meta": {
        "name": "cumulus-api",
        "stack": "lpdaac-cumulus",
        "table": "reconciliationReport",
        "limit": 2,
        "page": 1,
        "count": 7
    },
    "results": [
        {
            "createdAt": 1589824212102,
            "name": "inventoryReport-20200518T175012101",
            "location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/inventoryReport-20200518T175012101.json",
            "type": "Inventory",
            "status": "Generated",
            "updatedAt": 1589824218825,
            "timestamp": 1589824219329
        },
        {
            "createdAt": 1589596344305,
            "name": "ModisInternalReport",
            "location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/ModisInternalReport.json",
            "type": "Internal",
            "status": "Generated",
            "updatedAt": 1589596352010,
            "timestamp": 1589596352356
        }
    ]
}

Retrieve reconciliation report

Retrieve a single reconciliation report.

GET

/reconciliationReports/{name}

Example request

$ curl https://example.com/reconciliationReports/inventoryReport-20190305T153430508 --header 'Authorization: Bearer ReplaceWithTheToken'

Example responses

{
  "presignedS3Url": "https://example.amazonaws.com/example",
  "data": "report data, see examples below for each report type"
}

{
  "presignedS3Url": "https://example.amazonaws.com/example",
  "data": "Error: Report examplereport exceeded maximum allowed payload size"
}

Inventory Report

{
    "reportStartTime": "2019-03-05T15:34:30.508Z",
    "reportEndTime": "2019-03-05T15:34:37.243Z",
    "status": "SUCCESS",
    "error": null,
    "reportType": "Inventory",
    "filesInCumulus": {
        "okCount": 40,
        "okCountByGranule": {
          "MOD09GQ.A2016358.h13v04.006.2016360104606": 10,
          "MYD13Q1.A2017297.h19v10.006.2017313221303": 10,
          "MOD09GQ.A3518809.ln_rVr.006.7962927138074": 10,
          "MOD09GQ.A8768252.HC4ddD.006.2077696236118": 5,
          "MOD09GQ.A8722843.GTk5A3.006.4026909316904": 5
        },
        "onlyInS3": [
            "s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.cmr.xml",
            "s3://cumulus-test-sandbox-private/BROWSE.MYD13Q1.A2017297.h19v10.006.2017313221201.hdf"
        ],
    },
    "collectionsInCumulusCmr": {
        "okCount": 1,
        "onlyInCumulus": [
            "L2_HR_PIXC___000"
        ],
        "onlyInCmr": [
            "MCD43A1___006",
            "MOD14A1___006"
        ]
    },
    "granulesInCumulusCmr": {
        "okCount": 3,
        "onlyInCumulus": [
            {
                "granuleId": "MOD09GQ.A3518809.ln_rVr.006.7962927138074",
                "collectionId": "MOD09GQ___006"
            },
            {
                "granuleId": "MOD09GQ.A8768252.HC4ddD.006.2077696236118",
                "collectionId": "MOD09GQ___006"
            }
        ],
        "onlyInCmr": [
            {
                "GranuleUR": "MOD09GQ.A0002421.oD4zvB.006.4281362831355",
                "ShortName": "MOD09GQ",
                "Version": "006"
            },
            {
                "GranuleUR": "MOD09GQ.A0007443.H16AHq.006.7078190437865",
                "ShortName": "MOD09GQ",
                "Version": "006"
            }
        ]
    },
    "filesInCumulusCmr": {
        "okCount": 11,
        "onlyInCumulus": [
            {
                "fileName": "MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
                "uri": "s3://cumulus-test-sandbox-public/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
                "granuleId": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
            }
        ],
        "onlyInCmr": [
            {
                "URL": "https://example.com/MYD13Q1.A2017297.h19v10.006.2017313221202.hdf",
                "Type": "GET DATA",
                "GranuleUR": "MOD09GQ.A4675287.SWPE5_.006.7310007729190"
            },
            {
                "URL": "https://cumulus-test-sandbox-public.s3.amazonaws.com/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904_ndvi.jpg",
                "Type": "GET DATA",
                "GranuleUR": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
            }
        ]
    }
}

Granule Not Found Report

{
  "createStartTime": "2020-08-28T18:33:50.525Z",
  "createEndTime": "2020-08-28T18:34:16.646Z",
  "status": "SUCCESS",
  "reportType": "Granule Not Found",
  "filesInCumulus": {
    "okCount": 20,
    "okCountByGranule": {
      "MOD09GQ.A2770300.W1_V5Z.006.7853319756315": 2,
      "MOD09GQ.A7682091.QtZjhI.006.1218763030745": 3,
      "MOD09GQ.A9536857.bV1q4A.006.0980635705136": 3,
      "MOD09GQ.A7790080.xLPpTZ.006.2292724247258": 3,
      "MOD09GQ.A3769578.tlqq7j.006.4434181201872": 3,
      "MOD09GQ.A5423294.RFOrMI.006.8677601711674": 3,
      "MOD09GQ.A8837603.fb2Vhw.006.3950746589733": 3
    },
    "onlyInS3": [
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf.met",
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf.met",
      "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593198955197/MOD09GQ.A7375038.gm1irM.006.1317280710645.hdf.met",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf",
      "s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122881446/MOD09GQ.A2708681.CFkGhW.006.7154000014360.hdf"
    ],
  },
  "collectionsInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [
      "MOD09GQ_test-test-data-1593122280799___006",
      "MOD09GQ_test-test-data-1593122597517___006",
      "MOD09GQ_test-test-data-1593122881446___006",
      "MOD09GQ_test-test-data-1593178175944___006",
      "MOD09GQ_test-test-data-1593198739991___006",
      "MOD09GQ_test-test-data-1593198955197___006",
      "MOD09GQ_test-test-data-1593199179316___006",
      "MOD09GQ_test-test-data-1593199379860___006",
      "MOD09GQ_test-test-data-1593199548454___006",
      "MOD09GQ_test-test-data-1593199742089___006",
      "MOD09GQ_test-test-data-1593199934835___006"
    ],
    "onlyInCmr": [
      "A2_RainOcn_NRT___0",
      "A2_SI12_NRT___0",
      "A2_SI25_NRT___0",
      "A2_SI6_NRT___0",
      "AST_L1A___003",
      "CMR44JK___001",
      "L2_HR_PIXC___000",
      "L2_HR_PIXC___1",
      "MCD43A1___006",
      "MOD09GQ___006",
      "MOD11A1___006",
      "MOD14A1___006",
      "MUR-JPL-L4-GLOB-v4.1___1",
      "MYD09A1___006",
      "MYD13A1___006",
      "MYD13Q1___006",
      "hs3avaps___1",
      "hs3wwlln___1"
    ]
  },
  "granulesInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [],
    "onlyInCmr": []
  },
  "filesInCumulusCmr": {
    "okCount": 0,
    "onlyInCumulus": [],
    "onlyInCmr": []
  }
}

Internal Report

{
  "reportType": "Internal",
  "createStartTime": "2020-08-28T19:01:50.636Z",
  "createEndTime": "2020-08-28T19:02:08.211Z",
  "reportStartTime": "2020-01-28T18:21:31.119Z",
  "reportEndTime": "2020-08-28T18:21:31.119Z",
  "status": "SUCCESS",
  "collections": {
    "okCount": 10,
    "withConflicts": [
      {
        "es": {
          "createdAt": 1591902043153,
          "granuleId": "^.*$",
          "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
          "dataType": "L2_HR_PIXC",
          "name": "L2_HR_PIXC",
          "files": [
            {
              "bucket": "private",
              "regex": ".*.h5$",
              "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5"
            }
          ],
          "granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
          "reportToEms": true,
          "version": "000",
          "duplicateHandling": "error",
          "updatedAt": 1591902043153,
          "timestamp": 1591902043840
        },
        "db": {
          "duplicateHandling": "error",
          "granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
          "version": "000",
          "dataType": "L2_HR_PIXC",
          "files": [
            {
              "bucket": "private",
              "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
              "regex": ".*.h5$"
            }
          ],
          "updatedAt": 1591902043153,
          "createdAt": 1591902043153,
          "reportToEms": false,
          "granuleId": "^.*$",
          "sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
          "name": "L2_HR_PIXC"
        }
      }
    ],
    "onlyInEs": [
      "MCD43A1___006",
      "MOD09GQ___006"
    ],
    "onlyInDb": [
      "MYD13QA___005"
    ]
  },
  "granules": {
    "okCount": 77,
    "withConflicts": [
      {
        "es": {
          "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
          "processingEndDateTime": "2020-05-18T19:49:04.769Z",
          "published": false,
          "error": {},
          "productVolume": 253949,
          "timeToPreprocess": 0,
          "duration": 1.005,
          "createdAt": 1589831343764,
          "granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
          "processingStartDateTime": "2020-05-18T19:49:03.831Z",
          "provider": "s3_provider",
          "timeToArchive": 0,
          "files": [
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "size": 233840,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "type": "data",
              "key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf"
            },
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "size": 14297,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "type": "metadata",
              "key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met"
            },
            {
              "bucket": "cumulus-test-sandbox-internal",
              "fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "size": 5812,
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "type": "browse",
              "key": "test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg"
            }
          ],
          "collectionId": "MOD14A1___006",
          "status": "running",
          "timestamp": 1589831367756,
          "updatedAt": 1589831344769
        },
        "db": {
          "published": true,
          "endingDateTime": "2017-11-08T23:59:59Z",
          "status": "completed",
          "timestamp": 1589831361039,
          "createdAt": 1589831343764,
          "processingEndDateTime": "2020-05-18T19:49:20.023Z",
          "productVolume": 258297,
          "timeToPreprocess": 0.263,
          "timeToArchive": 0.942,
          "productionDateTime": "2018-07-19T12:01:01Z",
          "execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
          "files": [
            {
              "bucket": "cumulus-test-sandbox-protected",
              "key": "MOD14A1___006/2017/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "size": 233840,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
              "type": "data"
            },
            {
              "bucket": "cumulus-test-sandbox-private",
              "key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "size": 14297,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
              "type": "metadata"
            },
            {
              "bucket": "cumulus-test-sandbox-public",
              "key": "MOD14A1___006/BRO/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "size": 5812,
              "fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
              "type": "browse"
            },
            {
              "bucket": "cumulus-test-sandbox-protected-2",
              "key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
              "size": 4348,
              "fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
              "type": "metadata"
            }
          ],
          "cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1234771093-CUMULUS",
          "processingStartDateTime": "2020-05-18T19:49:03.831Z",
          "updatedAt": 1589831361039,
          "beginningDateTime": "2017-10-24T00:00:00Z",
          "provider": "s3_provider",
          "granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
          "collectionId": "MOD14A1___006",
          "duration": 17.275,
          "error": {
            "Error": "Unknown Error",
            "Cause": "None"
          },
          "lastUpdateDateTime": "2018-04-25T21:45:45.524053"
        }
      }
    ],
    "onlyInEs": [
      {
        "granuleId": "MYD13Q1.A0323210.Yujnv6.006.1973526114346",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1589913461833,
        "updatedAt": 1589913522990
      }
    ],
    "onlyInDb": [
      {
        "granuleId": "MYD13Q1.A5822875.qHI8hK.006.8698738069249",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1598552797312,
        "updatedAt": 1598552827611
      }
    ]
  }
}

ORCA Backup Report

{
  "collectionIds": [
    "MYD13Q1___006",
    "MOD09GQ___006"
  ],
  "createEndTime": "2022-02-11T19:13:41.986Z",
  "createStartTime": "2022-02-11T19:13:41.153Z",
  "granuleIds": [
    "MYD13Q1.A3194547.tnYsne.006.8400707913298",
    "MYD13Q1.A2655880.GOWVT9.006.3531712476486",
    "MOD09GQ.A8858216.Y6HJnu.006.6168319936421"
  ],
  "providers": [
    "modas_provider",
    "s3_provider"
  ],
  "reportEndTime": "2022-02-11T19:12:25.000Z",
  "reportType": "ORCA Backup",
  "status": "SUCCESS",
  "granules": {
    "okCount": 0,
    "cumulusCount": 2,
    "orcaCount": 2,
    "okFilesCount": 5,
    "conflictFilesCount": 6,
    "withConflicts": [
      {
        "okFilesCount": 4,
        "granuleId": "MYD13Q1.A3194547.tnYsne.006.8400707913298",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1644606673827,
        "updatedAt": 1644606718024,
        "conflictFiles": [
          {
            "fileName": "MYD13Q1.A3194547.tnYsne.006.8400707913298.cmr.xml",
            "bucket": "cumulus-test-sandbox-protected-2",
            "key": "MYD13Q1___006/MYD/MYD13Q1.A3194547.tnYsne.006.8400707913298.cmr.xml",
            "reason": "shouldBeExcludedFromOrca"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg2",
            "bucket": "cumulus-test-sandbox-public",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg2",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg",
            "bucket": "cumulus-test-sandbox-public",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A3194547.tnYsne.006.8400707913298.1.jpg",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          }
        ]
      }
    ],
    "onlyInCumulus": [
      {
        "okFilesCount": 1,
        "granuleId": "MYD13Q1.A2655880.GOWVT9.006.3531712476486",
        "collectionId": "MYD13Q1___006",
        "provider": "s3_provider",
        "createdAt": 1644606673656,
        "updatedAt": 1644606683360,
        "conflictFiles": [
          {
            "fileName": "MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "bucket": "cumulus-test-sandbox-protected",
            "key": "MYD13Q1___006/2017/MYD/MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "bucket": "cumulus-test-sandbox-private",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.hdf",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.1.jpg",
            "bucket": "cumulus-test-sandbox-public",
            "key": "MYD13Q1___006/BRO/BROWSE.MYD13Q1.A2655880.GOWVT9.006.3531712476486.1.jpg",
            "reason": "onlyInCumulus"
          },
          {
            "fileName": "MYD13Q1.A2655880.GOWVT9.006.3531712476486.cmr.xml",
            "bucket": "cumulus-test-sandbox-protected-2",
            "key": "MYD13Q1___006/MYD/MYD13Q1.A2655880.GOWVT9.006.3531712476486.cmr.xml",
            "reason": "onlyInCumulus"
          }
        ]
      }
    ],
    "onlyInOrca": [
      {
        "granuleId": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421",
        "provider": "s3_provider",
        "collectionId": "MOD09GQ___006",
        "createdAt": 1643920768281,
        "conflictFiles": [
          {
            "bucket": "cumulus-test-sandbox-protected",
            "key": "MOD09GQ___006/2017/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421.hdf",
            "fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421.hdf",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          },
          {
            "bucket": "cumulus-test-sandbox-public",
            "key": "MOD09GQ___006/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421_ndvi.jpg",
            "fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421_ndvi.jpg",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          },
          {
            "bucket": "cumulus-test-sandbox-protected-2",
            "key": "MOD09GQ___006/MOD/MOD09GQ.A8858216.Y6HJnu.006.6168319936421.cmr.xml",
            "fileName": "MOD09GQ.A8858216.Y6HJnu.006.6168319936421.cmr.xml",
            "orcaBucket": "cumulus-test-sandbox-orca-glacier",
            "reason": "onlyInOrca"
          }
        ]
      }
    ]
  }
}

Granule Inventory Report

"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published"
"MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false
"MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true

Create reconciliation report

Create a new reconciliation report.

POST

/reconciliationReports

Request body

parameter	value	required	description
`reportName`	`string`	`false`	Report name.
`reportType`	`"Granule Inventory"`\|`"Granule Not Found"`\|`"Internal"`\|`"Inventory"`\|`"ORCA Backup"`	`false`	Report type (default Inventory)
`startTimestamp`	`string`	`false`	Any input valid for a JS Date contstructor. Data older than this will be ignored in the generated report.
`endTimestamp`	`string`	`false`	Any input valid for a JS Date contstructor. Data newer than this will be ignored in the generated report.
`collectionId`	[`string \| array`]	`false`	collectionId (or array of collectionIds) for comparison of collection and granule holdings.
`granuleId`	[`string \| array`]	`false`	granuleId (or array of granuleIds) for use for comparison of collection and granule holdings.
`provider`	[`string \| array`]	`false`	provider name (or array of providers) for comparison of granule holdings
`status`	`string`	`false`	status filter for Granule Inventory reports

NOTE: Adding a startTimestamp or endTimestamp value to the POST request will result in one way comparisons for some fields for Inventory and Granule Not Found reports.

NOTE: Adding a granuleId input will result in an one way report for collections for Inventory and Granule Not Found reports.

NOTE: Inventory and Granule Not Found reports only allow one of the parameters (collectionId, granuleId, provider) in same request.

NOTE: Granule Inventory reports can be filtered with the following parameters: (collectionId, granuleId, status). If granuleId is a string or single value array, it will search for granules with granuleIds containing that substring.

Example request

$ curl --request POST https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
  "reportName": "ModisInventoryReport",
  "reportType": "Inventory",
  "startTimestamp": 1269993600000,
  "endTimestamp": 1350000000000,
  "collectionId": "MOD09GQ___006",
  "provider": "MODIS",
  "granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
}'

Example response

{
  "id": "bb7059f4-0cd8-4205-8857-fb0e6b68b3e4"
}

Delete reconciliation report

Delete a reconciliation report from Cumulus.

DELETE

/reconciliationReports/{name}

Example request

$ curl --request DELETE https://example.com/reconciliationReports/inventoryReport-20180620T205838883 --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{
  "message": "Report deleted"
}

Create EMS reports

Create EMS reports and send them to EMS. For more information on EMS reporting and types of reports see the Cumulus EMS Reporting documentation.

Overview of the request fields:

Field	Required	Value	Description
`reportType`	`Y`	`"metadata"`\|`"ingest"`\|`"distribution"`	type of report
`startTime`	`N`	`string`	report startTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss If startTime and endTime are not specified, the reports for previous day will be generated
`endTime`	`N`	`string`	report endTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss
`collectionId`	`N`	`string`	collectionId of the report records This field can be specified when report type is `metadata` or `ingest`

POST

/ems

Example requests

$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
    "reportType": "metadata"
}'

$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
    "reportType": "ingest",
    "startTime": "2019-06-27",
    "endTime": "2019-06-29",
    "collectionId": "MOD14A1___006"
}'

Example response

{
    "message": "Reports are being generated",
    "status": 202
}

Instance Metadata

The Cumulus API can provide information about its configuration.

GET requests to the instance metadata endpoint return a json object with information about how the Cumulus stack is configured. It returns the CMR provider and environment as well as the stackName (prefix).

GET

/instanceMeta

Example Request

$ curl https://example.com/instanceMeta --header 'Authorization: Bearer ReplaceWithTheToken'

Example Response

{
  "cmr": {
    "provider": "CUMULUS",
    "environment": "UAT"
  },
  "cumulus": {
    "stackName": "cumulus-stack-prefix"
  }
}

Elasticsearch

These endpoints provide an interface into the Elasticsearch SDK to perform functions like Elasticsearch reindex.

Cumulus uses index aliases to allow reindexing with no downtime.

Reindex

The reindex command creates a new index and reindexes the source index to the new, destination index. This uses the Elasticsearch reindex functionality, but includes setting up the new index.

Note that there will now be two copies of your index. Your Cumulus instance will still point to the old copy until you perform the change-index function described below.

An alias should not be specified unless you have a specific alias configured. If a source index is not specified, it will default to the index from the alias. If you want to name the destination index something particular, you can specify a name, otherwise the destination index name will default to 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019). If you specify a destination index name, it must be an index that does not already exist in your cluster. The destination index must be different than the source index or the operation will fail.

Since reindex can be a long running operation, the reindex-status endpoint must be queried to see if the operation has completed.

Overview of request fields:

parameter	value	required	description
`aliasName`	`string`	`false`	Alias to reindex from. Will be used if `sourceIndex` is not provided. Will default to the default alias if not provided.
`sourceIndex`	`string`	`false`	Index to reindex from
`destIndex`	`string`	`true`	Index to reindex to

Example request

$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-4-12-2019. Check the reindex-status endpoint for status." }

Example request

$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "aliasName": "cumulus-alias",
  "sourceIndex": "cumulus-4-4-2019",
  "destIndex": "cumulus-new-index"
}'

Example response

{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-new-index. Check the reindex-status endpoint for status." }

Reindex Status

Reindexing can be a long running operation, so use this endpoint to get the status of your reindex and the status of your indices.

reindexStatus.nodes will be populated if there is a reindex operation currently running, nodes will be empty if not.

The elasticsearchStatus shows the number of documents in each index which can be used as a sanity check for the reindex operation before switching over to the new index. Note that the active index will continue to index logs, so the number of documents in the active index will be slightly higher and continue to grow.

Example request

$ curl https://example.com/elasticsearch/reindex-status --header 'Authorization: Bearer ReplaceWithTheToken'

Example responses

{
    "reindexStatus": {
        "nodes": {
            "TKDzeZ_TQVWkwKZYtCtv_g": {
                "name": "TKDzeZ_",
                "roles": [
                    "master",
                    "data",
                    "ingest"
                ],
                "tasks": {
                    "TKDzeZ_TQVWkwKZYtCtv_g:546691": {
                        "node": "TKDzeZ_TQVWkwKZYtCtv_g",
                        "id": 546691,
                        "type": "transport",
                        "action": "indices:data/write/reindex",
                        "start_time_in_millis": 1555073365557,
                        "running_time_in_nanos": 75424890,
                        "cancellable": true
                    }
                }
            }
        }
    },
    "indexStatus": {
        "_shards": {
            "total": 32,
            "successful": 16,
            "failed": 0
        },
        "_all": {
            "primaries": {
                "docs": {
                    "count": 52213,
                    "deleted": 0
                }
            },
            "total": {
                "docs": {
                    "count": 52213,
                    "deleted": 0
                }
            }
        },
        "indices": {
            ".kibana": {
                "primaries": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-12": {
                "primaries": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-9": {
                "primaries": {
                    "docs": {
                        "count": 25644,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25644,
                        "deleted": 0
                    }
                }
            }
        }
    }
}

{
    "reindexStatus": {
        "nodes": {}
    },
    "indexStatus": {
        "_shards": {
            "total": 22,
            "successful": 11,
            "failed": 0
        },
        "_all": {
            "primaries": {
                "docs": {
                    "count": 51210,
                    "deleted": 0
                }
            },
            "total": {
                "docs": {
                    "count": 51210,
                    "deleted": 0
                }
            }
        },
        "indices": {
            ".kibana": {
                "primaries": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 1,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-12": {
                "primaries": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25569,
                        "deleted": 0
                    }
                }
            },
            "cumulus-2019-4-9": {
                "primaries": {
                    "docs": {
                        "count": 25640,
                        "deleted": 0
                    }
                },
                "total": {
                    "docs": {
                        "count": 25640,
                        "deleted": 0
                    }
                }
            }
        }
    }
}

Change Index

Change index switches the Elasticsearch index to point to the new index, rather than the current index. You may choose to delete your current index during this operation using the deleteSource parameter, which defaults to false. If you are using the default index created by Cumulus, this will switch your Cumulus instance to the new index and you will see those changes immediately via the API and dashboard. If newIndex does not exist, an index with that name will be created.

currentIndex and newIndex are required parameters.

In the context of reindex, you'd call change-index, following reindex completion to start using the new index with no downtime.

Overview of the request fields:

parameter	value	required	description
`aliasName`	`string`	`false`	Alias to use for `newIndex`. Will be the default index if not provided
`currentIndex`	`string`	`true`	Index to change the alias from
`newIndex`	`string`	`true`	Index to change alias to
`deleteSource`	`boolean`	`false`	If set to `true` it will delete the index provided in `currentIndex`

Example request

$ curl --request POST https://example.com/elasticsearch/change-index --header 'Content-Type: application/json' --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
  "aliasName": "cumulus-alias",
  "currentIndex": "cumulus-12-4-2019",
  "newIndex": "cumulus-4-12-2019",
  "deleteSource": false
}'

Example response

{ "message": "Reindex success - alias cumulus now pointing to cumulus-4-12-2019" }

Index from Database

In case of corruption of your Elasticsearch index, you can reindex your data from the database. This will include collections, executions, granules, pdrs, providers, and rules. Logs will not be indexed since they are not stored in the database.

You can specify an index (should be empty) or if no index name is specified, a default with the format of 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019) will be used. If the specified index does not exist, it will be created.

You may optionally specify the following request fields:

parameter	value	required	description
`postgresResultPageSize`	`number`	`false`	Number of records to pull per database query for index to Elasticsearch Defaults 1000
`postgresConnectionPoolSize`	`number`	`false`	Max number of connections to postgres database to allocate. Defaults to 10
`esRequestConcurrency`	`number`	`false`	Maximum concurrency of writes to Elasticsearch Defaults to 10
`indexName`	`string`	`false`	Name of index to re-index to. Defaults to `cumulus-dd-mm-yyyy`

It is recommended that workflow rules be turned off, as any data ingested into the database during this operation cannot be guaranteed to make it into the new index. Following the completion of the index, you will need to use the change index operation to switch your Elasticsearch to point to the new instance.

Indexing is an async operation, so an operation id will be returned. You can query the status using the asyncoperations GET endpoint with the operation id.

Example request

$ curl --request POST https://example.com/elasticsearch/index-from-database --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
  "indexName": "recovery-index"
}'

Example response

{"message":"Indexing database to recovery-index. Operation id: 8f4d35ba-c858-40ae-93f0-4465f54a911d"}

Example request - operation status

$ curl --header 'Authorization: Bearer ReplaceWithTheToken' https://example.com/asyncoperations/8f4d35ba-c858-40ae-93f0-4465f54a911d

Example response - operation status

{
   "id":"8f4d35ba-c858-40ae-93f0-4465f54a911d",
   "status":"SUCCEEDED",
   "taskArn":"arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task/dc41ff2b-e610-4f7a-8c63-48b48ceb7160",
   "output":"\"Index from database complete\""
}

Indices Status

Use the indices-status endpoint to view information about your elasticsearch indices. This endpoint will return the information for the indices call in the Elasticsearch API.

Example request

$ curl  https://example.com/elasticsearch/indices-status --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

yellow open .kibana           yxBUTswHSr2ecec8y1szEg 1 1      1   0   3.1kb   3.1kb
yellow open cumulus-2019-6-21 ELpdxYVKSc-VvNxlNjmMUA 5 1    700 225   2.5mb   2.5mb
yellow open cumulus-2019-6-24 lmylLkTsQJWIVYFJcftMqQ 5 1    700   0     4mb     4mb
yellow open cumulus           jQLXsb8yS4aEmLAAyHcmCw 5 1 217298 588 124.3mb 124.3mb

Current Index

Get the current aliased index being used by the Cumulus Elasticsearch instance.

Example request

$ curl  https://example.com/elasticsearch/current-index --header 'Authorization: Bearer ReplaceWithTheToken'

Example response

["cumulus-2019-6-24"]

Serve the dashboard from a bucket

Serve the dashboard from an S3 bucket.

This is a way to serve the Cumulus dashboard in a browser from an S3 bucket without making the bucket or files public.

To use this:

Your dashboard bucket must be in the bucket definitions in your Cumulus terraform.tfvars, otherwise you will not be able to access the bucket.
Deploy the dashboard to your bucket using the instructions in the Cumulus dashboard README.
In a browser, use the example request below, but replace example.com with your Cumulus backend API and dashboard-bucket with your bucket name.

https://example.com/dashboard/dashboard-bucket/index.html

GET

/dashboard/{bucket}/{key}

Example request

ORCA

This endpoint authenticates and forwards requests to the ORCA private API, and returns the response from the ORCA API. Please refer to ORCA API reference on how to use ORCA API.

POST

/orca

Example request

$ curl --request POST https://example.com/orca/recovery/granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'  --data '{
  "granuleId": "MOD14A1.061.H5V12.2020312.141531789"
}'

Migration Counts

This endpoint triggers a run of the postgres-migration-count-tool as an async operation of type Migration Count Report.

For more information on the tool, see the README

The query uses the following optional parameters:

parameter	description
cutoffSeconds	Sets the number of seconds prior to this execution to 'cutoff' reconciliation queries. This allows in-progress or other in-flight operations time to complete and propagate to ElasticSearch/Postgres. Default is 3600
dbConcurrency	Sets max number of parallel collections reports the script will run at a time. Default 20
dbMaxPool	Sets the maximum number of connections the database pool has available. Modifying this may result in unexpected failures. Default is 20
reportPath	Sets the path location for the tool to write a copy of the lambda payload to S3
reportBucket	Sets the bucket used for reporting. If this argument is used, a `reportPath` must be set to generate a report

POST

/migrationCounts

Example request

$curl -X POST https://$API_URL/dev/migrationCounts -d 'reportBucket=someBucket&reportPath=someReportPath&cutoffSeconds=60&dbConcurrency=20&dbMaxPool=20' --header 'Authorization: Bearer $TOKEN'

Example response

{"id":"7ccaed31-756b-40bb-855d-e5e6d00dc4b3","status":"RUNNING","taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789","description":"Migration Count Tool ECS Run","operationType":"Migration Count Report"}

Recover cumulus messages

Endpoint provides a mechanism for recovery of S3 sfEventSqsToDbRecords dead letter objects (created as described in the Core Documentation). The endpoint will invoke an async operation that will attempt to process all of the objects in the specified location.

The endpoint by default will process all records contained in the S3 objects from the default storage location on the S3 system bucket under the prefix of <stackName>/dead-letter-archive/sqs/. However, it is likely useful to process a subset of those objects, which you can do by moving the desired subset of objects to a new path on S3 and then specifying that new path using the path parameter to the endpoint request.

The query uses the following optional parameters:

parameter	description
bucket	The bucket to read records from. Defaults to the Core system bucket
path	The S3 prefix (path) to read DLQ records from. Defaults to `<stackName>/dead-letter-archive/sqs/`

POST

/deadLetterArchive/recoverCumulusMessages

Example request

curl -X POST https://cumulus.podaac.sit.earthdata.nasa.gov/deadLetterArchive/recoverCumulusMessages --header "Authorization: Bearer $TOKEN" --header 'Content-Type: application/json' --data '{
    "bucket": "some-cumulus-bucket",
    "path": "some/path/to/records/"
}'

Example response

{
    "createdAt":1646861517957,
    "updatedAt":1646861517957,
    "id":"a0cd2cf0-a677-e82a-27d1-0a09271aa37d",
    "status":"RUNNING",
    "taskArn":"arn:aws:ecs:us-west-2:xxxxxxxxxx:task/stack-CumulusECSCluster/{SHA}",
    "description":"Dead-Letter Processor ECS Run",
    "operationType":"Dead-Letter Processing" 
}

Show examples in:

cURL Python JS

Cumulus API