The Cumulus API allows developers to interact with the Cumulus Framework, such as monitoring status or creating, editing, and deleting records. This is the same API that powers the Cumulus dashboard.
By utilizing this API, a developer can integrate with the Cumulus framework in any language or environment; although interacting with Cumulus through the Cumulus dashboard may be appropriate for many end users, for some use cases it's best to have the flexibility of a web-accessible API.
The API accepts and responds with JSON payloads at various HTTPS endpoints.
In order to use these endpoints, you must include authentication information in your HTTPS request; authentication is explained in the following section.
The following table lists the query string parameters that can be used with most of the Cumulus API endpoints. {fieldName} is a stand-in for any of the fields in the record, and for nested objects dot notation can be used; for example, valid fieldNames include: pdrName, status, and recipe.processStep.description.
| query string parameter | description |
|---|---|
limit={number} |
number of records to be returned by the API call; default is 1, maximum is 100 |
page={number} |
page number, 1-indexed; default is 1 |
sort_by={fieldName} |
which field to sort by; default is timestamp |
order={asc|desc} |
whether to sort in asc or desc order |
sort_key[]={-fieldName1}&sort_key[]={fieldName2} |
One or more sort keys can be specified using the sort_key[] parameter. The order used impacts searching. Fields can be prepended with a - to sort in descending order or a + to sort in ascending. Ascending order is the default. The + must be escaped with %2B |
prefix={value} |
startsWith search of the granuleId, status, pdrName, name, error, id and msg fields |
infix={value} |
includes search of the granuleId, status, pdrName, name, error, id and msg fields |
fields={fieldName1, fieldName2} |
which fields to return, separated by a comma |
{fieldName}={value} |
exact value match for the given field |
{fieldName}__from={number} |
for numeric fields, field value must be greater than the given number |
{fieldName}__to={number} |
for numeric fields, field value must be less than the given number |
{fieldName}__not={value} |
field does not match the given value |
{fieldName}__in={value1, value2} |
field matches one of the values in the comma-separated list |
{fieldName}__exists={true|false} |
field exists or doesn't exist in the record |
q="fieldName:[1 TO 2] AND fieldName2:[3 TO 4]" |
arbitrary Apache Lucene query syntax, not needed for most uses of the API; if the q parameter is used, all other query parameters will be ignored, besides limit, page, and fields |
The Cumulus API is versioned and the current version is v1. Retrieve the latest API version from Cumulus.
To use any version, include the version number in the path before the query endpoint. {API_URL}/{version}/{query}
$ curl https://example.com/version
{
"response_version": "v1",
"api_version": "1.1.1"
}
Returns a bearer token using oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).
| query string parameter | description |
|---|---|
state={string} |
The URI to redirect to after if oAuth was successful |
$ curl https://example.com/token
{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}
Refreshes a bearer token received from oAuth with Earthdata Login service. The token will be returned as a JWT (JSON Web Token).
| parameter | type | required | description |
|---|---|---|---|
token |
string | true |
The JWT received from the /token endpoint to refresh |
$ curl --request POST https://example.com/refresh --header 'Content-Type: application/json' --data '{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY"}'
{"token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDcxMjk4ODEzIn0.vO6RlSRo47kkH15_muoUYNvv74fRzFxs7FlmVaarHlc"}
Delete the record for an access token received from oAuth with Earthdata Login service.
token is the JWT containing access token information to delete
$ curl --request DELETE https://example.com/tokenDelete/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NUb2tlbiI6IjIyMzc0YWE4MDM1M2E3ODFkYWJjYmFhZGJhOGE3ZmMwZmE1MWYzYjQzNWYxNTc4MjU2NjA0ZjFiNGQ0NTE2ODYiLCJleHAiOiIxNTQ0NDY1MDk3ODczIn0.SxFtZ7dqp9KsUSn1uTXhWis8Il8Hig8mwLANGU3cXhY
{"message": "Token record was deleted"}
When a request is made to the Cumulus API, it must contain a bearer token generated by the API.
The token is generated after the user login with Earthdata Login service.
The token is included in requests using the Authorization header.
If no token is provided, the Cumulus API server will respond with an error, requesting credentials. If an incorrect token is provided, the server will respond with a separate error noting this.
#! /bin/sh
ORIGIN=$(dirname $CUMULUS_BASEURL)
LOGIN_URL="$CUMULUS_BASEURL/token"
# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)
# Request the Earthdata url with client id and redirect uri to use with Cumulus
AUTHORIZE_URL=$(curl -s -i ${LOGIN_URL} | grep location | sed -e "s/^location: //");
# Request an authorization grant code
TOKEN_URL=$(curl -s -i -X POST \
-F "credentials=${AUTH}" \
-H "Origin: ${ORIGIN}" \
${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")
# Request the token through the CUMULUS API url that's returned from Earthdata
# Response is a JSON object of the form { token: String }
# This uses the cli tool jq to parse the JSON and get the token string
# More info on jq: https://stedolan.github.io/jq/
TOKEN=$(curl -s ${TOKEN_URL%$'\r'} | jq -r '.message.token')
echo $TOKEN
List providers in the Cumulus system.
$ curl https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "daac-cumulus",
"table": "provider",
"limit": 1,
"page": 1,
"count": 2
},
"results": [
{
"id": "HTTP_MODIS",
"globalConnectionLimit": 10,
"protocol": "http",
"host": "https://data.modis.gov/",
"timestamp": 1508861082226
}
]
}
Retrieve a single provider.
$ curl https://example.com/providers/HTTP_MODIS --header 'Authorization: Bearer ReplaceWithTheToken'
{
"createdAt": 1508861081785,
"id": "HTTP_MODIS",
"host": "https://data.modis.gov/",
"globalConnectionLimit": 10,
"updatedAt": 1508861081785,
"protocol": "http"
}
Create a provider. For more information on creating providers and the contents of a request see the Cumulus setup documentation.
Overview of the schema fields:
| Field | Value | Description |
|---|---|---|
id |
string |
provider id/name |
protocol |
"s3"|"http"|"https"|"ftp" |
file transfer (sync) protocol |
host |
string |
provider host endpoint |
port |
number |
provider host port |
globalConnectionLimit |
number |
limit to number of concurrent connections |
username |
string |
provider connection username |
password |
string |
provider connection password |
privateKey |
string |
filename assumed to be in s3://bucketInternal/stackName/crypto |
cmKeyId |
string |
AWS KMS Customer Master Key arn or alias |
allowedRedirects |
Array<string> |
Only hosts in this list will have the provider username/password forwarded for authentication. Entries should be specified as host.com or host.com:7000 if redirect port is different than the provider port. |
certificateUri |
string |
Optional SSL Certificate S3 URI for custom or self-signed SSL (TLS) certificate |
$ curl --request POST https://example.com/providers --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"host": "https://www.example.gov",
"id": "MY_DAAC_SATELLITE",
"protocol": "http",
"globalConnectionLimit": 10
}'
{
"message": "Record saved",
"record": {
"createdAt": 1491941727851,
"host": "https://www.example.gov",
"id": "MY_DAAC_SATELLITE",
"protocol": "http",
"globalConnectionLimit": 10,
"timestamp": 1513956151186
}
}
Update/replace an existing provider. Expects payload to specify the entire provider object, and will completely replace the existing provider with the specified payload. For a field reference see "Create provider".
Returns status 200 on successful replacement, 400 if the id property in the
payload does not match the corresponding value in the resource URI, or 404 if
there is no provider with the specified ID.
$ curl --request PUT https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"id": "MY_DAAC_SATELLITE",
"host": "https://www.example.co.uk",
"globalConnectionLimit": 10,
"protocol": "http"
}'
{
"createdAt": 1491941727851,
"id": "MY_DAAC_SATELLITE",
"host": "https://www.example.co.uk",
"globalConnectionLimit": 10,
"updatedAt": 1513956150733,
"protocol": "http",
"timestamp": 1513956555713
}
Delete a provider from Cumulus. The related PDRs and granules remain in the Cumulus and CMR systems.
$ curl --request DELETE https://example.com/providers/MY_DAAC_SATELLITE --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List collections in the Cumulus system.
If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink
If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.
$ curl https://example.com/collections --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "collection",
"limit": 1,
"page": 1,
"count": 3
},
"results": [
{
"name": "MOD11A1",
"version": "006",
"dataType": "MOD11A1",
"process": "modis",
"provider_path": "/",
"granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
"granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
},
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
}
],
"timestamp": 1513020427284,
"createdAt": 1510761441174,
"updatedAt": 1513020427162,
"edpa": true,
"some_other_field": "field",
"duplicateHandling": "skip",
}
]
}
List collections in the Cumulus system that have active associated granules.
If time parameters are specified, the query will return collections that have granules that have been updated in that time frame.
If the query includes a value of true for getMMT, cumulus will search the CMR for each collection and insert the link to the Metadata Management Tool (MMT) into each result under MMTLink
If the query includes a value of true for includeStats, cumulus will compute and include the granule aggregation statistics in each collection's return value.
$ curl https://example.com/collections/active?includeStats=true --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "collection",
"limit": 1,
"page": 1,
"count": 3
},
"results": [
{
"name": "MOD11A1",
"version": "006",
"dataType": "MOD11A1",
"process": "modis",
"provider_path": "/",
"granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
"granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$",
"sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf"
},
{
"bucket": "private",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met"
},
{
"bucket": "protected",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg"
},
{
"bucket": "public",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg"
}
],
"timestamp": 1513020427284,
"createdAt": 1510761441174,
"updatedAt": 1513020427162,
"edpa": true,
"some_other_field": "field",
"duplicateHandling": "skip",
"stats": {
"running": 0,
"completed": 6,
"failed": 1,
"total": 7
}
}
]
}
Retrieve a single collection.
$ curl https://example.com/collections/MOD11A1/006 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"process": "modis",
"granuleIdExtraction": "(MOD11A1\\..*)\\.hdf",
"version": "006",
"dataType": "MOD11A1",
"some_other_field": "field",
"createdAt": 1510761441174,
"edpa": true,
"name": "MOD11A1",
"duplicateHandling": "skip",
"provider_path": "/",
"files": [
{
"bucket": "protected",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
},
{
"bucket": "private",
"sampleFileName": "BROWSE.MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
"regex": "^BROWSE\\.MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf$"
},
{
"bucket": "private",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf.met",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.hdf\\.met$"
},
{
"bucket": "protected",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.cmr.xml",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}\\.cmr\\.xml$"
},
{
"bucket": "public",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_2.jpg",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_2\\.jpg$"
},
{
"bucket": "public",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104_1.jpg",
"regex": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}_1\\.jpg$"
}
],
"updatedAt": 1513020427162,
"granuleId": "^MOD11A1\\.A[\\d]{7}\\.[\\S]{6}\\.006.[\\d]{13}$",
"sampleFileName": "MOD11A1.A2017025.h21v00.006.2017034065104.hdf",
}
Create a collection. For more information on creating collections and the contents of a request see the Cumulus setup documentation. A collection generally includes, but is not limited to, the fields listed below.
Overview of the schema fields:
| Field | Value | Description |
|---|---|---|
name |
string |
collection name |
version |
string |
collection version |
dataType |
string |
matches collection with PDR datatype |
duplicateHandling |
"replace"|"version"|"skip"|"error" |
duplicate handling protocol |
process |
string |
process choice step variable |
provider_path |
string |
path of remote files to sync |
granuleId |
string (regex) |
regex to match granule IDs |
granuleIdExtraction |
string (regex) |
regex to extract ID from files |
sampleFileName |
string |
sample filename for granule ID |
url_path |
string |
s3 prefix template |
files |
array |
array of file specification objects |
-- file.bucket |
string |
file destination bucket |
-- file.regex |
string (regex) |
regex to match file names |
-- file.sampleFileName |
string |
sample filename |
-- file.url_path |
string |
s3 prefix template |
-- file.fileType |
string |
granule file fileType mapping |
$ curl --request POST https://example.com/collections --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"process": "modis",
"duplicateHandling": "replace",
"provider_path": "cumulus-test-data/pdrs",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
]
}'
{
"message": "Record saved",
"record": {
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"duplicateHandling": "replace",
"process": "modis",
"provider_path": "cumulus-test-data/pdrs",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
],
"createdAt": 1491946535919
}
}
Update/replace an existing collection. Expects payload to specify the entire collection object, and will completely replace the existing collection with the specified payload. For a field reference see "Create collection".
Returns status 200 on successful replacement, 400 if the name or version
property in the payload does not match the corresponding value in the resource
URI, or 404 if there is no collection with the specified name and version.
$ curl --request PUT https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"duplicateHandling": "error",
"newNeededField": "myCustomFieldValue",
"process": "modis",
"provider_path": "new_path/test-data",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
]
}'
{
"name": "MOD09GQ",
"version": "006",
"dataType": "MOD09GQ",
"duplicateHandling": "error",
"newNeededField": "myCustomFieldValue",
"process": "modis",
"provider_path": "new_path/test-data",
"granuleId": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}$",
"granuleIdExtraction": "(MOD09GQ\\..*)(\\.hdf|\\.cmr|_ndvi\\.jpg)",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{substring(file.name, 0, 3)}",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"files": [
{
"bucket": "protected",
"regex": "^MOD09GQ\\.A[\\d]{7}\\.[\\S]{6}\\.006\\.[\\d]{13}\\.hdf$",
"sampleFileName": "MOD09GQ.A2017025.h21v00.006.2017034065104.hdf",
"url_path": "{cmrMetadata.Granule.Collection.ShortName}___{cmrMetadata.Granule.Collection.VersionId}/{extractYear(cmrMetadata.Granule.Temporal.RangeDateTime.BeginningDateTime)}/{substring(file.name, 0, 3)}"
}
],
"createdAt": 1491946535919,
"updatedAt": 1514304825894
}
Delete a collection from Cumulus, but not from CMR. All related granules in Cumulus must have already been deleted from Cumulus.
$ curl --request DELETE https://example.com/collections/MOD09GQ/006 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List granules in the Cumulus system.
If the query includes a value of true for getRecoveryStatus, recoveryStatus will be included in each granule's return value when applicable.
$ curl https://example.com/granules --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "granule",
"limit": 1,
"page": 1,
"count": 8
},
"results": [
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "completed",
"provider": "LP_TS2_DataPool",
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
"files": [
{
"bucket": "cumulus-devseed-protected",
"checksum": 964704694,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 1447347,
"fileType": "data",
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 121318124,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
"fileSize": 22559,
"fileType": "metadata",
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 2188150664,
"key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 18118,
"fileType": "data",
"checksumType": "CKSUM",
"fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
"fileType": "browse",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
},
{
"bucket": "cumulus-devseed-protected",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
"fileType": "metadata",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
"fileType": "browse",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
}
],
"error": null,
"createdAt": 1513020455831,
"timestamp": 1513020462156,
"published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
"duration": 6.325,
"cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS"
}
]
}
Retrieve a single granule.
If the query includes a value of true for getRecoveryStatus, the returned granule will include recoveryStatus when applicable.
$ curl https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "completed",
"provider": "LP_TS2_DataPool",
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:7f071dae1a93c9892272b7fd5",
"files": [
{
"bucket": "cumulus-devseed-protected",
"checksum": 964704694,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 1447347,
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 121318124,
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met",
"fileSize": 22559,
"checksumType": "CKSUM",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf.met"
},
{
"bucket": "cumulus-devseed-private",
"checksum": 2188150664,
"key": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileSize": 18118,
"checksumType": "CKSUM",
"fileName": "BROWSE.MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-protected",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.hdf"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_2.jpg"
},
{
"bucket": "cumulus-devseed-protected",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755.cmr.xml"
},
{
"bucket": "cumulus-devseed-public",
"key": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg",
"fileName": "MOD11A1.A2017137.h20v17.006.2017138085755_1.jpg"
}
],
"error": null,
"createdAt": 1513020455831,
"timestamp": 1513020462156,
"published": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
"duration": 6.325,
"cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1220753758-CUMULUS",
"_id": "MOD11A1.A2017137.h20v17.006.2017138085755"
}
Create a granule. A granule can have the following fields.
| Field | Type | Required | Description |
|---|---|---|---|
beginningDateTime |
string |
no |
The time when the granule's temporal coverage begins |
cmrLink |
string |
no |
link to CMR |
collectionId |
string |
yes |
Collection associated with the granule |
createdAt |
integer |
no |
Time granule record was created (now) |
duration |
number |
no |
Ingest duration milliseconds |
endingDateTime |
string |
no |
The time when the graule's temporal coverage ends |
error |
object |
no |
The error details for this granule |
execution |
string |
no |
Step Function Execution URL |
files |
array |
no |
Files associated with the granule |
granuleId |
string |
yes |
Granule ID |
lastUpdateDateTime |
string |
no |
The date/time that data provider last updated the granule info on data provider's database |
pdrName |
string |
no |
PDR associated with the granule |
processingEndDateTime |
string |
no |
Time processing of granule ends. Usually a StepFunction's stop time |
processingStartDateTime |
string |
no |
Time processing of granule began. Usually a StepFunction's start time |
productVolume |
number |
no |
Sum of the granule's file's sizes in bytes |
productionDateTime |
string |
no |
The date and time a specific granule was produced by a PGE |
provider |
string |
no |
Granule's provider |
published |
boolean |
no |
If granule is published to CMR |
queryFields |
object |
no |
Arbitrary query fields assigned to the granule |
status |
string |
yes |
Ingest status of the granule one of ['running', 'completed', 'failed'] |
timeToArchive |
number |
no |
Time to post to CMR in seconds |
timeToPreprocess |
number |
no |
Time to sync granule in seconds |
timestamp |
integer |
no |
Timestamp for granule record (now) |
updatedAt |
integer |
no |
Update Time for granule (now) |
$ curl --request POST https://example.com/granules \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--data '{
"granuleId": "granuleId.A20200113.006.1005",
"collectionId": "alpha___006",
"status": "running",
"beginningDateTime": "1995-01-01T00:00:00.000Z",
"cmrLink": "https://mmt.uat.earthdata.nasa.gov/collections/C1237256734-CUMULUS",
"createdAt": 1631547286190,
"duration": 60000,
"endingDateTime": "2021-09-13T00:00:00.000Z",
"error": {},
"execution": "https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestStepFunction2:cff1266e-ef36-664f-ff00-3a4d26bd1735",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.hdf",
"fileName": "granuleId.A20200113.006.1005.hdf"
}
],
"lastUpdateDateTime":"2021-09-12T15:10:01.000Z",
"pdrName": "aPdrName",
"processingEndDateTime": "2020-10-13T23:59:59.999Z",
"processingStartDateTime": "2020-10-13T00:00:00.000Z",
"productVolume": 59632353,
"productionDateTime": "2020-10-14T15:40:05.546Z",
"provider": "s3-local",
"published": true,
"queryFields": {"custom": "values can go here"},
"timeToArchive": 5001,
"timeToPreprocess": 3240,
"timestamp":1631547675248,
"updatedAt":1631547675248
}'
{
"message": "Successfully wrote granule with Granule Id: granuleId.A20200113.006.1005"
}
Update/replace an existing granules. Expects payload to contain the modified parts of the granule and the existing granule values will be overwritten by the modified portions. The same fields are available as are for creating a granule..
Returns status 200 on successful replacement, 404 if the granuleId can not be
found in the database, or 400 when the granuleId in the payload does not match the
corresponding value in the resource URI.
$ curl --request PUT https://example.com/granules/granuleId.A19990103.006.1000 \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--data '{
"granuleId": "granuleId.A20200113.006.1005",
"files": [
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.hdf",
"fileName": "granuleId.A20200113.006.1005.hdf"
},
{
"bucket": "stack-protected",
"key": "granuleId.A20200113.006.1005.jpg",
"fileName": "granuleId.A20200113.006.1005.jpg"
}
],
"duration": 1000,
"status": "completed"
}'
{
"message": "Successfully updated granule with Granule Id: granuleId.A20200113.006.1005"
}
Associate an execution with a granule. Returns status 200 on successful association, 404 if the granule or execution can not be found in the database, or 400 when the granuleId in the payload does not match the corresponding value in the resource URI.
The request should have the following fields.
| Field | Type | Description |
|---|---|---|
collectionId |
string |
Collection associated with the granule (e.g. <name>___<version> where <name> is the collection name and <version> is the collection version) |
granuleId |
string |
Granule ID |
executionArn |
string |
Execution arn |
$ curl --request POST https://example.com/granules/granuleId.A19990103.006.1000/executions \
--header 'Authorization: Bearer ReplaceWithToken' \
--header 'Content-Type: application/json' \
--data '{
"granuleId": "granuleId.A19990103.006.1000",
"collectionId": "MOD09GQ___006",
"executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}'
{
"message": "Successfully associated execution arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 with granule granuleId granuleId.A19990103.006.1000 collectionId MOD09GQ___006"
}
Reingest a granule. This causes the granule to re-download to Cumulus from source, and begin processing from scratch. Reingesting a granule will overwrite existing granule files.
You trigger the reingest by posting with the data's action set to reingest.
A reingest request may also include one optional parameters, either
executionArn or workflowName. Use of one of these will cause the reingest
occur, but before running the ingest, a different original payload is found
from existing executions and used in place of the most recent execution. This
has the result of allowing you to reingest a granule with any of it's previous
inputs. If executionArn is specified, the originalMessage is pulled directly
from that execution. When workflowName is specified, the database is search
to find all of the executions with that workflowName that were run on the
granuleid, and the most recent originalMessage is pulled and used for the
reingest. Remember only to supply either executionArn or workflowName, if
both are present, workflowName is ignored and executionArn is used to determing
the input message to the reingest.
A warning message is included in the response if the collection's duplicateHandling is not set to 'replace'.
$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h20v17.006.2017138085755
--header 'Authorization: Bearer ReplaceWithTheToken'
--header 'Content-Type: application/json'
--data '{"action": "reingest",
["executionArn": "arn:aws:states:us-east-1:123456789012:execution:stack-lambdaName:9da47a3b-4d85-4599-ae78-dbec2e042520"],
["workflowName": "TheWorkflowName"] }'
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"action": "reingest",
"status": "SUCCESS",
"warning": "The granule files may be overwritten"
}
Apply the named workflow to the granule. Workflow input will be built from template and provided entire Cumulus granule record as payload.
$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{ "action": "applyWorkflow", "workflow": "inPlaceWorkflow" }'
{
"granuleId": "MOD11A1.A2017137.h20v17.006.2017138085755",
"action": "applyWorkflow inPlaceWorkflow",
"status": "SUCCESS"
}
Move a granule from one location on S3 to another. Individual files are moved to specific locations by using a regex that matches their filenames.
$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{ "action": "move", "destinations": [{ "regex": ".*.hdf$", "bucket": "s3-bucket", "filepath": "new/filepath/" }]}'
{
"granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
"action": "move",
"status": "SUCCESS"
}
Remove a Cumulus granule from CMR.
$ curl --request PUT https://example.com/granules/MOD11A1.A2017137.h19v16.006.2017138085750 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "removeFromCmr"}'
{
"granuleId": "MOD11A1.A2017137.h19v16.006.2017138085750",
"action": "removeFromCmr",
"status": "SUCCESS"
}
Delete a granule from Cumulus. It must already be removed from CMR.
$ curl --request DELETE https://example.com/granules/1A0000-2016121001_002_001 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"detail": "Record deleted"
}
Apply a workflow to the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
workflow |
Y |
string |
Worfklow to be applied to all granules |
queueUrl |
N |
string |
URL of SQS queue to use for scheduling granule workflows (e.g. https://sqs.us-east-1.amazonaws.com/12345/queue-name) |
ids |
yes - if query not present |
Array<string> |
List of IDs to process. Required if there is no Elasticsearch query provided |
query |
yes - if ids not present |
Object |
Query to Elasticsearch to determine which Granules to go through given workflow. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.
$ curl --request POST \
https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"workflowName": "HelloWorldWorkflow",
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"status": "FAILED"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
curl -X POST
https://example.com/granules/bulk --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "workflowName": "HelloWorldWorkflow"}'
{
"createdAt": 1574730504000,
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"updatedAt": 1574730504762,
"status": "RUNNING",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/d481e76e-f5fc-9c1c-2411-fa13779b111a"
}
Bulk delete the provided granules. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
forceRemoveFromCmr |
N |
bool |
Whether to remove published granules from CMR before deletion. You must set this value to true to do bulk deletion of published granules, otherwise deleting them will fail. |
ids |
yes - if query not present |
Array<string> |
List of IDs to process. Required if there is no Elasticsearch query provided |
query |
yes - if ids not present |
Object |
Query to Elasticsearch to determine which Granules to delete. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.
$ curl --request POST \
https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"forceRemoveFromCmr": true,
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"status": "FAILED"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
curl -X POST
https://example.com/granules/bulkDelete --header 'Authorization: Bearer ReplaceWithTheToken' --data '{"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"], "forceRemoveFromCmr": true}'
{
"createdAt": 1574730504000,
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"updatedAt": 1574730504762,
"status": "RUNNING",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/d481e76e-f5fc-9c1c-2411-fa13779b111a"
}
Reingest the granules provided. Granules can be sent as a list of IDs or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
ids |
yes - if query not present |
Array<string> |
List of IDs to process. Required if there is no Elasticsearch query provided |
query |
yes - if ids not present |
Object |
Query to Elasticsearch to determine which Granules to be reingested. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
workflowName |
no |
string |
optional workflow name that allows different workflow and initial input to be used during reingest. See below. |
An optional data parameter of workflowName is also available to allow you to override the input message to the reingest. If workflowName is specified, the original message is pulled directly
by finding the most recent execution of the workflowName assocated with the granuleId.
Use the Retrieve async operation endpoint with the id in the response to determine the status of the async operation.
$ curl --request POST \
https://example.com/granules/bulkReingest --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"status": "FAILED"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
curl -X POST
https://example.com/granules/bulkReingest
--header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json'
--data '{
"ids": ["MOD09GQ.A2016358.h13v04.006.2016360104606"],
"workflow": "workflowName",
}'
{
"createdAt": 1574730504000,
"description": "Bulk granule reingest run on 2 granules",
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"operationType": "Bulk Granule Reingest",
"status": "RUNNING",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
"updatedAt": 1574730504762
}
List PDRs in the Cumulus system.
$ curl https://example.com/pdrs --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "pdr",
"limit": 1,
"page": 1,
"count": 8
},
"results": [
{
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "failed",
"provider": "LP_TS2_DataPool",
"progress": 0,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:433612427488:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
"PANSent": false,
"PANmessage": "N/A",
"stats": {
"processing": 0,
"completed": 0,
"failed": 0,
"total": 0
},
"createdAt": 1514305411204,
"timestamp": 1514305424036,
"duration": 12.832
}
]
}
Retrieve a single PDR.
$ curl https://example.com/pdrs/7970bff5-128a-489f-b43c-de4ad7834ce5.PDR --header 'Authorization: Bearer ReplaceWithTheToken'
{
"pdrName": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR",
"collectionId": "MOD11A1___006",
"status": "failed",
"provider": "LP_TS2_DataPool",
"progress": 0,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:433612427488:execution:LpdaacCumulusIngestGranuleStateMachine-N3CLGBXRPAT9:6ef0c52f83c549db58b3a1e50",
"PANSent": false,
"PANmessage": "N/A",
"stats": {
"processing": 0,
"completed": 0,
"failed": 0,
"total": 0
},
"createdAt": 1514305411204,
"timestamp": 1514305424036,
"duration": 12.832,
"_id": "7970bff5-128a-489f-b43c-de4ad7834ce5.PDR"
}
Delete a PDR from Cumulus. Its granules will remain, and the PDR may be re-discovered and re-ingested/re-processed from scratch in the future.
$ curl --request DELETE https://example.com/pdrs/good_25grans.PDR --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List rules in the Cumulus system.
$ curl https://example.com/rules --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "rule",
"limit": 1,
"page": 1,
"count": 7
},
"results": [
{
"name": "repeat",
"workflow": "DiscoverPdrs",
"provider": "local",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"timestamp": 1511232462534,
"state": "DISABLED"
}
]
}
Retrieve a single rule.
$ curl https://example.com/rules/repeat --header 'Authorization: Bearer ReplaceWithTheToken'
{
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"updatedAt": 1511232462507,
"createdAt": 1510903518741,
"provider": "local",
"name": "repeat",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "DISABLED"
}
Create a rule. For more information on creating rules and the contents of a request see the Cumulus setup documentation.
Overview of the schema fields:
| Field | Value | Description |
|---|---|---|
name |
string |
rule name (letters, numbers, underscores only) |
state |
"DISABLED"|"ENABLED" |
rule state (default: ENABLED) |
workflow |
string |
name of workflow started by the rule |
rule |
Object |
rule object |
-- rule.type |
"onetime"|"scheduled"|"kinesis"|"sns"|"sqs" |
rule trigger type |
-- rule.value |
onetime: N/Ascheduled: cron-type or rate expressionkinesis: Kinesis stream ARNsns: SNS topic ARNsqs: SQS queue URL |
required value differs by type |
-- rule.arn |
string |
kinesis scheduled event arn |
-- rule.logEventArn |
string |
kinesis scheduled log event arn |
provider |
string |
provider record provided to workflow (optional) |
collection |
Object |
collection record provided to workflow (optional) |
-- collection.name |
string |
collection name |
-- collection.version |
string |
collection version |
meta |
Object |
contents to add to workflow input's meta field |
tags |
array |
Optional tags (for search) |
$ curl --request POST https://example.com/rules --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"provider": "local",
"name": "repeat_test",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"meta": { "publish": false },
"state": "DISABLED"
}'
{
"message": "Record saved",
"record": {
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"createdAt": 1510903518741,
"provider": "local",
"name": "repeat_test",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"meta": { "publish": false },
"state": "DISABLED",
}
}
Update/replace an existing rule. Expects payload to specify the entire rule object, and will completely replace the existing rule with the specified payload. For a field reference see "Create rule".
Returns status 200 on successful replacement, 400 if the name property in the
payload does not match the corresponding value in the resource URI, or 404 if
there is no rule with the specified name.
Special case:
| Field | Value | Description |
|---|---|---|
action |
"rerun" |
rerun rule (onetime rule only) |
$ curl --request PUT https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"name": "repeat_test",
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"provider": "local",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "ENABLED"
}'
{
"name": "repeat_test",
"workflow": "DiscoverPdrs",
"collection": {
"name": "AST_L1A",
"version": "003"
},
"updatedAt": 1521755265130,
"createdAt": 1510903518741,
"provider": "local",
"rule": {
"type": "scheduled",
"value": "rate(5 minutes)"
},
"state": "ENABLED"
}
onetime-rule special case)$ curl --request PUT https://example.com/rules/my_onetime_rule --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"action": "rerun"}'
Delete a rule from Cumulus.
$ curl --request DELETE https://example.com/rules/repeat_test --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
Retrieve a summary of statistics around the granules in the system. The collections returned are the number of distinct collections for the granules active during the given time period, defaulted to the last day if none is specified.
$ curl https://example.com/stats --header 'Authorization: Bearer ReplaceWithTheToken'
{
"errors": {
"dateFrom": "1970-01-18T12:36:59+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": 2,
"aggregation": "count",
"unit": "error"
},
"collections": {
"dateFrom": "1970-01-01T12:00:00+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": 3,
"aggregation": "count",
"unit": "collection"
},
"processingTime": {
"dateFrom": "1970-01-18T12:36:59+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": null,
"aggregation": "average",
"unit": "second"
},
"granules": {
"dateFrom": "1970-01-18T12:36:59+00:00",
"dateTo": "2017-12-26T04:38:15+00:00",
"value": 8,
"aggregation": "count",
"unit": "granule"
}
}
Count the value frequencies for a given field, for a given type of record in Cumulus. Requires the following query parameters, and may include the regular filter parameters:
| query string parameter | description |
|---|---|
type={providers|collections|granules|pdrs|logs} |
type of Cumulus record to query |
field={fieldName} |
which field to count frequencies for; no default |
curl 'https://example.com/stats/aggregate?field=status&type=pdrs' --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"count": 52,
"field": "status.keyword"
},
"count": [
{
"key": "failed",
"count": 43
},
{
"key": "completed",
"count": 5
},
{
"key": "parsed",
"count": 3
}
]
}
Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.
List processing logs from the Cumulus engine. A log's level field should be specified by their string value and be one of:
$ curl https://example.com/logs?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "logs",
"limit": 5,
"page": 1,
"count": 750
},
"results": [
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "https://ba008ffc.ngrok.io/ not found",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "HostNotFound: https://ba008ffc.ngrok.io/ not found\n at discover.discover.then.catch.e (/var/task/index.js:9006:22)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"message": "https://ba008ffc.ngrok.io/ not found",
"v": 1,
"timestamp": 1513881407867
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"file": "discover-pdrs/index.js",
"message": "Received a 404 error from undefined. Check your endpoint!",
"details": {
"host": "ba008ffc.ngrok.io",
"path": "/",
"port": "",
"protocol": "https",
"uriPath": "/",
"url": "https://ba008ffc.ngrok.io/",
"depth": 1,
"fetched": true,
"status": "notfound",
"stateData": {
"requestLatency": 66,
"requestTime": 66,
"contentLength": 34,
"contentType": "text/plain",
"code": 404,
"headers": {
"content-length": "34",
"connection": "close",
"content-type": "text/plain"
}
},
"id": 0
},
"v": 1,
"timestamp": 1513881407867
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "Cannot read property 'provider_path' of undefined",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "TypeError: Cannot read property 'provider_path' of undefined\n at HttpDiscover.Discover (/var/task/index.js:10098:33)\n at HttpDiscover._class (/var/task/index.js:96368:255)\n at new HttpDiscover (/var/task/index.js:10342:241)\n at handler (/var/task/index.js:8968:23)\n at Promise (/var/task/cumulus-sled/index.js:82:5)\n at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"v": 1,
"timestamp": 1513881031694
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "Provider info not provided",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "ProviderNotFound: Provider info not provided\n at handler (/var/task/index.js:8962:20)\n at Promise (/var/task/cumulus-sled/index.js:82:5)\n at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"message": "Provider info not provided",
"v": 1,
"timestamp": 1513879654040
},
{
"pid": 1,
"hostname": "ip-10-29-4-186",
"name": "cumulus",
"level": 50,
"msg": "Provider info not provided",
"file": "discover-pdrs/index.js",
"type": "Error",
"stack": "ProviderNotFound: Provider info not provided\n at handler (/var/task/index.js:8962:20)\n at Promise (/var/task/cumulus-sled/index.js:82:5)\n at invokeHandler (/var/task/cumulus-sled/index.js:72:10)\n at then.then.then (/var/task/cumulus-sled/index.js:112:14)\n at process._tickDomainCallback (internal/process/next_tick.js:135:7)",
"message": "Provider info not provided",
"v": 1,
"timestamp": 1513879435010
}
]
}
$ curl https://example.com/logs?level=error --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "logs",
"limit": 1,
"page": 1,
"count": 181
},
"results": [
{
"level": 50,
"msg": "Unexpected HTTP status code: 404",
"pid": 1,
"hostname": "ip-10-26-125-174",
"name": "cumulus",
"file": "sync-granule/index.js",
"type": "Error",
"stack": "Error: Unexpected HTTP status code: 404\n at ClientRequest.<anonymous> (/var/task/sync-granule/index.js:125136:29)\n at ClientRequest.g (events.js:292:16)\n at emitOne (events.js:96:13)\n at ClientRequest.emit (events.js:188:7)\n at HTTPParser.parserOnIncomingClient (_http_client.js:473:21)\n at HTTPParser.parserOnHeadersComplete (_http_common.js:99:23)\n at TLSSocket.socketOnData (_http_client.js:362:20)\n at emitOne (events.js:96:13)\n at TLSSocket.emit (events.js:188:7)\n at readableAddChunk (_stream_readable.js:176:18)",
"code": 404,
"v": 1,
"timestamp": 1515446810226
}
]
}
Note: This endpoint will only work if logs are being delivered to the Metrics system. Otherwise it will return a 400 Error.
Retrieve all logs from a specific execution.
$ curl https://example.com/logs/93e5e049-89aa-47e5-900d-0eec87327260?limit=5 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "test-src-integration",
"table": "logs",
"limit": 5,
"page": 1,
"count": 10
},
"results": [
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:57.752Z",
"sender": "test-src-integration-PostToCmr",
"version": "$LATEST",
"message": "Published MOD09GQ.A8009365.7JQhni.006.6594428626875 to the CMR. conceptId: G1223210348-CUMULUS",
"RequestId": "05712fb1-5593-4640-b5a3-a99db41d8003"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:42.054Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:41.913Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "Finishing downloading s3://cumulus-test-sandbox-internal/cumulus-test-data/pdrs/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:41.888Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875_ndvi.jpg",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
},
{
"level": 30,
"executions": "93e5e049-89aa-47e5-900d-0eec87327260",
"timestamp": "2018-08-15T19:23:41.886Z",
"sender": "test-src-integration-SyncGranuleNoVpc",
"version": "$LATEST",
"message": "uploaded s3://cumulus-test-sandbox-internal/file-staging/test-src-integration/MOD09GQ___006/MOD09GQ.A8009365.7JQhni.006.6594428626875.hdf.met",
"RequestId": "a20698f3-a9a1-4e64-acbf-09ef77827fbc"
}
]
}
Get a CSV file of all the granule in the Cumulus database.
$ curl https://example.com/granule-csv --header 'Authorization: Bearer ReplaceWithTheToken'
"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published" "MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false "MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true
List executions.
$ curl https://example.com/executions?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "test-src-integration",
"table": "execution",
"limit": 3,
"page": 1,
"count": 447
},
"results": [
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"duration": 1.693,
"createdAt": 1534443719571,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443721559,
"updatedAt": 1534443721265
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"duration": 1.718,
"createdAt": 1534443706867,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"name": "bee18782-e987-4bd4-b726-41669e489e2f",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443709043,
"updatedAt": 1534443708585
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"duration": 29.397,
"createdAt": 1534443668185,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
"error": {
"Cause": "None",
"Error": "Unknown Error"
},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "completed",
"timestamp": 1534443698322,
"updatedAt": 1534443697582
}
]
}
Retrieve details for a specific execution.
$ curl https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"updatedAt": 1534441782302,
"status": "completed",
"timestamp": 1534441782302,
"tasks":
{
"MoveGranuleStep":
{
"name": "test-src-integration-MoveGranules",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
"version": "$LATEST"
},
"ProcessingStep":
{
"name": "test-src-integration-FakeProcessing",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
"version": "$LATEST"
},
"StopStatus":
{
"name": "test-src-integration-SfSnsReport",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
"version": "$LATEST"
}
},
"createdAt": 1534441752026,
"duration": 30.276,
"name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"collectionId": "MOD09GQ___006",
"error":
{
"Error": "Unknown Error",
"Cause": "None"
},
"type": "IngestGranule",
"originalPayload": { "somePayloadKey": "object containing payload at execution start" },
"finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" }
}
Retrieve details and status of a specific execution. This also returns execution history and details of the state machine when the execution exists in Step Function API.
$ curl https://example.com/executions/status/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl6sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"execution":
{
"executionArn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv:d0a6584b-bea6-476e-a745-c1feb2ad00b2",
"stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
"name": "d0a6584b-bea6-476e-a745-c1feb2ad00b2",
"status": "SUCCEEDED",
"startDate": "2018-08-16T18:39:32.209Z",
"stopDate": "2018-08-16T18:39:59.089Z",
"input":
{
"foo": "input"
},
"output":
{
"bar": "output"
}
},
"executionHistory":
{
"events":
[
{
"taskName": "foo bar 1"
},
{
"taskName": "foo bar 2"
}
]
},
"stateMachine":
{
"stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
"name": "TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl4sgv",
"status": "ACTIVE",
"definition":
{
"Comment": "Ingest Granule",
"StartAt": "First State",
"States":
{
"First State": {}
}
},
"roleArn": "arn:aws:iam::123456789012:role/test-src-integration-steprole",
"creationDate": "2018-06-11T16:08:17.533Z"
}
}
Return executions associated with specific granules. Granules can be sent as a list of granule objects containing granuleIds and collectionIds or as an Elasticsearch query to the Metrics' Elasticsearch.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
granules |
yes - if query not present |
Array<Object> |
List of granules to process. Each granule must contain a granuleId and collectionId |
-- granule.granuleId |
yes - if using granules |
string |
GranuleId for a granule |
-- granule.collectionId |
yes - if using granules |
string |
CollectionId for a granule |
query |
yes - if granules not present |
Object |
Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
curl -X POST
https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'
$ curl --request POST \
https://example.com/executions/search-by-granules?limit=3 --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"collectionId": "MOD09GQ__006"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
{
"meta": {
"count": 3
},
"results": [
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"duration": 1.693,
"createdAt": 1534443719571,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:08aa40f8-7d91-41b4-986e-d12ce495aec5",
"name": "08aa40f8-7d91-41b4-986e-d12ce495aec5",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443721559,
"updatedAt": 1534443721265
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"duration": 1.718,
"createdAt": 1534443706867,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:bee18782-e987-4bd4-b726-41669e489e2f",
"name": "bee18782-e987-4bd4-b726-41669e489e2f",
"error": {},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "running",
"timestamp": 1534443709043,
"updatedAt": 1534443708585
},
{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"duration": 29.397,
"createdAt": 1534443668185,
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:55d62b27-50cd-4cc1-81f9-e425cf10532e",
"name": "55d62b27-50cd-4cc1-81f9-e425cf10532e",
"error": {
"Cause": "None",
"Error": "Unknown Error"
},
"type": "IngestGranule",
"collectionId": "MOD09GQ___006",
"tasks": {},
"status": "completed",
"timestamp": 1534443698322,
"updatedAt": 1534443697582
}
]
}
Return the workflows that have run on specific granules. If multiple granules are specified, it will return the intersection of workflows that have run on ALL the granules. That is, only a workflow that has been run on every inputted granule will be returned
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
granules |
yes - if query not present |
Array<Object> |
List of granules to process. Each granule must contain a granuleId and collectionId |
-- granule.granuleId |
yes - if using granules |
string |
GranuleId for a granule |
-- granule.collectionId |
yes - if using granules |
string |
CollectionId for a granule |
query |
yes - if granules not present |
Object |
Query to Elasticsearch to determine which Granules with which to search. Required if no IDs are given. |
index |
yes - if query is present |
string |
Elasticsearch index to search with the given query |
curl -X POST
https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{"granules": [{ "granuleId":"MOD09GQ.A2016358.h13v04.006.2016360104606", "collectionId": "MOD09GQ___006"}]}'
$ curl --request POST \
https://example.com/executions/workflows-by-granules --header 'Authorization: Bearer ReplaceWithTheToken' \
--header 'Content-Type: application/json' \
--data '{
"index": "index-in-es",
"query": {
"size": 500,
"query": {
"filter": [
{
"bool": {
"filter": [
{
"bool": {
"should": [{"match": {"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"}}],
"minimum_should_match": 1
}
},
{
"bool": {
"should": [{"match": {"collectionId": "MOD09GQ__006"}}],
"minimum_should_match": 1
}
}
]
}
}
],
"should": [],
"must_not": []
}
}
}'
[
"DiscoverGranules",
"IngestGranules"
]
Create an execution. An execution generally includes the fields listed below.
Overview of the schema fields:
| Field | Type | Required | Description |
|---|---|---|---|
arn |
string |
yes |
execution arn (this field is unique) |
asyncOperationId |
string |
no |
id of the associated async operation |
collectionId |
string |
no |
collectionId of the associated collection (e.g. <name>___<version> where <name> is the collection name and <version> is the collection version) |
cumulusVersion |
string |
no |
cumulus version for the execution (e.g. 9.0.0) |
duration |
number |
no |
duration of the execution |
error |
object |
no |
the error details in case of a failed execution |
execution |
string |
no |
the execution page url on AWS console |
finalPayload |
object |
no |
the final payload of this execution |
name |
string |
yes |
execution name |
originalPayload |
object |
no |
the original payload for this execution |
parentArn |
string |
no |
arn of the parent execution |
status |
string |
yes |
the execution status, possible values: running, completed, failed, unknown |
tasks |
object |
no |
the tasks for this execution |
type |
string |
no |
the workflow name, e.g. IngestGranule |
timestamp |
number |
no |
the timestamp for this execution |
$ curl --request POST https://example.com/executions --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
"collectionId": "MOD09GQ___006",
"cumulusVersion": "1.2.3",
"error": {},
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
"originalPayload": { "somePayloadKey": "object containing payload at execution start" },
"parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
"status": "running",
"tasks":
{
"MoveGranuleStep":
{
"name": "test-src-integration-MoveGranules",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
"version": "$LATEST"
},
"ProcessingStep":
{
"name": "test-src-integration-FakeProcessing",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
"version": "$LATEST"
},
"StopStatus":
{
"name": "test-src-integration-SfSnsReport",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
"version": "$LATEST"
}
},
"type": "IngestGranule",
"timestamp": 1513020462156
}'
{
"message": "Successfully wrote execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}
Update/replace an existing execution. Expects payload to specify the entire execution object, and will completely replace the existing execution with the specified payload. For a field reference see "Create execution".
Returns status 200 on successful replacement, 400 if the arn
property in the payload does not match the corresponding value in the resource
URI, or 404 if there is no execution with the specified arn.
$ curl --request PUT https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"arn": "arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"asyncOperationId": "c4ece1f9-dca0-42f8-be77-da43b9b25918",
"collectionId": "MOD09GQ___006",
"cumulusVersion": "1.2.3",
"duration": 30.276,
"error": {},
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735",
"finalPayload": { "somePayloadKey" : "object containing the last reported payload from an execution" },
"name": "cff1266e-ef36-664f-a649-3a4d26bd1735",
"originalPayload": { "somePayloadKey": "object containing payload at execution start" },
"parentArn": "arn:aws:states:us-east-1:123456789012:execution:test-src-integration-ParsePdr:59edd6a4-5187-4865-b3c3-71ebcf76d7a1",
"status": "completed",
"tasks":
{
"MoveGranuleStep":
{
"name": "test-src-integration-MoveGranules",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-MoveGranules",
"version": "$LATEST"
},
"ProcessingStep":
{
"name": "test-src-integration-FakeProcessing",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-FakeProcessing",
"version": "$LATEST"
},
"StopStatus":
{
"name": "test-src-integration-SfSnsReport",
"arn": "arn:aws:lambda:us-east-1:123456789012:function:test-src-integration-SfSnsReport",
"version": "$LATEST"
}
},
"type": "IngestGranule",
"timestamp": 1513020462156
}'
{
"message": "Successfully updated execution with arn arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735"
}
Delete an execution from Cumulus.
$ curl --request DELETE https://example.com/executions/arn:aws:states:us-east-1:123456789012:execution:TestSrcIntegrationIngestGranuleStateMachine-UhCSmszl2sgv:cff1266e-ef36-664f-a649-3a4d26bd1735 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Record deleted"
}
List workflows in the Cumulus system.
$ curl https://example.com/workflows --header 'Authorization: Bearer ReplaceWithTheToken'
[
{
"name": "TestLambdaVersionWorkflow",
"template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/TestLambdaVersionWorkflow.json",
"definition": {
"Comment": "Tests Lambda update after redeploy",
"StartAt": "StartStatus",
"States": {
"StartStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Next": "WaitForDeployment"
},
"WaitForDeployment": {
"Type": "Task",
"Resource": "${WaitForDeploymentLambdaAliasOutput}",
"Next": "VersionUpTest"
},
"VersionUpTest": {
"Type": "Task",
"Resource": "${VersionUpTestLambdaAliasOutput}",
"Next": "StopStatus"
},
"StopStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "WorkflowFailed"
}
],
"End": true
},
"WorkflowFailed": {
"Type": "Fail",
"Cause": "Workflow failed"
}
}
}
},
{
"name": "HelloWorldWorkflow",
"template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
"definition": {
"Comment": "Returns Hello World",
"StartAt": "StartStatus",
"States": {
"StartStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Next": "HelloWorld"
},
"HelloWorld": {
"Type": "Task",
"Resource": "${HelloWorldLambdaAliasOutput}",
"Next": "StopStatus"
},
"StopStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "WorkflowFailed"
}
],
"End": true
},
"WorkflowFailed": {
"Type": "Fail",
"Cause": "Workflow failed"
}
}
}
}
]
Retrieve a single workflow.
$ curl https://example.com/workflows/HelloWorldWorkflow --header 'Authorization: Bearer ReplaceWithTheToken'
{
"name": "HelloWorldWorkflow",
"template": "s3://cumulus-test-sandbox-internal/cumulus/workflows/HelloWorldWorkflow.json",
"definition": {
"Comment": "Returns Hello World",
"StartAt": "StartStatus",
"States": {
"StartStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Next": "HelloWorld"
},
"HelloWorld": {
"Type": "Task",
"Resource": "${HelloWorldLambdaAliasOutput}",
"Next": "StopStatus"
},
"StopStatus": {
"Type": "Task",
"Resource": "${SfSnsReportLambdaAliasOutput}",
"Catch": [
{
"ErrorEquals": [
"States.ALL"
],
"Next": "WorkflowFailed"
}
],
"End": true
},
"WorkflowFailed": {
"Type": "Fail",
"Cause": "Workflow failed"
}
}
}
}
Async operations are long-running requests serviced by the Cumulus API.
These tend to be bulk operations. Examples include bulk granule operations and replaying ingest notifications.
This endpoint lists async operations in the Cumulus system.
$ curl https://example.com/asyncOperations --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "asyncOperation",
"limit": 1,
"page": 1,
"count": 8
},
"results": [
{
"output": "{\"deletedGranules\":[\"granule-id-f02a53418f\"]}",
"createdAt": 1591384094512,
"taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
"description": "Bulk granule deletion",
"operationType": "Bulk Granule Delete",
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"status": "SUCCEEDED",
"updatedAt": 1591384094512,
"timestamp": 1591384095235
}
]
}
Retrieve information about a single async operation. Useful for determining the status of an async operation.
$ curl https://example.com/asyncOperations/0eb8e809-8790-5409-1239-bcd9e8d28b8e --header 'Authorization: Bearer ReplaceWithTheToken'
{
"id": "0eb8e809-8790-5409-1239-bcd9e8d28b8e",
"updatedAt": 1574730504762,
"status": "RUNNING",
"taskArn": "arn:aws:ecs:us-east-1:111111111111:task/d481e76e-f5fc-9c1c-2411-fa13779b111a",
"description": "Bulk granule deletion",
"operationType": "Bulk Granule Delete"
}
The Cumulus API supports requests to replay ingest notifications. The schema below describes the expected fields in a replay request
| Field | Type | Required | Description |
|---|---|---|---|
type |
string | required | Currently only accepts kinesis. |
kinesisStream |
string | for type kinesis |
Any valid kinesis stream name (not ARN) |
kinesisStreamCreationTimestamp |
* | optional | Any input valid for a JS Date constructor. For reasons to use this field see AWS documentation on StreamCreationTimestamp. |
endTimestamp |
* | optional | Any input valid for a JS Date constructor. Messages newer than this timestamp will be skipped. |
startTimestamp |
* | optional | Any input valid for a JS Date constructor. Messages will be fetched from the Kinesis stream starting at this timestamp. Ignored if it is further in the past than the stream's retention period. |
$ curl -X POST https://example.com/replays --header 'Authorization: Bearer ReplaceWithTheToken' --data '{ "type: "kinesis", "kinesisStream": "my-stream", "endTimestamp": 1567890123456, "startTimestamp": "2019-08-31T22:22:03.456Z"}'
{
"asyncOperationId": "208a463a-e096-4dd9-b006-e09345319ae6"
}
Retrieve the data schema for a particular type of Cumulus record.
This schema describes the expected format of a record's JSON object when retrieving from Cumulus, as well as a summary of what each field may contain. The schema response can also be used to determine which fields are required when creating a new record using the API.
Supported type values are provider, collection, granule, and pdr.
$ curl https://example.com/schemas/provider --header 'Authorization: Bearer ReplceWithTheToken'
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Provider Object",
"description": "Keep the information about each ingest endpoint",
"type": "object",
"properties": {
"name": {
"title": "Title",
"description": "A title for the provider record",
"type": "string",
"pattern": "^([\\w\\d_\\-]*)$"
},
"providerName": {
"title": "Provider, e.g. MODAPS",
"description": "Name of the SIP",
"type": "string"
},
"protocol": {
"title": "Protocol",
"type": "string",
"enum": [
"http",
"ftp"
],
"default": "http"
},
"host": {
"title": "Host",
"type": "string"
},
"path": {
"title": "Path to the PDR/files folder",
"type": "string"
},
"config": {
"title": "Configuration",
"type": "object",
"properties": {
"username": {
"type": "string"
},
"password": {
"type": "string"
},
"port": {
"type": "string"
}
}
},
"status": {
"title": "Status",
"type": "string",
"enum": [
"ingesting",
"stopped",
"failed"
],
"default": "stopped",
"readonly": true
},
"isActive": {
"title": "Is Active?",
"type": "boolean",
"default": false,
"readonly": true
},
"regex": {
"type": "object",
"patternProperties": {
"^([\\S]*)$": {
"type": "string"
}
},
"readonly": true
},
"lastTimeIngestedAt": {
"title": "Last Time Ingest from the Provider",
"type": "number",
"readonly": true
},
"createdAt": {
"type": "number",
"readonly": true
},
"updatedAt": {
"type": "number",
"readonly": true
}
},
"required": [
"name",
"providerName",
"protocol",
"host",
"path",
"isActive",
"status",
"createdAt",
"updatedAt"
]
}
List reconciliation reports in the Cumulus system.
$ curl https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithTheToken'
{
"meta": {
"name": "cumulus-api",
"stack": "lpdaac-cumulus",
"table": "reconciliationReport",
"limit": 2,
"page": 1,
"count": 7
},
"results": [
{
"createdAt": 1589824212102,
"name": "inventoryReport-20200518T175012101",
"location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/inventoryReport-20200518T175012101.json",
"type": "Inventory",
"status": "Generated",
"updatedAt": 1589824218825,
"timestamp": 1589824219329
},
{
"createdAt": 1589596344305,
"name": "ModisInternalReport",
"location": "s3://lpdaac-internal/lpdaac-cumulus/reconciliation-reports/ModisInternalReport.json",
"type": "Internal",
"status": "Generated",
"updatedAt": 1589596352010,
"timestamp": 1589596352356
}
]
}
Retrieve a single reconciliation report.
$ curl https://example.com/reconciliationReports/inventoryReport-20190305T153430508 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"presignedS3Url": "https://example.amazonaws.com/example",
"data": "report data, see examples below for each report type"
}
{
"presignedS3Url": "https://example.amazonaws.com/example",
"data": "Error: Report examplereport exceeded maximum allowed payload size"
}
{
"reportStartTime": "2019-03-05T15:34:30.508Z",
"reportEndTime": "2019-03-05T15:34:37.243Z",
"status": "SUCCESS",
"error": null,
"reportType": "Inventory",
"filesInCumulus": {
"okCount": 40,
"okCountByGranule": {
"MOD09GQ.A2016358.h13v04.006.2016360104606": 10,
"MYD13Q1.A2017297.h19v10.006.2017313221303": 10,
"MOD09GQ.A3518809.ln_rVr.006.7962927138074": 10,
"MOD09GQ.A8768252.HC4ddD.006.2077696236118": 5,
"MOD09GQ.A8722843.GTk5A3.006.4026909316904": 5
},
"onlyInS3": [
"s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.cmr.xml",
"s3://cumulus-test-sandbox-private/BROWSE.MYD13Q1.A2017297.h19v10.006.2017313221201.hdf"
],
"onlyInDynamoDb": [
{
"uri": "s3://cumulus-test-sandbox-protected/MOD09GQ.A2016358.h13v04.006.2016360104606.hdf",
"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
},
{
"uri": "s3://cumulus-test-sandbox-private/MYD13Q1.A2017297.h19v10.006.2017313221303.hdf.met",
"granuleId": "MYD13Q1.A2017297.h19v10.006.2017313221303"
}
]
},
"collectionsInCumulusCmr": {
"okCount": 1,
"onlyInCumulus": [
"L2_HR_PIXC___000"
],
"onlyInCmr": [
"MCD43A1___006",
"MOD14A1___006"
]
},
"granulesInCumulusCmr": {
"okCount": 3,
"onlyInCumulus": [
{
"granuleId": "MOD09GQ.A3518809.ln_rVr.006.7962927138074",
"collectionId": "MOD09GQ___006"
},
{
"granuleId": "MOD09GQ.A8768252.HC4ddD.006.2077696236118",
"collectionId": "MOD09GQ___006"
}
],
"onlyInCmr": [
{
"GranuleUR": "MOD09GQ.A0002421.oD4zvB.006.4281362831355",
"ShortName": "MOD09GQ",
"Version": "006"
},
{
"GranuleUR": "MOD09GQ.A0007443.H16AHq.006.7078190437865",
"ShortName": "MOD09GQ",
"Version": "006"
}
]
},
"filesInCumulusCmr": {
"okCount": 11,
"onlyInCumulus": [
{
"fileName": "MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
"uri": "s3://cumulus-test-sandbox-public/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904.jpeg",
"granuleId": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
}
],
"onlyInCmr": [
{
"URL": "https://example.com/MYD13Q1.A2017297.h19v10.006.2017313221202.hdf",
"Type": "GET DATA",
"GranuleUR": "MOD09GQ.A4675287.SWPE5_.006.7310007729190"
},
{
"URL": "https://cumulus-test-sandbox-public.s3.amazonaws.com/MOD09GQ___006/MOD/MOD09GQ.A8722843.GTk5A3.006.4026909316904_ndvi.jpg",
"Type": "GET DATA",
"GranuleUR": "MOD09GQ.A8722843.GTk5A3.006.4026909316904"
}
]
}
}
{
"createStartTime": "2020-08-28T18:33:50.525Z",
"createEndTime": "2020-08-28T18:34:16.646Z",
"status": "SUCCESS",
"reportType": "Granule Not Found",
"filesInCumulus": {
"okCount": 20,
"okCountByGranule": {
"MOD09GQ.A2770300.W1_V5Z.006.7853319756315": 2,
"MOD09GQ.A7682091.QtZjhI.006.1218763030745": 3,
"MOD09GQ.A9536857.bV1q4A.006.0980635705136": 3,
"MOD09GQ.A7790080.xLPpTZ.006.2292724247258": 3,
"MOD09GQ.A3769578.tlqq7j.006.4434181201872": 3,
"MOD09GQ.A5423294.RFOrMI.006.8677601711674": 3,
"MOD09GQ.A8837603.fb2Vhw.006.3950746589733": 3
},
"onlyInS3": [
"s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf.met",
"s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf.met",
"s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593198955197/MOD09GQ.A7375038.gm1irM.006.1317280710645.hdf.met",
"s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122280799/MOD09GQ.A6921412.znLBqe.006.9673856807617.hdf",
"s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122597517/MOD09GQ.A1201557.d6tP2Y.006.7482431709753.hdf",
"s3://cumulus-sandbox-protected/MOD09GQ___006/2017/MOD/test-data-1593122881446/MOD09GQ.A2708681.CFkGhW.006.7154000014360.hdf"
],
"onlyInDynamoDb": [
{
"uri": "s3://cumulus-sandbox-private/MOD09GQ___006/MOD/test-data-1593178175944/MOD09GQ.A2770300.W1_V5Z.006.7853319756315.hdf.met",
"granuleId": "MOD09GQ.A2770300.W1_V5Z.006.7853319756315"
}
]
},
"collectionsInCumulusCmr": {
"okCount": 0,
"onlyInCumulus": [
"MOD09GQ_test-test-data-1593122280799___006",
"MOD09GQ_test-test-data-1593122597517___006",
"MOD09GQ_test-test-data-1593122881446___006",
"MOD09GQ_test-test-data-1593178175944___006",
"MOD09GQ_test-test-data-1593198739991___006",
"MOD09GQ_test-test-data-1593198955197___006",
"MOD09GQ_test-test-data-1593199179316___006",
"MOD09GQ_test-test-data-1593199379860___006",
"MOD09GQ_test-test-data-1593199548454___006",
"MOD09GQ_test-test-data-1593199742089___006",
"MOD09GQ_test-test-data-1593199934835___006"
],
"onlyInCmr": [
"A2_RainOcn_NRT___0",
"A2_SI12_NRT___0",
"A2_SI25_NRT___0",
"A2_SI6_NRT___0",
"AST_L1A___003",
"CMR44JK___001",
"L2_HR_PIXC___000",
"L2_HR_PIXC___1",
"MCD43A1___006",
"MOD09GQ___006",
"MOD11A1___006",
"MOD14A1___006",
"MUR-JPL-L4-GLOB-v4.1___1",
"MYD09A1___006",
"MYD13A1___006",
"MYD13Q1___006",
"hs3avaps___1",
"hs3wwlln___1"
]
},
"granulesInCumulusCmr": {
"okCount": 0,
"onlyInCumulus": [],
"onlyInCmr": []
},
"filesInCumulusCmr": {
"okCount": 0,
"onlyInCumulus": [],
"onlyInCmr": []
}
}
{
"reportType": "Internal",
"createStartTime": "2020-08-28T19:01:50.636Z",
"createEndTime": "2020-08-28T19:02:08.211Z",
"reportStartTime": "2020-01-28T18:21:31.119Z",
"reportEndTime": "2020-08-28T18:21:31.119Z",
"status": "SUCCESS",
"collections": {
"okCount": 10,
"withConflicts": [
{
"es": {
"createdAt": 1591902043153,
"granuleId": "^.*$",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
"dataType": "L2_HR_PIXC",
"name": "L2_HR_PIXC",
"files": [
{
"bucket": "private",
"regex": ".*.h5$",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5"
}
],
"granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
"reportToEms": true,
"version": "000",
"duplicateHandling": "error",
"updatedAt": 1591902043153,
"timestamp": 1591902043840
},
"db": {
"duplicateHandling": "error",
"granuleIdExtraction": "^(.*)(\\.h5|\\.cmr)",
"version": "000",
"dataType": "L2_HR_PIXC",
"files": [
{
"bucket": "private",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
"regex": ".*.h5$"
}
],
"updatedAt": 1591902043153,
"createdAt": 1591902043153,
"reportToEms": false,
"granuleId": "^.*$",
"sampleFileName": "L2_HR_PIXC_product_0001-of-4154.h5",
"name": "L2_HR_PIXC"
}
}
],
"onlyInEs": [
"MCD43A1___006",
"MOD09GQ___006"
],
"onlyInDb": [
"MYD13QA___005"
]
},
"granules": {
"okCount": 77,
"withConflicts": [
{
"es": {
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
"processingEndDateTime": "2020-05-18T19:49:04.769Z",
"published": false,
"error": {},
"productVolume": 253949,
"timeToPreprocess": 0,
"duration": 1.005,
"createdAt": 1589831343764,
"granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
"processingStartDateTime": "2020-05-18T19:49:03.831Z",
"provider": "s3_provider",
"timeToArchive": 0,
"files": [
{
"bucket": "cumulus-test-sandbox-internal",
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"size": 233840,
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"type": "data",
"key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf"
},
{
"bucket": "cumulus-test-sandbox-internal",
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"size": 14297,
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"type": "metadata",
"key": "test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met"
},
{
"bucket": "cumulus-test-sandbox-internal",
"fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"size": 5812,
"source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"type": "browse",
"key": "test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg"
}
],
"collectionId": "MOD14A1___006",
"status": "running",
"timestamp": 1589831367756,
"updatedAt": 1589831344769
},
"db": {
"published": true,
"endingDateTime": "2017-11-08T23:59:59Z",
"status": "completed",
"timestamp": 1589831361039,
"createdAt": 1589831343764,
"processingEndDateTime": "2020-05-18T19:49:20.023Z",
"productVolume": 258297,
"timeToPreprocess": 0.263,
"timeToArchive": 0.942,
"productionDateTime": "2018-07-19T12:01:01Z",
"execution": "https://console.aws.amazon.com/states/home?region=us-east-1#/executions/details/arn:aws:states:us-east-1:123456789012:execution:IngestAndPublishGranule:edb21d63-b62d-42e9-8879-f39d53314f16",
"files": [
{
"bucket": "cumulus-test-sandbox-protected",
"key": "MOD14A1___006/2017/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"size": 233840,
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf",
"type": "data"
},
{
"bucket": "cumulus-test-sandbox-private",
"key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"size": 14297,
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"source": "s3://cumulus-test-sandbox-internal/test-data/files/MOD14A1.A4313371.RQSeyH.006.7795385650931.hdf.met",
"type": "metadata"
},
{
"bucket": "cumulus-test-sandbox-public",
"key": "MOD14A1___006/BRO/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"size": 5812,
"fileName": "BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"source": "s3://cumulus-test-sandbox-internal/test-data/files/BROWSE.MOD14A1.A4313371.RQSeyH.006.7795385650931.1.jpg",
"type": "browse"
},
{
"bucket": "cumulus-test-sandbox-protected-2",
"key": "MOD14A1___006/MOD/MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
"size": 4348,
"fileName": "MOD14A1.A4313371.RQSeyH.006.7795385650931.cmr.xml",
"type": "metadata"
}
],
"cmrLink": "https://cmr.uat.earthdata.nasa.gov/search/granules.json?concept_id=G1234771093-CUMULUS",
"processingStartDateTime": "2020-05-18T19:49:03.831Z",
"updatedAt": 1589831361039,
"beginningDateTime": "2017-10-24T00:00:00Z",
"provider": "s3_provider",
"granuleId": "MOD14A1.A4313371.RQSeyH.006.7795385650931",
"collectionId": "MOD14A1___006",
"duration": 17.275,
"error": {
"Error": "Unknown Error",
"Cause": "None"
},
"lastUpdateDateTime": "2018-04-25T21:45:45.524053"
}
}
],
"onlyInEs": [
{
"granuleId": "MYD13Q1.A0323210.Yujnv6.006.1973526114346",
"collectionId": "MYD13Q1___006",
"provider": "s3_provider",
"createdAt": 1589913461833,
"updatedAt": 1589913522990
}
],
"onlyInDb": [
{
"granuleId": "MYD13Q1.A5822875.qHI8hK.006.8698738069249",
"collectionId": "MYD13Q1___006",
"provider": "s3_provider",
"createdAt": 1598552797312,
"updatedAt": 1598552827611
}
]
}
}
"granuleUr","collectionId","createdAt","startDateTime","endDateTime","status","updatedAt","published" "MOD14A1.A9506271.IvEJsu.006.8359924290786","MOD14A1___006","2020-05-18T20:15:54.525Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-05-18T20:16:02.473Z",false "MYD13Q1.A9663671.0zkwKH.006.9812354158395","MYD13Q1___006","2020-07-06T19:46:19.957Z","2017-10-24T00:00:00Z","2017-11-08T23:59:59Z","completed","2020-07-06T19:46:57.054Z",true
Create a new reconciliation report.
| parameter | value | required | description |
|---|---|---|---|
reportName |
string |
false |
Report name. |
reportType |
"Granule Inventory"|"Granule Not Found"|"Internal"|"Inventory" |
false |
Report type (default Inventory) |
startTimestamp |
string |
false |
Any input valid for a JS Date contstructor. Data older than this will be ignored in the generated report. |
endTimestamp |
string |
false |
Any input valid for a JS Date contstructor. Data newer than this will be ignored in the generated report. |
collectionId |
[string | array] |
false |
collectionId (or array of collectionIds) for comparison of collection and granule holdings. |
granuleId |
[string | array] |
false |
granuleId (or array of granuleIds) for use for comparison of collection and granule holdings. |
provider |
[string | array] |
false |
provider name (or array of providers) for comparison of granule holdings |
status |
string |
false |
status filter for Granule Inventory reports |
NOTE: Adding a startTimestamp or endTimestamp value to the POST request will result in one way comparisons for some fields for Inventory and Granule Not Found reports.
NOTE: Adding a granuleId input will result in an one way report for collections for Inventory and Granule Not Found reports.
NOTE: Inventory and Granule Not Found reports only alllow one of the parameters (collectionId, granuleId, provider) in same request.
NOTE: Granule Inventory reports can be filtered with the following parameters: (collectionId, granuleId, status). If granuleId is a string or single value array, it will search for granules with granuleIds containing that substring.
$ curl --request POST https://example.com/reconciliationReports --header 'Authorization: Bearer ReplaceWithToken' --header 'Content-Type: application/json' --data '{
"reportName": "ModisInventoryReport",
"reportType": "Inventory",
"startTimestamp": 1269993600000,
"endTimestamp": 1350000000000,
"collectionId": "MOD09GQ___006",
"provider": "MODIS",
"granuleId": "MOD09GQ.A2016358.h13v04.006.2016360104606"
}'
{
"createdAt": 1623256106616,
"updatedAt": 1623256106616,
"id": "bb7059f4-0cd8-4205-8857-fb0e6b68b3e4",
"status": "RUNNING",
"taskArn": "arn:aws:ecs:us-east-1:123456789012:task/stackname-CumulusECSCluster/656b2c8203c4488e9feffff38beb4be5",
"description": "Create Reconciliation Report",
"operationType": "Reconciliation Report"
}
Delete a reconciliation report from Cumulus.
$ curl --request DELETE https://example.com/reconciliationReports/inventoryReport-20180620T205838883 --header 'Authorization: Bearer ReplaceWithTheToken'
{
"message": "Report deleted"
}
Create EMS reports and send them to EMS. For more information on EMS reporting and types of reports see the Cumulus EMS Reporting documentation.
Overview of the request fields:
| Field | Required | Value | Description |
|---|---|---|---|
reportType |
Y |
"metadata"|"ingest"|"distribution" |
type of report |
startTime |
N |
string |
report startTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss If startTime and endTime are not specified, the reports for previous day will be generated |
endTime |
N |
string |
report endTime in format YYYY-MM-DD or YYYY-MM-DDTHH:mm:ss |
collectionId |
N |
string |
collectionId of the report records This field can be specified when report type is metadata or ingest |
$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
"reportType": "metadata"
}'
$ curl https://example.com/ems --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
"reportType": "ingest",
"startTime": "2019-06-27",
"endTime": "2019-06-29",
"collectionId": "MOD14A1___006"
}'
{
"message": "Reports are being generated",
"status": 202
}
The Cumulus API can provide information about its configuration.
GET requests to the instance metadata endpoint return a json object with information about how the Cumulus stack is configured. It returns the CMR provider and environment as well as the stackName (prefix).
$ curl https://example.com/instanceMeta --header 'Authorization: Bearer ReplaceWithTheToken'
{
"cmr": {
"provider": "CUMULUS",
"environment": "UAT"
},
"cumulus": {
"stackName": "cumulus-stack-prefix"
}
}
These endpoints provide an interface into the Elasticsearch SDK to perform functions like Elasticsearch reindex.
Cumulus uses index aliases to allow reindexing with no downtime.
The reindex command creates a new index and reindexes the source index to the new, destination index. This uses the Elasticsearch reindex functionality, but includes setting up the new index.
Note that there will now be two copies of your index. Your Cumulus instance will still point to the old copy until you perform the change-index function described below.
An alias should not be specified unless you have a specific alias configured. If a source index is not specified, it will default to the index from the alias. If you want to name the destination index something particular, you can specify a name, otherwise the destination index name will default to 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019). If you specify a destination index name, it must be an index that does not already exist in your cluster. The destination index must be different than the source index or the operation will fail.
Since reindex can be a long running operation, the reindex-status endpoint must be queried to see if the operation has completed.
Overview of request fields:
| parameter | value | required | description |
|---|---|---|---|
aliasName |
string |
false |
Alias to reindex from. Will be used if sourceIndex is not provided. Will default to the default alias if not provided. |
sourceIndex |
string |
false |
Index to reindex from |
destIndex |
string |
true |
Index to reindex to |
$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken'
{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-4-12-2019. Check the reindex-status endpoint for status." }
$ curl --request POST https://example.com/elasticsearch/reindex --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"aliasName": "cumulus-alias",
"sourceIndex": "cumulus-4-4-2019",
"destIndex": "cumulus-new-index"
}'
{ "message": "Reindexing from cumulus-4-4-2019 to cumulus-new-index. Check the reindex-status endpoint for status." }
Reindexing can be a long running operation, so use this endpoint to get the status of your reindex and the status of your indices.
reindexStatus.nodes will be populated if there is a reindex operation currently running, nodes will be empty if not.
The elasticsearchStatus shows the number of documents in each index which can be used as a sanity check for the reindex operation before switching over to the new index. Note that the active index will continue to index logs, so the number of documents in the active index will be slightly higher and continue to grow.
$ curl https://example.com/elasticsearch/reindex-status --header 'Authorization: Bearer ReplaceWithTheToken'
{
"reindexStatus": {
"nodes": {
"TKDzeZ_TQVWkwKZYtCtv_g": {
"name": "TKDzeZ_",
"roles": [
"master",
"data",
"ingest"
],
"tasks": {
"TKDzeZ_TQVWkwKZYtCtv_g:546691": {
"node": "TKDzeZ_TQVWkwKZYtCtv_g",
"id": 546691,
"type": "transport",
"action": "indices:data/write/reindex",
"start_time_in_millis": 1555073365557,
"running_time_in_nanos": 75424890,
"cancellable": true
}
}
}
}
},
"indexStatus": {
"_shards": {
"total": 32,
"successful": 16,
"failed": 0
},
"_all": {
"primaries": {
"docs": {
"count": 52213,
"deleted": 0
}
},
"total": {
"docs": {
"count": 52213,
"deleted": 0
}
}
},
"indices": {
".kibana": {
"primaries": {
"docs": {
"count": 1,
"deleted": 0
}
},
"total": {
"docs": {
"count": 1,
"deleted": 0
}
}
},
"cumulus-2019-4-12": {
"primaries": {
"docs": {
"count": 25569,
"deleted": 0
}
},
"total": {
"docs": {
"count": 25569,
"deleted": 0
}
}
},
"cumulus-2019-4-9": {
"primaries": {
"docs": {
"count": 25644,
"deleted": 0
}
},
"total": {
"docs": {
"count": 25644,
"deleted": 0
}
}
}
}
}
}
{
"reindexStatus": {
"nodes": {}
},
"indexStatus": {
"_shards": {
"total": 22,
"successful": 11,
"failed": 0
},
"_all": {
"primaries": {
"docs": {
"count": 51210,
"deleted": 0
}
},
"total": {
"docs": {
"count": 51210,
"deleted": 0
}
}
},
"indices": {
".kibana": {
"primaries": {
"docs": {
"count": 1,
"deleted": 0
}
},
"total": {
"docs": {
"count": 1,
"deleted": 0
}
}
},
"cumulus-2019-4-12": {
"primaries": {
"docs": {
"count": 25569,
"deleted": 0
}
},
"total": {
"docs": {
"count": 25569,
"deleted": 0
}
}
},
"cumulus-2019-4-9": {
"primaries": {
"docs": {
"count": 25640,
"deleted": 0
}
},
"total": {
"docs": {
"count": 25640,
"deleted": 0
}
}
}
}
}
}
Change index switches the Elasticsearch index to point to the new index, rather than the current index. You may choose to delete your current index during this operation using the deleteSource parameter, which defaults to false. If you are using the default index created by Cumulus, this will switch your Cumulus instance to the new index and you will see those changes immediately via the API and dashboard. If newIndex does not exist, an index with that name will be created.
currentIndex and newIndex are required parameters.
In the context of reindex, you'd call change-index, following reindex completion to start using the new index with no downtime.
Overview of the request fields:
| parameter | value | required | description |
|---|---|---|---|
aliasName |
string |
false |
Alias to use for newIndex. Will be the default index if not provided |
currentIndex |
string |
true |
Index to change the alias from |
newIndex |
string |
true |
Index to change alias to |
deleteSource |
boolean |
false |
If set to true it will delete the index provided in currentIndex |
$ curl --request POST https://example.com/elasticsearch/change-index --header 'Content-Type: application/json' --header 'Authorization: Bearer ReplaceWithTheToken' --data '{
"aliasName": "cumulus-alias",
"currentIndex": "cumulus-12-4-2019",
"newIndex": "cumulus-4-12-2019",
"deleteSource": false
}'
{ "message": "Reindex success - alias cumulus now pointing to cumulus-4-12-2019" }
In case of corruption of your Elasticsearch index, you can reindex your data from the database. This will include collections, executions, granules, pdrs, providers, and rules. Logs will not be indexed since they are not stored in the database.
You can specify an index (should be empty) or if no index name is specified, a default with the format of 'cumulus-year-month-day' with today's date (e.g. cumulus-4-12-2019) will be used. If the specified index does not exist, it will be created.
It is recommended that workflow rules be turned off, as any data ingested into the database during this operation cannot be guaranteed to make it into the new index. Following the completion of the index, you will need to use the change index operation to switch your Elasticsearch to point to the new instance.
Indexing is an async operation, so an operation id will be returned. You can query the status using the asyncoperations GET endoint with the operation id.
$ curl --request POST https://example.com/elasticsearch/index-from-database --header 'Authorization: Bearer ReplaceWithTheToken' --header 'Content-Type: application/json' --data '{
"indexName": "recovery-index"
}'
{"message":"Indexing database to recovery-index. Operation id: 8f4d35ba-c858-40ae-93f0-4465f54a911d"}
$ curl --header 'Authorization: Bearer ReplaceWithTheToken' https://example.com/asyncoperations/8f4d35ba-c858-40ae-93f0-4465f54a911d
{
"id":"8f4d35ba-c858-40ae-93f0-4465f54a911d",
"status":"SUCCEEDED",
"taskArn":"arn:aws:ecs:us-east-1:XXXXXXXXXXXX:task/dc41ff2b-e610-4f7a-8c63-48b48ceb7160",
"output":"\"Index from database complete\""
}
Use the indices-status endpoint to view information about your elasticsearch indices. This endpoint will return the information for the indices call in the Elasticsearch API.
$ curl https://example.com/elasticsearch/indices-status --header 'Authorization: Bearer ReplaceWithTheToken'
yellow open .kibana yxBUTswHSr2ecec8y1szEg 1 1 1 0 3.1kb 3.1kb yellow open cumulus-2019-6-21 ELpdxYVKSc-VvNxlNjmMUA 5 1 700 225 2.5mb 2.5mb yellow open cumulus-2019-6-24 lmylLkTsQJWIVYFJcftMqQ 5 1 700 0 4mb 4mb yellow open cumulus jQLXsb8yS4aEmLAAyHcmCw 5 1 217298 588 124.3mb 124.3mb
Get the current aliased index being used by the Cumulus Elasticsearch instance.
$ curl https://example.com/elasticsearch/current-index --header 'Authorization: Bearer ReplaceWithTheToken'
["cumulus-2019-6-24"]
Serve the dashboard from an S3 bucket.
This is a way to serve the Cumulus dashboard in a browser from an S3 bucket without making the bucket or files public.
To use this:
terraform.tfvars, otherwise you will not be able to access the bucket.example.com with your Cumulus backend API and dashboard-bucket with your bucket name.https://example.com/dashboard/dashboard-bucket/index.html
This endpoint lists ORCA recovery request status.
Query must include one or both of the following query parameters:
| query string parameter | description |
|---|---|
granuleId={granuleId} |
query the recovery requests for a given granuleId |
asyncOperationId={asyncOperationId} |
query the recovery requests for a given asyncOperationId |
$ curl https://example.com/orca/recovery?asyncOperationId=0eb8e809-8790-5409-1239-bcd9e8d28b8e --header 'Authorization: Bearer ReplaceWithTheToken'
$ curl https://example.com/orca/recovery?granuleId=123456 --header 'Authorization: Bearer ReplaceWithTheToken'
[
{
"request_id": "832f322f-c64c-4f47-8a74-7f3a5ffc5bd1",
"request_group_id": "1b48171d-7b07-4fb0-8c15-fbac7d7da5c6",
"granule_id": "MYD13Q1.A0319630.zMiLs8.006.2186263431231",
"object_key": "MYD13Q1___006/MYD/MYD13Q1.A0319630.zMiLs8.006.2186263431231.cmr.xml",
"job_type": "restore",
"restore_bucket_dest": "bucket-orca-glacier",
"archive_bucket_dest": "bucket-protected",
"job_status": "complete",
"request_time": "2021-02-24 21:43:56.210106+00:00",
"last_update_time": "2021-02-24 21:43:56.210106+00:00",
"err_msg": null
},
{
"request_id": "0f8514cf-c8fb-403d-8dbb-071b31787489",
"request_group_id": "1b48171d-7b07-4fb0-8c15-fbac7d7da5c6",
"granule_id": "MYD13Q1.A0319630.zMiLs8.006.2186263431231",
"object_key": "MYD13Q1___006/MYD/MYD13Q1.A0319630.zMiLs8.006.2186263431231.hdf.met",
"job_type": "restore",
"restore_bucket_dest": "bucket-orca-glacier",
"archive_bucket_dest": "bucket-private",
"job_status": "inprogress",
"request_time": "2021-02-24 21:43:54.006079+00:00",
"last_update_time": "2021-02-24 21:43:54.006079+00:00",
"err_msg": null
},
{
"request_id": "615cdcd4-8fcd-4c59-882e-700cb3ab7772",
"request_group_id": "1b48171d-7b07-4fb0-8c15-fbac7d7da5c6",
"granule_id": "MYD13Q1.A0319630.zMiLs8.006.2186263431231",
"object_key": "MYD13Q1___006/2017/MYD/MYD13Q1.A0319630.zMiLs8.006.2186263431231.hdf",
"job_type": "restore",
"restore_bucket_dest": "bucket-orca-glacier",
"archive_bucket_dest": "bucket-protected",
"job_status": "inprogress",
"request_time": "2021-02-24 21:43:53.120706+00:00",
"last_update_time": "2021-02-24 21:43:53.120706+00:00",
"err_msg": null
}
]
This endpoint triggers a run of the postgres-migration-count-tool as an async operation of type Migration Count Report.
For more information on the tool, see the README
The query uses the following optional parameters:
| parameter | description |
|---|---|
| cutoffSeconds | Sets the number of seconds prior to this execution to 'cutoff' reconciliation queries. This allows in-progress or other in-flight operations time to complete and propagate to ElasticSearch/Dynamo/Postgres. Default is 3600 |
| dbConcurrency | Sets max number of parallel collections reports the script will run at a time. Default 20 |
| dbMaxPool | Sets the maximum number of connections the database pool has available. Modifying this may result in unexpected failures. Default is 20 |
| reportPath | Sets the path location for the tool to write a copy of the lambda payload to S3 |
| reportBucket | Sets the bucket used for reporting. If this argument is used, a reportPath must be set to generate a report |
$curl -X POST https://$API_URL/dev/migrationCounts -d 'reportBucket=someBucket&reportPath=someReportPath&cutoffSeconds=60&dbConcurrency=20&dbMaxPool=20' --header 'Authorization: Bearer $TOKEN'
{"id":"7ccaed31-756b-40bb-855d-e5e6d00dc4b3","status":"RUNNING","taskArn":"arn:aws:ecs:us-east-1:AWSID:task/$PREFIX-CumulusECSCluster/123456789","description":"Migration Count Tool ECS Run","operationType":"Migration Count Report"}