Cumulus Distribution API

The Cumulus Distribution API allows users to access protected data from Cumulus when authenticated via the configured OAuth provider. The supported OAuth providers are Earthdata Login and Cognito.

Versioning

The Cumulus Distribution API is versioned and the current version is v1. Retrieve the latest API version from Cumulus.

To use any version, include the version number in the path before the query endpoint. {API_URL}/{version}/{query}

GET

/version

Example Request

$ curl https://example.com/version

Example Response

{
    "response_version": "v1",
    "api_version": "1.1.1"
}

Distribution

File downloads through the Cumulus Distribution API are available to the authenticated users via the configured OAuth provider. Links to these protected files can be found in the metadata from the Common Metadata Repository (CMR). Protected URLs look like: https://cumulus-distribution-api/path/to/science-file.hdf. For any request of protected files, the distribution API ensures the user is authenticated with the OAuth provider then redirects the user to a signed S3 url that will download the requested science file.

HEAD

The Cumulus API wraps the S3 HeadObject method

HEAD

/path/to/science-file.hdf

Example Request

curl --location --head 'https://example.com/path/to/science-file.hdf' \
--header 'Range: bytes=0-1023' \
--header 'Cookie: accessToken=<access-token>'

Example Response

HTTP/1.1 200 OK
x-amz-id-2: uLwBLVcaGUv5mDgTQLJTSXrjqxNveiVJN5DRR2oHMIHLS6cLHT8q47UQeX2Y370KK+R7kafhmzE=
x-amz-request-id: 6QVAWF6HMTJ11FCK
Date: Wed, 16 Jun 2021 23:06:20 GMT
Last-Modified: Thu, 10 Jun 2021 18:23:19 GMT
ETag: "8d1ec5c0463e59d26adee87cdbbee816"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Server: AmazonS3
Content-Length: 1098034

Example Response Body

No Response Body Returned

Supported Headers

Name	Value	Passed to HeadObject	Example Usage
`Cookie`	(optional) bytes=(min byte)-(max byte)	`false`	`curl --location --head 'https://example.com/path/to/science-file.hdf' --header 'Cookie: accessToken=<access-token>'`
`Range`	(optional) bytes=(min byte)-(max byte)	`true`	`curl --location --head 'https://example.com/path/to/science-file.hdf' --header 'Range: bytes=0-1023'`

GET

The Cumulus API wraps the S3 GetObject method

GET

/path/to/science-file.hdf

Example Request

curl --location --request GET 'https://example.com/path/to/science-file.hdf' \
--header 'Cookie: accessToken=<access-token'

Example Response

HTTP/1.1 200 OK
x-amz-id-2: uLwBLVcaGUv5mDgTQLJTSXrjqxNveiVJN5DRR2oHMIHLS6cLHT8q47UQeX2Y370KK+R7kafhmzE=
x-amz-request-id: 6QVAWF6HMTJ11FCK
Date: Wed, 16 Jun 2021 23:06:20 GMT
Last-Modified: Thu, 10 Jun 2021 18:23:19 GMT
ETag: "8d1ec5c0463e59d26adee87cdbbee816"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Server: AmazonS3
Content-Length: 1098034

Example Response Body

<binary>

Supported Headers

Name	Value	Passed to HeadObject	Example Usage
`Cookie`	(optional) bytes=(min byte)-(max byte)	`false`	`curl --request GET 'https://example.com/path/to/science-file.hdf' --header 'Cookie: accessToken=<access-token>'`
`Range`	(optional) bytes=(min byte)-(max byte)	`true`	`curl --request GET 'https://example.com/path/to/science-file.hdf' --header 'Range: bytes=0-1023'`

Root

Displays a basic welcome page with login or logout link.

GET

Validates code against OAuth provider's API, adds an entry to the access tokens table, and redirects to the URI as specified by state parameter or / endpoint. It is used as a redirect_uri to pass into OAuth provider's authentication service API.

GET

/login

Query Parameters

query string parameter	description
`code={string}`	The authentication code from OAuth provider
`state={string}`	The URI to redirect to after if authentication was successful

Example request

$ GET https://example.com/login?code=somecode&state=somestate

Logout

Logs out the API, and removes the access token from the access tokens table.

GET

/logout

Example request

$ GET https://example.com/logout

Bucket Map

The Cumulus Distribution API supports an S3 bucket map. The /locate endpoint fetches bucket map paths for a given bucket. Information about S3 bucket map is available here

GET /locate

Query Parameters

query string parameter	description
`bucket_name={string}`	The s3 bucket name

Example Request

$ curl https://example.com/locate?bucket_name={bucket}

Example Response

["/path1/to/bucket", "/path2/to/bucket"]

Temporary S3 Credentials

The Cumulus API can provide temporary credentials that provide read-only, same-region, direct access to S3 objects.

For Cumulus deployments in NGAP[^ngap] , the /s3credentials endpoint can be configured to request temporary credentials from the NGAP lambda function: gsfc-ngap-sh-s3-sts-get-keys. Additionally, these deployments may be configured to limit the scope of the dispensed credentials only to bucket/keypaths that match the user's CMR[^cmr] ACL[^acl] permissions. Check with your Cumulus deployer to discover what types of credentials are dispensed by this endpoint.

GET requests with a valid cookie to the endpoint return a credentials object that can be used to make direct S3 requests. The easiest way to get a set of credentials is to visit the endpoint in a browser to handle authentication and redirects. If you wish to use temporary credentals in AWS, you can find examples on the /s3credentialsREADME endpoint in both javascript and python.

GET

/s3credentials

Example Request

https://example.com/s3credentials

Example Response

{
  "accessKeyId": "AKIAIOSFODNN7EXAMPLE",
  "secretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "sessionToken": "LONGSTRINGOFCHARACTERSnBeoImkYlERDDHhmwZivcKPd63LUp1uhuZ9bhhIHUjvt++hgRSk9HIMZDEHH9crnukckEZ+FGYrSiwndzjBQ==",
  "expiration": "2019-02-27 23:26:56+00:00"
}

Command line credentials request

In order to script the request for credentials you must provide an accessToken cookie in your curl request. This is how the distribution api determines if you are authenticated with Earthdata Login. If no cookie is provided the authentication workflow is begun. Here is a sample script to get credentials from the command line.

Sample script to recieve credentials.

#! /bin/sh

COOKIEJAR=cookie.txt
rm -f $COOKIEJAR

ORIGIN=$(dirname $CUMULUS_DISTRIBUTION_URL)
CREDENTIALS_URL="$CUMULUS_DISTRIBUTION_URL/s3credentials"

# create a base64 hash of your login credentials
AUTH=$(printf "$EARTHDATA_USERNAME:$EARTHDATA_PASSWORD" | base64)

# Request the Earthdata url with client id and redirect uri to use with Cumulus
AUTHORIZE_URL=$(curl -s -i ${CREDENTIALS_URL} | grep location | sed -e "s/^location: //");

# Request an authorization grant code
REDIRECT_URL=$(curl -s -i -X POST \
  -F "credentials=${AUTH}" \
  -H "Origin: ${ORIGIN}" \
  ${AUTHORIZE_URL%$'\r'} | grep Location | sed -e "s/^Location: //")

# Set the correct cookie via the redirect endpoint with grant code.
curl -i -c ${COOKIEJAR} -s ${REDIRECT_URL%$'\r'} | grep location | sed -e "s/^location: //"

# Call the s3credentials endpoint with correct cookies
CREDS=$(curl -i -b ${COOKIEJAR} -s $CREDENTIALS_URL)

echo $CREDS

Non-NGAP deployments

For non-NGAP deployments that wish to provide temporary credentials, you must provide the name of a lambda available to your stack either by overriding the default sts_credentials_lambda in your Cumulus deployment configuration or by setting the environment variable STSCredentialsLambda on your API. Your lambda function must take an payload object as described below and return AWS.Credentials appropriate to your use case probably via the AWS Security Token Service.

[^ngap]: NASA-compliant General Application Platform [^cmr]: Common Metadata Repository [^acl]: Access Control List

sample `sts_credentials_lambda` payload:

{
 accesstype: 'sameregion',
 duration: '3600', // one hour max allowed by AWS.
 rolesession: username,
 userid: username
}

Temporary S3 Credentials README

The Cumulus API also provides a convenient documentation page deployed alongside of the /s3credentials endpoint.

Un authenticated GET requests return a webpage that describes the basic use of the /s3credentials endpoint and displays a small example of how to use the returned credential object.

image of example response

GET

/s3credentialsREADME

Example Request

https://example.com/s3credentialsREADME

Example Response

See image to left.