AWS REST API Authentication Using Node.js

I’ve been learning as much as I can on Amazon Web Services over the last couple of months; the looming shadow of it over traditional IT finally got too much, and I figured it was time to make the leap. Overall it’s been a great experience, and the biggest takeaway I’ve probably had is how every service, and the way in which we consume them, are application-centric.
Every service is fully API first, with the AWS Management Console basically acting as a front end for the API calls made to the vast multitude of services. I’ve done a fair amount of work with REST APIs over the last 18 months, and it’s always good to fire up Postman (if you don’t know what this is, there is a post here I did about REST APIs, and the use of Postman), and throw a few API calls at a new technology to see how it works.
Now while AWS services are all available via REST APIs; there are a tonne of tools available for both administrators and developers, which abstract away the nitty gritty, we have:
  • AWS CLI – a CLI based tool for Windows/Linux/OSX (available here)
  • AWS Tools for Windows PowerShell – the PowerShell module for consuming AWS services (available here)
  • SDKs (Software Development Kits) for the following (all available here):
    • Android
    • Browsers (basically a JavaScript SDK you can build web services around)
    • iOS
    • Java
    • .NET
    • Node.js
    • PHP
    • Python
    • Ruby
    • GoLang
    • C++
    • AWS IoT
    • AWS Mobile
Combined, these provide a wide variety of ways to use pre-built solutions to speak to AWS based resources, and there should be something that any developer or admin can use to introduce some automation or programability into their work, and I would recommend using one of these if at all possible to abstract away the heavy lifting of working with a very broad and deep API.
I wanted to get stuck in from a REST API side though, which basically means building things from the ground up. This turned out to take a fair amount of time, but I learned a heck of a lot about the authentication and authorisation process for AWS, and how this helps to prevent unauthorised access.
The full authentication process is described in the AWS Documentation available here. There are pages and pages describing the V4 authentication process (the current recommended version), and this gets pretty complicated. I’m going to try and break it down here, showing the bits of code used to create each element; this should hopefully make it a bit clearer.
One post I found really useful on this was by Lukasz Adamczak (@lukasz_adamczak), on how to do the authentication with Curl, which I used as the basis for some of what I did below. I couldn’t find anything where someone was doing this task via the REST API in JavaScript.

Pre-requisites

The following variables need to be set in the script before we start:
// our variables
var access_key = 'ACCESS_KEY_VALUE'
var secret_key = 'SECRET_KEY_VALUE'
var region = 'eu-west-1';
var url = 'my-bucket-name.s3.amazonaws.com';
var myService = 's3';
var myMethod = 'GET';
var myPath = '/';
In addition to this, we have these package dependencies:
// declare our dependencies
var crypto = require('crypto-js');
var https = require('https');
var xml = require('xml2js');
The crypto-js and https modules were built into the version of Node I was using (v6.9.5), but I had to use NPM to install the xml2js module.

Amazon Date Format

I started with getting the date format used by AWS in authentication, this is based on the ISO 8601 format, but has the punctuation, and milliseconds removed from it. I created the below function, and use it to create the two variables shown below:
// get the various date formats needed to form our request
var amzDate = getAmzDate(new Date().toISOString());
var authDate = amzDate.split("T")[0];

// this function converts the generic JS ISO8601 date format to the specific format the AWS API wants
function getAmzDate(dateStr) {
  var chars = [":","-"];
  for (var i=0;i<chars.length;i++) {
    while (dateStr.indexOf(chars[i]) != -1) {
      dateStr = dateStr.replace(chars[i],"");
    }
  }
  dateStr = dateStr.split(".")[0] + "Z";
  return dateStr;
}
We’ll go into this later, but the reason there are two variables for the date (amzDate, authDate) is that in generating the headers for our REST call we will need both formats at different times. One is in the ‘YYYYMMDDTHHmmssZ’ format, and one is in the ‘YYYYMMDD’ format.

Our payload

The example used in this script is to use a blank payload, for which we calculate the SHA256 hash. This is obviously always the same when calculated against a blank string (e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, if you were interested :p), but I included the hashing of this in the script so the logic is there later if we wanted to do different payloads.
// we have an empty payload here because it is a GET request
var payload = '';
// get the SHA256 hash value for our payload
var hashedPayload = crypto.SHA256(payload).toString();
This hashed payload is used a bunch in the final request, including in the ‘x-amz-content-sha256’ HTTP header to validate the expected payload.

Canonical Request

This is where things got a bit confusing for me; we need to work out the payload of our message (in AWS’ special format), and work out what the SHA256 hash of this is. First we need to know the formatting for the canonical request. Ultimately this is a multi-line string, consisting of the following attributes:
HTTPRequestMethod
CanonicalURI
CanonicalQueryString
CanonicalHeaders
SignedHeaders
HexEncode(Hash(RequestPayload))
These attributes are described as:
  • HTTPRequestMethod – the HTTP method being used, could be GET, POST, etc. In our example this will be GET
  • CanonicalURI – the relative URI for the resource we are accessing. In our example we access the root namespace of our bucket, so this is set to “/”
  • CanonicalQueryString – we can build a query for our request, more information on this is available here. In our example we don’t need a query so we will leave this as a blank line
  • CanonicalHeaders – a carriage return separated list of the headers we are using in our request
  • SignedHeaders – a semi-colon separated list of the header keys we are including in our request
  • HexEncode(Hash(RequestPayload)) – the hash value as calculated earlier. As we used the ‘toString()’ method on this, it should already be in hexadecimal
We construct this request with the following code:
// create our canonical request
var canonicalReq =  myMethod + '\n' +
                    myPath + '\n' +
                    '\n' +
                    'host:' + url + '\n' +
                    'x-amz-content-sha256:' + hashedPayload + '\n' +
                    'x-amz-date:' + amzDate + '\n' +
                    '\n' +
                    'host;x-amz-content-sha256;x-amz-date' + '\n' +
                    hashedPayload;
This leaves us with the following as an example:
GET
/

host:my-bucket-name.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20170213T045707Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Note the blank CanonicalQuery line here, as we are not using that functionality, and the blank like after Canonical Headers, these are required for the string to be accepted when we hash it.
So now we can hash this multi-line string:
// hash the canonical request
var canonicalReqHash = crypto.SHA256(canonicalReq).toString();
This becomes another long hashed value.
c75b55ba2d959baf99f2c4976c7a50c7cd79067a726c21024f4a981ae2a90b50

String to Sign

Now, similar to the Canonical Request above, we create a new multi-line string which is used to generate our authentication header. This time it is in the following format:
Algorithm
RequestDate
CredentialScope
HashedCanonicalRequest

These attributes are completed as:

  • Algorithm – for SHA256, which is what we always use with AWS, this should be set to ‘AWS4-HMAC-SHA256’
  • RequestDate – this is the date/time stamp in the ‘YYYYMMDDTHHmmssZ’ format, so we will use our stored ‘amzDate’ variable here
  • CredentialScope – this takes the format ‘///aws4_request’. We have the date stored in this format already as ‘authDate’, so we can use that here, our region name can be found in this table, and the service name here is ‘s3’, further details of other namespaces can be found here
  • HashedCanonicalRequest – this was calculated above

With this information we can form our string like this:

// form our String-to-Sign
var stringToSign =  'AWS4-HMAC-SHA256\n' +
                    amzDate + '\n' +
                    authDate+'/'+region+'/'+myService+'/aws4_request\n'+
                    canonicalReqHash;
This generates a string like this:
AWS4-HMAC-SHA256
20170213T051343Z
20170213/eu-west-1/s3/aws4_request
c75b55ba2d959baf99f2c4976c7a50c7cd79067a726c21024f4a981ae2a90b50

Signing Key

We need a signing key now; this embeds our secret key, along with some other bits in a hash which is used to sign our ‘String to sign’, giving us our final hashed value which we use in the authentication header. Luckily here AWS provide some sample JS code (amongst other languages) for creating this hash:
// this function gets the Signature Key, see AWS documentation for more details
function getSignatureKey(Crypto, key, dateStamp, regionName, serviceName) {
    var kDate = Crypto.HmacSHA256(dateStamp, "AWS4" + key);
    var kRegion = Crypto.HmacSHA256(regionName, kDate);
    var kService = Crypto.HmacSHA256(serviceName, kRegion);
    var kSigning = Crypto.HmacSHA256("aws4_request", kService);
    return kSigning;
}

This can be found here.So into this function we pass our secret access key, the authDate variable we calculated earlier, our region, and the service namespace.

// get our Signing Key
var signingKey = getSignatureKey(crypto, secret_key, authDate, region, myService);

This will again return a long hash value:

9afc364e2eb6ba46f000721975d32bc2042058f80b5a8fd69efe422e7be5090d

Authentication Key

Nearly there now! So we need to take our String to Sign, and our Signing Key, and hash the string to sign with the signing key, to generate another hash which will be used in our request header. To do this we again use the CryptoJS library, with the order of the inputs being our string to hash (stringToSign), and then the key to hash it with (signingKey).
This returns another hash:
31c6a42a9aec00390317f9c714f38efeba2498fa1996cecb9b4c714b39cbc05a90332f38ef

Creating our headers

Right, no more hashing needed now, we have everything we need. So next we construct our Authentication header value:
// Form our authorization header
var authString  = 'AWS4-HMAC-SHA256 ' +
                  'Credential='+
                  access_key+'/'+
                  authDate+'/'+
                  region+'/'+
                  myService+'/aws4_request,'+
                  'SignedHeaders=host;x-amz-content-sha256;x-amz-date,'+
                  'Signature='+authKey;
This is a single line, multi-part string consisting of the following parts:
  • Algorithm – for SHA256, which is what we always use with AWS, this should be set to ‘AWS4-HMAC-SHA256’
  • CredentialScope – as used in our String To Sign above
  • SignedHeaders – a semi-colon separated list of our signed headers
  • Signature – the authentication key we hand crafted above
When we place all these together, we end up with a string like this:
WS4-HMAC-SHA256 Credential=A123EXAMPLEACCESSKEY/20170213/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=31c6a42a9aec00390317f9c714f38efeba2498fa1996cecb9b4c714b39cbc05a90332f38ef
Now we have everything we need to create our headers for our HTTP request:
// throw our headers together
headers = {
  'Authorization' : authString,
  'Host' : url,
  'x-amz-date' : amzDate,
  'x-amz-content-sha256' : hashedPayload
};
Here we use a hash array for simplicity, with our various headers added to the array, to end up with an array like this:
Key
Value
Authorization
WS4-HMAC-SHA256 Credential=A123EXAMPLEACCESSKEY/20170213/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=31c6a42a9aec00390317f9c714f38efeba2498fa1996cecb9b4c714b39cbc05a90332f38ef
Host
x-amz-date
20170213T051343Z
x-amz-content-sha256
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
These are now ready to use in our request.

Our request

Now we can send our request. I split this into a function to do the REST call:
// the REST API call using the Node.js 'https' module
function performRequest(endpoint, headers, data, success) {

  var dataString = data;

  var options = {
    host: endpoint,
    port: 443,
    path: '/',
    method: 'GET',
    headers: headers
  };

  var req = https.request(options, function(res) {
    res.setEncoding('utf-8');

    var responseString = '';

    res.on('data', function(data) {
      responseString += data;
    });

    res.on('end', function() {
      //console.log(responseString);
      success(responseString);
    });
  });

  req.write(dataString);
  req.end();
}
And the call to this function, which also processes the results, in the body:
// call our function
performRequest(url, headers, payload, function(response) {
  // parse the response from our function and write the results to the console
  xml.parseString(response, function (err, result) {
    console.log('\n=== \n'+'Bucket is named: ' + result['ListBucketResult']['Name']);
    console.log('=== \n'+'Contents: ');
    for (i=0;i<result['ListBucketResult']['Contents'].length;i++) {
      console.log(
        '=== \n'+
        'Name: '          + result['ListBucketResult']['Contents'][i]['Key'][0]           + '\n' +
        'Last modified: ' + result['ListBucketResult']['Contents'][i]['LastModified'][0]  + '\n' +
        'Size (bytes): '  + result['ListBucketResult']['Contents'][i]['Size'][0]          + '\n' +
        'Storage Class: ' + result['ListBucketResult']['Contents'][i]['StorageClass'][0]
      );
    };
    console.log('=== \n');
  });
});
This essentially passes our headers with the payload to the URL we specified in our variables, and processes the resulting XML into some useful output.
This is what this returns as output:
Tims-MacBook-Pro:GetS3BucketContent tim$ node get_bucket_content.js

===
Bucket is named: virtualbrakeman
===
Contents:
===
Name: file_a_foo.txt
Last modified: 2017-02-05T11:19:36.000Z
Size (bytes): 10
Storage Class: STANDARD
===
Name: file_b_foo.txt
Last modified: 2017-02-05T11:19:36.000Z
Size (bytes): 10
Storage Class: STANDARD
===
Name: file_c_foo.txt
Last modified: 2017-02-05T11:19:36.000Z
Size (bytes): 10
Storage Class: STANDARD
===
Name: file_d_foo.txt
Last modified: 2017-02-05T11:19:36.000Z
Size (bytes): 10
Storage Class: STANDARD
===
Name: foobar
Last modified: 2017-02-04T22:01:38.000Z
Size (bytes): 0
Storage Class: STANDARD
===

Tims-MacBook-Pro:GetS3BucketContent tim$

Conclusion

This is a fairly trivial example of data to return, but the real point behind this was building the authentication code, which proved to be very laborious. Given the wide (and growing) variety of SDKs available, it seems overly complex to try and construct these requests in this way every time. I have played with the Python and JavaScript SDKs and both take this roughly 150 line script, and achieve the same result in around 20 lines.
Regardless, this was a good learning exercise for me, and it may come in useful for people trying to interact with the AWS API in ways which are not covered by the SDKs, or via other languages where an SDK is not available.
The final script is shown below, and is also available on my GitHub library here: https://github.com/railroadmanuk/awsrestauthentication
// declare our dependencies
var crypto = require('crypto-js');
var https = require('https');
var xml = require('xml2js');

main();

// split the code into a main function
function main() {
  // this serviceList is unused right now, but may be used in future
  const serviceList = [
    'dynamodb',
    'ec2',
    'sqs',
    'sns',
    's3'
  ];

  // our variables
  var access_key = 'ACCESS_KEY_VALUE';
  var secret_key = 'SECRET_KEY_VALUE';
  var region = 'eu-west-1';
  var url = 'my-bucket-name.s3.amazonaws.com';
  var myService = 's3';
  var myMethod = 'GET';
  var myPath = '/';

  // get the various date formats needed to form our request
  var amzDate = getAmzDate(new Date().toISOString());
  var authDate = amzDate.split("T")[0];

  // we have an empty payload here because it is a GET request
  var payload = '';
  // get the SHA256 hash value for our payload
  var hashedPayload = crypto.SHA256(payload).toString();

  // create our canonical request
  var canonicalReq =  myMethod + '\n' +
                      myPath + '\n' +
                      '\n' +
                      'host:' + url + '\n' +
                      'x-amz-content-sha256:' + hashedPayload + '\n' +
                      'x-amz-date:' + amzDate + '\n' +
                      '\n' +
                      'host;x-amz-content-sha256;x-amz-date' + '\n' +
                      hashedPayload;

  // hash the canonical request
  var canonicalReqHash = crypto.SHA256(canonicalReq).toString();

  // form our String-to-Sign
  var stringToSign =  'AWS4-HMAC-SHA256\n' +
                      amzDate + '\n' +
                      authDate+'/'+region+'/'+myService+'/aws4_request\n'+
                      canonicalReqHash;

  // get our Signing Key
  var signingKey = getSignatureKey(crypto, secret_key, authDate, region, myService);

  // Sign our String-to-Sign with our Signing Key
  var authKey = crypto.HmacSHA256(stringToSign, signingKey);

  // Form our authorization header
  var authString  = 'AWS4-HMAC-SHA256 ' +
                    'Credential='+
                    access_key+'/'+
                    authDate+'/'+
                    region+'/'+
                    myService+'/aws4_request,'+
                    'SignedHeaders=host;x-amz-content-sha256;x-amz-date,'+
                    'Signature='+authKey;

  // throw our headers together
  headers = {
    'Authorization' : authString,
    'Host' : url,
    'x-amz-date' : amzDate,
    'x-amz-content-sha256' : hashedPayload
  };

  // call our function
  performRequest(url, headers, payload, function(response) {
    // parse the response from our function and write the results to the console
    xml.parseString(response, function (err, result) {
      console.log('\n=== \n'+'Bucket is named: ' + result['ListBucketResult']['Name']);
      console.log('=== \n'+'Contents: ');
      for (i=0;i<result['ListBucketResult']['Contents'].length;i++) {
        console.log(
          '=== \n'+
          'Name: '          + result['ListBucketResult']['Contents'][i]['Key'][0]           + '\n' +
          'Last modified: ' + result['ListBucketResult']['Contents'][i]['LastModified'][0]  + '\n' +
          'Size (bytes): '  + result['ListBucketResult']['Contents'][i]['Size'][0]          + '\n' +
          'Storage Class: ' + result['ListBucketResult']['Contents'][i]['StorageClass'][0]
        );
      };
      console.log('=== \n');
    });
  });
};

// this function gets the Signature Key, see AWS documentation for more details, this was taken from the AWS samples site
function getSignatureKey(Crypto, key, dateStamp, regionName, serviceName) {
    var kDate = Crypto.HmacSHA256(dateStamp, "AWS4" + key);
    var kRegion = Crypto.HmacSHA256(regionName, kDate);
    var kService = Crypto.HmacSHA256(serviceName, kRegion);
    var kSigning = Crypto.HmacSHA256("aws4_request", kService);
    return kSigning;
}

// this function converts the generic JS ISO8601 date format to the specific format the AWS API wants
function getAmzDate(dateStr) {
  var chars = [":","-"];
  for (var i=0;i<chars.length;i++) {
    while (dateStr.indexOf(chars[i]) != -1) {
      dateStr = dateStr.replace(chars[i],"");
    }
  }
  dateStr = dateStr.split(".")[0] + "Z";
  return dateStr;
}

// the REST API call using the Node.js 'https' module
function performRequest(endpoint, headers, data, success) {

  var dataString = data;

  var options = {
    host: endpoint,
    port: 443,
    path: '/',
    method: 'GET',
    headers: headers
  };

  var req = https.request(options, function(res) {
    res.setEncoding('utf-8');

    var responseString = '';

    res.on('data', function(data) {
      responseString += data;
    });

    res.on('end', function() {
      //console.log(responseString);
      success(responseString);
    });
  });

  req.write(dataString);
  req.end();
}