Invoke AWS Services Cross Account from Lambda (with AWS CDK and AWS SDK v3)

Recently we started a big refactor of an application that was running on another team’s AWS Account.
Our plan is to move to a serverless architecture using our own account to gradually dismiss the old one.

In order to achieve that, we deployed a Lambda on one account which was in charge of forwarding every request to other AWS Services in another account.
Some of the services we invoke cross-account are SSM ( to retrieve credentials and other configuration stuff), SQS ( to push POST and PATCH requests to a queue to handle batch processing and retries, DynamoDB, to read data in case of GET requests.

Create a Cross Account Role

To access resources CrossAccount we need to a Role that one account can Assume to “impersonate” the other account, and also create a Policy with the right permissions about the service and actions you want to trigger.

In the new account / CDK Stack,

Here the only interesting part is that we need to add to our cdk.context.json the reference to otherAccountID.

Add Policies

Then assign Policies for every service we will be using:

The first policy allows the role to getParameters from SSM.
The resource ARN must be specified.
If SSM parameter was created via CDK it is possible to reference it like this: mySSM.parameterArn but hardcoding the reference again in the cdk.context.json and using it like this arn:aws:ssm:${this.region}:${this.account}:parameter${ssmKey} is also a possibility.

Similarly, here we are assigning Permissions to retrieve items from DynamoDB.

While in the following we grant rights to push items to a queue.

Having this granular policies in your cross account role are very important to expose access only to the services you plan to use and not have security issues.
It might be a bit harder at the beginning when you are not sure if you are doing everything right, or confusing when you in few weeks or months add another service ( or like in our example, try to Put an Item in Dynamo) and it does not work until you update the policies but it is better to stick to the The Principle of Least Privilege.

a subject should be given only those privileges needed for it to complete its task. If a subject does not need an access right, the subject should not have that right.

Now that we have our role and policies set up in our main account, let’s go to our Lambda in the other account and assume the CrossAccount Role so that we can init our AWS Services with the right Credentials.

First implementation (kinda working, until..)

In our handler we were first checking if we had stored the credentials and if not we invoke STS AssumeRole.
Then we passed the loaded credentials to our AWS Client.

After deploying and testing everything worked, until we realised that when the Lambda Container stays up for a long time ( due to frequent requests), the Credentials expire.

Just check for Expire?

First quick fix was checking in the handler if the Credentials were expired and eventually reloading it.

But it didn’t feel right and especially with multiple services and AWS clients we wanted to have some better code in place in order to refresh the credentials automatically.

Extend Credentials class to handle refresh

By checking the docs we found some interesting methods:

which could be used by subclasses overriding them.

Therefore we changed our method into a Class:

The key part here is that in the assumeRole we don’t return directly the Parent Class, but an object which contains also ExpiredTime information

Also, instead of manually checking if Credentials exist and are expired and eventually reload them, we could safely every time invoke await credentials.getPromise() which does that for us.

Not yet optimal

This solution was working fine, but there was something that was really bothering me.

We were storing the Credentials and refresh them, but we were, at every run of the lambda handler, instantiate a new AWS Client in order for it to receive valid credentials.

I can’t tell how much this is a problem, but it is a general recommendation to initialize SDK clients outside of the function handler. Subsequent invocations processed by the same instance of your function can reuse these resources. This saves cost by reducing function run time.

Therefore I dug in the docs even more, I checked the source code of AWS SDK Client in my IDE and and also thanks to Typescript, I noticed something interesting.

Every Client accepts a configuration object which contains the Credentials to be used. Either the Credentials, or a Provider of Credentials!

What does that Provider do?

A function that, when invoked, returns a promise that will be fulfilled with a value of type T.
Example:A function that reads credentials from shared SDK configuration files, assuming roles and collecting MFA tokens as necessary.

How to use it?
By looking up the CredentialsProvider in the new AWS SDK V3 docs I immediately found what I was looking for.fromTemporaryCredentials
that returns a CredentialProvider that retrieves temporary credentials from STS AssumeRole API.

I throw away most of the code that was written previously to extend the Credential Class and handle the refresh method and I simply passed the ARN of my CrossAccountRole to the fromTemporaryCredentials to get a Provider<Credentials> which will take care of the refresh automatically.

By simply passing that CredentialsProvider in the ServiceConfig, I can initiate my services outside the Lambda handler and reuse them, but be sure that when the Credentials are expired, they will be refreshed, without me even knowing it!

It took us a bit to get to this final version (unless someone has some hints or different approaches — feel free to leave a comment! ) but now it shines in its simplicity.

Of course when developing a feature it is important to get shit done and deliver, and as this story shows, we did go live with the first working implementation but then we iterate on it until we found the proper way of doing it.

Never stop at the first quick solution that works, try to improve your code, and most importantly read the docs to really understand what you are using.

Hope it helps!

Photo by Marc-Olivier Jodoin on Unsplash

Originally published at https://dev.to on April 27, 2022.

--

--

--

Sport addicted, productivity obsessed, avid learner, travel enthusiast, expat, 2 kids. Technical Lead (NodeJs Serverless)

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Davide de Paolis

Davide de Paolis

Sport addicted, productivity obsessed, avid learner, travel enthusiast, expat, 2 kids. Technical Lead (NodeJs Serverless)

More from Medium

Deploy Cognito Triggers Using Serverless Framework

AWS Serverless : A New Way to Manage Your WebApps

Mail Server Using Serverless Framework On AWS [Lambda + SES +IAM]

Unique indexes in DynamoDB