Strengthen Your AWS Security by Protecting App Credentials and Automating EC2 and IAM Key Rotation

One of the biggest headaches faced by developers and DevOps professionals is the problem of keeping the credentials used in application code secure. It’s just a fact of life. We have code that needs to access network resources like servers and databases, and we have to store these credentials somewhere. Even in the best of circumstances this is a difficult problem to solve, but the messy realities of daily life further compound the issue. Looming deadlines, sprawling technology and employee turnover all conspire against us when we try to keep the build pipeline secure. The result is “credential detritus”: passwords and security keys littered across developer workstations, source control repos, build servers and staging environments.

Use EC2 Instance Profiles

A good strategy for minimizing credential detritus is to reduce the number of credentials that need to be managed in the first place. One effective way to do this in AWS is by using EC2 Instance Profiles. An Instance Profile is an IAM Role that is assigned to an EC2 instance when it’s created. Once this is in-place, any code running on the EC2 instance that makes CLI or SDK calls to AWS resources will be made within the security context of the Instance Profile. This is extremely handy because it means that you don’t need to worry about getting credentials onto the instance when it’s created, and you don’t need to manage them on an ongoing basis—AWS automatically rotates the keys for you. Instead, you can spend your time fine tuning the security policy for the IAM Role to ensure that it has the least amount of privileges to get its job done.

Get Credentials Out of Source Control

EC2 Instance Profiles are a big help, but they won’t completely eliminate the need to manage credentials in your application. There are plenty of non-AWS resources that your code requires access to, and EC2 Instance Profiles won’t help you with those. For these credentials, we need another approach. This starts by making sure that credentials are NEVER stored in source control. A good test I’ve heard is this: if you were forced to push your source code to a public repository on GitHub today, then you should be able to sleep well tonight knowing that no secret credentials would be revealed. How well would you sleep if this happened to you?

The main problem is that source code repositories typically have a wide audience, including people who shouldn’t have access to security credentials. And once you check in those credentials, they’re pretty much up there forever. Moreover, if you use a distributed SCM like Git, then those credentials are stored along with your source code on all of your developers’ machines, further increasing your exposure. The more breadcrumbs you leave lying around, the more likely rats will end up infesting your home. This appears to be what happened in the Ashley Madson hack that took place earlier this year.  Hard-coded credentials stored in source control were implicated as a key factor in the attack. Apparently, their source code was littered with the stuff. Not good.

So how do you avoid becoming the next Ashley Madison? First, separate configuration from code. Hardcoding any configuration information in source code is a bad idea. Second, after configuration information has been eradicated from source code, make sure your credentials are removed from source control. One of the best ways to do this is by storing credentials in a separate property file that is explicitly excluded from source control (e.g. by adding the property file name to the .gitignore file). These property files can then be securely distributed to the developers and applications that require them, e.g. by distributing them via an encrypted S3 bucket secured with a strong IAM security policy. Just make sure that whatever you do, don’t send them via email, since email is not encrypted.

Also, be careful when passing credentials to programs via command line arguments, as these credentials will appear in the process list. This can be a problem when using Docker, for example. If you pass in security credentials by calling docker run with the –e environment variable option, this will be viewable by anyone on the machine who can see the process list. Instead, use an environment variable file using the --env-file option to pass in credential information—this leaves the variables themselves out of the process list.

Automate AWS Key Rotation

After we’ve reduced the number of credentials we’re managing manually by using Instance Profiles and have removed the other credentials from source control, then we need a way to keep the remaining credentials secure on an on-going basis. One effective way to do this is by rotating access keys frequently. This is a well-established security best practice recommended by Amazon and others. Frequent key rotation ensures that if a key is compromised, the length of time the key is valid is limited. It also helps protect us when developers leave the organization because it means that former employees are less likely to have active credentials in their possession. A more subtle benefit—but one that is arguably more important—is that when you rotate keys frequently, it forces you to identify everywhere credentials are being used in your code and who has access to them. If you don’t know this information, then it’s impossible to ensure your credentials are safe.

But just because we know that we should rotate our keys regularly, it doesn’t mean that we do. Key rotation is like going to the dentist: it’s time consuming, unpleasant and potentially painful. Key rotation takes time to perform and can be a risky operation if we’re not sure how all the keys are being used in the code. So, like many non-emergency, high-risk maintenance tasks, it tends to fall to the bottom of the to-do list.

The best way to overcome this is by automating the key rotation process itself. By automating key rotation, you’ll do it more frequently, which means it is less likely to cause a major problem, which makes it less risky, which means you’ll do it more frequently. Automation could mean manually running a script once a month, or scheduling a script to run as a cron job on a weekly basis. When key rotation is automated, why not do it weekly, or even daily?

Having the ability to rotate keys automatically also gives you the super power to change keys immediately in the event a key is compromised or a disgruntled developer leaves the company. The day may come when this ability makes you look like the smartest guy in the room.

But how do we get started? AWS has a bewildering array of credential types which can make the prospect of automating key rotation for AWS resources seem like a daunting task. Fortunately, the bulk of use cases can be covered by rotating IAM user access keys and EC2 SSH keys, which are both addressed below.

Rotating IAM User Access Keys

Rotating IAM keys is a bit more straightforward than rotating EC2 SSH keys so we’ll start here. An IAM access key is a set of credentials, comprised of a key ID and a secret key, that is assigned to an IAM user (Roles and Groups, in contrast, don’t have their own credentials). Applications should use access keys assigned to an application-specific, non-Administrator user. Applications should not use access keys for the regular accounts used to login into the AWS console, and they should ABSOLUTELY NOT use the key for the AWS root account (the root account should never even have an access key in the first place). By using a separate, application-specific user, you can lock down its IAM policy to give it the least amount of privileges required to do its work.

AWS has designed IAM user access keys to be rotated easily. Each user can have up to two access keys, and these keys can be activated or deactivated with a single API call. The general pattern to rotate keys is as follows:

1. Create a new (second) access key for the user.
2. Test your application code with the new key.
3. If nothing breaks, de-activate the old key.
4. Re-test.
5. If nothing breaks, delete the old key.

Rotating EC2 SSH Key Pairs

Rotating EC2 SSH key pairs is a bit trickier because you have to deal with OS-specific differences. Moreover, there is a great deal of confusion about how EC2 key pairs work which compounds the issue.  So, before we discuss how to implement automated EC2 key rotation, let’s walk through the EC2 public-private key pair architecture.

Not surprisingly, Amazon has a lot of control over the management of AWS-specific resources like IAM users and S3 buckets that it manages from cradle to grave. For these types of resources, Amazon can provide lots of management support to make administration easier. EC2 instances, however, do not fall into this category. Amazon may be responsible for the VM infrastructure that underlies EC2 instances, but you are on your own when it comes to managing the server inside. Things like the management of the root password is a server administration responsibility, so it’s your problem, not Amazon’s.

The one circumstance in which Amazon “intrudes” on server administration is when it creates the EC2 instance in the first place. It does this out of necessity: AWS has to create a root account on the server so you can access it after instantiation. This is why the EC2 key pair exists. But that’s as far as Amazon is willing to go in the server administration business—it doesn’t even keep a copy of the private key after you create the key pair.

The confusion arises because it seems as though the EC2 key pair is a permanent fixture linked to the root account of your server. While it’s true that the key pair is permanent (it can’t be altered after the instance has been created), it is not eternally linked to your root account. In fact, you are free to do whatever you want with your root credentials, including adding other keys to the root account and removing the original key used during instance creation—AWS will be none the wiser. After all, server administration is your domain, not Amazon’s.

Despite this fact, there are some crazy suggestions out there about how to change an EC2 key pair. This advice usually entails creating new instances, detatching volumes, attaching volumes and performing various other administrative machinations. However, none of this is actually necessary if you take EC2 key pairs for what they really are: a one-time credential used to bootstrap new instances. Once you understand this concept, you gain the freedom to ignore the AWS key pair altogether after EC2 inception and return to the old fashion business of server administration. This is why I name my AWS EC2 key pairs something like “OneTimeEC2InstantiationKeyPair” to make it clear that the EC2 key pair serves a specific, one-time purpose.

Based on this approach, we can rotate EC2 SSH keys by simply updating the ~./ssh/authorized_keys file for the root account to include the new key. (Note: when I use the term “root” account here, I really mean the main admin account. Most Linux images have a primary non-root admin account that can sudo su into the actual root account as needed. E.g. the “ubuntu” user for the Ubuntu AMI.) The authorized_keys file contains the public half of each of the SSH key pairs used to authenticate the user. Changing this file will have no impact whatsoever on the Key Pair field of the EC2 instance. However, it will allow you to modify the credentials for the root user on the server.

The one downside with to approach is that after you change the root user SSH key to something other than the original AWS key pair, then you are on your own for managing the keys. This means that you need to keep track of your keys and know which keys grant access to which EC2 instances. Fortunately, a simple trick can make this administrative chore much easier. Just add a tag that contains the name of the SSH key for the root user to each EC2 instance. This allows you to lookup the current key for each instance so you can locate the right private key file to access it.

With a better understanding of how EC2 SSH keys work, we’re now ready to define a process for EC2 key rotation:

1. Create a new key pair using ssh-keygen (not using AWS’s key pair functionality).
2. Add the public key portion of the SSH key pair to the ~./ssh/authorized_keys file for the root/admin account on the EC2 instance.
3. Test by logging into the instance with the new key.
4. If everything works, remove the original key.
5. Re-test to confirm it still works.
6. Create/update a tag called “EC2KeyName” that contains the name of the new SSH key.

This process works fine for most Linux distros like Ubuntu and Amazon Linux (which is CentOS-based). But if you’re a Docker fan like me, and you’re running CoreOS, the process is a little different. The difference lies in the way CoreOS uses cloud-config. Cloud-config is CoreOS’s version of cloud-init. Cloud-init is the de facto standard for initializing VMs created in a cloud environment. Cloud-init is used to run bootstrap scripts and pass configuration information into a newly created instance.

Typically, cloud-init is only run on the first boot of the server. However, CoreOS’s implementation of cloud-init—cloud-config—runs on every reboot. In and of itself, this is not a problem. The challenge arises because cloud-config re-writes the ~/.ssh/authorized_keys file every time it is run (i.e. on every reboot) using the AWS key pair stored in the EC2 meta-data. This means that after you rotate the SSH key, CoreOS will revert back to the AWS key pair after reboot. Since you can’t change the AWS key pair after you’ve created the EC2 instance, you are essentially blocked from rotating EC2 keys on CoreOS if you’re using cloud-config.

Fortunately, the smart folks at CoreOS are already hard at work developing an alternative/replacement for cloud-config. This new provisioning/initialization system—called Ignition—is designed to offer sys admins and DevOps pros more flexibility during server initialization by engaging earlier in the boot sequence. More importantly for us, Ignition only runs once, during the first boot, which unlocks the ability for us to rotate SSH keys.

So, assuming you’re using Ignition instead of cloud-config, the process to rotate keys on CoreOS is substantially the similar to the one above, except that step #4 is as follows:

4. If everything works, remove the original key by replacing the SSH key in the ~/.ssh/authorized_keys.d/coreos-ignition file and run update-ssh-key to enable the change.

Summary

As cyber security becomes an increasing area of concern for businesses and technology professionals, it’s imperative that we integrate security best practices into our development workflow. Locking down application credentials throughout the development process is an important first step. We’ve outlined three ways here to bolster the security of the build pipeline in an AWS environment:

1. Use EC2 Instance Profiles to reduce the number of credentials that need to be managed manually.
2. Eliminate all hard-coded configuration from code, and remove all credentials from source control.
3. Automate the rotation of the access keys, and perform key rotation frequently. Although we focused on AWS-specific credentials here, the same general pattern can and should be applied to other, non-AWS technologies used by the application.