In the last post, John Martinez wrote about how Autoscaling can help an application deployed on AWS survive an attack. While that is great, there is an actual fiscal cost to that type of mitigation.
Granted, the cost is most likely worth the mitigation, but what else could be done before making that investment? In this post we will review a less costly way to mitigate an attack before we scale up to absorb it.
Basically, we want to show you how to block traffic before it triggers a need to scale.
AWS provides the tools necessary to help control what traffic is allowed, and in this blog, we are going talk about Security Groups and their relatives, the Network Access Control List (NACL).
While these groups are not traditional firewalls, they are very effective at controlling network and port traffic. If you do anything on AWS other than store data on S3 or use Route53 for DNS, you will are more than likely to run into thea need to configure Security Groups.
Much like a basic firewall, the principle purpose of a Security Group is to allow only the traffic you want, in the direction you want. Rather than be negative, it may help to focus on what these groups allow.
An old network sage once told me, “Block everything, only allow in what you need,” and in many cases he was spot on. If you try to figure out what to block all the time, that may be your full time job and a pretty negative one at that.
It is much brighter to focus on what to allow, assuming that if you don’t allow it, it is blocked. Yes, the glass is half full.
So, do you need to allow all traffic from 0.0.0.0/0? In AWS Network terms, that means everyone, every machine, everywhere has the ability to make a connection to your AWS resources.
Everyone and everything on the Internet can establish a connection to your resources! Oh, and by default, this is the access you are granting all of your systems to make outbound connections to, everyone, everything, everywhere.
Another way to look at is is as if each zero was a wild card, so instead of 0.0.0.0/0 see it as *.*.*.*/*. In some cases, this may be exactly what you want. But in many, it may be exactly what you don’t.
Let’s break this down into two directions.
For the sake of this article, inbound connections are those allowed into your application. Customers would be an example of inbound connections to a web server, but you may have other applications that need to make inbound connections like the web server to your database.
The question here is, who are you inviting into your house when they ring the doorbell? Everyone? If a delivery person arrives with a package at your front door for someone who is in the other room, do they just open the door and invite themselves in? Insert 0.0.0.0/0 and you have basically opened the door to the world.
For your basic web application, you may actually want to allow everyone access. Allowing this traffic could be detrimental to your business. In this case, you should consider allowing access only to the ports your application will respond on.
The most common would be port 80 for unencrypted web traffic and port 443 for the encrypted traffic. This would cut the allow rule down to two lines. This one change has effectively reduced the threat surface from thousands of ports down to two.
Not bad. In this case, you may also want to question if you want to leave port 80 open. In today’s security landscape, more and more applications are trending toward always encrypting traffic, so only allowing in port 443 is completely acceptable in these cases.
Now that your customers can gain access to just your application, you want to be able to administer the web server, right? Well, some would say absolutely not, you must deploy infrastructure as code and never actually admin a box manually.
Ok, for those that can, I completely agree. There is no need for administrative access to the instance. However, for many, at some point in time, they will need to do something on the instance and need remote access. This is where you should do two things.
- Only allow the access from the origin IP and port where you will admin your instance from. Be very explicit here. http://checkip.amazonaws.com/ is one way to determine your origin IP and then only allow for the specific port. For example, by default, SSH would be 22 and RDP would be 3389. Again, we discourage the defaults, but this would be an allow rule something like 22.214.171.124/32 to that one port.
- The other recommended security best practice is to only turn this on when needed and remove it when not. Yes, this adds an additional step and removes the convenience, but if you only allow access when access is needed, you have very effectively not only reduced the attack surface with the explicit rule above, you have limited the time. This can all be scripted and if you are going through the steps to admin an instance, you should factor in turning on and off remote access to “only when needed.”
So, where should 0.0.0.0/0 be configured?
Yes, that was a trick question. A good answer would be, “Do not allow 0.0.0.0/0 unless you mean it!” However, my question was actually about where in AWS you can configure access. There are a total of four places.
Security Groups and Network Access Control Lists (NACL) both have inbound and outbound rules. They are like defensive layers in getting to your application.
As you can see, Security Groups are closest to your application, while NACL are closest to the inbound traffic. So, from a defense in depth perspective, you want to limit the broadest amount of traffic, the furthest away from your application. With each layer in, you become more and more granular in access.
This is actually not different than many security implementations you may encounter on a daily basis.
The next time you walk into a building, notice how there are broad security boundaries on the exterior and as you proceed through, security becomes more and more granular with some rooms you can freely access, while others are more controlled, and yet others you are not allowed entry. Inbound access into your application is similar.
In the context above, where should 0.0.0.0/0 be configured? If you must allow world access to your application, you will need to configure it both on the NACL and the Security Group. However, you can limit the ports the world has access to with the Security Group.
If you only need to allow specific network access to your application, you can limit it on the NACL, thus preventing anything you did not define making it into your security group.
Think of a port scanner – If denied at the NACL, the scanner cannot discover what ports you have open, since those are configured on the Security Group. The NACL can also be thought of as the perimeter boundary, or your first checkpoint.
While we are talking about NACL, this is the one place where you can configure an explicit deny rule. Security Groups are very focused on what you allow, denying everything that is not allowed.
However, NACL can be configured to deny traffic, and contrary to the first paragraph, that is a good thing. Why?
Well, first, if you discover you are being attacked and the origin IPs are very specific, you can quickly and easily block them at the NACL. This is for a DoS where the origin IPs are known.
For a DDoS, this is not so easy, but maybe they are not evenly distributed and you could put up some deny rules to help. In any event, this is where having familiarity with your logs and partnering with AWS Support to help you identify the attack can help.
So much of the discussion thus far has been about inbound rules, but what about outbound? Configuring outbound allow rules can be done both with Security Groups and NACLs.
Deciding how to configure them is no different than inbound. The exception here is that you own the application and you know what it needs to do.
Both Security Groups and NACLs allow all outbound traffic by default. This means that your application servers will have no problem browsing the web or making any connections needed out of the box.
But the question you should ask yourself is, should they? Again, there are no limits of what your applications can do, they can browse the internet, they can send email, they can, well, they can do anything.
In many cases, as a steps to prevent the applications from doing things you may not want them to, you can limit the outbound communications.
Remember that once you configure a Security Group or NACL, anything not configured will be denied. Another aspect to carefully consider here is that Security Groups are stateful while NACLs are not.
This means that is you configure outbound rules on your security groups, it will not impact inbound sessions, but if you configure outbound rules on a NACL, you will need to allow the outbound traffic back to the origin IP to establish a session.
As a best practice, it may be easier to think of using Security Groups for ports to access, while NACLs are limited to Networks. In this example, what ports does your application need to make connections to (and why?)
Configure this per application group on the Security Group, while leaving NACLs open for web applications and implement explicit rules for applications.
For a two tiered web application, with web servers in one group and database in another, security could be implemented by inbound NACL that allows connections to the web servers from the world, while the databased only had inbound connections allowed from the web servers.
For inbound Security Groups, the web servers would only allow port 443 connections, while the database would only allow inbound 3306 (for MySQL) from the web server Security Group (yes, you can use the security groups themselves so you don’t need to keep track of the instance IPs!)
For outbound connections, you could remove both the web server and the database Security Groups outbound rule to prevent them from initiating connections on the internet, while for the web servers allow all outbound traffic to ensure sessions are able to be established and the database would only allow outbound connections to the web server private subnet IP range.
More examples can be found on the AWS web site for Security in your VPC.
So, the question is now, when should 0.0.0.0/0 be configured?
A quick recap of our past AWS Best Practice posts:
- Disable Root API Access Key and Secret Key
- Enable MFA Tokens Everywhere
- Reduce Number of IAM Users with Admin Rights
- Use Roles for EC2
- Least Privilege: Limit what IAM Entities Can Do with Strong Policies
- Rotate all the Keys Regularly
- Use IAM Roles with STS AssumeRole Where Possible
- Use AutoScaling to Dampen DDoS Effects
- Do Not Allow 0.0.0.0/0 Unless You Mean It
- Watch World-Readable and Listable S3 Bucket Policies