AWS Certified SysOps Administrator - Associate

Welcome to IT Pro TV. I'm your host Don Pezet. [CROSSTALK] You're watching IT Pro TV. Greetings everyone, and welcome back to another exciting episode of IT Pro TV. I'm your host Justin Dennison, and boy am I excited because we're getting started with the AWS Certified Sys Ops Administrator Associate. And I'm a little winded after saying that. But that's all right. In this particular episode, we're gonna be talking about AWS security. And here to lead us along is the one and only Mr. Don Pezet. How are you doing today, Don? I am doing great, Justin. And when we kicked off this series, I had my choice of any number of topics to lead in with, but I thought it would be a really good idea for us to lead in with security. Because if you're opening up a brand new AWS account there's a lot of mistakes you can make early on that you will certainly regret down the road. And security is probably number one amongst them. So what we're going to do is we're going to spend the next handful of episodes talking about some of the security features that are found inside AWS. How you can protect your account, how you can protect your resources, and more importantly how to understand what Amazon protects for you and what you are responsible for protecting yourself? Because many, many people come into cloud services with, [LAUGH] a very big misunderstanding of what's being taken taken care of for them. So we're gonna lay all that out, so we got a good understanding of what's available, and get a chance to see all that right across these next couple of episodes. Well, Don, I'm glad we're starting here because, as someone who's set up maybe some databases, we're the first set of decisions that you make are actually security decisions and I'll be very candid here, right? I've set up a monitor database that does not have encrypted connections, [LAUGH]. [LAUGH] Doesn't have a username and password, I'm like looks good enough for me. But these decisions are easily made up front, and actually pay dividends on the backend if you know about them, right? Absolutely, because if you design something secure from day one, then it's already done, right? But if on day 200 you've got customers in there you've got tons of data, and then you say, [SOUND] I should really secure that. Now you are looking at migrations or down time, updating clients, it becomes a nightmare. Would have been nice if you would have done it in the beginning. How about a worse scenario, right? How about you didn't secure things in the beginning and then you get a data breach, that's awesome, right? So now you get to do a nice little PR announcement all of your customers, saying whoops, data got compromised my bad, and you lose customers potentially damaging your business. So those are all bad things that we can really avoid if we just have a fair understanding of what exactly is Amazon taking care of for us? Now, Amazon in their defense they've actually done a really good job of documenting a lot of this, but you have to look for it. If you don't look for it, what most people do is they go sign up for a demo account or their initial account, right? Maybe you're already got an Amazon account cuz you bought batteries or whatever. And then you go and activate an AWS account, same account, it's already activated, it's done. Then it drops you into, well, the console we've been looking at. If you look at mine, it just drops us in, and here's all these services get to work, right? And it doesn't really call out the fact that way down here at the bottom of my screen is security, identity, and compliance. There's this whole section dedicated to security, way down there, right? I remember when I first got into AWL, where was the first place I went? EC2, I just jumped in and started spitting up instances, because that's what I wanted to do. But today is far more complex, Amazon actually does have a really good web page. Now I'll put the links for this in the show page but if you wanna check it out it's aws.amazon.com/security. And the purpose of this page is to basically educate you on what they take care of, and what they don't. Because you'll find there's a lot of stuff that they do take care of, right? When you sign up for AWS, they're offering several different services that cross the lines of the different service types. So you probably heard platform as a service. Infrastructure as a service, software as a service, all right? Software as a service, that's a big favorite for people. An example of software as a service would be like Gmail, if you sign up for Gmail, Justin do you use Gmail? I do use Gmail. It's my personal email, it's good to go. All right, so how do you back up your mailbox with Gmail? Well, we are talking about security, I don't actually back up my inbox. You don't? Because Google's taking care of that for you, right? How do you upgrade your Gmail client? Well, I cross my fingers when I login and I hope that it's updated [LAUGH]. [LAUGH] Right, so that's the whole point with software as a service, is that it's all taken for of for you. You don't have to back up your Gmail box. You don't have to upgrade a client, it's all just done. You really don't have to worry about security besides your own user credentials, which we're gonna talk about here in a little bit. But when we deploy solutions in AWS, they are rarely like that. In fact, I can think of really only one or two examples like that. The majority of the services, we control certain aspects, and that's where we get into things like infrastructure as a service, and platform as a service. For example with an EC2 instance, if I deploy an EC2 instance, and I pick a Linux AMI, right? Amazon will try and make sure it's up to date as far as that initial bit that rolls out, right? So they mentioned somewhere here on the page that it's up to date, right? But at the end of the day, it's up to date when you deployed it. Once it's deployed, now that responsibility drops into your hands. You're responsible for keeping it up to date. Amazon won't update those EC2 instances for you because it might break your software. You have to test the updates and make sure that you're ready. If you don't understand that, you end up in a situation that a lot of people bump into, which is where they are deploying instances like crazy, they are bringing up new services and then they never update them. And year later they get a bleach, the other day they get broken up and wide, and it's because they have a service that haven't been updated over a year. And they might have a highly scalable highly luxuriant infrastructure, but without updates being applied, it's like leaving the front door open to your house, right? You become vulnerable. So let's talk about what Amazon actually does provide and does an amazing job with. The main thing is infrastructure. Amazon has probably the most phenomenal data center infrastructure of any organization in the world, better than most governments, right? Well, the US military and other organizations like that might have some really good stuff. But Amazon has some amazing infrastructure in place. And they maintain every aspect of that physical infrastructure. Things like power, water, air conditioning, right? Data centers require a lot of air conditioning to keep them cool, otherwise the servers overheat. The servers themselves, power supplies, hard drives, the warranties on the physical servers. That's all stuff Amazon provides. If you ever wanna [COUGH] get a peek inside of the infrastructure side, Amazon used to keep it all secret, right? They don't want to give away secrets about how their infrastructure is built out. But they actually launched this really cool thing, it's the AWS data center tour. You can go to the AWS websites aws.amazon.com/compliance/data-center, and they've got a tour that you can go through and see what it's like inside one of their data centers. So if we go in here to our data centers, they show some pictures of what one of their typical data centers looks like, and they're pretty, pretty impressive. You have row after row after row of servers, you have air conditioners, you have generators, you have a secured facility. The generators that are outside, actually I think these are air handlers, nope those are generators. So some of the generators that are outside, see how they're walled and fenced so people can't get to them from the outside, that's all stuff that cost a lot of money to do. And Amazon does that, they manage that aspect for you so you don't have to. Protecting the perimeter, protecting your data, even the environmental side of things, they're taking care of all of that. So when I deploy and instance on EC2 and I tell it, yeah, I want SSD storage and I deploy it. Or I fire up an S3 bucket and I start throwing a bunch of data in there. I don't have to worry if a physical hard drive dies. If a physical hard drive dies, the Amazon team takes care of it. I don't have to worry about it. I don't have to worry about if it's RAID 5 or RAID 10. I just tell it, hey, this is a database server, and I need a commitment for a certain amount of IOPS, Input/Output Operations per Second. I need that performance I don't care what kind of drive it is, just give me that performance and they handle all of the stuff underneath. And it's that level of service that can start to give us a full sense of security. Where we say, what I don't worry about this, Amazon they're going to protect me, right? And they are certain things that yes they will protect you from. But there are others things that are still left on your plate. And it's your responsibility when you go and deploy inside of AWS to take stock of that, to take inventory and say, all right, I've deployed some instances. All the physical infrastructure is taken care of for me, now I need to maintain the rest. Where it starts to get confusing are services like Elastic Beanstalk or Far Gate, applications like that. Some of the container services where Now it becomes platform as a service. Amazon takes care of the hardware and they take care of the operating system. You don't see the operating system but you do see the software that's on top of it. You're maintaining the softwarre. Or, sometimes it even gets a little more blured than that. For example, RDS is a great example, right? If I were to go into RDS and spin up a relational database server I can create a relational database server. And I can pick, I can be very specific with what I want. I can say, look I want a MySQL database, here I'll pick that one. And as I go forward, it'll let me pick how I'm deploying that. Whether I want it to be a multi-availability zone deployment, so if one AZ goes down, the other link keeps it up. Or if I just wanna go to single, like in dove test, where if one AZ goes down, I might lose the database. I can actually weaken the security of this system right here. Amazon will let me do it. Amazon assumes you know what you're doing and, well, hopefully you do. Typically we want multi easy. And then as we go forward, now it's asking me to pick the database engine version, right? 5.7.22. I can pick older versions if I want, right? Now Amazon does a pretty good job of making sure that they don't have versions with documented security vulnerabilities. Notice how with MySQL we've got 5.7.17. And there's 5.7.18, 5.7.19. Where's .18? Well, there's a documented security vulnerability in it. And so they've removed that one and you can upgrade to 5.7.19 with a click of a mouse. Nice and easy, right? But, you can't deploy at .18 anymore because there's a vulnerability. So Amazon is making sure that when you deploy your secure but once you're deployed, there's a lot of things that are still gonna fall in your lap to maintain, like staying with the latest version of MySQL. They'll do minor updates for you. They usually won't do major updates. So if I'm running 5.6, they won't upgrade me to 5.7. It's up to me to come in and do that if there's some feature I need. But for security updates, occasionally there are critical security updates where they'll notify you and say, look we're updating your server for you. It's either that or shut you down and so they'll update you. They do that on services like RDS. They don't do that on EC2. On EC2 you can bring up a Linux AMI or an AMI you built yourself with a custom operating system on it. And you could have a three-year-old known remote exploit on that system. And Amazon will allow it, as long as it doesn't impact other customers. If you started impacting other customers, they'd pick up on that and those turn your instances down and that maybe the first note you get about that. If you haven't been thinking about security and all of the sudden, you see an instance goes down, you check your mailbox and here's a note from Amazon saying particular service have been disabled, that happens. And it's all a part of not actually taking ownership of that part of security. All right so, where it gets a little ambiguous is what exactly is Amazon protecting? And I think they have a really good part here on the security page. So if I go back to that page I showed you guys a moment ago and scroll all the way down, well I guess not all the way, just a bit down, there's a nice little section right here that talks about some of the things they offer. And we have to be careful when we read this, cuz some of it, if you just take it at a glance, you might misinterpret. So for example, penetration testing, right? It's right here on the list. I might think, wow, Amazon penetration tests my software, right? They don't, that's not something they do. If you click on that, it'll explain, you can get permission from Amazon to allow your own hired PenTester to test the system. If you just try and pen test your own system, Amazon will detect your attacks and block them. The pen test might fail, right? So you've gotta get permission before you do it. Otherwise, you set up red flags with them. But some other things they do offer, distributed denial service protection, right? It's not 100%. They actually have a certain protection that's in place that you can use throughout 53. That's not perfect. It will block some of the well known attacks, the ones that they've got documented and they know happened. But if there's a DDoS attack that's been crafted to your site, they don't know how to identify that. They don't know your site and how your site works. And so those attacks still get through. The DDoS mitigation is very generic mitigation, So we'll typically engage in third parties or other solutions. The Amazon WAF, the web application firewall, was actually really effective at stopping things like this because you get to configure custom rules for your site, and dictate how to prevent that traffic from overwhelming your site, or what's legitimate traffic and what isn't. That can be implemented pretty easily but it requires intimate knowledge of your application and that's knowledge that Amazon doesn't have. All right let's see, so some of the other things to think about, hypervisors. When we deploy instances inside of a VC2 for example, those instances are virtual machines and they're running on top of a hypervisor. That hypervisor for the most part is Zen or KVM. Amazon actually has deployments across both. We never see that, we never see the Zen hypervisor or the KVM whatever. Amazon could switch, they could switch to VMware tomorrow, you wouldn't know it, right? Cuz they mask all of that behind the user interface and the API. They mask it And they're in charge of it. They control the hypervisor behind the scene. They keep it secure. And they have to do that because your instances will get deployed on hardware alongside other customers, right? And when you deploy that way, if there was some vulnerability in the hypervisor and somebody could do a virtual machine escape attack, They might be able to access memory from VMs that are being used by other customers. This was in the news quite a bit with the spector and meltdown attacks, where somebody could spin up an EC2 instance, start to flood the CPU with certain types of activity, and start to intercept calls that were being made from other virtual machines. One customer Could get a data from the other. Amazon was one of the first to patch for that, to protect it, they knocked it out. As a customer, we didn't have to worry about it. But, let's say we do worry about it. Let's say that's a big deal, and that was an exploit that we knew about. What if there are ones we didn't know about? What if I was really worried about that? Like, man, I want to deploy inside AWS but I just don't want to share hardware with other customers. Well, you've got options. Inside of EC2, for example, I'll jump back over to EC2. When you create an instance, and you start going through the wizard here to create one, one of the questions that it asks, I'm just kind of moving through pretty quickly here to get to this screen right here. This is where I configure like instance details, and as I scroll down right here, I can find Tenancy. And it's warning me that it's a shared hardware instance, and if I look at the little note here, it's telling me that, yeah, this will be running on a physical server that you share with other customers. And if you're not comfortable with that, if you really need to make sure you're the only customer on this hardware, you can change it from shared to dedicate it, right? You can even go to a full, dedicated host where only your virtual machines are on their hardware. Now, that's expensive unless you fill it up, right? If I'm gonna fill up an entire server, then it's actually the same cost as if I was deploying individual instances anyway. But if I just deploy one instance on there, and I'm only using part of the resources, because I went dedicated it's now locked that whole host to me to gets expensive. But I have that ability to ensure that I don't have to worry about VM escape attacks, or memory intercepts, or anything like that because I went with dedicated hardware. If I knew to look for that option. This is a default option that most people just glance right by, they don't even think about the fact that you might have virtual machines running on hardware. And your competitor might have virtual machines on the same hardware. You wouldn't know an attacker, a malicious agent, could have VMs on that same hardware. All hidden from us, because that's stuff that Amazon manages. Now Don, it seems like if we have decisions that we're making, like we're setting up an EC2 instance, or we're setting up RDS. We need to be aware of selections that we can make that may impact security concerns. But what about things like Dynamo DB, where it's just click, click, and I don't see anything? [LAUGH] Is more or less that taken care of, or are there still concerns there? There are still concerns, but more of it is taken care of than in other service. So depending on which service you're deploying some of them are really, really just barely have to do anything. Elastic Beanstalk, I know it's kinda going out of favor, containers have killed it off. But the nice thing about Elastic Beanstalk was there are instances in the background and you just didn't see them. You just picked what language you were gonna be writing, and so what environment. I need Java, I need Python, or whatever and you throw your code on there. Your code was obviously your responsibility, but everything else was managed by Amazon. The key secret here is that Amazon has so many services. And they vary so much from whether they're just infrastructure as a service, to platform as a service, software as a service. It almost pains me to say this, you kind of have to read the manual, the old RTM. Let me show you an example here for EC2. So this is the Securing Amazon EC2 Instances page. And again, I'll put the links in the show notes. But it's aws.amazon.com/answers/security/aws-secur- ing-EC2-Instances, rolls right off the tongue. But basically, what it does is it walks through the general best practices. When you're deploying in EC2 instances, here's the things that you need to think about. You need to monitor your audit logs. You need to do change management and configuration management. You need to only give out the minimum privileges needed for someone to perform an activity. Don't give them root access on a VM if they don't need it. Just give them the bare minimum to be able to do what they need. Consider restricting your network access, that virtual machines when they're deployed, Justin you mentioned that MongoDB, right? If I deploy an EC2 instance, by default it's going to get a public IP address. It's going to be exposed to the Internet, and that means I've got a security group. I need to configure to filter who can get to what. During the deployment phase you'll likely want to lock that down to only allow your IP to access it until you get it all secured and configured. And then you open it up. So this article walks you through that process. And some of the decisions you make, like what data encryption, are really easy to turn on when you're deploying the instance. But they're really hard to turn on afterwards. In fact, a lot of them require you to back up your instance, terminate it, and then redeploy with the security options on from the backup that you made. That means down time. We don't want down time. So if we can deploy secure from the beginning we'll be in great shape. And ever board go to that /answers/security page/ and Amazon has write ups on all sorts of stuff. Accounts, applications, a lot of the different services. I focus on EC2, because I'm a sysadmin, it's kind of where I work. But as a developer you might be working in other areas and if you're dealing with Lambda functions and so on. They all have different security capabilities inside of them, and it comes down to reading the documentation. And they all have white papers and other publications that tell you here's the suggested security features of those. And from my viewpoint, that's incredibly helpful, right? There's a lot of documentation. But there's some things I don't know necessarily as a developer. A good best practice or hey, I never thought of it that way. Hey, maybe just don't have a public IP. The first time I learned that it kind of blew my mind. I was like well how does that work? And then, I went down this rabbit hole. But I see something here, not all of these security selections that are best practices like data encryption. It's easy to turn them on, but they're not necessarily on by default, right? We need to still kind of work through and see those individual pieces. Absolutely, and basically, Amazon's trying to create a secure and stable environment for the customer. But at the same time, they have to address the customers' needs, and many customers have different needs. It's easy to turn encryption on, it's hard to turn encryption off. And encryption also impacts performance. So when Amazon promises you a certain performance level. If they turn on encryption by default, they have to reduce that performance level and advertise it different. So the better option is to have it off by default and leave it up to the customer. And unfortunately, the customer doesn't always know best, and we end up in the situation that we are today. All right, let's talk about, cuz I know I'm running low on time, so let me talk about a couple of the other features that are available by default. IP spoofing protection, you have IP addresses. Well, makes sense people from the Internet shouldn't be coming in on IP's that you all ready have. That gets filtered out by default. If Amazon recognizes any spoofed IP traffic, they actually block before it gets to you. Which is a nice thing to have. They do some basic denial of service prevention. If they see network flooding coming in or anything like that, they've got ways to trim that off before it gets to you. But it's not perfect, keep that in mind. Let's see what else do we have. SMTP limits, right? If a server is compromised attackers usually have a few different things they do once they compromised an instance. They might start minding Bitcoin. While you've CPU limitations on any instance, so there's a limit on to what damage they can do. But another thing that they like to do is turn your server into spam bot, right? Just relaying spam through it, because they can generate click revenue that way. Well, Amazon actually has limits to how much SMTP email can be sent out, and it's actually very, very low. You can't send much email at all. It's really just designed for sending email alerts for server status. So if you actually want to put any kind of email operation in place, you've gotta open up a help desk to get with Amazon and say, I've got this server, it's gonna be sending this volume of email. And then, they can open that up to allow it, but they protect that. So unfortunately, you've kind of already been compromised by that point, if somebody is trying to relay. But at least it stops you from actively being a relay. Redundant routes, right? When we deploy, I mentioned with that RDS example. If I do a multi-AZ deployment while there's routes, they get people to those availability zones. What if a router goes down? Amazon creates redundancy there. They have, in each AZ, I believe it's a minimum of five ISPs that provide connectivity there. And it's all there inside of a BGP autonomous system which we don't have to worry about the technicalities on it. But that always that range of IP's to bounce between multiple ISP's. An entire ISP could go down, you wouldn't even know that your traffic just gets routed around. And there's inter AZ links as well and that's what you replicate across between. They're very, very fast links. Those are redundant and protected. They even have certain inter-region lengths to be able to replicate data between regions. You'll see that in some services, especially when you get into some of the more advanced cloudfront technologies. Now, you've got stuff spread across several different regions, nice to be able to have that in place too. So those are other areas Amazon provides. The key ones that I want to mention though are, any time you install a software, it's your responsibility. The software is your responsibility to keep updated, maintained and to deploy it securely. Security Groups are entirely your responsibility, you've got to configure them to only allow the minimal access for your customers to be able to access your service. If you have a database server that shouldn't be exposed to the public. That only the web front end should be able to see, it's up to you to build the VPCs to protect that. To create security groups that don't allow public connections to your database server, that's your responsibility. And then, operating systems. If you have a root login to whatever that system is, so like with most EC2 instances, updating the operating system is your responsibility. And any software you install on it will be yours. So those are key things to remember. The other part, though, that we'll talk about in the next episode, is securing your AWS credentials. If somebody gets a hold of your credentials it doesn't matter what security functionality you've put in. They've got the keys to the kingdom. They can log into your account, access all of your resources. And you can end up in trouble. And I wish I could say, this is a problem that newbies have, people that are just getting started, but this happens to a lot of people. And there was a high visibility example just a month ago where Tesla, the electric car company, right? Their AWS account got compromised because they, I think it was a phishing attempt on one of the developers. And the attackers managed to get credentials. What they did is they logged into that account and then, spun up a bunch of bit cleaning instances. And so, Tesla will have a slightly higher bill that month. If that happens to you, you should definitely call and engage Amazon's help services, and they kind of walk you through what to do. And sometimes, they can even forgive some of the billing which is nice but at the end of the day, it is your responsibility to secure that. Your AWS keys, your user credentials should be treated like a national government secret like you don't want those to get out. That allows somebody walk right into your environment, access your data and now you've got a. Well, Don, that's definitely something to be afraid of. I will tell you that I tended to get really hyper-focused paranoid about making sure, no, it's not on my GitHub. It's not here and it may even be as simple as, oops, I turned this S3 bucket public, and it shouldn't actually be public. And I've exposed other data that shouldn't be available. And then, maybe that had my keys in it. Which I'm, that's probably a bad idea and by probably, it is. And the only way we're gonna remedy that is, well, to learn more. And the only way we're gonna do that is we come on back. Any final parting words? Yeah, don't trust security to one person, right? No matter how big your team is, or how small your team is, security shouldn't be trusted to one person. You always need multiple sets of eyes. And if you don't have those resources internally, I mentioned pen testing earlier, that's penetration testing. You can hire penetration companies. There's a number of them out there. I mean, just look offhand, I can think of Black Hills, Semper Sec. There's a bunch that are out there. They can all help you with doing that. Where they come in, and they have trained professionals that evaluate the security of your system. And they see if you've left anything unsecured and can help you remediate that. If you go that route, one, you've got to find a company you trust, somebody you know that will handle your data properly. And then two, you have to engage Amazon's help services to let them know, hey, I'm gonna be pen testing between this time and this time. The pen testers IP addresses are this, this, and this, and you've got to make sure that's cleared ahead of time and schedule it. Any professional pen tester will make sure you've done that in advance. They know, cuz what they're doing is legal if they're authorized, it's illegal if they're not authorized. So they wanna make sure that it's legal and documented, so they'll push you through that process. But definitely, something to do, don't hesitate to get help, right? Get a second set of eyes to look over your security. Well, Don, thank you so much for giving us a nice little overview of AWS security. And well, there's some more things that we need to understand, especially about credentialing. So definitely join us on back. But for now, we're gonna go ahead and get out of here. So signing off for ITProTV, I've been your host Justin Dennison. And I'm Don Pezet. And we'll see you next time. [MUSIC] Thank you, for watching ITPRO.TV.