Transcipt

Whitney:

Hello everyone and welcome. My name is Whitney and I'll be your host for today's webinar: Learn How to Combine Load Testing with Continuous Integration. We're excited to have you guys all here for the next 30 minutes. I'd like to introduce our speakers. We've got Troy Presley, Solutions Engineer at Apica. Troy has over 15 years of experience helping some of the world's biggest brands test and scale their digital infrastructure. Then we've got Daniel Freij, Senior Performance Engineer and Community Manager at Apica. Daniel's role is to tackle the most complex, low-testing scenarios and oversee our growing community. He's going to give you a brief intro of the community during the presentation as well. 

 

This webinar will be recorded and sent to everyone who has registered within the next week ... And I'm sure, as always, there will be some questions throughout the presentation. Please ask any questions in the questions box and we'll make sure to answer them live today. This is a very conversational type of webinar so we encourage you guys to participate and join in the conversation as well. We would also love to hear from you guys regardless if you have anything to say so feel to follow us on Twitter, @apicasystems and LinkedIn, you'll find us at Apica Systems as well. So without further ado, let's bring our speakers in. Welcome guys.

Troy:

Hi everybody. Thank you for joining the webinar today. Today, as Whitney had mentioned, we're going to talk about load testing in the context of continuous integration and why it's important to do load testing as part of your deployments. So Daniel and I are going to take a conversational approach to this so we'll kind of go through the topics and we'll go back and forth on the different thoughts we have about the different topics. Feel free to join in and ask questions and we will answer them as we can.

 

All right. Getting started. Hang on, there we go. So when we're talking about continuous integration, why is load testing important? And the answer to that question is it's very similar to why would you load test at all? So if you have important websites or if you have customer facing websites, especially if they're revenue generators or brand websites that are important to your business, you need to understand how your applications are going to respond to the load that is going to come in from your customers. If you don't understand that, then you don't know where your application's going to fail and that is already a failure.

 

If you're not load testing, then basically this could always potentially be your application. You release what you think is a perfectly engineered application and all of a sudden, more people come in than you were expecting or there was some slight error in the code or the way it's deployed or how it interacts, and all of a sudden, your application crawls to a halt. And if that happens, this is what your dev ops team looks like, and nobody's happy if we're in that situation. 

Daniel:

Yeah. You never want to look like this.

Troy:

So, of course, when people think about load testing, most people think about load testing in terms of responding to upcoming major events, either a product release or a major sales event, like a Black Friday or something like that. But, typically, what we found is if you're responding to an upcoming event, you're probably already behind the curve. You'll never know, even if you start relatively early with your load testing regime, you never know if you're going to have enough time to solve the problems in time for the event that you're trying to get ready for. So although it's definitely better to do load testing to prepare for an upcoming event than it is to do no load testing at all. But we'd like to say the perfect situation is to take it a step further and just constantly load test.

 

Every time you deploy code, you should know how your application, your new changes to your application, are going to respond to different amounts of load. And that means making it a part of your deployment process. Of course, you can do this manually, and that's acceptable, but in today's world, we have a lot of tools that make it a lot easier to do these kind of steps in an automated fashion. Continuous integration, continuous deployment, things like Jenkins or TeamCity or what we're going to talk about today, which is Amazon CodePipeline. All of these tools make it very, very easy to integrate several things. Not just load test, but several different kinds of testing and verification steps into your deployment process. And we'd like to suggest that load testing should absolutely be a part of those. 

Daniel:

Absolutely. And you should even start thinking about load testing as a form of monitoring, when it comes to CI. So the amount that we usually use at the peak is that if we don't monitor it, you can never manage it. And it's the same thing with your scalability and performance when it comes to deploys. You never want to end up in a situation where you're not looking at how the performance is developing over time. Especially if you're doing continuous releases, say every month or every week.

Troy:

Yep. That's absolutely true. Absolutely true.

 

So one thing I'd like to point out, again, kind of playing into what Daniel just said, a lot of you probably have pretty decent UAT or automated testing, kind of functional testing or end-to-end testing that's built in, and hopefully that's even automated, but there are problems that UAT and automated testing can't catch that load testing can. And that's particularly when you're talking about modern applications that can be broken down into many, many microservices or interconnected services. Even if each of those services passes perfectly under the testing regime that you have, that doesn't mean that in combination from when you have realistic traffic coming in to those applications that it's going to perform in the same way or that there's not going to be a change in the way that the performance happens for different load amounts. And that's, again, where load testing as part of your deployment comes in.

 

Just kind of as a side note, and we've also got things like concurrency related bugs and code interplay from multiple rolled up commits, so if you think about like your code deployment process as you go through, you probably are testing the code that one developer, one team, is doing very well, but what happens when two or three teams are making different changes together and those commits get rolled together into a further step in your deploy? And all of a sudden, each of those individual changes played very well through your testing and there were no bugs found, but when they come together, all of a sudden, you have issues that are impacting performance that you wouldn't have been aware of otherwise. 

Daniel:

Yeah, and this is a very common problem in the business. You're doing all the QA and all of your tests with a single user on, so when you actually do get to the point where you get ten or a hundred users, the site will most likely go down, because the only knowledge you have about your information is "well, I don't have any functional bugs and it works for my user" ...

Troy:

Yep.

Daniel:

"So we get to deploy."

Troy:

Yep. 

Daniel:

And you deploy. Your one hundred users come and you quickly see that after ten users, the response time is six seconds. At one hundred, your site goes down, but you thought it was going to have ten thousand users.

Troy:

Yep.

Daniel:

This is a perfect example of why it's important, the load testing, in your deploy process.

Troy:

We just recently had a customer that was like that. They came in basically very confident that they were going to pass a very large load test and came in ... What was the concurrency that they were trying to do? It was like ... they were trying to do like five thousand or something like that. 

Daniel:

Yeah.

Troy:

Five thousand user concurrency and they were very confident that they were going to have no problems at that level. We hit like two hundred users and their site completely fell over. 

Daniel:

Yeah, and it wasn't even like a big e-commerce site or platform or anything like that. This was for booking museum tickets.

Troy:

Yeah, exactly. And, you know, obviously if they had not had the foresight to come test with us, then they would have been in a very bad position, but if they had had load testing as part of their continuous deployment process, which I think they are working towards getting to, once they have that in place, then they'll know exactly where they stand on every single code release. And then they're not going to be running around in emergency mode ever. Right? They don't have to be worried about preparing for a particular event or anything. They'll be ready for anything that comes and they'll know what's happening. 

Daniel:

No more Homer Simpson moments.

Troy:

Yeah, no more Homer Simpson moments. Exactly.

 

The other thing is, is like, there is no such thing as an application that can handle infinite users. There is always going to be a point at which your application falls over. What load testing is about is finding that point so you know A) at what point your application is going to start having problems and where it potentially will just not work at all anymore. Understanding those two points. And second, is understanding when you do have problems, where are those problems going to present. That way, you're not running around trying to troubleshoot that at the moment. You already know beforehand that when you get to a certain number of users, this part of your system is going to be the first thing to break. 

Daniel:

Yeah.

Troy:

This is going to be the first thing to start showing errors.

Daniel:

And something we noticed over the last few years at the peak is that you get to that point and you find that bottleneck and you remove that bottleneck, and all of a sudden, you can handle a higher load. But you find five more bottlenecks. Because now the bottleneck is no longer the web service, for example. Now it goes down that it's database or application logic.

Troy:

Yep.

Daniel:

So the more bottlenecks you remove, the more you're going to find. So that's why it's so important to have an integrated process with CI, to catch all of these cases. 

Troy:

And to be clear, there's never a situation where you don't have a bottleneck. It's just understanding what is the bottleneck that you have currently and what is the level of users that your application can handle based on the current bottleneck you have. Right. When you remove one bottleneck or improve, you know, expand one bottleneck, all that's going to do is expose you to the next one. But every time you do that, you're increasing the potential load that your application can handle. 

Daniel:

Yep. 

Troy:

So, kinda wanted to make this point to. For people who are on a budget and are thinking "I can't afford to do a massive load test," either from a staff time perspective or from a monetary perspective, "I can't afford to do that on every single deploy." Even smaller pre-deploy load tests have value. Because what you can do, if you're running a consistent number of users, let's say just a hundred users even, if you're running a hundred user load test every single time you deploy, that consistency of the number of users and then comparing the performance of that hundred users from one run to the next is still going to tell you whether your application has changed in its performance profile from run to run and from code deploy to code deploy. So not having enough staff time or not having enough money is not an excuse to not load test, 'cause you can ... even really, really small load tests can still have value. 

Daniel:

Yep. An alternative way to this is actually assigning the load that you're pushing to the system to a single leg of it. Where your load balances, and this way you don't have to close off your whole environment when you're doing a load test. You test one leg with a stream of users to be able to support that one leg and you find out what goes wrong there and how do we solve this when we scale up.

Troy:

Yep. And, of course, with a much smaller load test, you're not going to see that turnover point, you're not going to see where ... you're probably not going to see where your application is starting to fail or starting to show errors, that kind of thing. But it will show you the performance changes and that can be enough under certain circumstances.

Daniel:

Absolutely.

Troy:

So what we're going to do now is I'm just going to show you how easy it is, we're focusing on AWS CodePipeline today, but we integrate with TeamCity, Jenkins ...

Daniel:

Yes.

Troy:

Bamboo, Azure. We integrate with ...

Daniel:

Visual Studio.

Troy:

Visual Studio. We have many, many integrations with many, many continuous integration or continuous deployment tools. So this one is just an example of how easy it is to integrate. 

Daniel:

And worth mentioning here is that if you ever have a CI tool that we don't support, we do have APIs that you can build into to be able to bring Apica Load Testing into your continuous integration process. And that's very easy. In some cases, that's just a single script that needs to be added, for example, in the Jenkins process.

Troy:

Yep. Exactly. Jenkins, in particular ... Jenkins, specifically, we already have tight integration with. But if you are using one of the other continuous integration tools that we have not built integration for, first, we'd like to hear about it. So let us know. I'd love to know the tools you're using. But, second, there's not ... you know, because of the APIs we expose, there's never going to be a continuous integration tool that we can't get it working with.

Daniel:

Yep.

Troy:

So, gonna go through this relatively quickly, but the point is that it's extremely easy. If you're familiar with CodePipeline, then this should look familiar. If not, it still should be relatively easy to understand. CodePipeline, like any continuous integration tool, just has stages that you go through during the deployment. So if you want to add load testing, it's easy as adding a stage in the middle of your pipe. You can add an action to that stage. And then there's a drop box inside of CodePipeline and any of the ones that we have integration with, you're either just going to add a plugin or just choose from the list, Apica. In this case, you don't have to do anything extra. It's already populated in the test provider. You just hit the "Connect" button to connect to your Apica account. If you don't have an Apica account, we give you the opportunity, right after hitting that connect button, to create a test account right there. Making it easier to get started.

Daniel:

Yep. Feel free to sign up for trial and play along with this bit.

Troy:

Absolutely. But if you do have a username and password already, you just type that in here. And what that's going to do is now you're going to just choose your settings preset, scenario, and thresholds. So you have the ability to create new presets here. We're not going to go into test creation. That's a much deeper topic. But it's really, once we get into that webinar, it's not hard to create a basic load test. And you can get as deep as you want with it. So you can create very, very simple or very, very complex load tests without too much hassle. 

 

But, assuming you already have the test, you would just choose the settings that you want to run that test under, choose the actual test, which is the scenario, and then you're going to go, finally, into choosing the thresholds.

Daniel:

And this is your failure states. So I'm running the load test, but what is a bad result? When should I roll back and when should I continue?

Troy:

Yeah. So these are examples of thresholds. So basically you can add multiple thresholds or you can just have one or you can have no thresholds if you want, because you can just use this as a data collection. You can pass through this stage with no thresholds and just capture that information about it and manually compare it. But if you did want to have the ability to block a deployment, then you can add these thresholds and automatically cause the deploy to fail.

Daniel:

So what this means is that if you're doing daily or weekly deploys automatically without much oversight, then this would help you prevent the site from failing from a deploy. 'Cause if you get more than a five percent failure rate, in this case, we can actually tell CodePipeline to just roll back. This is a bad deploy. And you'll come in in the morning, not the version you want of the application, but it will be alive.

Troy:

Yep. Exactly. So this is, again, very convenient. Instead of looking at it from the kind of standard end-to-end or functional testing perspective where the tool or your application is failing tests, because from a functional standpoint, it's not doing what it's supposed to, this is from a performance standpoint.

Daniel:

A non-functional test.

Troy:

Yeah. A non-functional test. The ability to stop a deploy based on the actual performance of your application or a failure rate that's non-functional.

Daniel:

Yes.

Troy:

So, you've set up your first CodePipeline or other continuous integration tool with a load test from Apica and now you've run through that a couple times. What do you get out of that? Let's kind of take a look at that.

 

So this is if you log in to the Apica portal and look at the continuous integration section, you're going to see graphs that look like this.

Daniel:

So what we're looking at right here is when running continuous integration against a demo application, and we see a pretty fast move line up until 2015 where we actually had some fun with the code. Let's call it that. So the information that I'm presented with here tells me that all of a sudden, the session time is a minute and four seconds ... four hundred milliseconds actually, and it has a failure rate of a hundred percent. Not really where we want to be after deploy, right? But this information, what it tells us is we need to look at this data point, and we can do just that. So just below that graph, when you click that data point, you get presented with all the different steps in the scenario that you ran. And you can quickly isolate this way which step did we fail on? So is it just generally when we're starting the start page? When we're checking tickets? No? All of these look fine. But when we go into admin, we have one hundred percent failure rate and that's that. 

 

So that would mean that we deployed something, didn't work at all, and this is what happened. So when you reach this point, you can just keep drilling down, so if you click this data point, you would get the results and when you're looking at the results, you can see exactly what request we fail on. You can click that, dig down to the error messages and all the tag information. So if you have dynamics installed here, you can quickly see "oh, the deploy didn't go well, let's open it up in Apica load test, drill down, down to the error level" and from there you can just grab the tag information and search for it in AppDynamics or Dynatrace or any APM tool you have almost.

Troy:

Yep. So the theme of this webinar is, of course, kind of more aimed at the performance, but I think it's important to point out. So the graphs that we've shown here show from more of a performance standpoint. A) you've got kind of the overall, the whole test, and we're able to very quickly drill down in here and see each of the individual steps or pages in that test and you can see which one was actually causing the problem. And, to Daniel's point, this goes more into the troubleshooting aspect, but you can actually take this tool and take the information you're getting out of these load tests and start the troubleshooting process. Not only finding out what step caused the problem, but you can continue to click and go down ... 

Daniel:

Yep.

Troy:

Down the rabbit hole here, all the way to the request level inside of our tool. And if you want to continue that, you can go into AppDynamics or your other APM tools and continue that analysis on your [crosstalk 00:19:38].

Daniel:

You could call it like a five-click resolution in this case.

Troy:

Yeah. Exactly. Exactly. So the other thing, and this kind of ties back into small load tests still have value, we have the ability, the Apica infrastructure has the ability to handle anything from a single user test all the way up to multiple millions of users. We have yet to have a customer that's exhausted our supply of virtual users, so anywhere from the smallest test to the largest test in the world, we can help you.

Daniel:

And there's actually no limit, because we can scale up into Amazon, so ... 

Troy:

Yep. Exactly. Yep. And, just kind of as a side note for those of you who have sensitive applications, we have the ability to also augment our cloud-based load agents with your on-premise agents that are run inside of your data center or your own load agents that you just put anywhere on the Internet, so you can actually, rather than using our shared infrastructure if it's important to you, you can either create load agents that are inside your data center or that are just your agents and nobody else gets to use that are out on the Internet, and those can all be controlled via the same portal and the same continuous integration. 

Daniel:

Yeah, and after that, in terms of security, all the communication between agents and controllers also do encryption on the communication level, but also on the file level.

Troy:

Yep.

Daniel:

So even if you can break the network encryption, good luck in breaking the file encryption. So we feel very safe in being able to offer this service to enterprise and the bank industry. 

Troy:

Yep. And we have several customers in both the banking and the medical industries.

Daniel:

Yep.

Troy:

So, kind of going back over this, we've kind of talked about why it's extremely important to do load testing in general. You know, you should do some form of load testing no matter what. If it's manual and only for an event, that's at least a first step. It's better than no load testing. You should absolutely automate that process. And I think that it's important to do that as part of your actual automated code deploy. And that's going to put you in the best position to understand your application, understand if you do have an unexpected spike in users that's beyond what you were expecting, knowing where your application's going to break so that your dev ops team or your programmers are going to know exactly where to go to start working on the issue when it does have a problem. And Daniel and I have kind of shown you how easy it is to implement these tests into a tool like Amazon CodePipeline. And that all rolls into as easy as it is to set this up and as important as it is and the fact that you can do small load tests to fit your budget, if you're not doing load tests today, it's because you just didn't do it.

Daniel:

Yeah. And that's a really good point. Nowadays, there is really no excuse to not do load tests. We've shown a bit with the continuous integration slides that we've shown you this time and also other webinars that we've done on how to create load testing and the simplicity of it, you basically just need to start with your landing page and then work yourself through the system. So when you're starting load testing, all you need is one URL, and you can create that test in a minute in that portal. And even with a trial subscription that we have for Apica Load Test portal, you can run almost up to five hundred users I think.

Troy:

Yep.

Daniel:

So this should be able to just get you started really quick without any big investments outside of your time, of course, and just get started. So let's avoid the "just didn't do it" ...

Troy:

Yeah.

Daniel:

And Homer Simpson moments.

Troy:

And, at the risk of being repetitive, any load testing is better than no load testing. So a lot of you might be afraid that you've maybe come in contact with some load testing tools in the past that have been very complex and you're afraid of the time investment of building complex scripts like that. Any load testing, even a simple URL check of your homepage is better than no load testing at all, so if you're worried about the complexity and the time investment, just get started with something simple.

Daniel:

The landing page, for example.

Troy:

Yeah.

Daniel:

That's the first page that your users will hit.

Troy:

Exactly. Just start with something simple and move from there. It's better to stick your toe in the water and get started than it is to just continue to not do any load testing at all.

 

So that is the end of the formal presentation. We're happy to take questions. Or we can kind of just keep talking.

Whitney:

Thanks, guys. I know that you answered a couple of those questions ... people's questions throughout the presentation, but if there are any last minute questions, feel free to ask them now. We can spend another minute or two on questions. And if not, Daniel and Troy are going to have a little bit more of the conversation. 

Daniel:

So one good way, if you don't have any questions, a good resource is always, if you want to learn things, how to get started with ZebraTester or Apica Load Test portal, you can always visit our community at community.zebratester.com, shown here. So if you have any questions around how to do load testing, how to create a script, or what plugins are available to enable you in your load testing, feel free to visit the site. It's continuously monitored by me and my colleagues, like Troy. So if you have any questions, just shoot them there and we're going to answer them and you might actually find some articles that already answered most of the questions.

Troy:

Yep.

Whitney:

I've got a question from Ranjit here. Sorry if I've just butchered your name. How long should you run a load test?

Daniel:

A load test doesn't really have a limit. It all depends on what questions you want answered. So, for example, you want to find out how many users per second can hit your landing page, then you can do that with a simple stress test running for five minutes. If you want to investigate your application for memory leak, you can run the same test, but for five hours. And then if you want similar concurrency, that's more of a one hour test. There's no given time limit. It's all depending on what is the question you want answered. You can do it in multiple ways with the exact same script by just changing the think times on the steps to either "I want to do as many requests as possible at this time" or "I want to simulate a load pattern according to my Google Analytics". 

Troy:

Yep. If you think about it, I like metaphors, if you think about it from the metaphor of architecture, right? Do you want to check if your building can withstand a tsunami or do you want to check if it can handle an earthquake? What the type of stress test you do is going to be dependent on what you're trying to look for in your application. There are many, many different types of load tests. Especially when you're doing manual load tests. When you're talking about in the context of a continuous integration or continuous deployment solution, you probably have a smaller subset of those you're interested in. Mostly, you're worried about performance at certain concurrency levels. So that would be usually something like a five to ten minute test is probably adequate for that purpose.

Daniel:

Yep. And looking at these two test types in a different way. So the stress test will mostly just give you answers of, as I mentioned, like how many requests of this type can I make per second, but you might reach your threshold of what your application can handle at even one hundred users in just pure [inaudible 00:27:20]. Now, the metric you look at in this case would be "I can do x amount of searches per minute". 

Troy:

Yep.

Daniel:

But rather, if you're doing a different test, you can reach maybe one thousand users if you're just slowing down and growing the think times a bit. And then it's more for how many sessions can I have active performing this action on my website.

Troy:

Yep.

Daniel:

So just changing the think time from the three second default we use for stress testing to something like thirty seconds to simulate a user waiting on a page before doing an action will have a dramatic effect on what questions are being answered by the test.

Troy:

Yep.

Whitney:

And I think that's all the time we've got for today. So I'd like to thank all of you for taking the time out of your day to join us and as well thanking our speakers, both Troy and Daniel. Thank you for taking the time to educate us all on AWS CodePipeline and integrations.

Daniel:

And, as I mentioned, if you have any follow-up questions, feel free to visit our community.

Whitney:

Awesome! Thanks, everyone! Have a great day!