Transcript

Troy Presley:

There's a lot of content we're gonna go over. We're gonna go over everything to do with why you would load test for holiday readiness, how you'd prepare for that, the scheduling that makes sense for load testing, and how to perform those. A lot to cover but we're gonna try and make it about the appropriate level of technical depth and the appropriate level of informative.

 

Without further adieu we'll get started here.

 

First, preparing just for [in a view 00:00:35] that are not veterans with the load testing world why would you want to load test in relation to holiday readiness or really any major event that you'd have with your application. There's a lot of reasons.

 

The first and foremost is all of you realized this but performance matters. We have some stats here that show how often applications are degraded and how those are found, but the main takeaway here is that in a very large percentage of time, in the performance problems that you have with your applications are actually found by your end-users or your employees and that's not the correct way to handle it. I think we all agree.

Daniel Freij:

Absolutely, it's usually because you haven't download testing of some kind. When the application needs to stay lock it doesn't. If you haven't done in the scale testing [that 00:01:32] gonna see that before you used to [hit 00:01:36] the application, but that's a very basic thing that most people can agree on.

Troy Presley:

The point of load testing is to understand where your application stands now. If you've given yourself enough time to allow you to have an iterative process to correct the performance problems that you find during load testing. 

 

Moving on, this is just some great stats around Black Friday and how performance problems can impact your actual sales. Not everyone of you is gonna be in E-commerce site, but anyway your application is important to your business and the performance is gonna have an impact.

 

For those of you who are in the E-commerce space, here's some basic statistics you see when we're looking at, for instance last year, there was an estimated 1.6 billion lost because of poor performance. Everybody knows that the famous stat about three seconds, if your site takes more than three seconds to load, three seconds is about the line at which nearly half of your users will abandon and move on to other sites. 

Daniel Freij:

I actually like to say that that figure is not true anymore. It's more like somewhere around 500 milliseconds on a second nowadays.

Troy Presley:

Yeah, that's a very good point. The three seconds number that you're throwing around was actually from a famous Wal-Mart study from several years ago and the patience level of end-users is getting tighter and tighter every year.

 

As we shift towards mobile, ironically, people's patience on mobile devices is less than on desktops, so even though it's harder to achieve faster load times on mobile, you're under tighter constraints, so it's even more important to understand exactly where you stand when it comes to mobile.

Daniel Freij:

But one thing to add here, I do suspect we have a lot of E-commerce people listening in here, but if you're also in the content business, you're having a big CMS. You can almost treat this in the same way, but every page is revenue for you. The less page is loaded, the less apps are gonna be shown and the less people are gonna sign up for your premium subscription. It can be exactly the same way like if your response time doubles, that's 50% less, sell is pretty much. 

Troy Presley:

Yup, that's exactly right.

 

Just to highlight this point and I promise we'll move on to the technical portion here soon but just to highlight this point. This is looking back post mortem on 2015. This was companies just from 2015 from one article, Forbes article, highlighting some of the major companies that had problems last year. 

 

This is just pointing out the fact that this isn't just small- and medium-sized businesses. Large companies have a significant amount of trouble at keeping their infrastructure up and being ready for the holidays and in any major event like I said before.

 

You're in good company with having trouble or trying to get better at handling the situations. 

 

Other reasons that you might want to load test and when you're preparing for major events.

 

Unit and functional tests don't find everything. I'm very happy that when I look out of the community today, unit and functional tests have become a major part of most organizations. Not everybody but we're definitely moving towards a significant adoption of testing as part of that one strategy for major applications and that's a very good thing.

 

Unit and functional tests don't find everything. There are several types of bugs that I've listed here. For instance Concurrency Bugs, Compositional bugs, DB Indexes and Locks. These are things that when you're running through one code path, it will run perfectly.

 

If you're running a unit or a functional test, it will run perfectly because you're only simulating one user, but if you start to throw multiple users, the way the code interacts with each other or how the code interacts with your database when multiple hits are coming in, you'll see problems that you wouldn't necessarily see if you're only simulating one user.

Daniel Freij:

I'm also doing mix testing. I'm testing multiple functions at the same time, will show things, your search function is slow, but it's actually so slow that the database [binds to hold 00:05:59], you know [inaudible 00:06:00]. 

Troy Presley:

This brings up a really good point. Really when it comes to any application, it is impossible to eliminate all bottlenecks. There's always going to be a bottleneck when you reach a certain number of users. Load testing is about helping you not to solve all bottlenecks because no one can do that. 

 

It's about understanding your application and also that you know where your bottlenecks are when you start to see your users, your actual users coming in and they reach a certain level, you're gonna know where your applications gonna start to slow down or where it's gonna start to break. You're gonna know whether to pull your teams to start making fixes because you're getting that.

 

It's understanding your application before problems happen as opposed to trying to figure as it happens.

Daniel Freij:

The great parallel here not doing load testing or any monitoring is pretty much a Rubik's cube blind. You can't really see like what is happening when you turn it around. You have no idea how it's gonna handle big loads or if the search function is gonna kill the site.

Troy Presley:

The other thing I'll say is, it may sound cliché, but at any successful company that has in the application is load testing, it's just are you doing it on purpose or is it because your end-users are doing it for you. It's better to do it purposely and to control that situation rather than letting your end-users be the people doing the load test for you.

Daniel Freij:

There's also about repeatability. If you're doing the [counter 00:07:34] solution where you ask direct a certain segment of the traffic, for example, you're now gonna get an exact replay of that. Still that's gonna be a certain portion of your user base that might get bug code, degradation, that five percent is gonna show up in the revenue.

Troy Presley:

Yeah, it's exactly right. 

 

The other thing I'll point out is a lot of you are aware application complexity is increasing. Although the applications that we're releasing are more and more powerful and a lot more interactive and responsive, along with that comes increasing complexity.

 

I know when I started building applications nearly 20 years ago, this is the typical layout you would see and nothing really got much more complex than that back then.

Daniel Freij:

Not exactly. You can still look at it like this. Usually when you're deploying your environment, you say, "I wanna add 50 more application servers," but you're just thinking about it as a layer and this image makes sense.

Troy Presley:

But with the advent of microservices and third party, basically third party connections, integrations, IoT all of these modern applications look a lot more like this with large amounts of interconnected systems that all have to be running in order for the application to do what it needs to do.

 

What that means is, testing this becomes a lot more complicated if you have a simple three layer application. Isolating and testing functions is a lot simpler. When you're testing an application like this, sure, you can use unit and functional tests to test each individual note here or each individual microservice, but when you fit it altogether, how that works in the performance profile that's gonna come out of all of the systems talking to each other, is gonna be a lot harder to reason about without actually putting realistic traffic through it.

Daniel Freij:

Not only that. To be able to test an environment like this, you need to actually get everything up to a certain point with load. You can't just put traffic in here and touch everything. You actually need to make sure that everything talks at the biggest [capacity of CAM 00:09:45] to find out if, "Well, I ran out of Oracle licenses in a DB". 

Troy Presley:

Load testing is really, again, it's about knowing what questions to ask. The first thing you're gonna ask when you're trying to prepare for any type of load testing but particularly for a major event of holiday testing is can you answer this basic questions about your application right now? 

 

Like we said, there's always gonna be bottlenecks on your application. Do you know what they are right now? Can you say when I hit X number of users that this part of my application that you know beforehand is gonna start having problems?

 

If you don't know that then load testing is something that you need to build into your repertoire here. Do you know what your performance is at normal and at peak usage? In other words do you know on a normal day you expect to see a thousand concurrent users and your page is gonna load in one and a half seconds and if you hit 5,000 users that's gonna go to three seconds? Do you know that profile beforehand so that you can know what's normal and know when you're not operating within what you know to be your application's performance?

Daniel Freij:

Another thing to consider as well is that if you have a scaled up, then you don't really know the answer to these questions again.

Troy Presley:

It's anytime you make any change whether it's infrastructure or code.

Daniel Freij:

Say that you've optimized all the images to decrease from 50% in size that's gonna increase the [profit 00:11:07] from your web servers and hit your database harder. 

 

As soon as you scaled the environment up, you always need to verify those simple things at least. It's all about do I know what's gonna happen now if I put load on it with 10 more web servers. 

Troy Presley:

Again, load testing is about knowing your application, knowing what it's gonna do, so do you know what the performance issues you're gonna have. If you have a good testing regime including a load testing regime, you should never really be surprised by anything. It's not to say that there won't be problems, but you should know what those types of problems are gonna be. You should know what the performance issues. 

 

When they come in, that should not be something that comes out of the blue for you. You should understand what that is. When you start to have failures, you should say, "Oh, that's because we've hit 1.2 million users and we knew that that was going to happen," and it's great that you have more than your expected users, but at least you know what the type of problem that's coming in and what you need to do.

Daniel Freij:

You've talked to and really get pointed what to do to fix it. You've done the practice before, you've seen the problem occur and you got the [Romberg 00:12:22] for it, so that decreases the time like instead of shifting in Black Friday, for example, and having it send to two hours of your nightmare, your operations team can without much interactions, just deploy the resolution from the [Romberg 00:12:42] and be up and running in two hours. 

Troy Presley:

The last one I'll just touch on very quickly is, are you using the right hardware? Especially when you're looking at Cloud Deployments, it's very easy to just throw large instances at the problem. Even with physical hardware it can be easy to order physical servers that are way above spec for what your application needs.

 

Load testing will allow you to, especially if you correlate that with an OnPrem monitoring solution like an APM solution, you can start to profile and see if you're really using the correct type of instance or the correct type of hardware or whether you're wasting money using a non-ideal image.

 

If you have an application that's CPU bound, but it doesn't use a lot of memory, there you can scale back on memory on a per computer basis, or if you're network bound but not using a lot of CPU, you could just use high network instances with low CPU and save a lot of money if you're picking the right type of hardware or the right type of instance.

 

Load testing can help you to understand what type of hardware you need at different levels of users and really optimize your infrastructure spend as well.

Daniel Freij:

You can pretty much say it enables you to put a price on a transaction.

Troy Presley:

Yeah and especially if a large percentage of your traffic is gonna come in during the holiday season, that can allow you to understand exactly what type of infrastructure you need to invest in before the holiday is coming so that you can really optimize your spend during the holiday season as opposed to just randomly picking instances 

Daniel Freij:

It can be other things as well like if your cluster in the Amazon Cloud, for example, can be really busy, you're gonna experience a lot of [bad name resemblance 00:14:22].

Troy Presley:

Hopefully, we've convinced you that you need to load test, so when should you load test?

 

The first thing I'll say is early is good. When performing load test, you're preparing for traffic so you wanna start early enough that you actually have time to fix the problems. If you're load testing two weeks before Black Friday, that's obviously not gonna give you enough time to fix more than the most trivial bugs.

 

You wanna start early enough to do that to fix the problems that you find with load testing. I put a couple points up here to highlight things that you should take into consideration when determining how early to start.

Daniel Freij:

Let me just add some to the context here. It's not usually load testing itself that will take up all the time. It's resolving everything that you find when you load test and then verifying that that is okay [on iterating 00:15:17] on that.

 

That is the long process, getting started with load testing can be made super easy and super quick. It's used [in a manner 00:15:25] like if you're not doing load testing, you can be up and running with load testing in 10 minutes, but it just means that you don't have to start everything with the biggest and most advanced testing suite ever. You don't need to test everything on your website straight away but the most important parts-

Troy Presley:

Yeah, critical path

 

I think we've all talked about that or we've all looked at that if you've done any unit or functional testing, you understand the critical paths or through your application and load testing can be approached the same way. 

 

If you're not gonna do full load test of every single function in your application, at least understand what your critical paths can do especially if you're coming up close to the holiday season that's a good thing to focus on.

 

When you're scheduling your load test how early you should start, things to keep take into consideration, make sure like we've been saying that you allow for several fix and test iterations.

 

You cannot assume that you're gonna fix everything in the first run, so understand that you need to schedule for multiple load tests with time in between each load test to actually fix the problems, and how many iterations is really dependent on the complexity of your application, but you do need to allow for several [length of 00:16:37] three to 10 depending on how complex.

Daniel Freij:

It might not even be the application that's complex, it might be your organization.

Troy Presley:

That's covered by the points underneath here. I do have understand your code lock date. If you have a lock down on your code, make sure that you take that into account. That should be a relatively obvious one but you wanna aim to be fixed. You don't wanna have to go get special exception to alter code passed your lock date. You want to be able to get everything ready for that, so account for it.

Daniel Freij:

Also make sure you have one.

Troy Presley:

Yes, if you don't have a code lock date, have one. A good target time for that is two weeks out from whatever you're expecting your peak load that's what I've seen as an industry standard but a minimum of a week out.

Daniel Freij:

Absolutely, that's minimum.

Troy Presley:

Then that gives you the ability to when you absolutely need you to go get that exception but to put a hard deadline on all of your development and infrastructure teams to be ready.

Daniel Freij:

Doing the duration load test one week before, example to verify the bugs, it might lead you doing and deploying it the day before Black Friday that you won't have time to load test.

Troy Presley:

That's exactly right. We've already talked about the complexity of the application to some degree, but I also want to give special focus to understand if you have external teams.

 

A lot of applications now depend on third parties or consultancies or some parts of their application. If that's the case with your application, make sure you understand that and make sure that that almost always will add time to affix [inaudible 00:18:05] resolution, the time, so understand that when you're scheduling for multiple iterations that there's going to be more time if you're having to deal with external teams.

Daniel Freij:

Yeah and also to invite them to be part of the load testing process if not only to read the results and observe, because it will increase each department's understanding.

Troy Presley:

We said early as good, I'd say always is better. When I say always is better what I mean is load testing is perfect if it's actually a part of your code deployment flow. Especially with most teams moving on to some form of continuous integration or continuous deployment solution, load testing tools like what we offer [the APICA 00:18:50] have the ability to integrate with almost anyone of these continuous integration, continuous deployment solutions or team, it is called Pipeline Jenkins. 

 

These solutions that you're using to push your software today, you can actually integrate this, the same way you do your functional test or your unit test, you can implement load test as a part of that and actually gate your deployments based on the performance you see. 

 

If you're doing that on a consistent basis, if you can work that into that flow then you understand your problems always. Now you know your problems way before you get up to any major customer influx event that you're gonna have whether that be holiday readiness or marketing push or anything else. 

Daniel Freij:

You can even treat it in such a way that you're on the load test daily or several times a day to get snapshots of how your application performs under normal loads. 

Troy Presley:

I put here on the second bullet point, but it's entirely possible to, during your normal code deployments, to run very small load test. Just running a consistent number of users on every push is gonna give you a performance profile, so even if you only did a 50 user or a hundred user load test, but you did it on every single code deploy, you're gonna have a consistent profile of how your application works. 

 

If you make a change to one of the supporting functions in your application and it has an impact on the performance, you're gonna see that instantly with even a hundred user load tests. If you're consistently running hundred user load tests every time you push, you're gonna see even a small change in that performance.

 

Then for the larger load tests, you can schedule those more, you don't have to do it on every code push, but schedule them once a quarter or schedule them once a month, whatever what makes sense for your organization, but again doing that consistently instead of just right before you're expecting load.

Daniel Freij:

Or once a year for checkbooks.

Troy Presley:

Yeah, exactly if it makes sense for your organization, even once a year scheduled well in advance of your holiday traffic is the right way to handle that rather than rushing and having to figure this out at the last second.

 

I just threw up this calendar as an example. It's really trying to point out that if you're looking at ... This is again the example of Black Friday but this could be any holiday or whatever holiday traffic is gonna be makes sense for your business. Using Black Friday as an example, you really need to be starting with your basic testing, I would say as early as January.

 

Getting started understanding thinking about a performance at that point is gonna take a way so many headaches when you get down there, but as you move closer, about halfway through the year, you're gonna start to really start to take this seriously. Like when you're talking in January to May, maybe you're just building a culture within your development team of performance, understanding that that is going to be an issue and not letting it be a surprise. 

Daniel Freij:

All it take is step back here. Look at this image and consider that there's a lot more holidays than just here.

Troy Presley:

Yes, of course. 

Daniel Freij:

That's where the whole continuous testing comes in here. You don't want to do this just once a year before Black Friday, [gonna 00:22:04] wanna to prepare before every major events that's gonna [vibe 00:22:07] your sales up to make sure that there's no risk to revenue.

Troy Presley:

That's exactly right.

 

Again, using the Black Friday example, I'm showing on this calendar, if you're already in September and haven't started to really do a regimented testing against your application to understand, you're really getting into the danger zone at that point. You're running out of time to fix any major issues. 

 

Imagine if you had infrastructure issues where it took actual forklift fix to correct your performance problem. If you find out about that in mid-October shooting for resolution by Black Friday, now you're in trouble and you won't know that without doing load test. 

Daniel Freij:

Something to maybe help you in the way here is that before Black Friday and if you're in the IT department, make sure you talk to the sales and marketing departments coz they might be running a really big campaign that you don't know about.

Troy Presley:

We're moving into the technical meat here. What kind of load test should you use? We understand that load tests are important. We understand that running them early is good, running them always is better, but what load test should you run? 

 

There's lots and lots of types of load tests out there more ... Load tests are not a single concept they are actually many different types. I'm gonna focus on three that are most critical for most organizations and especially for holiday readiness.

Daniel Freij:

Absolutely.

Troy Presley:

Those are stress, concurrency, and disaster recovery. What do these mean?

 

We'll start with stress test.

Daniel Freij:

Stress test is the core of a way to find out how many transactions of a certain type can do before response time degrades? What's the peak number? That's gonna give you some performance information about your application but it won't tell all the limits.

 

What we'll do is highlight a lot of pure transaction per second bottlenecks. It might be that, "Oh, you have under utilized web servers but your database is [spicy 00:24:15]".

Troy Presley:

Typical stress test, it's not a clean line between the different types of load test, but stress test typically are focused on a single function or a single use case in understanding how about work. Stress testing one particular for instance on an E-commerce site the cart and that would be absolutely a critical path for a E-commerce site coming up on holidays, how does my cart perform and that would be the focus for a stress test. 

 

Concurrency test on the other hand are more practical in terms of understanding practical limits based on realistic traffic. The goal here is not just testing the cart but also testing the category page also testing the marketing pages, also testing that creating several scripts that blend together like an orchestra to represent real traffic that would get your site. 

Daniel Freij:

This is usually where they doubt in the Google Analytics for several, several hours that they see them. It creates some load model. Because concurrency testing is all about simulating real life, so if you have the 200,000 concurrent users lost Black Friday, then you need to simulate at least 200,000 concurrent users this time, unless marketing is the main awesome job, it might be 400,000, but you still wanna simulate the usage as far as you understand your customers will do it.

 

If they're gonna click on four public pages before entering the cart with the public that needs to be added as well. It can be as simple as you have one cart transaction flow and you wanna do X transactions a minute at least and you can just have a single page load test that just refreshes a public page. 

Troy Presley:

That's exactly right.

 

Once you have this mix, once you understand what your realistic traffic looks like and you've started, you're running tests with those, this concurrency test also allow you to do what ifs scenarios.

 

What if the percent, the conversion from category pages to a cart pages increases? Right now, maybe it's 10% and it goes up to 15% because you've [re-flowed 00:26:29] your site and now you've got more people putting things in cart. This allows you to do those scenarios. You can alter the mix of the scenarios together that you're running and understand how your application is gonna perform [one of 00:26:40] those new what-ifs scenarios.

Daniel Freij:

This is also a great test to use when you're getting your auto-scale environment up and running to see how long does it take to start these instances [inaudible 00:26:53] continuously increase the load as you're going on to make sure that your instances that are auto-scaling have the right thresholds to start auto-scaling. 

 

How long does it take until we spent enough boxes to meet the new demand of a 10,000 users extra? Will those users [may abandon 00:27:15] the site because it took too long?

Troy Presley:

Yeah, that's exactly right.

 

Finally, the last one I wanted to focus on was disaster recovery test.

Daniel Freij:

This is a fun one.

Troy Presley:

Disaster recovery test, the purpose of a disaster recovery test is we understand that it's always possible that more users come in than you've anticipated, more users than you've had time to really get your infrastructure ready for.

 

What's going to happen under those scenarios or what if something in your infrastructure breaks? You have the correct number of users that you've anticipated but something goes wrong in your infrastructure and you wanna understand how your automated failover systems work or how your manual processes work, a disaster recovery.

 

The way that works is we create a consistent amount of traffic instead of ramping it up or increasing the load, we just send a static amount of traffic consistently over a long period of time and then you would go in and simulate failures in your own environment.

Daniel Freij:

For example a power failure in data 72. Will the traffics fix you over? It can also be, you gotta [hold 00:28:22] down by database and you wanna simulate that one taking over. How much is that gonna disrupt your site? When you actually know how much it's gonna disrupt your site, you can start taking actions to prevent it. 

 

Disaster recovery tells us all about playing around with environment and breaking things.

Troy Presley:

Yeah and it's not just testing your automated systems, it's testing your run books as well. Do your manual processes involve people? Do they work properly? Are they efficient? Is there any place to improve there?

 

A lot of companies overlook disaster recovery testing from a load testing standpoint, but it can be very, very helpful for preparing for events like Black Friday.

Daniel Freij:

Absolutely and if you're interested in knowing more about that, I would recommend that looking up some Netflix blogs and reading about Chaos Monkey which does exactly that [anonymous 00:29:14] environment, so you can do your load testing and have Chaos Monkey as it [wrecks 00:29: 18] all over the place to give your operations department enough information to start fixing things and documenting how to fix those things when they happen.

 

Troy Presley:

It's definitely one of the best named products out there.

 

Daniel Freij:

Absolutely, I love it.

 

Troy Presley:

Just to point it out, if you're doing a disaster recovery style test but you're not actually simulating any failures, we call that an endurance test and that can be useful just to see how does your application perform over a sustained amount of load for a long period of time. It may work perfectly at the beginning but after eight hours of the same amount of load, might start to see some wear and tear, might breakdown in places the you might not expect.

 

Daniel Freij:

This is usually where you'll find memory leaks.

Troy Presley:

Memory leaks are very, very common thing to find with, [interests us 00:30:02] for sure.

 

We've gotten through what type of load test you wanna run. In a lot of cases it maybe all three of those but it's gonna depend on your organization.

 

How do you create these tests? Again, you'll hear me say this a lot but load testing is all about question, so you'll see probably more questions in this presentation than statements. The reason for that is because again your load testing is about understanding the questions that you want answered and then the load test is there to answer those questions, but first you need to know what the questions are.

 

What type of questions should you start with? The most basic is what you want to know. Are you looking to understand your maximum concurrent users? Do you wanna know your peak? Do you want to know what third party content has? [Are you 00:30:47] able to impact that has? There's a lot of different things you can ask here.

Daniel Freij:

Yes, is the CDN working? What is the response time from Luxembourg in a load test as compared to Atlanta? 

Troy Presley:

These are common questions but there are no right or wrong questions. It's really dependent on what you wanna know. This is a good set to start with but there's plenty of other types of questions you may wanna know the answer to.

Daniel Freij:

For example, how many log ins can I do? 

Troy Presley:

Yeah, exactly.

 

Once you know that then you also are gonna want to put some thought into what requests matter. When you're creating a load test, not all requests that happened in your application are going to matter all the time.

Daniel Freij:

The perfect example about this is CDNs for example. You should have an agreement [assessment 00:31:36], you're not allowed to load test for example. Also it's not really testing your site, it's more of a verification thing that you maybe do once after delivery of the service to verify its meeting expected goals.

 

In terms of the application, that's just a big black hole. The only thing it does, it throttles you a bit. Loading all the images so long, the only thing you should think about in terms of when you load test unless you have a way of setup. It's that if you remove all the static images, because they're not gonna touch your server anyway, is make sure that you add enough post type to account for those action being downloaded.

Troy Presley:

If you're simulating real users, real users don't do things milliseconds of time. They have think time in there so understanding how to set that up properly is important to simulate real users.

Daniel Freij:

Third-party is also usually a big scare, because you don't really wanna load this Google or Facebook. Facebook might be a bit picky about it. The load testing Google is done by us all every second of the day almost.

Troy Presley:

Again, I wanna point out there are no right or wrong answers here. This is really just questions that you should think about when you're designing your load test because some organizations are gonna have some requirements and other organizations are gonna have different requirements.

Daniel Freij:

It's absolutely a good idea to test your CDN as long as you have an okay for it, as long as you know what questions you want answered with it.

Troy Presley:

Right, you need to understand would you load test with third-party content included. If you load test with CDN content included, understand what the results from those tests mean and understand what it means if you don't include those requests and make sure that those tests are answering the questions that you want the answers to. 

 

Next thing you wanna know, do you have any unique or dynamic data that needs to be included?

Daniel Freij:

Yeah so like if you're gonna record just a static test, pretty much anyone here at the [peak 00:33:45] it can do that in a minute, and some friends and my sister.

Troy Presley:

If you have an application that you're trying to load test that requires a log in, then you need to understand things like can the same user log in multiple times because if not your load test is gonna have to pull usernames and passwords from an external source.

 

Does it cost your database to act differently if you're constantly hitting the same product page or do you need to spread it out over multiple products and therefore need to have a random product ID in your load test?

Daniel Freij:

For example if you're logging in with the same user with 10,000 threats that's most likely gonna look up some tables.

Troy Presley:

In any good load testing tool including APICAs is gonna allow you to pull an external data so you need to understand where that's required, if it's required, and what type of data you're gonna need. 

Daniel Freij:

It can be simple things like just handling a bait to make sure that the request will function. It can be versioning of the JavaScript [inaudible 00:34:48], if you wanna have them in your test, so you don't run the test week after you create the test while you get a bunch of 404 and [send them to 00:34:55].

Troy Presley:

That's exactly right.

 

Then finally what constitutes a failure? Load test scripts do not ... They don't know your application so they're not gonna know what a failure is out the box. You have to tell them when something is not what you expected.

Daniel Freij:

You can even look at it in the totally opposite way, what constitutes a success. When you're running your test, you also have to verify that if you add something to the cart, then you load the cart, verify that what you've added is actually present in the cart that's the success criteria.

 

If it's not, they are to fail it. That way you will know without, "Okay, it's failed when I added it to the cart," rather than you're being in the [shuckle stat 00:35:41] and like, "Oh, I can buy it because I have no public [added 00:35:44]."

 

Verifying the successes of all requests is a really good idea not only from a test perspective but also from a debug perspective because you wanna be able to back track and see where did the test failed, where did the application failed. 

Troy Presley:

Almost as important as the actual script and the user scenario that you're running through is understanding the successes and failures and building that into the script. 

Daniel Freij:

I would say you it's 50% of the test at least.

Troy Presley:

Once you've got your set of questions, it's not always gonna be exactly like this but this is a general flow of what it takes to get a script recorded. Determining the user flow is the scenario that your use is gonna walkthrough. You could have many, many scripts so you're gonna have one user flow for each of those, one user flow for each of the scripts that you're gonna record.

Daniel Freij:

I usually ask for a marketing contact here that knows the Google Analytics.

Troy Presley:

This could also be SmartPhone applications as well. This is not all web pages, so if you have a SmartPhone application that you're gonna load test against, understand the click flow through the application in which mobile devices you wanna have recordings for, understanding all of that before you get started is key. 

Daniel Freij:

Because you can do through a proxy, it also means that you're not limited to a phone or a computer. You can plug in the device like a TV and record the traffic [dot though 00:37:18] sometimes it starts up for example or simulate an updating of cable box maybe not as relevant as-

Troy Presley:

It's starting to become more relevant. There's not a whole lot of setup box shopping going on, but it is starting to become more and more relevant, so things to think about there.

 

Second step is to record the script. There's a lot of different ways you can do this. I put using a proxy recorder which is the industry standard for load testing. There are other ways that it can be done.

Daniel Freij:

Hard files for example.

Troy Presley:

Yeah, hard files can be done. If you're using a web browser, you can export a hard file after having clicked around and there's a lot of different ways.

 

One of the easiest and time tested ways is to use a proxy recorder where if it's a mobile device or a browser or any other type of network traffic creator, you proxy the traffic through a recording tool, you record the script that way as you do the user flow, so you would just click around as your user would do based on the scenarios you came up with in step one. Then that's gonna produce a recording of all the network traffic that went back and forth during that user session.

 

Once you have that, the next step is to remove the unnecessary requests.

Daniel Freij:

Your Facebook requests, your analytic stuff because you don't wanna inflate your metrics and have sales departments and marketing departments of the champagne.

Troy Presley:

You remove the requests based on like we've talked about in the previous slide and then you're gonna handle your variables. 

 

Variables are about the dynamic data. When we talked about that if you needed multiple usernames and passwords this is where you would handle that. This could also be non-user data. This could be session data that's passed back and forth that needs to be handled or anything in the script that is gonna change on a play-by-play basis so you're gonna set that up. 

 

Then you're gonna handle the external data which is gonna be the files that provide that data.

Daniel Freij:

You can view this as all external data those things like if it was [narrated 00:39:12] in the JavaScript and you're not a browser, it may come from a server, so the testing that you should know where to get this value from. It's the same if you write in a search field that's a user input. That means that also it came from user that [can be 00:39:30] extract the value from somebody.

 

You have to consider those things and account for them. Maybe add your top 10 search terms or top 100 if you have a lots of content.

Troy Presley:

Finally, you're gonna configure your failure detection. This is where you look at the recording that you have now and you go through, you decide which URL calls are important, which ones are not important, which ones need to have content verification.

Daniel Freij:

It's gonna be criteria like. I expect the 200 okay. It's supposed to be text HTML and I'm expecting to see a specific [based on 00:40:12] pattern in here and a token value.

Troy Presley:

Your developers who should be a key part of creating these scripts or at least consulting with the load test creators are gonna understand each of these criteria as you go through this, so that your developers are gonna be invaluable in understanding what the correct responses for this are. 

 

Hopefully, at this point you would have a script that you could use for your load testing or multiple scripts if you're gonna do concurrency testing. How do you run those tests?

 

There's a lot of services if you're using whatever tool that you use to create the script. It could be potentially the tool that you're gonna use to run the load test if you're doing them internally, but if you're a larger organization, you're eventually gonna hit the point where you gonna need more infrastructure than you can do with a simple tool. That's where you gonna have to make these determinations.

 

Are you gonna run with this as provider? APICA has a Cloud-based SaaS, a solution that you can use our infrastructure to do a very large load test or you can do your own private installations and creating the load test clusters yourself and run it that way.

Daniel Freij:

You can even do your own ZebraTester local installation app to connect that to the SaaS portal.

Troy Presley:

Yup.

Daniel Freij:

So that you have your 10k testing environment but really need to reach 100K, you're gonna invite a thousand more users, then you source them from the Cloud. 

Troy Presley:

We're at 42 minutes here so I'm gonna start to speed it up a little bit.

 

The other things you wanna look at is the target environment. Do you wanna run against in a Dev environment or DR environment or Prod environment? You're gonna wanna be as close to your production environment as you can possibly be if you're preparing for an event like Black Friday. We expect that you would wanna run if not against your production at least against an environment as close to production in terms of the infrastructures you can get. 

Daniel Freij:

It not only means your environment, it also means the testing environment needs to be as close as possible. If you expect traffic from all over the US, you're gonna test from as many locations in the US as you can, not only east coast.

Troy Presley:

Yeah, that's exactly right.

 

You also want to consider whether or not you want to pull an information from other systems like APM solutions, for example, which would be like a Dynatrace or AppDynamics.

Daniel Freij:

New Relic.

Troy Presley:

These things which would give you the server perspective that you can correlate with the information that's coming from your load test, it gives you a better picture of what's going on from a full [crosstalk 00:42:44]. 

Daniel Freij:

We actually integrated all of them.

Troy Presley:

Database preparation refers to if you're using dynamic data, if you have a script that's going to register users, you may have to clear out users from previous test in order to be able to re-register them because if you try and register twice, you may have a problem and that's just an example but you may need to prepare your database for the test.

Daniel Freij:

That also has a second reason. You can almost describe [it is 00:43:09] impromptu mechanics. Whatever you observe is gonna change. If I do stuff in my test against the database while it's gonna grow that's gonna affect what results you get because your data's gonna become more and more fragment to them until your next database and so on. 

Troy Presley:

This is really not a technology thing, but it's almost critical anyways is that, how does your team communicate while you're running load test? When you're running load test, your application is gonna be made up of separate teams probably.

 

At the very least you're gonna have your infrastructure team and your development team. There could potentially be dozens of other teams that your application touches and may need to be in contact when the load test runs in order to be able to fix these things. Know what your communication lines are, know who's in charge of each function, and know how to get hold of them.

 

If it's a critical load test, potentially have them on a Phone Bridge as you do the load test or at least on a slot channel or something just to [how 00:44:07] know what your communication channels are, know who needs to be involved, and know how to get a hold of them.

Daniel Freij:

You can even [break the 00:44:15] application and have operations treat it as a training exercise.

Troy Presley:

Finally, criteria for aborting. Especially if you're going against production environments or anything that's gonna have any impact on your production environments, know what criteria under which you're gonna abort this test and know that before you get into the test. 

 

Especially, again, if you're going against the production environment, you're probably running off hours but you do have actual customers so you do wanna know that if under these conditions and it's gonna be different for every application and different for every team, but you do wanna know that if this happens or we reach this level or these alarms go off that we are going to hit the abort button and stop that that test.

Daniel Freij:

Speaking about abort criteria, make sure you always have a success criteria so you know when you don't.

Troy Presley:

Especially for endurance or long running tests. It's good to know we already know that we have succeeded on this test. We don't need to continue for the next four hours.

 

Interpreting results, if you've run this in either a SaaS provider or locally on your environment, you're gonna get a whole bunch of data back from those tests and you need to know what does this mean.

 

The primary output from load testers, a lot of information but the primary surface level stuff that you need to understand on the most basic level is load curves.

 

Load curves are always going to be a number of users on the X axis and some metric on the Y axis. I've highlighted three that I think are some of the most critical load curves that we'd look at and that would be the session duration, how long each session lasts as you're going through the load test when you're simulating a user, the network throughput, which is just simply how much network traffic is going through, and the number of errors that have happened or failures that have happened during the load test.

 

This is actually showing ideal results. Ideal results are if you had this theoretical perfect application that never had any impact no matter how many users you threw at it. Your session duration would never increase so it will be a flat line straight across. Your network throughput would go up perfectly diagonally with the number of users and would increase in perfect ratio with the number of users and you'd have zero failures.

 

We all know that there is no such thing as perfect application so what your real world results look like?

 

Something like this.

Daniel Freij:

I would actually freak out more seeing the previous curves than these.

Troy Presley:

Actually, I'll point this out in a second but if you have the perfect load curves that just means you haven't gone high enough in the number of users.

Daniel Freij:

Your environment is not really alive. 

Troy Presley:

With the realistic application, what you're gonna look at is session duration, at some point it's gonna start to go up. Once you've hit whatever bottleneck in your application that hasn't been fixed yet, you're gonna start to see a slow down in your application which means that each individual session is gonna increase in time.

Daniel Freij:

There can be cases where session duration is flat out. You get a bunch of errors but- 

Troy Presley:

Throughput is gonna flatten out as well because as the application is not going as fast, and another traffic going at any particular amount of time is gonna slow down. Finally, the number of failures and errors you're eventually gonna hit a point where failures and errors happen.

Daniel Freij:

Good to know about the failures and errors is that, and this is common in all tools not just our tools, you're gonna hit certain points where you get so many errors that just abort straight away rather than wait for the results because you're not really gonna see anything because of all the error messages.

Troy Presley:

I've thrown up the safe, peak, and unstable zones on these graphs. What Daniel was just talking about if you look at the first graph with session duration that's the reason that curve originally starts to go up when performance degrades, but then actually goes back down again is because when the application starts to fail, like when you look at the red section of the failures and errors graph where that's gone up, failures typically happen much, much faster than real requests. 

 

You'll see that the performance actually gets looks like on the graph that's getting better but the reason it's getting better is because you're having a bunch of errors that are happening [inaudible 00:48:26].

Daniel Freij:

I don't wanna make a distinction. When we say that faster, like errors get to be used faster, it's true. You get the error much, much quicker, but they're more costly for CPU because error message you cannot trigger some log in function, maybe a trace of some kind being sent out. You really wanna avoid these cases where all of a sudden, [they see 00:48:52] as the bunch of errors because this can be really expensive server wise not just [wrecked it use 00:49:00] experience.

 

If you go back actually to one more slide, as you load test your application more, it's like when you look at slides or graphs like these, it's good to know that you will start seeing patterns and those patterns are just gonna tell much more than the results sometimes.

Troy Presley:

That's exactly right.

 

Finally I just wanna talk very quickly about what environments you can test and this is really talking about the different stages, different development staging, pre-production and production environments. There's reasons to test against each of this. Dev environments are great for those type of concurrency bugs and database bugs that can happen with multiple users that don't happen with single users. Dev environments are great place to do very small tests.

 

Staging and QA environments are great as a point where multiple Dev teams roll their code together and being able to catch those bugs before they go out towards production using load test to do again multiple user tests to find concurrency bugs and database bugs from multiple roll [togethers 00:50:10] from multiple Dev teams. 

Daniel Freij:

With ZebraTester for example you can have a locally ZebraTester load generator.

 

Just do a 50 user load test which is the free trial and that's usually good for the local Dev environment test.

Troy Presley:

Then Pre-Production environments, this is really talking about if you have the ability, especially if you're running primarily on a Cloud provider it should be very easy to do to produce the exact same infrastructure you use for your production environment with the same sites of instances, the same size databases, these kinds of things. Running a load test against that is gonna give you a perfect baseline of what it's gonna look like in production without impacting your actual users.

 

Daniel and I actually disagree on the [site 00:50:55] but the perfect place to run load test is Pre-Production. Daniel actually leans more towards running against production.

Daniel Freij:

Both.

Troy Presley:

We're both right I think. In production the reason to run there is that is actually the environment you're running on.

Daniel Freij:

It's more of a readiness test rather than we should not run only the load testing in production that's gonna be not only expensive but introduce risks, but you should always have verified your production environment.

Troy Presley:

I think that points out it again every organization is gonna be different and you need to understand where you are as an organization and then make a decision about whether your large scale load testing happens against a pre-production environment or your actual production environment.

 

You can do both just to understand the conditions under which you would do it against one versus the other. 

 

That's the end of the presentation we had together. I wanted to touch on one last thing that wasn't in the slides, which is this presentation is now happening about a month out from Black Friday. Earlier in the presentation we've talked about that really the best time to get started is several months before to give you iteration time.

 

I did wanna point out that it is still not too late to start. The perfect time to start is much earlier but starting now is still valuable.

Daniel Freij:

Still better than not doing it at all.

Troy Presley:

It's still better than not doing it at all. Being ready, at the very least with the month out, most of the organizations will have at least enough time for one iterative maybe even two iterative cycles for their code fixed.

 

Even if you didn't, if you're a larger organization and couldn't even get one iterative cycle, at least you're gonna understand what your limits are so you know when to start worrying, you also are gonna know where you're likely to break and be able to deploy your teams appropriately, so it's still has a lot of value even this one month out.

Daniel Freij:

Best case it's just configuration issues. You should have solved in a day or two, best case.

Troy Presley:

That being said, that is the end of the official presentation here. We'd love to answer any questions that you have.

Speaker 3:

Yeah so thank you. We do have a couple of questions here again. I know it's been a number of questions coming in. We will be sending out that infographic that was on one of the slides.

 

If you guys have any further questions after the webinar, Troy and Daniel did their contact information on that first slide. We will show you right now and then [here 00:53:24] you go. 

 

Yeah, we look forward to see you next time.

 

Firstly, I've got Matt who asks what's the best way to record how visitors use our website?

Daniel Freij:

I would say Google Analytics is the tool of preference here maybe.

Troy Presley:

There are several different ways Google Analytics is amazing for visualizing. It has a lot of visual tools that will show you the percentage flows through your site. I highly recommend it for that purpose. If you're not using Google Analytics and if you have technical capability to go through your logs, you can do it with your logs and there are tools out there to do a log analysis.

Daniel Freij:

Make use of your test to some product elements as much as possible because the product owner will know a lot of information about your application and your testers will usually know what the primary flows are and can easily replicate these quickly for you. At that point of time, it's just a matter of maybe passing a browser via proxy or have [there and 00:54:26] save on a hard file for you.

Troy Presley:

Yeah, that's exactly right.

 

There's a lot of different ways to approach it is to grab hard files and can do analysis that way. There's a lot of services out there that do hit map and on very sophisticated ways of tracking how users use your application including the timing between different pages. I found, and I think Daniel agrees with me, that for the level of understanding that you need to produce a good load test, Google Analytics really hits [after the 00:54:56] it's most of the check marks. 

Daniel Freij:

But that's usually the first things I ask for when asked to build a [load level 00:55:00]. 

Speaker 3:

Then I've got [Hearty Dan 00:55:04] here who asks how can we record the scenario for post request with input headers?

Daniel Freij:

I recognize that question.

Speaker 3:

I tried adding the headers but unable to find a place to add the input headers.

Daniel Freij:

If you can reach out to me on the community, we can do a quick session on [technical edit 00:55:31], but this is more of I need to understand what you're trying to do exactly.

Troy Presley:

This is a little bit more technical and technically specific so we're happy to help. You can go to the community or e-mail either myself or Daniel. The e-mail address is on the screen and we're really happy to take that.

Daniel Freij:

I may post on the ZebraTester community. We'll pretty much light up my phone, email, and chat.

Speaker 3:

[Hearty Dan 00:55:53] we'll return to you after this webinar and get your questions answered.

 

One last question, I've got from [Carly 00:56:00]. Can APICA integrate with Jenkins for performance testing?

Troy Presley:

Yes. 

Daniel Freij:

Yes. We integrate with Jenkins for [and a deploy 00:56:08] so you can add a script Jenkins that executes a load test or API and then [boost 00:56:14] down the results and allows you to configure success rate area. The same can be done in team CD, AWS code Pipeline and with Visual Studio, I think.

Troy Presley:

Yes, visual studio.

Daniel Freij:

It's just as simple as if you have now the tool you can actually reuse the Jenkins script for this.

Speaker 3:

Awesome so that was a longer webinar for us, all of you are still here, so we hope you enjoyed it [certainly 00:56:42]. We'll see you next time and thank you again Daniel and Troy for taking time out of your [inaudible 00:00:56] this day. See you guys next time.

Daniel Freij:

Thank you.

Troy Presley:

Thank you.