Get the right load mix out of a few numbers

When testing ecommerce applications on SaaS environments, you often do not get enough numbers from clients because they simply do not know these numbers or only a few. One reason for that is, that the client simply have not had any only presence before. Often the client also does not have detailed numbers, because the previous hoster or the IT department just holds them back or simply cannot get to these numbers.

So what to do, when you do not know every detail about the current or future load pattern? We are describing one approach below that was very successful so far and always yielded satisfying results.

What we need

  • Visits per peak hour (example 10k)
  • Page views per peak hour (example 100k)
  • Orders per peak hour (example 200 orders)
  • Optionally we can use the conversion rate to get from visits to orders or vice versa.
  • Optionally we can take searches, “add to cart” operations, user registrations, and so on into account.

The mentioned scenarios are typical ecommerce scenarios and look like that. We will not talk about smaller scenarios such as address editing for a registered user.

  • TSingleClickVisit: Enters the store only, does not move beyond the start page
  • TBrowsing: TVisitor plus category and product browsing
  • TSearch: TVisitor plus keyword search plus browsing of the result
  • TAdd2Cart: TBrowsing plus add to cart operations
  • TGuestCheckout: TAdd2Cart plus checkout without an order placement (anonymous user)
  • TGuestOrder: TAdd2Cart plus full checkout (anonymous user)
  • TRegisteredCheckout: TAdd2Cart plus checkout without an order placement (registered customer)
  • TRegisteredOrder: TAdd2Cart plus full checkout (registered customer)
  • TRegistration: Account creation

What we assume

Ecommerce sites follow similar patterns and with a few exceptions, such as special promotions, certain behavioral patterns are nearly identical. So for instance, about 50% of all checkouts are stopped before the order is placed. About 20 to 50% of all created carts aren’t checked out at all.

What we calculate

Based on these assumptions, we put together a fairly simple but sufficiently accurate load mix. Of course, we can also analyze the current log files and try to come up with something more precise, but that will be a snapshot only. Traffic is very volatile and so we should be very generous when setting up this mix.

Since we do not take any daily averages as base but the peaks, we will have a pretty comfortable buffer for our daily ecommerce life anyway.

Bottom-Up

Let’s say, 200 orders are set as goal. Splitting them 50/50  between registered and anonymous users, we get 100 visits of each type. All numbers are per hour of course.

  • TGuestOrder = 100
  • TRegisteredOrder = 100

As a next step, we take our 50% checkout abandonment rate into account. We have 200 checkouts per hour that are stopped and 200 that run through and turn up as orders (as counted previously). So we need to add 200 visits. And because these visitors can either run with their preset account or without, we split them up in 100 guest and 100 registered checkout attempts.

  • TGuestCheckout = 100
  • TRegisteredCheckout = 100
  • TGuestOrder = 100
  • TRegisteredOrder = 100

This gives us 400 visits per hour that go into the checkout. We now assume a low cart to checkout conversion rate, about 20% for instance, and so we take 400 checkout visits * 5 and get 2,000 visits that involve cart usage. Since we already have 20% converted into checkouts, we have 2,000 minus 400 visits that use the cart.

  • TAdd2Cart = 1,600
  • TGuestCheckout = 100
  • TRegisteredCheckout = 100
  • TGuestOrder = 100
  • TRegisteredOrder = 100

We also know that many users do not continue after hitting the home page or any landing page. Let’s add some of these users now.

  • TSingleClickVisitor = 1,000
  • TAdd2Cart = 1,600
  • TGuestCheckout = 100
  • TRegisteredCheckout = 100
  • TGuestOrder = 100
  • TRegisteredOrder = 100

But wait, what are we missing? Well, we have not registered any new accounts yet. Didn’t we? We did, because the registered checkout creates accounts if required and reuses them several times. But to get a more substantial customer growth, we simply add 200 visits that run registrations:

  • TRegistration = 200
  • TSingleClickVisitor = 1,000
  • TAdd2Cart = 1,600
  • TGuestCheckout = 100
  • TRegisteredCheckout = 100
  • TGuestOrder = 100
  • TRegisteredOrder = 100

What is left to do? Well, we do not have any “I am just looking around”-visitors yet. We know that our total visit count is 10,000 and we already assigned 3,200 of these to cart, checkout, and registration, so we have 6,800 visits left we can now use for something else. Depending on the shop type (large store, small store etc), people tend to use search more or less. To put enough stress on search and refinements, we simply assume 50% of all people like to search. Thus the missing 6,800 visits will be 3,400 catalog browser visits and 3,400 visits with usage of search before browsing the search result.

The total mix is:

  • TBrowsing = 3,400
  • TSearch = 3,400
  • TRegistration = 200
  • TSingleClickVisitor = 1000
  • TAdd2Cart = 1,600
  • TGuestCheckout = 100
  • TRegisteredCheckout = 100
  • TGuestOrder = 100
  • TRegisteredOrder = 100

Wait… where are my concurrent users? This is simple: “concurrent users” is an inaccurate way of describing traffic, so we have not used that number yet. Why is that?

To get to the bottom of that, we simply check how long a visit takes. Depending on the shop, an average visit might take 2 to 4 minutes. Successfully shopping might take 15 minutes. If we expect about 10 page views per visit and a page view takes 1 second to load and 20 seconds to read it (already a really really high number for an average), a visit would take 10 * 1 second + 9 * 20 seconds = 190 seconds.

Let’s go with the 190 seconds for a visit on average. If we just could serve one visitor at a time, we could serve 60 minutes (3600 seconds) / 190 seconds per visits = 19 visitors per hour. But because we would like to serve 10,000 per hour, we have to deal with 10,000 / 19 = 526 visitors at the same time. This is the famous concurrent user number.

If we now double the think time, we have 1,052 concurrent users/visitors. If we cut it down to 1 second think time, we will get a visit length of 19 seconds and therefore 10,000 visits / (3600 seconds / 19) = 53 concurrent visitors.

So we already have three different “concurrent user” numbers and are still simulating the same traffic. This shows that the number of concurrent users is a pretty questionable way of describing traffic.

It does not matter which number we take, because most of the time the servers will see the same traffic. Because we run against a SaaS environment that serves a multiple of other customers at the same time and is sized to serve the peak traffic for all customers at the same time, we have plenty of comfortable room around us. This permits us to run with 53 concurrent visitors for most of the testing. This will save us client hardware resources for the load generation. e.g. saves us money. We are basically only interested in the runtime of requests and not if the environment can handle that, because it can.

The goal of this test is to demonstrate that the implementation on the SaaS platform is efficient, not that the SaaS platform itself is fast and stable, because this is guaranteed by design and contract. Testing this would require way more traffic and generate huge costs, because the environment would suddenly no longer be a shared one but exclusively used for this testing purpose.

When finalizing the entire test and all tests turned out good, we are going to turn up the concurrent user count to 530 users and compare the result with the previous measurements. Just to satisfy the traditional test expectations.

Does that work for you?

Hope that gives you an idea how to come up with a nice user mix for testing without having too much data in the first place. Comments welcome.

Author:
Date: Tuesday, 7. June 2011 13:49
Trackback: Trackback-URL Categories: Performance, Testing, XLT
Article feed: RSS 2.0
Add a comment

4 comments

  1. 1

    Cool article. Thanks for sharing. I do have a question though: you have the concurrent users (either 526 or 53), but how do you calculate the tps rate to hit the pages? So where I’m still confused is the TPS rate those X users (all in the same time) will hit the system. Is it pageviews/visits (in seconds) – 100K/10K/3600=0.002TPS with X users (whatever the concurrent user count is)?

  2. 2

    Hi, thanks for your comment! Regarding your question, hopefully interpreting your question in the right way:
    Transactions per second are calculated depending on the meaning of “transaction”: A transaction as we see it is equivalent to a visit. In our example you would calculate 10000/3600. Others see a transaction equal to a page view – in that case it would be 100000/3600.
    Basically tps is not a key figure in our reports, both tps and concurrent users are merely a result of the defined load profile, which is defined by the number of users or performed scenarios per hour.
    Hope this helps to clarify your question.

  3. 3

    Thank you for your answer Andrea. What I’m really interested in is the load you are going to apply as a peak based on your given visit and page view information.
    So if we have the visits/page views/time spent on site we can calculate the concurrent user number (526 in your example). The questions I have:
    a) How do you split the number of users if you want to divide the tests in separate scenarios (X users TBrowsing, Y users TSearch … and so on), doing the percentage (X=179users)?
    b) If you want to simulate the peak load for A you need the TPS rate. In this case you want to hit the server totally with page views/3600 (TBrowsing, TSearch and all the others should be hit with 100k/3600)?

  4. 4

    We are using a feedback based model here (arrival rate model, see http://blog.xceptance.com/2012/11/27/understand-and-choose-a-load-test-model/). Key point: You don´t define a fixed number of users or page view in advance, but create a mix of typical user scenarios covering realistic page flows.

    So what we do:
    1. Define the scenarios.
    2. Put some randomness into the scenarios (not each real visit has the same number of page views).
    3. Define the number of visits per hour for each scenario like shown in the example (e.g. TBrowsing 3,400, TGuestOrder 100) – these are the target numbers we want to reach.
    4. Define a maximum number of concurrent users per scenario (which are not necessarily used if not needed to reach the target arrival rates).
    5. Run the test.

    Result:
    As we run a feedback based test the number of concurrent users and the transactions per second are not input parameters, but a result of the test run depending on response times of the site. You will be able to get these numbers from the created reports.

    So regarding your questions:
    a) Actually we don´t have to split the number of users for each scenario. We only define the number per scenario and hour. The number of users will be determined during the test run.
    b) For simulating a peak load we will increase the arrival rate (number of performed visits per hour). It is not necessary to calculate in detail the number of users or tps.

    It would be a different approach to use the user count model. Here there number of concurrent users is an input parameter. But even here you don´t define tps in advance but get them as a result depending on the (random) number of page views that are included within your scenarios.

    Conclusion: I think it´s a matter of the chosen approach: Run static tests with defined user and transaction parameters, or use a more complex one which simulates real user scenarios in which the number of performed visits is the main input parameter and concurrent users and tps are output ones.

Add comment