Improvements

• Script Developer: Breakpoints can be set at a module call now. They have been moved to the first command of the module before.
• Runtime Engine: WebDriver has been updated to version 2.16.1.
• Script Developer: Now it supports new HTML 5 input elements, such as email and date, during record and playback.

Fixes

• Runtime Engine: When running multiple test cases at once from Eclipse or with Ant, the values measured for each test case were always stored in the timers.csv file of the first test case. This was a development mode issue only.
• Master controller: Agent controller urls are now checked to ensure that they are unique.
• Script Developer: The base url text box was not correctly updated when the url was changed via the Edit Dialog before.
• Script Developer: WaitFor commands included a small pause before checking the condition even though the condition was already fulfilled. This pause has been removed.
• Script Developer: Using a mouse wheel for scrolling now works in Firefox 9.
• Runtime Engine: When getting the value of an option element, the HtmlUnitDriver does not fall back to the option’s text content in case the option does not specify a value attribute. This behavior is mandatory according to the HTML specification. Fixed in the XltDriver until a fix is provided by the underlying project.
• Runtime Engine: For input elements of type hidden, HtmlUnit’s API method isDisplayed() returned true although hidden inputs are never visible to the user.

All details and the full download can be found here: https://lab.xceptance.de/releases/xlt/4.1.7/

All details and the full download can be found here: https://lab.xceptance.de/releases/xlt/4.1.6/

If you read and like the above information, you should not miss the OWASP web security guidelines. This is a must read for every tester and developer. OWASP Guide Project:

Web application security is an essential component of any successful project, whether open source PHP applications, web services such as straight through processing, or proprietary business web sites. Hosters (rightly) shun insecure code, and users shun insecure services that lead to fraud. The aim of this Development Guide is to allow businesses, developers, designers and solution architects to produce secure web applications. If done from the earliest stages, secure applications cost about the same to develop as insecure applications, but are far more cost effective in the long run.

Unlike other forms of security (such as firewalls and secure lockdowns), web applications have the ability to make a skilled attacker rich, or make the life of a victim a complete misery. At this highest level of the OSI software map, traditional firewalls and other controls simply do not help. The application itself must be self-defending. The Development Guide can help you get there. The Development Guide has been written to cover all forms of web application security issues, from old hoary chestnuts such as SQL Injection, through modern concerns such as AJAX, phishing, credit card handling, session fixation, cross-site request forgeries, compliance, and privacy issues…

Good news: very powerful free testing tools are available for Web designers today. Some are more user-friendly than others, and some have significantly better user interfaces. Don’t expect much (if any) support with these tools. But if you’d rather not spend extra money on testing, some great options are here as well.

Read the full article…

By the way, our own tool Xceptance LoadTest (XLT) offers a way to run cross-browser functional tests. XLT leverages WebDriver, a multi-browser API for automation. WebDriver does not support all browser and does not equally support all browser well, but we tried to iron out as much as possible. On top of it, you can use the XLT Script Developer to easily create automation scripts and run them either using our own scripting language or export them to Java to directly run them on the WebDriver-API.

You can download Xceptance LoadTest for free with no strings attached from our web site: www.xceptance-loadtest.com.

So what to do, when you do not know every detail about the current or future load pattern? We are describing one approach below that was very successful so far and always yielded satisfying results.

What we need

• Visits per peak hour (example 10k)
• Page views per peak hour (example 100k)
• Orders per peak hour (example 200 orders)
• Optionally we can use the conversion rate to get from visits to orders or vice versa.
• Optionally we can take searches, “add to cart” operations, user registrations, and so on into account.

The mentioned scenarios are typical ecommerce scenarios and look like that. We will not talk about smaller scenarios such as address editing for a registered user.

• TSingleClickVisit: Enters the store only, does not move beyond the start page
• TBrowsing: TVisitor plus category and product browsing
• TSearch: TVisitor plus keyword search plus browsing of the result
• TAdd2Cart: TBrowsing plus add to cart operations
• TGuestCheckout: TAdd2Cart plus checkout without an order placement (anonymous user)
• TGuestOrder: TAdd2Cart plus full checkout (anonymous user)
• TRegisteredCheckout: TAdd2Cart plus checkout without an order placement (registered customer)
• TRegisteredOrder: TAdd2Cart plus full checkout (registered customer)
• TRegistration: Account creation

What we assume

Ecommerce sites follow similar patterns and with a few exceptions, such as special promotions, certain behavioral patterns are nearly identical. So for instance, about 50% of all checkouts are stopped before the order is placed. About 20 to 50% of all created carts aren’t checked out at all.

What we calculate

Based on these assumptions, we put together a fairly simple but sufficiently accurate load mix. Of course, we can also analyze the current log files and try to come up with something more precise, but that will be a snapshot only. Traffic is very volatile and so we should be very generous when setting up this mix.

Since we do not take any daily averages as base but the peaks, we will have a pretty comfortable buffer for our daily ecommerce life anyway.

Bottom-Up

Let’s say, 200 orders are set as goal. Splitting them 50/50  between registered and anonymous users, we get 100 visits of each type. All numbers are per hour of course.

• TGuestOrder = 100
• TRegisteredOrder = 100

As a next step, we take our 50% checkout abandonment rate into account. We have 200 checkouts per hour that are stopped and 200 that run through and turn up as orders (as counted previously). So we need to add 200 visits. And because these visitors can either run with their preset account or without, we split them up in 100 guest and 100 registered checkout attempts.

• TGuestCheckout = 100
• TRegisteredCheckout = 100
  
• TGuestOrder = 100
• TRegisteredOrder = 100

This gives us 400 visits per hour that go into the checkout. We now assume a low cart to checkout conversion rate, about 20% for instance, and so we take 400 checkout visits * 5 and get 2,000 visits that involve cart usage. Since we already have 20% converted into checkouts, we have 2,000 minus 400 visits that use the cart.

• TAdd2Cart = 1,600
• TGuestCheckout = 100
• TRegisteredCheckout = 100
• TGuestOrder = 100
• TRegisteredOrder = 100

We also know that many users do not continue after hitting the home page or any landing page. Let’s add some of these users now.

• TSingleClickVisitor = 1,000
• TAdd2Cart = 1,600
• TGuestCheckout = 100
• TRegisteredCheckout = 100
• TGuestOrder = 100
• TRegisteredOrder = 100

But wait, what are we missing? Well, we have not registered any new accounts yet. Didn’t we? We did, because the registered checkout creates accounts if required and reuses them several times. But to get a more substantial customer growth, we simply add 200 visits that run registrations:

• TRegistration = 200
• TSingleClickVisitor = 1,000
• TAdd2Cart = 1,600
• TGuestCheckout = 100
• TRegisteredCheckout = 100
• TGuestOrder = 100
• TRegisteredOrder = 100

What is left to do? Well, we do not have any “I am just looking around”-visitors yet. We know that our total visit count is 10,000 and we already assigned 3,200 of these to cart, checkout, and registration, so we have 6,800 visits left we can now use for something else. Depending on the shop type (large store, small store etc), people tend to use search more or less. To put enough stress on search and refinements, we simply assume 50% of all people like to search. Thus the missing 6,800 visits will be 3,400 catalog browser visits and 3,400 visits with usage of search before browsing the search result.

The total mix is:

• TBrowsing = 3,400
• TSearch = 3,400
• TRegistration = 200
• TSingleClickVisitor = 1000
• TAdd2Cart = 1,600
• TGuestCheckout = 100
• TRegisteredCheckout = 100
• TGuestOrder = 100
• TRegisteredOrder = 100

Wait… where are my concurrent users? This is simple: “concurrent users” is an inaccurate way of describing traffic, so we have not used that number yet. Why is that?

To get to the bottom of that, we simply check how long a visit takes. Depending on the shop, an average visit might take 2 to 4 minutes. Successfully shopping might take 15 minutes. If we expect about 10 page views per visit and a page view takes 1 second to load and 20 seconds to read it (already a really really high number for an average), a visit would take 10 * 1 second + 9 * 20 seconds = 190 seconds.

Let’s go with the 190 seconds for a visit on average. If we just could serve one visitor at a time, we could serve 60 minutes (3600 seconds) / 190 seconds per visits = 19 visitors per hour. But because we would like to serve 10,000 per hour, we have to deal with 10,000 / 19 = 526 visitors at the same time. This is the famous concurrent user number.

If we now double the think time, we have 1,052 concurrent users/visitors. If we cut it down to 1 second think time, we will get a visit length of 19 seconds and therefore 10,000 visits / (3600 seconds / 19) = 53 concurrent visitors.

So we already have three different “concurrent user” numbers and are still simulating the same traffic. This shows that the number of concurrent users is a pretty questionable way of describing traffic.

It does not matter which number we take, because most of the time the servers will see the same traffic. Because we run against a SaaS environment that serves a multiple of other customers at the same time and is sized to serve the peak traffic for all customers at the same time, we have plenty of comfortable room around us. This permits us to run with 53 concurrent visitors for most of the testing. This will save us client hardware resources for the load generation. e.g. saves us money. We are basically only interested in the runtime of requests and not if the environment can handle that, because it can.

The goal of this test is to demonstrate that the implementation on the SaaS platform is efficient, not that the SaaS platform itself is fast and stable, because this is guaranteed by design and contract. Testing this would require way more traffic and generate huge costs, because the environment would suddenly no longer be a shared one but exclusively used for this testing purpose.

When finalizing the entire test and all tests turned out good, we are going to turn up the concurrent user count to 530 users and compare the result with the previous measurements. Just to satisfy the traditional test expectations.

Does that work for you?

Hope that gives you an idea how to come up with a nice user mix for testing without having too much data in the first place. Comments welcome.

• EU-West: ami-772b1d03
• US-East: ami-52649b3b
• US-West: ami-29bfec6c

Images can be used free of charge. The EU image is brand new and features Ubuntu 11.04. It has also a smaller disk of only 8GB compared to 15GB before. This helps to make it eligible for a free tier micro instance. Of course this instance type is not recommended for load testing, but you can easily test deployment and remote execution of XLT before you move up to more expensive setups.

The Problem

The Java application was behaving strangely in 4 out of 10 runs. It did not process all data available and assumed that the data input already ended. The application features several producer-consumer patterns, where one thread offers preprocessed data to the next one, passing it into a buffer where the next thread reads it from.

The consumer or producer fall into a wait state in case no data is available or the buffer is full. In case of a state change, the active threads notifies all waiting threads about the new data or the fact that all data is consumed.

On 2-core and 8-core machines, the application was running fine but when we moved it to 24-cores, it suddenly started to act in an unpredictable manner.

The Cause

After a lot of debugging I found out that threads wake up without having been notified by their partner thread. In this case the consumer was woken up despite the fact that data was unavailable aka the producer has not delivered and therefore not notified anyone. But the consumer was awake…

The debugging nightmare finally revealed a rare behavior of any POSIX based application. This is the quote from the JDK6 doc:

A thread can also wake up without being notified, interrupted, or timing out, a so-called spurious wakeup. While this will rarely occur in practice, applications must guard against it by testing for the condition that should have caused the thread to be awakened, and continuing to wait if the condition is not satisfied. In other words, waits should always occur in loops.

The Verdict

So never trust your notification chains. Threads might wake up even though nobody directly notified it. Additionally you should never exclude the possibility that the move from a small box to a big box does not influence the application behavior. More cores mean more trouble.

Besides a few defect fixes, release 4.0.5 delivers four great improvements to speed up test case creation and maintenance with Script Developer and make your work more productive.

• The XLT Script Developer runs on Firefox 4 and 3 now.
• Test variables are now resolved recursively, so you can use variables within resolved content.
• There is no need to open modules anymore if you want to edit a line or two of it. Also enabling/disabling of module code can be done easily from the main test case. This saves time and aids script maintenance.
• During script debugging and script execution, you can now evaluate assertions instantly to see whether or not your verification expression will match.

The full set of release notes can be viewed directly in the release area. You will also find documentation and the download link there. As usual, th

I believe in the power, speed and “update-ability” of CSS3. Not having to load background images as structural enhancements (such as PNGs for rounded corners and gradients) can save time in production (i.e. billable hours) and loading (i.e. page speed). At our company, we’ve happily been using CSS3 on client websites for over a year now, and I find that implementing many of these properties right now is the most sensible way to build websites.

Until today, all of that was based on an assumption: that I can produce a pixel-perfect Web page with CSS3 quicker than I can with older image-based CSS methods, and that the CSS3 page will load faster, with a smaller overall file size and fewer HTTP requests. As a single use case experiment, I decided to design and code a Web page and add visual enhancements twice: once with CSS3, and a second time using background images sliced directly from the PSD. I timed myself each round that I added the enhancements, and when finished, I used Pingdom to measure the loading times.

More…

Enjoy reading.

• Parallel and concurrent garbage collectors
• How does garbage collection work?

Enjoy reading.

It is modern, unique, and resembles our company colors. The font is distinct and the X is pretty eye-catching, because it has been carefully designed and crafted to express the uniqueness of Xceptance and it’s product and service offers.

As always, updates are free for all customers and all people using XLT despite the license.

Of course, the next line is the only right place. A curly bracket is a hermit and does not like to be put next to any other character… 

What is your opinion?

Cartoon courtesy of Geek and Poke under CC-BY-ND-2.0

An update of the AWS cloud images will follow in the next hours.

Up to 40 Commonwealth Bank Automatic Teller Machines are spewing cash across NSW just hours after suffering a computer error this morning.

Most modern programming languages are virtual machine based and these machines have knobs you can turn to adjust their behavior according to your requirements. XLT runs on Java and so all the things you might have already learnt from tuning your Java-based servers apply to XLT as well. If you do not have experience in tuning your Java-based servers, you will learn a lot that can be applied to your servers and help you to increase performance.

So, let’s take a look at the upcoming XLT feature that provides you with more details about garbage collection (GC) without the need to use any external tool to monitor or analyze logs or JVMs runtime behavior.

Perfect GC behavior

This is a chart displays nice memory statistics that indicate absolutely no problems with the garbage collection. As you can see the memory usage (first chart in the image) is alternating but never touching the upper limit. The second chart gives you the times when full garbage collection occurred, basically these are the bad events. Events that block the virtual machine from continuing to run. We should avoid these at any cost… nearly, because a full GC is costly. There was only one full GC right at the beginning at the test. This is a normal JVM behavior. So we seem to have done a pretty good job. No major collection of garbage. Sweat!

There are still the minor collections. Minor collections occur frequently and they reclaim small portions of memory with short living objects. The third charts shows us, that minor collection took about 10 to 30 ms, some spikes up to 70 ms. This is acceptable.

GC with Issues

Ok, let’s have a bad example in comparison. It runs with almost default VM settings. This means just min and max heap were set (Xms256MB/Xmx512MB). You can see clearly that a lot of major collections were running for up to 800 ms each. Also the minor collections were longer.

Clearly, the new charts help you see and fix misbehavior quickly. “What about the settings?” you might ask now. Yes, this example is not very helpful without revealing the difference between both test runs. So, we will give you the settings of the good one. The bad run had only Xms and Xmx set, as mentioned before.

## Set minimum memory to use
-Xms512m

## Set maximum permitted memory
-Xmx512m

## Enable concurrent old GC to avoid sudden long pauses
-XX:+UseConcMarkSweepGC

## When to start GC for the tenured/old area of the memory.
## This has to be low enough to avoid that threads need
## memory and can not get any before the GC has finished.
## This will lower the wait time.
-XX:CMSInitiatingOccupancyFraction=70

That’s really all. The use of the Concurrent-Mark-Sweep-Collector (CMS) is the most important setting. The CMS does not block the main VM worker threads, but collects the garbage concurrently. Because there might be situations were the CMS collector is not fast enough in cleaning up, this means the old space runs out of free memory, we give the CMS a hint with the last line. This line explicitly tells the CMS to start early with the clean up. In this case at a 70% fill level. The default is 96% and way too high.

One last thing. Setting Xms and Xmx to the same number helps the VM to relax because it will not try to get back to the minimal memory limit (Xms) and so it cleans less aggressively.

If you want to lean more about Java Garbage Collection, take a look at this Oracle documents: GC Tuning.

You might be especially interested in the new Script Developer. Our main feature of XLT 4.0.

The script developer is our approach to write and execute scripts efficiently within Firefox. It is a tool to quickly automate web application, share scripts without the hassle of complicated installations, while maintaining full control over possible other ways to execute scripts. The script developer lays the foundation to run test within the browser, execute scripts during builds, create and run test-driven tests, and, if required, export scripts into Java to unleash the power of a modern programming language.

Enjoy the screencasts and of course feedback is always welcome.