Today, we released XLT 4.0.1. This is an minor update to XLT 4.0 that fixes five defects. Additionally, it provides some documentation enhancements. You can download the release from here: https://lab.xceptance.de/releases/xlt/4.0.1/. At the same location, you will find the documentation as well as the release notes.

We just published four brand-new screencasts about XLT 4.0, its features, and how to work with them. This is our first attempt to use screencasts as a way of documenting our software. They do not replace the written documentation, of course, but they do provide a quick and easy way to become familiar with XLT.

You might be especially interested in the new Script Developer. Our main feature of XLT 4.0.

The script developer is our approach to write and execute scripts efficiently within Firefox. It is a tool to quickly automate web application, share scripts without the hassle of complicated installations, while maintaining full control over possible other ways to execute scripts. The script developer lays the foundation to run test within the browser, execute scripts during builds, create and run test-driven tests, and, if required, export scripts into Java to unleash the power of a modern programming language.

Enjoy the screencasts and of course feedback is always welcome.

There are new public Amazon EC2 images (AMI) available each running 4 agentcontrollers of XLT 4.0.0.r6019 on ports 8500 to 8503. Feel free to use them for your load testing purposes.

• EU-West: ami-9b5064ef
• US-East: ami-5647b63f
• US-West: ami-539dcd16

The usage of the images is free or charge but you have to pay your AWS usage costs of course. Please keep in mind that you need a valid license to run a load test with more than 5 virtual users.

We just released Xceptance LoadTest 4.0. This release of our load test software got some really nice feature enhancements to make your regression testing easier. So we stick to our general software approach: One tool for regression and load testing. One set of scripts for both purposes.

Script Developer

As an alternative to writing test cases in Java, you can now use the XLT Script Developer to create script test cases. Script test cases are based on a simple syntax and a reduced set of operations, which makes them a perfect fit for non-programmers. Only the Script Developer, which is an extension to Firefox, is necessary to create, edit, and manage basic script test cases.

To create a new script test case, the test designer simply uses the application under test. All interactions with the application are recorded in the background and stored to an XML script file as a sequence of script commands. While recording, assertion commands to validate the web pages may be inserted manually. From the Script Developer, script test cases can be replayed in Firefox at any time to quickly check whether the test case still runs successfully.

Existing script test cases can be modified later on, for example, to add new or delete obsolete commands. Common command sequences, which could be reused in other test cases as well, can be refactored to parameterizable script modules. Finally, any recorded value can be extracted out of the script into a test data file to separate test data from script code.

Script files can also be run outside of the browser, via the XLT framework, which simulates a head-less browser. This mode is suitable for unattended test case execution, during functional or load tests. When saving scripts, the Script Developer also creates JUnit test case classes as “wrappers” around script test cases, which serve as a bridge between the XLT framework and the script world. This way, from the framework’s point of view, script test cases are in no way different from test cases written in Java.

More Data to Query

For improved tests accuracy, you can now query the request and response data and run assertions on it. This permits checks on the communication because not all requests are reflected in the DOM tree.

Improved EC2 Handling

AWS (Amazon Web Services) added the ability to tag EC2 resources to simplify the administration of your cloud infrastructure. As a form of meta data, tags can be used to create user-friendly names and improve coordination between multiple users. The XLT EC2 administration tool ec2_admin features an additional menu which lets you select your EC2 resources based on the tag name.

Better Automation

To improve automation of tests, we added the ability to pass properties on the mastercontroller command line. Additionally the test definition file for the test suite can be redefined on the command line as well.

Faster Work Flow

When test goes wrong or a logging is turn up, the data to download from all agents can be pretty big. To get a fast or selective result, you can now decide how much data you want to download.

JDK Compatibility

Beginning with v4.0, XLT requires a Java virtual machine 6 or above to run. Java 5 is not supported any longer. The reason is the end-of-life announcement for JDK 5.

Misc

We refreshed HtmlUnit and updated it to version 2.8, Ruby got updated to 1.5.1, and WebDriver is now v2.0a6. The event API got simplified and is now easier to use.

Where to get it

More information about the release, the quick start guide, and the manual can be found in the release area. Of course, the full download of XLT 4.0 is available there too. 

We are looking forward to your feedback, comments, and of course… Happy testing!

Today, I rediscovered this nice article about browser cache usage: Performance Research, Part 2: Browser Cache Usage – Exposed!. It gives you a pretty good idea about the average cache usage. Bottom line: Optimize your site for no cache hits at all and you are good.

40-60% of Yahoo!’s users have an empty cache experience and ~20% of all page views are done with an empty cache. … It says that even if your assets are optimized for maximum caching, there are a significant number of users that will always have an empty cache. …reducing the number of HTTP requests has the biggest impact on reducing response time. The percentage of users with an empty cache for different web pages may vary, especially for pages with a high number of active (daily) users. However, we found in our study that regardless of usage patterns, the percentage of page views with an empty cache is always ~20%.

If you write a performance testing software, your first obligation is a performance test for and with that software. So before we can ship XLT 4.0, we have to make sure that it can hold up to its promises. Test software has to be tested too, so to speak. Today we provisioned a bunch of Amazon-EC2 boxes with 30 test agents in total. These charts are from a short test to see where we can get to, when ignoring any target numbers. This was a short and violent test. Just hammer the system under test and see if it can and will recover.

A short ramp up period in the beginning of the test and afterwards we kept a steady load factor. For the steady phase we reached more than 11,000 hits per second.

The target system was seeing about 2,200 concurrent users during the peak of the test.

During that test, the network was transporting about 95 Mbytes/s inbound traffic, this is a network utilization of about 760 Mbit/s. Amazing that the EC2 boxes in the EU data center can handle that traffic. We used just 10 boxes and each box has a 100Mbit network, so the overall limit must have been reached we think.

By the way, the target system recovered easily and was able to serve its normal duties without any problems. But this test clearly showed the limits in terms of throughput. But this test did not show any limits of XLT 4.0, because neither the load boxes in terms of CPU nor the reporting had any problems with this test size.

People asked for a rough guide for running load tests against their live site and whether this is a good idea at all.

Well, let me first say that test environments exist for exactly that purpose. So if you already have something live, of course you used it before going live for load testing, and now you cannot run another test there… yes, you need a test environment. This is well spent money for several reasons:

• Created data will not pollute your installation aka log entries, analytics data, orders in the database, created users, and so on.
• You do not have to take your live site down for testing.
• Problems during testing will not leave your live site down or take it down.
• Your hosting company or IT department might charge for all test traffic against the live site as it would have been real traffic (revenue share, bandwidth utilization), so having your own testing realm makes expenses more predictable.
• You can easily change code and retest when changes are necessary. You can profile, you can debug.
• You do not risk problems with traffic going to live systems, such as payment due to configuration or testing errors. If you already had items from test orders piling up in your office, you know what I mean.

So, get yourself a test environment. Spend the money and enjoy the freedom to measure, debug, and tune.

P.S. Of course, there are situations, where you cannot replicate a live environment or you cannot find application problems in a test setup due to totally different timing behavior… Well, this indicates only that your live environment is already screwed.

We just released an update for XLT 3.3. The version 3.3.4 fixes a problem with the onblur event in HtmlUnit that might have led to a recursive loop. More information and the download link can be found in the release notes.

A new Amazon Machine Image (AMI) running XLT has been made available in the EU region. Use the command line tool of version 3.3.3 to discover it. If you want to provision it manually, please refer to

• AMI ID: ami-82d1fbf6
• Name: xceptance-ubuntu-9.10-64bit-011-xlt-3.3.3.r4407

The AMI runs Ubuntu 9.10 64bit with a Sun JDK 6u20. The file limits have been bumped up to permit plenty of sockets for large tests with many users. XLT 3.3.3.r4407 runs on port 8500 and can be used directly. Make sure that your security group includes port 8500 as an open port.

Because I do not know what is the current state of the Java G1 Garbage Collector, I decided to try G1 with JDK6u20. Somehow I was disappointed because after a short moment of predictable GC performance, the entire VM stopped and some major collection was running. You can easily see that in the charts of that run. Right around 20:09:45, the threads were stopped and the entire VM behaved ugly.

So, the G1 is not yet ready for production, of course nobody stated that it is ready for production. If I read the release notes of JDK6u21 correctly, it delivers plenty of G1 changes, so I might try that soon.