How to Speed Up Manual Testing with Test Automation

Tuesday, 21. May 2013 14:45

How to Speed Up Scripted Testing with Test Automation
Image © Juja Schneider

This article is not about setting up a sophisticated environment for automated tests since the relevance and importance of test automation is covered elsewhere. I rather want to give you my personal answer on how to speed up scripted testing.

Many of our e-commerce projects contain test scenarios such as the following: search for an item, select some refinements, put the product into the cart, and proceed to checkout where you need to enter your data and test another feature on the receipt page. In fact, this isn’t very complicated but it certainly takes a lot of time – even the more so when you have to go through this procedure not only once but many times a day which, on top of that, might get pretty boring in the long run.

Based on my experiences as a tester, I can honestly say that test automation tremendously facilitates scripted testing. I don’t use it to actually work off test cases assigned to me, but it always turns out to be a convenient tool that saves me from having to repeat the same steps over and over again.

Here are my pros of using test automation for scripted testing:

  • I can center my attention on the most important features in the current test case.
  • I save much time – time that I can well use for additional, exploratory testing.
  • I’m able to stay focused much longer because I don’t have to bother with monotonous and repeating test steps.
  • The minimal effort of test automation. I use XLT Script Developer that works as a Firefox Add-on and lets me record and replay test steps easily. If the project requires testing in Firefox only, I’m almost instantly done. Most projects include testing in multiple browsers, however. See here for a possible solution: WebDrivers in XLT: How to Run Test Cases in Multiple Browsers
  • My helpers can be a nice starter for future test automation.
  • It’s actually not a big deal if the previous point doesn’t apply as the automation tool still served its purpose of making my test life a little easier. ;-)

And here are some tips to take it easy:

  • Don’t be shy to throw away your helper after the project finished.
  • Don’t over engineer your scripts. They are meant to help you right now and not in the long run.
  • Don’t get lost in trying to get it to work. If you cannot get it working, return to manual testing even though it is painful at this point in time.

Category: Automation, Testing | Comments (0)

Release Xceptance LoadTest 4.2.8

Thursday, 16. May 2013 17:21

XLT_transparentWe just released a minor XLT update that brings you an improvement and a fix.

  • Script Developer: Mouse events dispatched by SD use target element coordinates now.
  • Load Test Environment: The result target directory can now be specified on the mastercontroller command line.

As always, this release update is free of charge and you can download it here: XLT Release 4.2.8. Do not forget to sign up to our release email list. To purchase licenses and support, please visit the XLT Portal.

XLT is free for up to five virtual users for load and performance testing. Regression testing usage is always free of charge.

Category: XLT | Comments (0)

Terms explained: Visits, Page views, Sessions, Requests, Hits

Tuesday, 14. May 2013 8:46

Terms explained: Visits, Page views, Sessions, Requests, Hits
Image © Juja Schneider

When discussing possible load test setups with our clients, we usually need to refer to these key terms: visits, sessions, requests, hits, page impressions, and page views. Actually, we don’t need to discuss all of them, but some are occasionally brought up by the customer, some are requested by us, depending on the context (and complex enough to be discussed in a separate article).

Many clients of ours have told us that it’s sometimes hard not to confuse these terms as they seem to denote the exact same thing. Today’s article is thus meant to give you an overview on their definition and help you distinguish them. Here it comes.

Visits

Basically, a visit occurs when you send a request to a server and, as a response, the website you requested is displayed. The display of the page is what we call the page view (which is covered in more detail below). Take, for instance, www.amazon.com: you enter the URL, that is you send a request to that server. As a result, you arrive on the shop’s homepage. In doing so, you generate a visit and the website owner now knows that someone visited their site. Accordingly, a visit consists of at least one page view. Typically, though, you’ll decide to browse through the shop, thus producing further page views with every single one of them adding up to your visit regardless of what you actually do while staying on the site.

Your visit ends when you become inactive for whatever reason, for example, when you stop clicking links or when you overall close your browser. The server then deletes or deactivates the data that has been collected during your visit. Depending on the web application, the time period before this actually happens varies from 30 minutes up to 24 hours (see below for more details). Compared to the real world, a virtual visit is therefore not much different from a visit in an actual store: regardless of how you spend your time in there, the store owner will consider you a visitor and they will eventually forget about you when you decide to leave their store for a longer period of time.

A visit defines three metrics that are important for us to know: the visit duration, the number of page views per unit of time, and the time period between two page views (thinktime). By the way, the Internet is crowded by an insane number of machines that also generate visits and that cause statistics to get messed up. There are a couple of technical measures trying to filter out these irrelevant visits. Let me know if you’re interested in how that can be accomplished and I’ll talk about it in another article.

Page view

As mentioned above already, a page view, or page impression, is the display of a website you trigger by sending a request to a server. It was not long ago that one request was sufficient for the page you requested to be displayed; via further requests, the browser then added images, CSS, and Javascript. In the modern web, the page view term is used in broader contexts as AJAX and recent user concepts don’t exchange complete websites but only modify small parts anymore. Thus, a page view is actually an interaction or action most of the time.

But let’s stick with the old-fashioned page view for now. It gets initiated by a request and terminated by the response from the server. This sequence of events is also referred to as hit. A site may contain other elements, such as images and CSS. Each of these elements is again processed by a request/response sequence and leads to another hit. Thus, a page view is made up of an HTML component and many embedded elements with each element and the HTML component causing further hits. Important metrics are, for instance: page load time, page size, and view time, namely the time period until the next click.

Hit is often the business term for a request that, in turn, is the technical term for a single data transfer.

To sum this up: A visit consists of one or more page views and has a certain duration while a page view has a runtime and consists of elements referred to as request or hit.

Sessions

Now, how does a session differ from a visit? In simple terms, a session is the technical picture behind a visit. The software you take advantage of while browsing through a webshop has to remember which requests belong together so that functionalities such as a login or a cart can actually do the things they’re supposed to do.

Sessions consist of data that summarize certain information concerning your visit which is why they are also called session information. Usually, these information have a limited lifetime as they are subject to a session-timeout. In most cases, this time-out amounts to something between 30 minutes to 2 hours. As soon as your visit ends, time is ticking down and, upon reaching the session-timeout, all data is deleted. If you continue your visit before the time has elapsed, the countdown resets to 0. To illustrate this with our real-world example, imagine you decided to buy something in the store you walked into (visited) but at the register you suddenly realize you didn’t bring any money. You can leave your cart at the register for a while to go get some cash but if you don’t return in time, the cashier will assume you’re not coming back and empty your cart.

Technically speaking, the number of sessions equals the number of visits. Due to business-related aspects, however, visits are counted differently than sessions so that the number of visits is usually lower than the number of sessions per unit of time.

Summary

In a few words:

  • Visits mostly equal sessions, where session is the technical term and visit is the business term. A visit consists of at least on page view.
  • Hits and requests are the same, request is the technical, hit is the business term.
  • A page view is a single complete page delivered. At least one request is needed to serve it, mostly many requests are fired to assemble a page. Nowadays, page views tend to be interactions because often full pages aren’t loaded anymore; instead, only pieces are dynamically changed (the famous Ajax magic).

Confused? Feel free to ask!

Category: Software, Testing | Comments (1)

Blog Performance – We Kicked it up a Notch

Monday, 13. May 2013 9:58

Usually, we just  measure the performance of our customer’s applications and talk about it, but from time to time we have to set an example ourselves.

In the last couple of weeks, we increasingly felt that our blog isn’t loading fast enough to deliver a satisfying experience. You know that when you can feel it, it might be too late already. Additionally, SEO is about content and performance and our blog is an important marketing tool for us.

That’s why we went on a quest for improving the performance of our company’s WordPress-based blog. Our motto: “Don’t just complain about the lack of performance, do something about it!”

Step 0 – Measuring

Our initial blog performanceMeasuring is believing and so we started with this WebPageTest result. As you can see, the initial performance is bad, a lot of content is not properly cached, and rendering started after 2.6 sec. Time for some serious tuning.

Step 1 – Reading

Tuning requires you to know what to tune. Thus, we read the famous Best Practises for Speeding Up Your Web Site by Yahoo. A similar article by Google can be found here. If you deal with web site performance in any way, you should read this. We consider it mandatory for performance and web testers.

Step 2 – Wordfence

One of the first things we did was disabling Wordfence because we had the strange feeling that the performance degradation of the blog had started just recently – a couple of weeks ago, we’d installed Wordfence in response to the WordPress login probing attacks. Yes, the disabling made us faster: first byte time came down from 1.7 sec to 0.7 sec. This was awesome… but who wants to run WordPress without protection? So we tuned Wordfence and disabled the firewall and live traffic detection, basically just keeping the login security and scanning features.

Step 3 – JavaScript

So, what was next? We realized that Slimbox had an impact because of the MooTools library it used. Not really a huge impact, but the JavaScript held back the rendering and further loading. We picked wp-jquery-lightbox as a replacement which didn’t change that much but got a little faster; Jquery seems to be a little better optimized in terms of JavaScript performance and size.

We also updated the SyntaxHighlighter since we’d been using an older version that was no longer under active development. Yet no relevant change in performance here.

Step 4 – Thinking

That’s basically all you can do, provided that your content is okay (properly compressed images) and that you have a fairly small CSS file for your theme. Because changing the hoster, tuning the database, or even replacing the webserver is a little overkill for a low-traffic blog, we could only take the caching route next. You can go back to the Yahoo and Google article and check what is possible without starting to reinvent the wheel aka writing a new blog, changing the theme, rewriting all content…

Step 5a – WordPress Caching

When you look up tuning tips for WordPress, you immediately find W3 Total Cache and WP Super Cache. Based on our reading of reviews, Super Cache seemed to be a little easier to handle, so we tried that. Somehow we failed though: it didn’t do anything and the performance stayed the same. As it’s not that easy to use, we switched to W3 Total Cache.

Of course, Total Cache is overwhelming as well and it took a while for us to realize that it couldn’t save its settings due to the file system permission handling of our hoster. Thus, we updated the .htaccess files manually. Since we also pull content from our Piwik installation, we added some of the .htaccess settings there as well. These settings basically refer to  serving the content as cacheable to the browser, avoiding refetching as well as delivering the content compressed. CSS, JS, and HTML can be compressed quite well and so it’s recommended to run mod_deflate for reducing bandwidth consumption.

Step 5b – W3 Total Cache

We had to go through a lot of trial and error to get the performance where we wanted it. Total Cache offers so many options that we had to try over and over again. Don’t forget to measure the  settings several times since also the measurement tool can have a bad minute as well! Also, if you run on a shared hosting environment, you’re not alone and other traffic might influence your measurements.

Step 6 – Measuring Again

Blog Performance TunedAfter all this was done, we measured the final performance. The first bytes now came in after 0.22 sec and rendering started after 0.7 sec. Wow! We’d begun with 1.7 and 2.6 sec!

Step 7 – Enjoying

This basically means that we improved the performance of our blog by factor 4 to 8! Google and Bing, feel invited to drop by and index our content! We hope that our visitors enjoy the improved performance as much as we do. If we wanted to further enhance it, we had to move to a better server that can be tuned by dealing with database and network settings on a lower level. However, this doesn’t make any sense right now, micro-tuning doesn’t do any good here.

Always keep in mind: you can only test what you know and understand. It is this knowledge that allows you to evaluate client performance the next time you have to test it, and it also provides useful feedback on how a problem can be solved or in which direction you have to look for a solution.

We’re looking forward to hearing about your client site performance tuning and measuring experiences. Please share your WordPress tuning and operational procedures as well. Happy performance blogging!

Category: Performance, Software | Comments (0)

This is Just a Data Issue

Wednesday, 8. May 2013 14:17


Image © Juja Schneider

Many projects have their closed or “won’t fix” bugs commented with remarks such as:

  • “Content-related issue”,
  • “Won’t happen on live system”, or
  • “This is just a data issue”.

Both the development of new features and data maintenance may be accomplished in different environments. For testers, it’s thus difficult to decide on the cause for a certain issue: is a required text missing because pages aren’t implemented the right way or is it simply not there because the corresponding product data aren’t available in this particular test system? Very often testers work on development systems without having access to latest data. The following is a typical scenario:

  1. Testers report a bug.
  2. The development resolves this bug as content-related and won’t fix.
  3. Testers report further bugs of that sort.
  4. The development gets irritated: “We’ve told them that these are content issues!”
  5. The relationship between testers and developers might really go downhill from there, seriously threatening the project’s success: the credibility of the test team is damaged, developers  don’t take reported bugs seriously anymore, testers get overly cautious and stop reporting content-related problems…

Inevitably, potential bugs don’t get reported and reported bugs never get retested before the project goes live. This is even more true in poorly managed projects because no one really feels responsible to solve this problem.

Here are a couple of guidelines we advise you to keep in mind if the descriptions above somehow sound familiar to you:

For testers:

  • A bug is a bug is a bug. When you see it, report it. Let your test or project manager know that you’ll report these kind of issues even if they are potentially content-related. Only this way, you can be sure not to miss anything important.
  • Draw conclusions from previously reported issues and comments.
  • Try to get a feeling for your project and make sure not to address problems you’re actually causing yourself.

For project and test managers:

  • Don’t dismiss these issues as minor troubles you don’t want to bother with. They can easily turn into big ones at any stage of your project. Before go-live, you want everything to be fixed, not only functional problems.
  • Problems that appear to be data-related at first sight might in fact be functional. Handling them within the regular test process facilitates their correct fixing.
  • Don’t hesitate to tell your testers if they are causing the problem themselves. Granted, this might be an uncomfortable situation at first, but it will definitely trigger an Aha! effect everyone in the team can benefit from.
  • Give content-related issues a low severity. This way, you can focus on troubles that are more pressing and, at the same time, ensure that nothing gets lost and QA will still retest everything.
  • Inform your test team about the system’s architecture and how content and date influence its behavior.
  • Also let your test team know when certain data aren’t available yet and when you don’t want the resulting problems reported. Testing is way easier when having such information in advance.

Category: Testing | Comments (0)

WebDrivers in XLT: Test Case Design that Compensates for Inconsistencies across WebDrivers

Monday, 6. May 2013 9:56

Still remember the first post of our article series? It talked about how you can run XLT test cases in different browsers. If you’ve done so already, you might have noticed that the behavior of test cases developed in Script Developer sometimes differs depending on the WebDriver you’ve chosen. This article is meant to help you resolve such issues.

First of all, what’s the reason for these inconsistencies? Web browsers differ in their characteristics, such as site representation or functionality, due to their varying support of web technologies like CSS or HTML. You probably know that there is much more we could list, but the major point is pretty obvious already: using WebDrivers for test case execution calls real browser instances. Logically, you’re faced with the same differences as in real web browsers. It’s impossible to achieve a completely consistent web browser behavior; yet you can design your test case as outlined below to at least reduce the differences:

Time: Enlarging time properties can prevent timing problems with dynamically loaded content, like iframes or page content added with JavaScript. Script Developer lets you set an ‘Implicit Wait Timeout’. It allows to specify how long the test should wait for a command to fail while trying to find elements.

Implicite Wait Tmeout

Implicit Wait Timeout

A corresponding setting is also available for WebDriver test execution in the XLT properties files. The following property defines the WebDriver’s implicit wait timeout used when finding elements:

com.xceptance.xlt.scripting.defaultImplicitWaitTimeout=10000

Another way to control the execution speed is to set the action think time, that is a delay between the execution of two subsequent actions. Note that this property is only relevant when using the XltDriver.

com.xceptance.xlt.thinktime.action=500

Incomplete database processes or unfinished AJAX operations may also cause certain problems. If you encounter such problems with the XltDriver, check whether the following property is set to default:

com.xceptance.xlt.js.backgroundActivity.waitingTime=0

This is the time (in ms) to wait for JavaScript background jobs to finish.
When this time has elapsed, all pending jobs will be removed. Default value is 0, if not set. If set to -1, the engine will neither wait for running jobs to finish nor remove pending jobs. Note that the other WebDrivers don’t have this property. That’s why you should prevent possible problems by taking care of them when you actually design your tests with Script Developer. waitFor… commands are a very good way to achieve this. They’ve been developed for XLT and wait much longer than other assertions. You can configure their waiting time, or command timeout, in both the Script Developer settings and in the XLT properties for WebDriver test execution:

Command Timeout

Command Timeout

com.xceptance.xlt.scripting.defaultTimeout=30000

Make sure to use waitFor… commands for dynamically loaded content, even if, for instance, an assert… command appears to be sufficient in Script Developer. If you want the test execution to wait for a specific time period, you can also define a pause. Avoid using …andWait commands for dynamically loaded content because these commands will wait for a new page load which will not happen when content is loaded dynamically.

CSS/link selectors: Using CSS-selectors may be comfortable and is also supported by Script Developer. Keep in mind, though, that the CSS interpretation is different for each WebDriver. For example, the CSS text-transform feature modifies the text specified in HTML. A text validation might fail if this CSS feature is supported in one browser only. Link locators may also cause problems in this context. For now, it’s best for you to avoid these selection strategies when creating test cases in Script Developer and instead use XPath referencing where possible.

Redundant clicks: Text and number validation in forms is often done using the onBlur event that is fired when you leave the input field by clicking somewhere outside the form.

Imagine you are testing input field validation. A JavaScript function displays an error message onBlur, you leave a field with an invalid input, and click somewher else so that the onBlur event is fired. If this click hits the submit button, Script Developer and the XltDriver will forward to the next page whereas WebDrivers need two clicks, one for firing the event and another for submitting the page. A helpful workaround is the seemingly redundant click on a static page element to force the JavaScript event right before submitting. This also helps in case of hiding UI-elements, that mask other elements. This scenario is a known issue in ChromeDriver.

Conclusion:

Being third-party products, WebDrivers aren’t consistent and never will be. Nevertheless, modifying certain XLT properties as outlined above may help you cope with these differences and reduce them whenever you want to go for multiple browsers.

Category: Automation, Testing, XLT | Comments (0)

Release Xceptance LoadTest 4.2.7

Thursday, 2. May 2013 21:29

XLT LogoXLT 4.2.7 has been released. It delivers improvements as well as defect fixes. We recommend to upgrade as soon as possible to stay up to date.

Script Developer

  • Fix: Obsolete entries from the base url drop down can be removed now.
  • Improvement: With special characters in the link name, the Script Developer sometimes recorded an XPath locator instead of the more appropriate link locator.

Load Test Environment

  • Fix: Custom samplers stopped working for the rest of the load test when an exception was thrown. Now the sampler continues to run.
  • Fix: The Mastercontroller used a previously entered comment when the current comment was left empty.
  • Improvement: The Mastercontroller returns non-zero exit code for all error conditions now.

Framework

  • Fix: HtmlUnit did not sent the basic HTTP authentication header for follow-up requests correctly, only when challenged again.

Of course, we are providing a set of fresh Amazon Machines Images as well. You can find all information on our product page. Make sure you stay informed by subscribing to our XLT release email list.

Category: XLT | Comments (0)

Funny Bugtracker Findings: The Cookie Issue

Tuesday, 30. April 2013 12:03

Title: Need John to bring in more oreo stuffed chocolate chip cookies

Type: Bug
Status: Approved
Priority: High
Environment: Meeting Room
Description: Cookies were so great we need more. And would come in handy right about now.
Comments: John Doe added a comment:
Steps to repeat: http://crowningvictoria.com/2012/12/oreo-stuffed-cookies/

 

Category: Funny Bugtracker Findings | Comments (0)

WebDrivers in XLT: How to Run Test Cases in Multiple Browsers

Tuesday, 23. April 2013 15:06

Have you heard of XLT Script Developer yet? If you have, you’ll probably agree that it’s a convenient tool to record and run automated test cases. However, with it being a Firefox plugin, you’re basically bound to run your test cases in Firefox. Wouldn’t it be nice to reuse Script Developer test cases in multiple browsers? You can actually do so by taking advantage of the WebDriver API that is part of the XLT framework.

If you don’t know much about WebDrivers, you should continue reading this article – the first one of a little article series on WebDrivers that hopefully gives you a good introduction to the topic.

WebDriver is a tool for automating web application testing that was originally introduced by Google. The API permits control of real web browsers, such as Firefox, Internet Explorer, Chrome, Safari, and the HtmlUnit headless browser.

Let’s start with the basics: How to use WebDrivers in the XLT context.

Script Developer test cases are stored in XML files. If you want to execute your test cases in multiple browsers, you need to run them as part of an XLT testsuite in your Java IDE, e.g. Eclipse. You can do so in two ways:

  1. Reuse the original Script Developer file (XML) within a JUnit wrapper class
  2. Export the test case to Java

Both options differ in the opportunities they offer: while the first one forces you to stick with Script Developer for test case editing and maintenance, the second option lets you make full use of Java.

If you go for the first option, select Generate JUnit wrapper class for test cases in the Script Developer Settings and you end up with a JUnit wrapper class for each test case.

XLT Script Developer Menu

XLT Script Developer Menu

XLT Script Developer Settings

XLT Script Developer Settings

You can now find this generated java class as part of your XLT testsuite in a folder defined by your Script Developer package structure.

In the following, this generated Java class needs manual editing. To prevent that your changes will be overwritten, uncheck “Generate JUnit wrapper class for test cases” because the wrappers are automatically regenerated after each modification in the related test case.

You can also make a copy of the wrapper class right after its creation and edit the copy. In that case, you don’t need to uncheck the checkbox after generating the wrappers.

The original wrapper looks like this:

@ScriptName
("com.xceptance.xlt.samples.tests.scripting.TBlogVisitor")
public class TBlogVisitor extends AbstractScriptTestCase
{
}

You can now execute the test by selecting “Run as JUnit test” from the Eclipse context menu of this specific file. By default, the XltDriver is used to run the test in a headless HtmlUnit-based browser.

Now, here comes the interesting part: to set your preferred WebDriver, add a constructor to the wrapper class. The example below makes use of the Chrome WebDriver:

@ScriptName
("com.xceptance.xlt.samples.tests.scripting.TBlogVisitor")
public class TBlogVisitor extends AbstractScriptTestCase
{
 public TBlogVisitor()
 {
  //Set WebDriver here
  super.setWebDriver(new ChromeDriver());
 }
}

To make this work smoothly, you need to pay attention to the WebDrivers for Chrome and Internet Explorer. Download and install the driver files from the locations given above. Also make sure you’ve installed the latest versions of these browsers. Last not least, a system property setting is required right before calling the setWebDriver method.

Use this code line for Chrome:

System.setProperty("webdriver.chrome.driver",PATH TO DRIVER);

Use this one for Internet Explorer:

System.setProperty("webdriver.ie.driver",PATH TO DRIVER);

The placeholder PATH TO DRIVER has to be replaced with the location of the driver file. In Linux and MAC OSX systems, the path ends with /chromedriver, in Windows with /chromedriver.exe or /IEDriverServer.exe.

Pretty easy, isn’t it? But what about the second option? Why export test cases to Java code? The commands in Script Developer test cases are strictly sequential and thus may at times restrict your programming ambitions. If you want to take advantage of all the opportunities offered by Java, such as implementing loops, adding randomization, or forcing a conditional execution, you should convert your test case into Java code.

Right-click your test case or test package in Script Developer, choose “Export” from the context menu, select XLT Scripting API from the dropdown, and then confirm the export. Again, you’ll find the exported java class as part of your XLT testsuite in a folder defined by your Script Developer package structure.

An exported Java class looks like this:

public class TBlogVisitor extends AbstractWebDriverScriptTestCase
{
	public TBlogVisitor()
	{
    	super(new XltDriver(true), "http://example.com/");
	}

	@Test
	public void test() throws Throwable
	{

    	//
    	// ~~~ OpenHomePage ~~~
    	//
    	startAction("OpenHomePage");
    	open("/pebble/");
    	assertTitle("My blog");
    	assertElementPresent("link=<< Previous");
    	assertNotElementPresent("link=Next >>");
    	final Paging _paging = new Paging();
    	_paging.execute();

    	//
    	// ~~~ ViewArticleDetails ~~~
    	//
    	startAction("ViewArticleDetails");
    	clickAndWait("//div[@id='content']/div[1]/h1/a");
    	assertElementPresent("link=Send a TrackBack");
    	assertElementPresent("//div[@id='content']/div/div[3]/a[1]");

    	//
    	// ~~~ AddComment ~~~
    	//
    	startAction("AddComment");
    	click("//div[@id='content']/div/div[3]/a[1]");
    	type("id=commentBody", "A comment");
    	clickAndWait("//*[@id='commentForm']//input[@name='submit' and @type='submit' and @value='Add Comment']");
    	assertElementPresent("//*[@id='content']//p");
    	clickAndWait("name=submit value=Confirm");
    	assertElementPresent("//*[@id='content']//div[contains(@class,'contentItemBody')]");

    	//
    	// ~~~ ReturnToHomePage ~~~
    	//
    	startAction("ReturnToHomePage");
    	clickAndWait("link=Home");
    	assertTitle("My blog");
    	assertElementPresent("link=<< Previous");
    	assertNotElementPresent("link=Next >>");
	}

	@After
	public void after()
	{
    	// Shutdown WebDriver.
    	getWebDriver().quit();
	}
}

In the given example, the XltDriver is called by default (at line 5). You can change this into new InternetExplorerDriver() to use Internet Explorer WebDriver or one of the other available ones instead (Firefox, Chrome, Opera, iPhone, Android, Safari).

To help you understand the whole issue, the example below illustrates the complete procedure for the JUnit wrapper class, including the required imports, the property setting, and a useful way to automatically close the browser after test execution.

package com.xceptance.xlt.samples.tests.scripting;
import org.junit.After;
import org.openqa.selenium.chrome.ChromeDriver;
import com.xceptance.xlt.api.engine.scripting.AbstractScriptTestCase;
import com.xceptance.xlt.api.engine.scripting.ScriptName;

@ScriptName
("com.xceptance.xlt.samples.tests.scripting.TBlogVisitor")
public class TBlogVisitor extends AbstractScriptTestCase
{
 public TBlogVisitor()
 {
  //Set driver path property here
  System.setProperty("webdriver.chrome.driver","C:/drivers/chromedriver.exe");

  //Set WebDriver here
  super.setWebDriver(new ChromeDriver());
 }

 //Automatically shutdown Web Driver after execution
 @After
 public void after()
 {
  super.getWebDriver().quit();
 }
}

Now that you know how to run automated Script Developer test cases in all major browsers, you’re well prepared for the next part of this series: the challenges of test case design in different WebDrivers.

Category: Automation, XLT | Comments (2)

The Art of Reading Performance Test Charts

Monday, 22. April 2013 9:36

Powerful load and performance test tools don’t only retrieve pages of your website randomly with zillions of users at the same time, but they also cover realistic scenarios simulating the real-world user. It’s a given that they can deliver lots of useful information and plot interesting charts. To fully take advantage of these benefits, however, you need to be able to interpret this information and draw the right conclusions.

It is this need for the correct interpretation of test results, the mapping of all you see against actual application behavior that makes performance and load testing a non-trivial task. It requires much experience to decide on the right actions, make the right assumptions, or simply come up with a reasonable explanation of why something happened this way and not the other.

In today’s article, we’d like to present you a couple of charts displaying typical response time patterns, and discuss what they could indicate.

Disclaimer: Of course, the reasons for a certain behavior vary a lot, depending on your application and testing. However, as there’s no fixed manual for the interpretation of load testing charts, we want to provide you with a couple of basic guidelines to help you get better in interpreting them yourself and make the most out of your test results. Feel free to comment whether or not you agree with the ideas and explanations we come up with.

The Warm-up

These charts might indicate a system with a cold cache, for instance, when the system has just been started and the caches aren’t filled yet.

The basic characteristics of such a behavior are high response times in the beginning, followed by gradually lower response times until eventually a  certain degree of runtime stability is reached. This time frame is often referred to as the system’s warm-up period. Throughout this period, a couple of things can happen under the surface. If you know the system under test well, you’ll probably come up with the following: database and file system caches are filled, proxies learn about the data and store them, the system under test scales up because it sees traffic, page snippets are cached and so the computing overhead reduces… you name it.

Also keep in mind that it might be the testing process itself that causes such a response time profile. If the system is perfectly warmed up and you hit it, your sent traffic might be too uniform in the beginning. That being the case, randomization kicks in so that the traffic eventually distributes better over time. Furthermore, take into consideration that your load software and hardware are possibly not warmed up either.

The Caching

These charts depict either a typical cache clean or job patterns. In case of a cache clean, system-internal caches expire every half hour. If that’s not the case, the charts may indicate  a running background job draining power from the database or consuming lots of system bandwidth.

Both charts display the same test; however, this test has been executed for different time periods. While two spikes could signify a random event (despite the fact that the temporal distance of 30 minutes is suspicious), the longer test run seems to prove our first assumption: something is going on every half hour.

In any case, make sure that such a behavior is not produced by the test machines themselves, for example, because they’re busy writing or backing up data.

The Spiking

This is what we call a forest of spikes: many spikes that don’t seem to follow a comprehensible pattern; longer runtimes just occur occasionally, often caused by requests accessing certain data or URLs that produce long runtimes. To get behind that mystery, you have to dig into the results in more detail, find the calls behind the spikes, and derive a pattern based on the information you find. Often you’ll come across similar URLs, request parameters, or maybe response codes. Don’t forget any application logs you might have access to, such as web server, error, information, or debug logs. In a perfect world, your application under test offers the necessary tools to get to the bottom of this problem.

XLT lets you easily find this information. All test result data are accessible as CSV files that are quickly readable and documented. Feel free to work with this information and go beyond the scope of the reports available.

The worst outcome here is a non-identifiable pattern and no information on the server side as to what might have happened. In such a case, you have to repeat the test and narrow down your test setup later on to exclude as many variables as possible to find the cause. This is actually also a good time to ask for development or tech-ops support.

The 3rd-party Calls

The first chart is typical for issues with 3rd parties, especially in the field of e-commerce. We’re not talking about direct calls to 3rd parties here, such as analytics vendors or recommendation engines, but calls from one server to the other. Thus, the response time we see is the response time of two systems. Of course, it’s good to know the area where 3rd party calls typically happen, but you have to know the application under test anyway to test it efficiently. So when the final order steps start to act weird, you can easily narrow down the potential reasons.

The second chart looks more like the cache clean or expiration problem described above, but since you know the application, you also know that this area doesn’t use the typical caching logic but is highly dynamic instead. This means that the errors occurring every 50 minutes point into a different direction: as we know that 3rd parties are attached and called during shipping, we can conclude that the 3rd party failed on us.

Verdict

Knowing typical response time patterns helps you specify a certain problem so that you can give hints to the development or further shape the path of testing. If you can read charts and derive the right conclusions or at least know which questions you have to address, you’ll be ahead of the crowd. Be aware that knowledge on the system under test is very important – the production and measurement of a certain load doesn’t make much sense when you’re not able to actually interpret and explain what you’ve measured. Always remember: 42 is not a valid answer for everything. :)

Category: Performance, Testing, XLT | Comments (0)