Category Archives: Testing

posts to do with software testing

dpxdt visual testing tool

A large amount of my time as a tester is spent checking web pages for differences whether they be due to functionality changes or layout and styling changes. Also when a site has  some underlying technology changed or has been moved to a new platform you need to check for changes to the front end look of a site.

I spent quite a lot of time looking for a way of automated this process so i could compare web pages from before and after a change so i only had to check pages that did look different. Its quite a tricky thing to do but i finally managed to find a tool that would do this and the best bit was its free and open source.

This tool once set up (You might need a friendly dev ops to help as its not super easy) you can point it at a page and it will crawl that page and all the links from the page to a crawl depth that you can specify and take screen shots of all of them. This creates your baseline that you will compare against. You then do the same thing against the updated page and once it has completed it will provide you with a list of all the same pages in the crawl that were different from each other. For each set of the same pages it will also identify the parts of the image that are different.

The tool makes a great addition to a continuous integration set up and thats really what its aimed to do but we have it set up just as a stand alone test tool.

It also makes regression testing a whole bunch easier!

Give it a whirl and let me know how your using it!

A very simple automation framework using selenium builder

In one of my previous posts i wrote that i was going explain how i set up a very simple test automation framework using selenium builder, well here goes!

These instructions are for windozzzzz by the way!

First thing you going to need is firefox and the selenium builder plug in. Once you have these installed you can record you first script, instructions on how to do this are here.

Next we need to grab Se Interpreter, this will allow us to run our new script created using selenium builder via the cmd line.

Create a new folder and pop in the files you downloaded for Se Interpreter into it, then also pop in this folder the script you created using the selenium builder.

Now we want to run our recorded script from the cmd line, to do this your going to need to run a cmd from the folder you have you script file and your Se-builder files in.  The cmd should look something like this:

java -jar SeInterpreter.jar <path_to_script.json>

If you run this you will see firefox start up the script will run and then Firefox will close. You should see the logging in the cmd window, this will tell you whether the test passed or failed.

The next step is to get a log file created when you run a script, if we are going to be scheduling and running these tests we will need to check the log files after the test run to check our results.

We are going to need to add something to the cmd line in order to this:

java -jar SeInterpreter.jar <path_to_script.json>>C:\Path to logfile\MyLog.log 2>&1

If you now run this you will get a log file created in the folder you stipulate in the cmd above with the name you have given the log.

You can also create a test suite file, this has details of individual test files in it. This means that you can run a bunch of scripts but you only have to tell selenium interpreter to run one test suite file. You just have to switch the script name in the cmd to the test suite file and let the tool do the rest.  In your log file you will get the results of each script that is run (The ones you have stipulated in your test suite file).

Now we have the scripts and our cmd line sorted we need to schedule the scripts to run at a certain time. I like my scripts to run over night against the latest development build so i can check the next morning how the build did. I am running my test on windows so i created a bat file using the cmd line and scheduled this to run at 11pm each weekday night. I could have used the windows scheduler to do this but i found that it ran the scripts in headless mode (You could not see firefox opening up when the scripts were running on the desktop) This caused me a problem as some of my scripts tested pop up error messages and these need the browser gui to be open. The free tool i used to do this is here.

You can add more test scripts and add them to your test suite and these will then get run at the time you have scheduled. You can also add more suites to the bat file as new lines e.g.:

java -jar SeInterpreter.jar <path_to_test_suite1.json>>C:\Path to logfile\MyLog1.log 2>&1
java - jar SeInterpreter.jar <path_to_test_suite2>>C:\Path to logfile\MyLog2.log 2>&1

Each one of these lines will create its own log file for you to use to check the results.

There you have it, possibly the easiest automation frame work possible and you can get something like this set up in a few hours or less then extend it as you think of more tests!


Web form validation testing part 1

Most web sites will have forms, places were a user can add details in order to provoke an action from the web site. Stuff like signing up to a customer account or a checkout were you have to add your address and payment details. Most forms will have some sort of validation so that the data entered by the user is usable by the web site and also to help the user to avoid mistakes.

The eCommerce site that my company users has lots of different forms and i wanted to write some comprehensive automated regression tests to check that the validation was working after we did any changes to the site. I was specifically interested in testing the input combinations that should be allowed and ones that should provoke an error that is displayed to the user.

I was not looking at how the form should handle the incorrect format of an input e.g. wrong format of email address, but rather the combinations of good data that could be entered into the form.

For example we have on the site a contact us form,  see screen shot below for the form fields:

Screen Shot 2014-09-06 at 18.56.41

There are a number of fields on this form that are compulsory and also there are different combinations of entries that are allowed and other ones that are not allowed.

The first thing i needed to do was to get all the possible combinations of inputs for this form. Not all of these will be valid but we can start to narrow down the test cases once we have created them all.

luckily there were a few tools available that will create a truth table of results when you give a number of different input fields. I used this one:

For the contact us form that has 9 input fields you just have to add nine fields into the truth table solver and hit solve. It will then spit out a matrix that should include every combination of inputs that are possible for the form. Next pop this into spreadsheet application, you should end up with something like this:

Screen Shot 2014-09-06 at 19.31.01

The ‘1’ in a cell indicates a valid input and the ‘0’ in the cell indicates no input.

You can see from the screen shot that the test results for these cells (This is just the first few rows in the sheet, there are 513 rows in the actual sheet!) are N/A that means that the combination of inputs for that row is not actually possible. You have to go thru your results and identify these cases along with the ones that will allow the form to be submitted and the ones that will provoke an error and shouldn’t allow the user to submit the form.

Now you will have a list of tests that will cover all the possible combinations of good data input, next thing is to create scripts that will test all your identifed tests that pass or fail.

Part 2 of this blog will focus on creating those automated scripts using the json format script that can be run using selenium builder!

Added benefits of bug tracking systems

If your involved in software development you need a bug tracker, there i said it. The reason for this post is to try to talk a little about the benefits of a good big tracking system, not the obvious  ones but some benefits that you may not have thought of.

1. They help you to put process into your bug fixing

By using status for the bug as it moves thru the fix process you can make sure that it passes through all the different gates before it gets released. Examples of different status that i use in my process are:

1. New = Ticket created but not yet looked at
2. Hard Shoulder = Tickets that cannot be moved due to something blocking development
3. Elaboration Doing = Ticket is in the process of being correctly spec’ed and time required estimated
4. Elaboration Done = Ticket has been spec’ed and time required estimated
5. Development Doing = Development work has started on the ticket
6. Development Done = Development work has been completed on the ticket
7. Peer Code Review Doing = New code is peer reviewed to check for errors (Static analysis and code should be run)
8. Peer Code Review Done = Review has been completed and any errors fixed
9. QA Doing = Functionality of new development tested by QA on beta
10. QA Done = Planned testing of new development functionality completed
11. UAT Doing = (UAT = User Acceptance Testing) End business users test the software to make sure it can handle required tasks in real-world scenarios, according to specifications. This is done on beta
12. UAT Done = UAT complete
13. Ready to deploy = Developer has prepared the new code to be deployed into the live environment
14. Done Live = Dev Op’s has deployed the new code to the live environment
15. Closed = All parties happy with new live functionality so ticket can be closed

This might seem like a rather long list but i have found there is definite value in each of these status and when followed the team has reassurance that any problems should be found before the functionality goes live.

2. They become a knowledge base

The first thing i do when a new bug is raised is do a search in the bug tracker for similar bugs, this provides to different things:

1. It tells me if the bug has already been raised

2. Its tells me if similar bugs have been raised. If this is the case i can look back on the issues recored in that bug and they may well help the developer diagnose the problem in the new similar bug. Also because i am logging what testing i did to check the bug fix i can often use a variation of this to carry out the fix test on the new similar bug, this can be a real time saver.

3. They help you identify troublesome areas of an application

Most bug trackers have reporting functionality, this used in conjunction with some custom bug status that are used to indicate areas of the application where the bug was found can help to identify weak points in the application. This information can then be used to work out areas that seem fragile where it would be useful to have a good set of regression tests to pick up problems early. Also it could help to flag up code areas where a refactor of the code should be done to make that area more robust.

4. Identify the dev’s who’s code has the most bugs..

Possibly a bit divisive this one but rather than use it as a stick to beat the dev with it should be used to help the dev get some extra training or mentoring to help them to improve there code.

5. See which dev is overloaded

Which dev has the most bugs assigned to them?, should some of the bugs assigned to them be spread round to other dev’s in the team?

Thats a few added benefits of bug tracking systems that you maybe haven’t thought about. I am sure there are more, if you have any examples please put some in the comments.



Some thoughts on automated web GUI testing…

I have used a quite a number of test automation tools during my career, some were expensive enterprise software packages such as QTP and Loadrunner whilst others were open source such as selenium.

Whether you think open source tools are better than paid for ones is a matter of what suits your needs but a powerful driver has and always will be cost. However thats only really half the story because even though some paid for tools may have a high up front cost there easy of use and ROI maybe better than open source tools.

Let me explain this statement, generally when using open source tools some coding knowledge is required, what if you test team does not have this knowledge? sure they can learn but this is going to be an investment and therefore a cost. They are going to take time to get up to speed, time that could be spent writing test scripts and automating tests if the testers could just get stuck in right away. The benefit of paid for tools is that they tend to be easier to get started with and can be used happily with no programming knowledge (QTP) and you should get good customer support if you have any issues.

However having just said all that i think there is finally a tool that is open source (Free) that is also very easy to use…please take a bow Selenium Builder!

I have been using Selenium builder for around a year now and its pretty great. I use it for regression testing the ecommerce site that my company uses to sell its stuff. The front end site is a mixture of html and javascript and Selenium builder seems to work pretty well when i use it.

Selenium builder is a firefox plug in, but before you all shout what about the other browsers i need to test ! It maybe worth considering that aside from visual differences most functional issues will be apparent in every browser so just using FF means that i am not going to miss many bugs at all, you of course might be testing a complex web app and in that case you will need to have good cross browser regression test coverage. You can also do this with selenium builder scripts but you will need to configure your environment to be running a selenium server.

One to the cool things about selenium builder is that is default format of script is JSON so they are easy readable and therefore easy to edit. Also the firefox plug in has a nice GUI that you can use for editing scripts without even having to open a script in an editor.

It’s one of the few record and playback tools that i have used that works well. In my experience recored scripts playback very consistently and of course this makes scripts very quick to create.

Thanks for reading!