All posts by julian

QA Reviews

As part of the development process we follow the developers have a peer code review. This is done after the developer is happy with their completed code. It involves another developer inspecting the code to look for stuff like:

  • Possible logic errors.
  • Areas of the code that appear to be unreachable.
  • Syntactic errors (Although a good IDE should pick most of these up).

It’s a great part of the process because it can often find problems even before the work has been given to the test team to look at.

I think that testers could also benefit from having a similar review but carried out on the testing that they have done and recorded (We can do the same peer code review on code we have written to test within the test team of course such as python code written to perform a selenium test).

In order to do this exercise the testers must document the testing they have carried out throughly, this should be a given !

The things we can look for in QA reviews are:

  • Has the basic happy path test been done, an example would be “If a user on a website signs up for a news letter email do they get an email ?”
  • Have any breakage tests been done? an example would be “If a user on a website signs up for a news letter but the email that the user tries to submit does not contain an ‘@'”
  • If the feature / change is on a web application have all the supported browsers been tested ?
  • If the feature change is on a web application has it been tested on mobile devices ? (If the application is designed to be used on mobile devices of course!)
  • Are there any areas of the application that could be impacted by the new feature / change? have these been regression tested ?
  • Is the reviewers understanding of the feature / change that has been tested the same as the testers who carried out the testing work ?, if it is not this might indicate that the tester has not quite understood what the feature / change was and due to this might not have performed the correct testing.
  •  If bugs were found during the testing process and fixed, was the feature / change then regression tested to check that the bug fix did not break another part of the feature / change.

Maybe you could think of some more good things to include in the QA reviews that you could do?

Happy testing!

 

Helpful Vs Unhelpful War stories

Unhelpful:

We have all heard them, someone in work starts telling you about how the last place they worked was so much better because of some reason, examples of unhelpful war stories are:

  • In my last job we used better technology (No real detail it was just better…)
  • In my last job they had free drinks and biscuits (So it was easier to put on weight?)
  • In my last job the pay was better (So why did you leave?)
  • In my last job they were a bigger company (Is bigger always better?)

I myself have not been immune to the lure of trotting one out an unhelpful war story on occasion but recently i have made a effort to stop. My reasons are:

  • It usually doesn’t help things.
  • It makes people wonder why you’re working there.
  • It can make you look silly. (So if your last job was that much better why did you leave?)
  • It can effect your teams moral and you can end up being the person seen by management as the bad apple.

Helpful:

If we are going to tell war stories we should make an effort to make them helpful and constructive, examples are:

  • In my last job we used an agile development process and we all really enjoyed working that way.
  • In my last job we used a great free testing tool that helped us find lots of bugs, could we try using that here?
  • In my last job I used python to write my test scripts i could help some other team member to learn it.

Remember to try to only tell helpful war stories!

 

 

How to pass in command line arguments when scripting selenium tests using python unittest

I have been using a python unittest framework to write my selenium scripts in for about 6 months now and i must say i am really enjoying it.

Not having been a developer learning how to write scripts in python has really opened my eyes to the more powerful features of selenium.

One of the things that i need to cover in my role is testing on a number of different ecommerce web sites. Our main brand site has 6 different variations for different countries. These are:

  • UK
  • USA
  • Australia
  • EU (Site is in English but uses Euros as the currency)
  • France
  • Germany

Many of the elements of these sites are shared such as the buttons and forms so I didn’t really want to create a script to test each, this would mean more maintenance if things change and more work to create.

I wanted to create a script that i could tell to run on any of these sites. After a bit of head scratching and googling i decided that the best way to do this would be to pass in command line arguments to tell the script to run for a particular site. The problem with this is that when you run a python unittest script it has its own set of command line arguments you can pass in so on the face of it creating and using my own seemed to be a non starter.

However i then found a way around this problem, what i needed to do was to extract my command line arguments into separate variables and remove them from the sys.arg before calling unittest.main()

Example of this at the end of a script.

Screen Shot 2015-09-17 at 23.23.55

 

And this is the line that uses that arg to get the passed in website url:

Screen Shot 2015-09-17 at 23.34.55

Now I can pass in the website i want the test to run on as the first sys.argv and then use this with the webdriver.get command to start my script on the right site.

Hope this helps if you’re looking for a similar solution for your python unitest selenium script. I will be posting more on how i have expanded my scripts in other ways by using command line arguments soon.

Thanks for reading!

Recruiting testers part 3

This is the third and final post on the subject of recruiting testers.

The selection of interview candidates and the interviews are done so its time to check out interview feed back to try to identify the strongest candidate. Back in the second post on this subject i mentioned that i was scoring the candidates out of 10 for each of the items in each sections 2 and 3 of the interview. There is a bit more to the feedback than just noting the candidates scores however, the best way to show this is to look at some

See below for the format and example content from an interview write-up:

The interview plan (Didnt quite keep the timings!)

  1. 20 minutes for questions on their CV
  2. 20 minutes on questions around their competencies based on the things we required on the job ad.
  3. 20 minute for the test site test that **** wrote for me to see how many bugs they find and then get them to pick there favourite and write a bug report for it. Will go thru these results after the interview is complete.

CV Questions (Scores out of 10 next to questions only questions that i managed to ask will be scored!):

  1. What sort of tests did you create with selenium IDE / TestNG / Webdriver, any Limitations you found? (0)
  1. Which do you prefer Agile or Waterfall (5)
  1. What did you get from doing the ISTQB certification (5)
  1. Can you explain some test cases based on Behaviour-driven development (5)
  1. How did you tackle cross browser testing (5)
  1. How do you create your test requirements (0)
  1. How did you use data driven testing (0)
  1. How do you tackle testing time estimation (5)

Competency based questions:

  1. Talk about what process you would use when a new feature is being development and you will need to test it. (5)
  1. Talk about a time when you worked with end users to successfully UAT a new feature / product
  1. Talk about a time where you dealt with a developer to fix a bug you had raised.
  1. Give an example of when you improved a process that you were working with. (5)
  1. Talk about a time when you had to flag up issues to management that were impacting the testing phase of a project.(5)

Score out of 80 = 40

Interview observations

  1. Lots of experience but sometimes they did not explain the things they had done very clearly.
  2. Even though selenium featured a great deal in his cv they did not really know much about it.
  3. Didnt seem to know much about the buzzword in their cv when questioned eg Behaviour driven development.
  4. Did have management experience and wanted to learn more, not sure they would be able to fulfill this in the short-term with the role on offer.
  5. They made positive improvements to processes and defined some when there were none in place.
  6. They had worked at the start of projects to improve requirements
  7. They had used to having a large amount of documentation around testing, but is fine with having less.
  8. They had suggested tools to use as they had experience of them in the past, not because they did a POC…
  9. Didnt get a feel of how they worked with developers, what their relationship was like with them…
  10. Couldn’t really get much of a feel for the applications they had tested in his previous roles.

QA web site test notes transposed from written notes of candidate:

  1. I gave card number as ‘AAAA’, then application raised a bug as card number is too short but as per functionality card number should accept numeric values. this is high severity bug
  1. Application focusing on only one bug at a time. Action: I gave card number as “12345678912345” and name on card as “1234” and Expiry date as “10/15”. I gave 3 fields with wrong data, but application focusing on card number only. It raised a bug “card number is invalid”, but it did not give errors on name on the card, Ti should contain numeric values and Expiry data, It should contain numeric values and Expiry date, it should be 11/15.
  1. Post code: For post code i gave input as “777” As per functionality the application should rise a bug. It is not the standard format.
  1. Email: I gave input as “uuuu”. Application accepting the input. As per functionality. The email should be xyz@gmail etc
  1. I kept the “last name” text field empty and clicked on the pay now. “Application raised a bug “Please enter first name” but as per functionality application should raise “Please enter first name” but as per functionality application should raise “please enter last name”
  1. Card No: I gave card NO as “ 4444 4444 4444 4444” as per functionality application should contain numbers but text field card NO Not working properly for the number as well.
  1. FIRST Name: Should accept numeric values, but accepting numbers. It is wrong
  1. Last name: Should accept numeric values only but accepted non numbers. This is wrong.

Example bug report :

Bug number

Requirement ID

Bug description

Status

Severity

Priority

 

My thoughts on QA Test output from candidate:

The output from the test shows a fairly detailed testing of the validation of the customer details page but there was very little focus on the other pages. There were bugs to find on those other pages. Didn’t see the 404 page when you click the link for contact us, should really of at least clicked all the buttons for the pages there were not many!. Also did not notice that the basket does not add up the products correctly and that the currency converter does not work.

Even though i asked the candidate to pick a bug found and create a bug report from it they didn’t do this and instead gave me some things that a bug report should contain.

Overall thoughts:

Candidate seemed to have good knowledge of testing processes but seemed to rather hide behind buzzwords which when asked about revealed a lack of knowledge of them. During explanations reiterated a lot of the same points again. They were  keen to expand into automation which is encouraging. Slightly surprised they missed some rather easy to spot bugs in the test but he did do some good testing of the form validation. My conclusion is that they didn’t quite match up to their CV but would still be able to do an adequate job.

After the all the interviews has taken place i had a number of these candidate feedback documents to go through. Having this level of feedback really helped to make a fair and informed decision on who we wanted to offer the position.

I hope this series of posts will help you if your recruiting for your team.

Thanks for reading!

Recruiting testers part 2

When i started to look at the best way to interview software testers the first thing i did was to do a google search…lots of trawling through hits later i had a good idea what other people had done.

I took some of the good stuff i found and mixed in some of the things that i thought were good based on my own experience of being interviewed. I also wanted to try and be as subjective as possible, so i  came up with this plan for each interview:

Part one, general questions:

Try to put them at ease with some general questions on there background and hobbies, lots of interviews completely miss the “what you do in your spare time” bit, probably because lots of CV edited by agents take that bit out. I think you can tell quite a lot about a person if you take the time to find out what there interests outside of work are. Also at this point i would ask them what they knew about the company, this shows if they have been engaged enough to do some research.  Don’t forget to ask them if they would like a glass of water, they are going to be doing the bulk of the talking so lets help them to not get a dry mouth!

Part two, questions about previous experience :

Questions relating to there previous experience as detailed in there CV. This is were i start to introduce a score for each of the answers given (Remember that bit about being as subjective as possible). This is where it can get interesting if your interviewing someone with limited testing experience because you will have to look harder to find experience that would be relevant to the role your employing for. I tend to look for evidence of these type of things in there work experience (Computer skills should be a given):

  • Planning
  • Attention to detail
  • Team skills
  • Analitical reasoning

Some examples of questions i have used in the past in this section of the interview are:

When the CV mentioned web testing:

  • What are the main issues you found when testing web based applications?

When the CV mentioned use of a bug tracking system:

  • What system have you been using for bug tracking and test management? What do you think the pros and cons are of those systems?

When the CV mentioned using SQL:

  • How did you use SQL in your testing?, what did you gain from using it?

You probably get the general idea by now. I then score the answers out of ten.

Part 3, scenario based questions:

Part 3 of the interview focuses on scenario based questions, these are to try to help you understand how the candidate will respond to different scenarios that play out in the working environment. Here are some examples i have used:

  • Talk about how you would work with a developer to fix a bug you had raised.
  • Talk how you would flag up issues to management that were impacting the testing phase of a project.
  • Talk about how you used automation to test a new feature.

Again I then score the answers out of ten.

Part 4, the test:

Part 4 of the interview process was a test on an example website. With the help of a friendly dev Op’s a test site was written that took the form of a very simple eCommerce site that had some product pages and a basket page and a section of a checkout. The tasks included in this test were:

  1. Candidate had 20 minutes going through the site trying to find any issues that were there. There were around 20 issues written into the site ranging from incorrect maths in the checkout to incorrect currency symbols and dead links.
  2. The candidate was then asked to note down each issue they found and give it a severity rating.
  3. Once the test was over they were asked them to write up one of the issues that they found into what they thought would be a good bug report format.

I should stress that the object of this test was not to find ALL the bugs but rather to see if they picked up on all the most serious issues. These were the ones that would cause a company the most problems if they were in the website code when it went live.

Part 5, any questions.

The final part of the interview would be when i ask if the candidate has any questions they they would like answered. If they have some good questions then this is a good sign that again shows that they have done some prep and are generally interested in the role on offer. After i had answered any questions the candidate was taken to  the departments office area and shown the development board and given a run through on the development process we are using.

After this we terminate the interview and they are taken to reception. I then like to give them an indication of how long they will have to wait for the result of the interview, i think its polite to do this so they know they won’t be left hanging on for an answer.

The 3rd and final post detailing the recruitment of testers will talk a little about how you should go about reviewing the candidates performance during the interview process and finally offering the job to the candidate that was the strongest.

 

Recruiting testers part 1

I recently got the chance to expand my test team, i already have one team member who was doing a great job but i needed more, we were holding up the development process as features were waiting to be tested and between us we were not getting through them in a timely manner.

There are a few ways you can go with this process, you can try your hand at posting a job ad on one of the job boards e.g. CW Jobs or Dice. This is cheap and you can get a job ad up in minutes. The other option is to use a recruitment agency, this is not as cheap but can, actually save some of your time which is expensive right?

I had tried the job posting option to get my first team member and i struck lucky, that team member had already proved to be excellent and i felt a little smug about the money i had saved the company by not using a recruitment agent. However what i had not remembered so well when i started to recruit for the next team member was how much of my time i have to spend going through CV’s that were really not suited to the role at all. Also the arranging of all the interview times with the candidates was a rather unwelcome distraction.

At this point after reviewing yet another completely unrelated to the job on offer CV i decided enough was enough and i would have to try a recruitment agent so i could actually get on with my day job.

Now here is the thing, agents generally work on a % of the first years salary your offering to the successful applicant. The rate they quote can be in my experience anything between 15 and 25 %. The thing to remember is there are a lot of agencies out there looking for business so use this to your advantage. Negotiate the rate with them, you should be able to get the rate down quite a bit from the figure they will first quote you, remember if they don’t want to budge walk away, there are plenty of others to try.

The other thing to consider is do you want a very experienced tester or a less experienced tester. The benefits of an experienced tester are they won’t take much work to get upto speed (provided their  experience is in the same area the job your offering happens to be in) and they should be able to work without too much handholding. However they are going to be more expensive. The other option is to look for less experienced testers who are keen to learn. It was also for me about giving these less experience testers a chance, after all if someone hadn’t given me a chance i wouldn’t be writing this!.

You don’t always have to have a background in IT to be a good tester, i came from a project analyst background, when i stumbled in to testing through doing some UAT as we were short staffed. To prove this point the first tester that i recruited had worked in a call centre but had funded themselves through the ISEB foundation certificate in software testing and was really set on becoming a software tester. There attitude was excellent and they have since become a real cornerstone of the team.

The other benefits i have found with this approach are:

  • Cheaper to recruit.
  • Easier to mould to the testing approach your using.

Of course the downsides are that they need more handholding and general management than an experienced tester.  This can be a big demand on your time and when i was the only tester getting the first member of my team up to speed caused a bit of a bottle neck with the jobs waiting to be tested. However is was worth it as once this tester was up to speed we managed to get through the backlog quickly and kept the job list moving smoothly, which was the point of expanding the test team in the first place…result!

The great thing is with this approach (Apart from the money saved!) is that once you have got your first less experienced tester up to speed you can use them to document there own learning and use this to help the next new team member to get unto speed.

If you stagger your recruitment well you can find that your team can share the training duties with you so for example my first recruit ended up doing the bulk of the training to get my second recruit up and running and the same with my third and forth recruits.

Now i have a great team all up to speed and contributing to improving the quality of the software that gets shipped.

When putting the job package together we offered an approach where once the tester proves themselves they get rewarded. I think its good to have something to aim for when you start a job so we did this:

  • A pay rise after successful completion of the probation period (3 months in our case)
  • Training budget to pick a course to go on, again after successful completion of the probation period.

In part 2 of this blog post i will cover how i tackled the interview process.