Recruiting testers part 3

This is the third and final post on the subject of recruiting testers.

The selection of interview candidates and the interviews are done so its time to check out interview feed back to try to identify the strongest candidate. Back in the second post on this subject i mentioned that i was scoring the candidates out of 10 for each of the items in each sections 2 and 3 of the interview. There is a bit more to the feedback than just noting the candidates scores however, the best way to show this is to look at some

See below for the format and example content from an interview write-up:

The interview plan (Didnt quite keep the timings!)

  1. 20 minutes for questions on their CV
  2. 20 minutes on questions around their competencies based on the things we required on the job ad.
  3. 20 minute for the test site test that **** wrote for me to see how many bugs they find and then get them to pick there favourite and write a bug report for it. Will go thru these results after the interview is complete.

CV Questions (Scores out of 10 next to questions only questions that i managed to ask will be scored!):

  1. What sort of tests did you create with selenium IDE / TestNG / Webdriver, any Limitations you found? (0)
  1. Which do you prefer Agile or Waterfall (5)
  1. What did you get from doing the ISTQB certification (5)
  1. Can you explain some test cases based on Behaviour-driven development (5)
  1. How did you tackle cross browser testing (5)
  1. How do you create your test requirements (0)
  1. How did you use data driven testing (0)
  1. How do you tackle testing time estimation (5)

Competency based questions:

  1. Talk about what process you would use when a new feature is being development and you will need to test it. (5)
  1. Talk about a time when you worked with end users to successfully UAT a new feature / product
  1. Talk about a time where you dealt with a developer to fix a bug you had raised.
  1. Give an example of when you improved a process that you were working with. (5)
  1. Talk about a time when you had to flag up issues to management that were impacting the testing phase of a project.(5)

Score out of 80 = 40

Interview observations

  1. Lots of experience but sometimes they did not explain the things they had done very clearly.
  2. Even though selenium featured a great deal in his cv they did not really know much about it.
  3. Didnt seem to know much about the buzzword in their cv when questioned eg Behaviour driven development.
  4. Did have management experience and wanted to learn more, not sure they would be able to fulfill this in the short-term with the role on offer.
  5. They made positive improvements to processes and defined some when there were none in place.
  6. They had worked at the start of projects to improve requirements
  7. They had used to having a large amount of documentation around testing, but is fine with having less.
  8. They had suggested tools to use as they had experience of them in the past, not because they did a POC…
  9. Didnt get a feel of how they worked with developers, what their relationship was like with them…
  10. Couldn’t really get much of a feel for the applications they had tested in his previous roles.

QA web site test notes transposed from written notes of candidate:

  1. I gave card number as ‘AAAA’, then application raised a bug as card number is too short but as per functionality card number should accept numeric values. this is high severity bug
  1. Application focusing on only one bug at a time. Action: I gave card number as “12345678912345” and name on card as “1234” and Expiry date as “10/15”. I gave 3 fields with wrong data, but application focusing on card number only. It raised a bug “card number is invalid”, but it did not give errors on name on the card, Ti should contain numeric values and Expiry data, It should contain numeric values and Expiry date, it should be 11/15.
  1. Post code: For post code i gave input as “777” As per functionality the application should rise a bug. It is not the standard format.
  1. Email: I gave input as “uuuu”. Application accepting the input. As per functionality. The email should be xyz@gmail etc
  1. I kept the “last name” text field empty and clicked on the pay now. “Application raised a bug “Please enter first name” but as per functionality application should raise “Please enter first name” but as per functionality application should raise “please enter last name”
  1. Card No: I gave card NO as “ 4444 4444 4444 4444” as per functionality application should contain numbers but text field card NO Not working properly for the number as well.
  1. FIRST Name: Should accept numeric values, but accepting numbers. It is wrong
  1. Last name: Should accept numeric values only but accepted non numbers. This is wrong.

Example bug report :

Bug number

Requirement ID

Bug description

Status

Severity

Priority

 

My thoughts on QA Test output from candidate:

The output from the test shows a fairly detailed testing of the validation of the customer details page but there was very little focus on the other pages. There were bugs to find on those other pages. Didn’t see the 404 page when you click the link for contact us, should really of at least clicked all the buttons for the pages there were not many!. Also did not notice that the basket does not add up the products correctly and that the currency converter does not work.

Even though i asked the candidate to pick a bug found and create a bug report from it they didn’t do this and instead gave me some things that a bug report should contain.

Overall thoughts:

Candidate seemed to have good knowledge of testing processes but seemed to rather hide behind buzzwords which when asked about revealed a lack of knowledge of them. During explanations reiterated a lot of the same points again. They were  keen to expand into automation which is encouraging. Slightly surprised they missed some rather easy to spot bugs in the test but he did do some good testing of the form validation. My conclusion is that they didn’t quite match up to their CV but would still be able to do an adequate job.

After the all the interviews has taken place i had a number of these candidate feedback documents to go through. Having this level of feedback really helped to make a fair and informed decision on who we wanted to offer the position.

I hope this series of posts will help you if your recruiting for your team.

Thanks for reading!

Recruiting testers part 2

When i started to look at the best way to interview software testers the first thing i did was to do a google search…lots of trawling through hits later i had a good idea what other people had done.

I took some of the good stuff i found and mixed in some of the things that i thought were good based on my own experience of being interviewed. I also wanted to try and be as subjective as possible, so i  came up with this plan for each interview:

Part one, general questions:

Try to put them at ease with some general questions on there background and hobbies, lots of interviews completely miss the “what you do in your spare time” bit, probably because lots of CV edited by agents take that bit out. I think you can tell quite a lot about a person if you take the time to find out what there interests outside of work are. Also at this point i would ask them what they knew about the company, this shows if they have been engaged enough to do some research.  Don’t forget to ask them if they would like a glass of water, they are going to be doing the bulk of the talking so lets help them to not get a dry mouth!

Part two, questions about previous experience :

Questions relating to there previous experience as detailed in there CV. This is were i start to introduce a score for each of the answers given (Remember that bit about being as subjective as possible). This is where it can get interesting if your interviewing someone with limited testing experience because you will have to look harder to find experience that would be relevant to the role your employing for. I tend to look for evidence of these type of things in there work experience (Computer skills should be a given):

  • Planning
  • Attention to detail
  • Team skills
  • Analitical reasoning

Some examples of questions i have used in the past in this section of the interview are:

When the CV mentioned web testing:

  • What are the main issues you found when testing web based applications?

When the CV mentioned use of a bug tracking system:

  • What system have you been using for bug tracking and test management? What do you think the pros and cons are of those systems?

When the CV mentioned using SQL:

  • How did you use SQL in your testing?, what did you gain from using it?

You probably get the general idea by now. I then score the answers out of ten.

Part 3, scenario based questions:

Part 3 of the interview focuses on scenario based questions, these are to try to help you understand how the candidate will respond to different scenarios that play out in the working environment. Here are some examples i have used:

  • Talk about how you would work with a developer to fix a bug you had raised.
  • Talk how you would flag up issues to management that were impacting the testing phase of a project.
  • Talk about how you used automation to test a new feature.

Again I then score the answers out of ten.

Part 4, the test:

Part 4 of the interview process was a test on an example website. With the help of a friendly dev Op’s a test site was written that took the form of a very simple eCommerce site that had some product pages and a basket page and a section of a checkout. The tasks included in this test were:

  1. Candidate had 20 minutes going through the site trying to find any issues that were there. There were around 20 issues written into the site ranging from incorrect maths in the checkout to incorrect currency symbols and dead links.
  2. The candidate was then asked to note down each issue they found and give it a severity rating.
  3. Once the test was over they were asked them to write up one of the issues that they found into what they thought would be a good bug report format.

I should stress that the object of this test was not to find ALL the bugs but rather to see if they picked up on all the most serious issues. These were the ones that would cause a company the most problems if they were in the website code when it went live.

Part 5, any questions.

The final part of the interview would be when i ask if the candidate has any questions they they would like answered. If they have some good questions then this is a good sign that again shows that they have done some prep and are generally interested in the role on offer. After i had answered any questions the candidate was taken to  the departments office area and shown the development board and given a run through on the development process we are using.

After this we terminate the interview and they are taken to reception. I then like to give them an indication of how long they will have to wait for the result of the interview, i think its polite to do this so they know they won’t be left hanging on for an answer.

The 3rd and final post detailing the recruitment of testers will talk a little about how you should go about reviewing the candidates performance during the interview process and finally offering the job to the candidate that was the strongest.