Monthly Archives: July 2019

When should we stop using Record and Playback in Test Automation Development?

On 17th July 2019, I presented my talk on Record and Playback in test automation. It was the first time I gave this talk and was thrilled at the positive response. I also enjoyed answering the many questions that followed the talk. This is one of my favourite questions which I want to answer in more detail.

How long did it take for you to get to the stage where you no longer needed to use Record and Playback?

When I started out using test automation I did have some programming experience, however this was in Java, not C# which I was expected to use. It had also been about a year and a half since I’d done any programming, so I was a little rusty. My previous experience meant I was able pick up C# quickly. It wasn’t long before I was only using Record and Playback as a guide rather than relying on it completely.

Using Record and Playback is a choice. I consider my C# programming skills sufficient enough to not have to use Record and Playback, however this doesn’t mean I’ve stopped using it. As well as a great learning tool for someone new to test automation, it also allows experienced programmers develop automated tests quickly.

Not everything can be recorded. You cannot automate using Record and Playback alone. Sometimes I use Record and Playback, sometimes I don’t. Depends entirely on what I’m trying to automate.

Should we stop using Record and Playback?

Lets assume you are working in test team that is full of experienced test automation developers who are proficient programmers. Does this mean you should stop using Record and Playback?

It is completely your choice. If you have the skills and confidence to develop automated test cases without the use of Record and Playback, then you don’t have to use it. However, there are a few things that need to be considered before disregarding it completely.

Help people who are new to test automation development

Occasionally, you may get new people join the test team. There may be an expectation for them to take part in test automation development. It may be some time before their skills align with your own.

Attempting test automation development for the first time can be a daunting prospect. Mistakes will be made, time will be wasted, but this should not put people off. Some practice and learning is required before any value can be gained. Inexperience should not be preventing people from developing automated test cases.

For the sake of integrating new team members, we should be providing the development team with the option to use record and playback if they choose. This will help encourage new developers to gain the confidence to develop robust, maintainable and reliable automated test cases.

A tool to speed up the creation of automated tests

Another thing to consider is how Record and Playback can be used to make our work more efficient. If there is a tool available that could improve the way you do your job, then there should be no shame in using it.

A calculator is one example of this. It is possible to solve basic arithmetic without one. However, choosing to use a calculator in no way undermines our own intelligence.

Test automation itself is another example. We can still run these tests manually. But automation can still drastically improve the our testing efforts.

Record and Playback can be used to create tests quickly by auto-generating code required for the test to run. This code can then be adapted to improve the maintainability and robustness of the tests.

Your Choice

I’ve always said that Record and Playback must never be the sole method for developing automated test cases. If you do choose to use it, then you must also take the time to examine the auto-generated code and adapt it accordingly. Tests developed using record and playback alone will be unreliable and difficult to maintain.

Record and Playback is a tool that can be used to help us develop automated tests. It is your own choice if you decide to use it or not. If you can develop automated tests without it then you don’t have to use it.

I will be presenting my talk ‘The Joy of Record and Playback in Test Automation’ at Test Bash Manchester and TestCon Europe later this year.

Main image from http://www.publicdomainpictures.net

Advertisements

Learning about Personas – What I Read Last Week (28th July 2019)

A few months ago I came across the term ‘Abuse Cases’ in a blog post by Nicola Owen. It was a term that I’d never come across before. This inspired me to write a blog post where I provided my own interpretation of Abuse Cases.

Abuse Cases – Understanding Motives

When Ministry of Testing announced their latest Power Hour event on Personas, I was eager to submit a question about Abuse Cases. Gem Hill beat me to it (although she didn’t use the term Abuse Cases). I see Abuse Cases as being examples of ways the application could be misused. This question didn’t just focus on ways the product could be misused, but also how a product could be attacked.

Check out The Club for a full list of questions and answers from the Power Hour.

Events

Personas – Power Hour
Cassandra H. Leung dedicated an entire hour to answering questions about Personas on The Club. Questions were asked about personas templates, edge cases and personas for those who would misuse an application.

Social Media Discussion

Superhuman discussion (Twitter)
Twitter discussion shared by Cassandra via the Persona Power Hour as an example of a persona created to show how an application could be misused.

Smoke Testing vs Sanity Testing
After finding the discussion about smoke and sanity testing on LinkedIn, I decided to setup another discussion on The Club to see if anyone else had some ideas to share.

Articles and Blog Posts

Learning from Failure: The tricky iOS Environment – Melissa Eaden – Testing and Movies and Stuff
This article contains a tale of a mistake that led to iOS issues and the lessons learned from this mistake.
“Issues…can give us an opportunity to change practices, habits, and better understand the system we are working with.”

“Cheating” Is Necessary – Melissa Eaden – Testing and Movies and Stuff
Is it cheating to look things up? Or ask for clarification?

How to form a regression testing plan with these 5 questions – Mike Kelly – Tech Target
There are many things to consider when setting up a regression test plan. Here, we look at questions about goals, coverage, techniques, maintenance, environment and reporting that should be asked while putting together a regression test plan.

It’s Automation Logs! Better Than Bad, They’re Good! – Paul Grizzaffi – Responsible Automation
In this article, we look at the importance of useful logging and what is required to make it useful.

Testers, Please speak to the developers
This week I published my write up of the 99 second talk I gave at the Birmingham test meetup last week. In this post I talk about the importance of speaking to the developers. Communication ensures that everything understands the requirements and identify ways to make the application more testable.

Other blogs that share lists of test related articles

https://5blogs.wordpress.com/ (daily)
http://blog.testingcurator.com/ (weekly)
http://thatsabug.com/ (weekly)
https://weapontester.com/tea-time (weekly)
https://www.ministryoftesting.com/feeds/blogs

Testing Conferences
https://testingconferences.org/

The Club
https://club.ministryoftesting.com/
A forum for discussing, asking questions, answering questions, and requesting help. Run by the Ministry of Testing.

Feel free to recommend anything that you think is of interest.
Main image taken from http://www.publicdomainpictures.net

Testers, Please Speak to the Developers!

This blog post is based on a 99 second talk I gave on 17th July 2019 at the Birmingham #MidsTest meetup.

Hands Holding Jigsaw
Developers and testers working together – www.publicdomainpictures.net

Once, a change had to be made to the software. This particular change was something that could not be controlled via the user interface. It was also not possible to observe the change via the user interface.

When we think about the testability, if something cannot be observed or controlled via the User Interface, this would mean that this change was untestable – right?

There are other ways.

Don’t rely on the User Interface

Sometimes a change might be made to the application that isn’t user facing. It might be something that the user should not change or view for safety or confidentiality reasons.

From a usability and user experience point of view, being able to see and interact with the user interface is essential. However, there is much more happening beneath the surface that the user never sees. There is likely to come a time where this needs to be tested.

Understanding Requirements

Contrary to popular belief, developers and testers should not be enemies. They can help each other. A Whole Team Testing approach is being discussed throughout the testing community. Testers can help developers and developers can help testers.

When there is a new work item or change request, encouraging testers and developers to discuss the requirements early on can avoid any misunderstandings later on. A failed test because either the developer or tester did not understand the requirements wastes time.

Making an application testable, making a defect fixable

Communication can also be used so both tester and developer fully understands what is required to complete a work item. A work item should not be complete until it has passed testing by a tester.

If something is not easily testable, then the developer needs to make it testable. To do this, the developer needs to know what the tester needs.

If a defect is found while testing, then the tester needs to provide enough information for the developer to fix the defect. To do this, the tester needs to know what information the developer needs.

Introducing extra logging

When told a change had to be made to the application that could not be controlled or observed via the user interface, I started out by talking with the developer.

We discussed the requirements to make sure we both understood the change and why it was needed.

We then discussed what we needed to fully implement this change. We decided that adding some additional logging would help with testing. We discussed what and when information was required.

Extra logging helps both the developer and the tester. Both can benefit for the information it provides. The tester benefits by having a better understanding of what is happening beneath the user interface. The developer can also use this information to help fix any defects found.

Mutual Understanding

By speaking to the developer before the change was implemented, we were able to reach a mutual understanding. We both agreed on the requirements and what was needed to make the the change testable.

Agreeing on requirements early on can reduce delays later on. Making sure that something is testable improves the quality of the application. Working together improves the efficiency of the entire development process.

Talking about Record and Playback at the Birmingham Test Meetup – What I Read Last Week (21st July 2019)

This week I attended the Birmingham #MidsTest meetup, where I presented my talk “The Joy of Record and Playback in Test Automation” for the first time. I am due to give this same talk at TestBash Manchester and Test Con Europe, both in October this year.

I was great to be able to give this talk before these conferences. I can practice and read through each slide as much as possible, but it is nothing like presenting in front of actual people. I was able to highlight a few areas where things did and didn’t work. This allows me to make some improvements before I next give this talk.

I also gave another 99 second talks – I love these. This time I talked about the importance of communicating with developers to help with testing efforts. I’ll write this up into a blog post at some point next week.

Future Events

Personas – Power Hour
Ministry of testing are hosting another power hour on The Club, this time about Personas. If you have any questions, post them by 7pm on 24th July (British Summer Time).

Social Media Discussions

Smoke Testing vs Sanity Testing
I was asked my opinion on a LinkedIn discussion which I was more than happy to share. It was regarding the use of smoke and sanity testing. Smoke testing is where we test the key features in an application, but only do basic checks. Sanity testing is where only a specific feature is tested, but we go a lot deeper into checking that the feature works exactly as expected. I prefer to do sanity testing when a change has been made to the application, and smoke testing every 2 – 4 weeks depending on how much the system has changed. What are your thoughts?
Thanks to Bharath Selvam for bringing this post to my attention.

Excuses, Blame and Fear
I started a discussion on The Club a couple of months ago on testing excuses and if they are evidence of testing incompetence, negative workplace culture, or something else. A few people have added some interesting responses to the discussion in the last week.

Difference Between Static and Dynamic Testing?
Never be afraid to ask a question. The Club is a great place to ask for help and clarify someone you fully know or understand. Here, someone asks a question of the different between static and dynamic testing. Do you know?

Testability – Power Hour
This event took place the week before last, and had some interesting responses. Testability is a subject that has really interested me in the last few months. Questions I posts included one on testability for manual and automated testing, what we should ask for when asking for improvements on Testability, and what to do when something cannot be controlled or observed via the UI.

Articles and Blog posts

Maybe You Don’t Need a Date Picker – Adrian Roselli
Picking a date can be a complex and tedious process, especially on smaller mobile devices. Surely having to scroll through seemingly endless lists isn’t the best way?

Two Rules That Make You Look Smart When You Ask Questions – Dan Rockwell – Leadership Freak
There is no such thing as a stupid question, but there is such a thing as asking a question in a stupid way.

What to say in a standup – One Man – The life of one man
What is a standup and what do we need to say? Who needs to attend?

Other blogs that share lists of test related articles

https://5blogs.wordpress.com/ (daily)
http://blog.testingcurator.com/ (weekly)
http://thatsabug.com/ (weekly)
https://weapontester.com/tea-time (weekly)
https://www.ministryoftesting.com/feeds/blogs

Testing Conferences
https://testingconferences.org/

The Club
https://club.ministryoftesting.com/
A forum for discussing, asking questions, answering questions, and requesting help. Run by the Ministry of Testing.

Feel free to recommend anything that you think is of interest.
Main image taken from http://www.publicdomainpictures.net

Shift Left or Shift Right – Discovering what is in the bottle

“The problem is not that testing is the bottleneck. The problem is that you don’t know what’s in the bottle. That’s a problem that testing addresses”.

Michael Bolton

On 11th June 2019, I watched the Ministry of Testing Ask Me Anything Webinar on Shift Left and Shift Right. This blog post is a summary of what I learnt from this webinar and my own interpretation of what shift left and shift right is.

A new feature has been completed and it is ready to be released. But, there is something preventing this from happening – testing!

In a lot of cases, there is this time between feature completion and feature delivery where testing takes place. The problem with this is that it leaves little time for fixing any defects found during testing. If the tester finds a critical defect, the team are faced with a difficult decision – either fix the bug and delay the release, or release the feature with the bug.

Graph showing how testing effort peaks just before the release of an application or feature. This is a scenario where there is no shift left or shift right.

If time and testing efforts were plotted on a graph, there would be a massive peak towards the end of the development process. Introducing shift left and shift right allows that peak to be smoothed out.

Shift left and shift right allows that peak to be smoothed out through process improvement.

Shift Left

Shift left involves introduces earlier testing tasks that will ease the burden of testing that tends to increase just before feature release. Tasks can include earlier reviews of requirements and documentation, plus earlier planning. There is also the additional of earlier testing, particularly through other layers of the testing pyramid (Service, API).

The main purpose of shift left, is improving the testing process. Speeding up the time between feature completion and feature release. This can be made possible through earlier testing, reviews and planning and the introduction of testing automation.

Shift Right

Unlike Shift left, shift right is not so much about actual testing. Shift Right is about gathering information post release. This information is then fed back into the testing process. Before watching the ask me anything webinar, I didn’t know anything about shift right. When first hearing the term, I imagined post production testing. I’ve always been wary of this as there can be issues when testing with live customer data.

In hindsight, it all makes sense that shift right isn’t about post-release testing. After-all, shift left isn’t just about testing earlier but gathering information, planning and preparing for testing earlier. We review requirements, build an understanding about the application and what needs to be tested, and develop test cases. This information can be used to improve the testing process – the tests can be run earlier, quicker and more efficiently.

So, similar to shift left, with shift right we are aiming to learn more about the application and understand how the application is used by actual customers. We ask questions, analyse what users are actual doing and identify what we missed. Shift left will have already improved the overall testing process, shift right can make it even better. We take the information gathered post-release and feed it back to the testing process.

Implementing shift left and shift right

Both concepts require communication. The difference is who we communicate with.

The concept of shift left is already well established throughout the software development industry. It is also something that should be relatively easy to implement, especially when compared to shift right. Shift left can be achieved by speaking to other people within the team – developer, testers, designers, product managers.

For shift right, we need to collaborate with colleagues who we may not directly interact with on a daily basis. As a result, they may not fully understand what information we need, and why we need it. The difficulties with engaging with colleagues who are not part of the immediate team is partly why shift right can be so hard to implement.

To implement shift right, we need information about how the end-user is actually using the system. What parts of the application are they using the most? How are they using it? The information is most likely already available, it is just a case of knowing what we need and asking for it (often easier said than done, especially in larger organisations). Customer support or help desk is a good place to start. Help desk technicians speak to customers on a daily basis. They can provide detailed accounts of customer issues. These are issues that were probably missed because they were not covered in the original test plan. Data scientists are also worth speaking to. Data retrieved through analytics and monitoring can be used to identify user behaviour.

The bottleneck

Why do we need shift right? We could probably achieve an efficient testing process with shift left alone. No matter how great something is – it can always be better. By using new information about the user behaviour, we can make an already perfect process more perfect.

As Michael Bolton says, testing is not the bottleneck – we just don’t know what is in that bottle.

No amount of testing will ever completely reveal what is in that bottle. With shift left and shift right, we can discover much more than we already know. We can use that information to reveal even more hidden information.

Additional Resources

Ministry of Testing Club – Shift Left and Shift Right discussion
Testing Ask Me Anything – Shift Left, Shift Right – Marcus Merrell

Main image taken from https://www.publicdomainpictures.net

Learning about SpecFlow at the London Tester Gathering Workshops – What I read last week (8th July 2019)

I was recently given the opportunity to attend one of the London Tester Gathering Workshops. These took place over 3 days and included a variety of full and half day workshops – between the 26th and 28th June 2019. I’d been lucky enough to have won a ticket in a LinkedIn competition. We were asked to choose the workshop we’d like to attend and give a reason why. When I saw that one of the workshops was about learning to ‘Automate BDD scenarios using SpecFlow’, I knew I had to enter the competition. My reason was simple, at work we were hoping to start using SpecFlow to improve our current test coverage.

I learnt a great deal attending this workshop. I was sat at a table with a develop and tester from the same company who had travelled all the way from Ulverston in Cumbria (a beautiful part of the country, I have family who live up there so I travel there often). Chatting to them about the exercises really helped enhance my learning.

There was also a definite advantage of being taught about SpecFlow from the creator himself – Gáspár Nagy. He went into great detail about Behaviour Driven Development (BDD), the understanding is essential to using SpecFlow. He explained the exercises fully so that we could complete them in the workshop, or complete them at home.

This was the first time I’d attended a workshop like this one. I’m so grateful to have had the opportunity. I definitely plan to attend more in the future. The opportunity to learn about the subject from an expert (especially the creator), and discuss with other attendees provided a really positive learning experience. We were provided with enough information to apply our new skills to our own testing projects, and given enough additional resources to expand our skills further.

I wrote a blog post about my experiences: https://louisegibbstest.wordpress.com/2019/07/03/automating-bdd-scenarios-using-specflow-london-tester-gathering-workshop-2019/

Lee Marshall also attended the London Tester Gathering Workshops. He attended the EventStorming; Deliberate, Collaborative Learning Between Multiple Disciplines workshop by João Rosa and Kenny Baas-Schwegler. He wrote about his experiences on his own blog:
https://thepiratetester.wordpress.com/2019/07/04/2019-07-04-eventstorming-at-ltg/

Podcasts

259 – New way to learn test automation with Angie Jones
Angie Jones talks about her role as a Developer Advocate at Applitools and the test automation university – a free, online platform that allows people to develop test automation skills from all over the world.

258 – Boozang A new approach to UI Testing with Mats Ljunggren
Mats Ljunggren introduces Boozang, a codeless automation tool that uses AI to improve maintenance and speed up the creation of automated tests.

Articles and Blog Posts

5 tips how software testers can collaborate with software developers – T. J. Maher – medium
” Developers aren’t opponents. We are teammates. We’re partners.”
A brilliant article about the importance of maintaining positive relationships between developers and testers. Key ideas include encourage both to participate in planning sessions and not letting your frustrations get to you when your work is criticised.

Why the three part user story template works so well – Mike Cohn – Mountain Goat Software
This article looks at the positives of the user story, that it provides context by informing us who it applies to, what they want and why they want it. It also looks at the negatives when user stories start with ‘as a user’. This can indicate lazy thinking and that the designers are unaware who the ‘users’ are.

Why I run a meetup – Lee Marshall – Pirate Tester
Lee Marshall talks about how and why he runs test meetups. Lee runs the #MIdsTest meetup, which takes place each month, alternating between Birmingham and Coventry (England). I regular attend these meetups and was given the opportunity to speak at one back in January this year. I will be speaking at the next meetup on 17th July.

You do not need to make the wrong assumption about your users anymore – Joel Montvelisky – qablog
If a tree falls in a forest and no one is around to hear it, does it make a sound?
I previously used this philosophical riddle in a lightning talk at the Fall 2019 OnlineTestConf. I looked at how the impact on bugs can dramatically increase when a bug is not ‘heard’. In this article, Joel Montvelisky provides an alternative take on this idea. If there is no sound, does this imply that the bug is not important? Should we be the ones making this assumption?

Other blogs that share lists of test related articles

https://5blogs.wordpress.com/ (daily)
http://blog.testingcurator.com/ (weekly)
http://thatsabug.com/ (weekly)
https://weapontester.com/tea-time (weekly)
https://www.ministryoftesting.com/feeds/blogs

Testing Conferences
https://testingconferences.org/

The Club
https://club.ministryoftesting.com/
A forum for discussing, asking questions, answering questions, and requesting help. Run by the Ministry of Testing.

Feel free to recommend anything that you think is of interest.
Main image taken from http://www.publicdomainpictures.net

Automating BDD Scenarios using SpecFlow (London Tester Gathering Workshop 2019)

On 26th June 2019, I attended the London Tester Gathering workshops. The workshop I’d chosen, Automate Scenarios with SpecFlow. I chose this workshop because I’m hoping to start using SpecFlow on my current test automation project.

The workshop was run by Gáspár Nagy, the creator of SpecFlow, self-proclaimed BDD addict and editor of the BDD monthly newsletter (I’ve already subscribed).

My current project

Image result for test pyramid
Testing pyramid. UI testing is only the tip, more testing levels exist. https://martinfowler.com/bliki/TestPyramid.html

I’m not new to test automation. I’ve already developed a series of automated end-to-end UI tests using Ranorex. These tests are designed to provide broad test coverage of the main features in the application. The steps are designed to mimic the process that a typical user is likely to follow. Running these tests allow us to find out if the most commonly used features in the application work as expected.

We also run a series of manual tests that cover individual features more deeply. Ranorex is great for end-to-end testing, but automating more in-depth test cases was inefficient and brought little value to the project. I believe that these tests would be better suited to behaviour driven development, which in turn can be automated using SpecFlow.

By attending the workshop, my main aim was to learn how to use SpecFlow. In addition to this, I hoped to understand how it can be used to improve my current testing strategy. I didn’t want to just include automated end-to-end UI tests. I wanted to dig a little deeper into the test pyramid and cover other testing levels with test automation.

What is BDD?

In order to use SpecFlow, you need to understand what BDD. Therefore it the workshop started with a discussion around what BDD is.

The scenarios are written and agreed on before the development takes place

BDD stands for behaviour driven development. It encourages collaboration between the testers, developers and other stakeholders. All requirements should be fully understood and agreed on before any development takes place. Allowing for an earlier feedback loop, where any questions or confusion is cleared up before the scenarios are formalised. This ensures that all parties fully understand what work needs to be done.

The advantage of BDD is that its tests are designed to show how the expected behaviour aligns with the product. The development of a feature is designed to focus on the user expectations.

The scenarios are written in a common language that allow anyone in the team to write tests, making it easier to document and verify the tests. It also ensures that there is a shared understanding of the requirements across the entire team, not just those who understand code.

How is BDD used in SpecFlow?

SpecFlow uses Gherkin language in its scenarios which are designed to show the features expected behaviour. Gherkin breaks down the scenario into Given, When,Then steps. This language ensures that the scenarios can be understood by anyone.

The format used in SpecFlow will look something like this:

Example of a scenario in SpecFlow. Taken from one of the exercises used in the workshop.

Then, for each statement in the scenario, some code is written that will run the steps required for that statement.

Automation

After covering the basics of BDD and scenarios, we then went through a series of exercises designed to encourage us to use SpecFlow and understand how it works. Each exercise had its own visual studio solution, containing all the resources needed to complete the exercise. The application being tested, a pizza website called GeekPizza.

We first created a basic test that checked the number of pizzas displayed on the menu. There was an additional bonus exercise to try out at home. We were also encouraged to think about how we would test that the automation worked correctly.

The second exercise looked at introducing a data table containing a list of items which need to be checked while the test is being run. The third exercise was designed to show how to split up files and step definition classes.

The next set of exercises showed us how to use SpecFlow for web automation. We only had time to work on the first exercise, but we have enough information to help us with the remaining exercises.

Final thought…

This workshop provided an excellent introduction to Behaviour Driven Development, which is essential for SpecFlow. All the exercises, even the ones we’d already done, included bonus tasks so there is plenty to work on at home. The workshop provided everything we needed to really practice and understand SpecFlow.

Gáspár is definitely the person to go to if you need help with writing BDD scenarios or automating them using SpecFlow. I strongly recommend going to one of his talks, workshops or courses if you need to learn more.

Main image taken from http://www.publicdomainpictures.net