A.N.: Actually, we’ll be talking about functional test automation for web applications. Here, I use the term “test automation” to keep it concise.
To be honest, I did not give the credit and appreciation Selenium IDE deserved. In fact, it is a great tool for introducing testers to automated testing.
For those who are not familiar with Selenium IDE, it is a Firefox extension built and donated to the Selenium project by Shinya Kasatani. Selenium IDE provides an easy-to-use interface for creating automated tests by capturing user actions as they are performed and exporting them as reusable scripts in many programming languages and frameworks.
Selenium IDE main GUI.
Some people might realize that it only works with Firefox (which is no longer as popular as it used to be), or worse, it only works with Firefox before version 55. There is a compatible replacement from the Katalon Studio team — Katalon Recorder for Firefox and Chrome — which aims at supporting all features of Selenium IDE on modern browsers. But that is another story.
Now, some people would argue that you should never use Selenium IDE, especially since there are many valid reasons for not using it in a large automated testing project, and I agree. However, just like programmers started their journey with plain text editors and command lines before getting their lives depend on advanced IDE like Eclipse or IntelliJ IDEA, we testers need some bite-size lessons on test automation before really getting into it.
And Selenium IDE did this job really well — that’s one of the reasons why lots of people are still using it every day.
A picture is worth a thousand words. Image courtesy: Brian Anderson.
Why should a tester learn test automation with Selenium IDE, instead of something trendy, sophisticated, and modern? Because this tool will help you:
Work through common concepts (assertion, locators, data-driven testing, etc.).
See the same tests in popular programming languages and frameworks (because sometimes taste matters — you need to know what is your favorite one).
And it does that without getting you involved in irrelevant things like complicated GUI, boring documents, and proprietary stuff that is only helpful for a single tool. Never look for “the best test automation tool” without knowing what you need, and what it can offer—do not get yourself burned only after investing too much time in a solution.
Here is a simple example to show you how Selenium IDE works. At first, you record your actions on a website and adding some assertion (checking).
Record & Play a test with Selenium IDE.
Then you play (execute) the test. You can see that basically, test automation involves making machines doing predefined actions and checking. Thanks to that you will have more time for doing exploratory testing (Point #1). However, do not expect it to magically catch more bugs for you unless the developers messed up some stable areas of your software (Point #2).
This test can also be exported to scripts in various programming languages and frameworks for advanced adjustments or learning purposes. Among them are Java, Groovy (used in Katalon Studio — my favorite), C#, Ruby, Robot Framework (also my favorite).
Trigger action: Click, Double Click, Type, Submit, etc.
Locate elements: both CSS and XPath are supported.
Verify & Assert: for checking if the AUT (application under test) is working correctly (for example, VerifyText, VerifyElementPresent, VerifyTitle, etc. — therefore, some people state that automated testing is really automated checking). Advanced pattern matching methods like Glob and Regex are supported.
Store & Echo: for calculating dynamic values, working with variables and logging.
Debug: for fixing the test scripts (See? It is not a dump tool).
Data-driven testing: for running a test multiple times with different inputs. This is an important factor to take into account if you want to design your own framework.
And so on.
They are fundamental functions that any good test automation tool must offer. You will need this checklist for tools evaluation or framework design.
Selenium IDE is not designed for executing tests in an unattended manner. If you are new to test automation, I suggest you run your tests several times to feel this pain “physically”. Easy execution and reporting is a crucial problem that many test automation tools have forgotten to solve it thoroughly.
I will elaborate on each of above points as well as the execution problem in next articles.
If you are a software tester and have been in this field for a while then you might have run into situations (let me call them software testing traps and hurdles) that limit your efficiency and effectiveness as a tester. It could be a common problem like lack of enough time and/or resources to finish testing or could be because you are surrounded by coworkers and colleagues who don’t realize the importance of your work. But if you’re like me who cannot work on projects and with people unless you have got credibility, respect and their confidence in the work you do, then you must be aware of these pitfalls, mistakes, traps and hurdles that any tester can face in their life.
I started writing this blog when I began my software testing career (exactly 9 years from today) and I don’t know about you but I have run into plenty such software testing traps while working on various testing projects at various stages of my career. And every time I ran into them, it gave me a chance to look for magic spells, ways, methods, techniques, tricks, tips and anything and everything that could help me come out of such situations. And today’s article is a compilation of some of those top 5 traps that I’ve ever run into in my software testing career and some of the ways that helped me overcome them, in my context. The following case points and suggested solutions can help you overcome many common real-life software testing problems.
#1 Running Out of Testing Ideas?
This is by far the most common problem that a tester can run into while on a project. How many times have you been in a situation where you didn’t know what else to test and how? I call this phenomenon as “tester’s block syndrome” [a condition, associated with testing as a profession, in which a tester may lose the ability to find new bugs and defects in the software that (s)he is testing]. If you’re curious, which you should be (if you are or aim to become a good tester), then you can read more about it in the article titled The Se7en Deadly Sins in “Software Testing” that I wrote a while back.
How to overcome this trap?
You can use Pair testing to your advantage to generate test ideas that seem to have dried up when you try alone. Pair testing is nothing but a testing technique where two testers work in pair to test the software under test.
BCA (Brute Cause Analysis):
Testers can employ this unique brainstrom technique when one tester thinks about a bug and the other tester thinks of all possible functions and areas where this bug can manifest.
Think ‘Out of the Box’:
Instead of thinking about the feature/function/application in front of you, rather try thinking in opposite directions. Take a step back and reassess the situation. Have you been trying to run functionality test when you ran out of ideas? How about performance, load and stress tests? How about tests involving data, structures, platforms, browsers, devices, operations?
How many times were you in a team meeting where your manager or someone from the dev. team was talking about this cool new/enhanced feature that needs testing and everybody else in the meeting room appeared to be ‘getting it’ whereas it was only you who had no idea what it was? When in such situation, nodding your head as if you are able to understand everything may seem like the natural (easy) path but trust me; it is not the best path to go unless you want to end up in trouble later in the test planning and execution phase of this feature!
How to overcome this trap?
Ask Relevant Questions:
The importance of good questioning skills can not be stressed enough if you plan to be an excellent tester. And this very skill can come to your rescue when you are trapped in a situation like the above. It’s okay to admit you don’t understand something and then get it clarified than to not admit and be ignorant for rest of your life.
Okay, so you have asked tons of relevant questions about the upcoming feature/application/product that needs testing and have taken notes. Now what? Now is the time to pull your testing team and brainstorm ideas to find all sorts of possible test ideas, strategies, plans etc for this test project by gathering a list of ideas that come spontaneously by the teammates.
Read between the lines:
More often than not, when starting working on a new product or technology or even a tool you can find some level of available documentation on the same to help you get started. But a word of advice — take everything that you read there with a pinch of salt. I’m not saying not to read them at all. But when you do, be careful about all those things that might not have been put down in words but are implied. Sometimes, proactively being able to find and underhand these implied messages in the project documents can help you in a big way to understand the testing goal.
#3 Suffering from In-attentional Blindness?
How many times have you missed a very obvious bug or a defect or an error that was right there on the screen, staring right back you and yet you missed it because you were busy ticking off the other test items from the testing checklist or executing the test case document? Situations like these can be very embarrassing not only because you missed something that is so basic and so obvious but also because it happened when you were actually busy religiously following the test cases to find things just like these!
How to overcome this trap?
Stop blindly following the Test Case and Test Matrix:
Before starting to use a test case for your testing always ask yourself the following questions and then adjust your test cases to fill any missing links.
– “Why is this test case important?” — “What are the things that are covered by this test case? What are not?” — “What portion of the product functionality does this test case cover?” — “Can this test case be tested in any other methods or ways? If yes, how?”
Change the Focal Length of Your Testing Approach:
When following the test cases and test matrix to test something, keep and open eye for anything else that might be going on during test execution. Explore other related areas even though they are not mentioned in your test case/matrix. A control object that flickers a little when you save your inputs in another section of the form, a ding sound coming from the speaker when certain button is clicked, a slight change in the color of a Submit button when you click inside another test area — all of these subtle looking actions may be an indication of an approaching catastrophic system failure.
#4 Not Sure if ‘It’ is Really Working… or Not?
How many times have you come across issues that you didn’t report as errors and bugs because you were not sure if it was really a bug or something that you did wrongly and later those same issues were found and picked up by a coworker or your manager or, god forbid, your clients or the customers?
How to overcome this trap?
Trust Your Tester’s Instinct:
IF your instinct is telling you that something is fishy and what you’re observing and experiencing could very well be a bug, then follow your instinct and report it to the devs. After all, what could be the worst case scenario? The devs might come back and say it is something that you did wrong (misconfiguration of certain settings, misunderstanding of the actual feature etc) and not a bug. It is still much more better than ignoring it thinking it might not be a bug and later your manager or customer finding it.
Start with a fresh set of eyes:
Fresh eyes find bugs, and if you are still unsure then take a short break and retest and confirm that what you’re seeing is really not a bug.
Have it tested by a fellow tester:
Pick one of your fellow testers and ask them to go through the same test scenario and see what they come up with.
#5 What to Test and What can be Skipped… Safely?
How many times have you been in a situation when you felt overwhelmed by the number of possibilities and choices to approach testing? With the complexity of software and technology becoming more complex day by day, often the number of things that a tester needs to consider while testing can be overwhelming. And with the project deadline approaching fast it can be very challenging to decide what to test, where to begin, how to begin and what can be skipped.
How to overcome this trap?
Gather Intelligence Data:
First of all, look at the existing bugs in your bug tracker tool and make a note of critical bugs. Talk to developers and ask them to think of top 10 most critical things in the product that affects majority of end-user functions and make a list of them too. Go though the review docs, user manuals, implementor’s guide and basically anything that can give you an idea of things that are going to be most important for your customers and end users.
DIQ approach (Dive In/Quit):
Now that you have the list of all these important things that need testing, let me introduce to you the magical DIQ approach (Dive In/Quit). In this testing approach, pick any of these most critical test items and just dive in and test. While testing, if it appears too hard for you then quit and take another item, dive in and test until you have exhausted all your test ideas on it. Repeat! So basically you take an item > dive in > quit when you can’t test it any further > repeat it with another item > come back to initial item when you have finished all other test items.
#And finally… Learn to Accept FAILURE, once in a while!
Due to the intrinsic nature of complexity of modern day software and communications systems, software testing is turning more complicated. As a result, more efficient and effective testing heuristics, techniques and methodologies needs to emerge. If you are not evolving fast enough as a tester then the chance of failure is exponentially high and you should be prepared to face failure once in a while. After all, we are testers; not magicians! But as long as you are learning from your past mistakes, upgrading your testing skills and updating your testing heuristics to accommodate those mistakes so they never happen again, I think you should be fine.
About the Author:
Debasis Pradhan has over a decade worth of exclusive experience in the field of Software Quality Assurance, Software Development and Testing. He writes here to share some of his interesting experiences with fellow testers.
The complexity of advanced technologies these days makes it a real challenge to deliver high-quality applications quickly, especially when your products need to go through intensive manual testing in the release process.
Taken into account different factors such as multi-platform support, cross-browser, responsive design and more, the manual testing of a website could face unexpected complications and extra effort. More time than not, these impediments prove to be considerable burdens and drag down delivery time as a consequence.
Automation testing, while initially seems to take more time, especially during the first few months of developing test scripts, will prove to be a more effective approach in the long run, both in regards of time and resources since by the time the automation solution is completed, it’s a matter of execution and maintenance with limited effort. The benefit is greatly multiplied in the project that requires intensive regression testing. The ability to reuse the test framework later by other teammates is also a big plus. Mature automation solutions typically do not require many staff for daily operations so that could save a bit budget for our project.
One of the key criteria for automation is that certain testing activities would prove to be impractical when carried out by human. Things such as validating the exact size & color of all controls on the application under test, processing a large amount of data or calculating the performance of the website are better executed by machine for consistency purpose. Besides, having such cases automated improves test coverage for many areas across projects.
What kind of tests should be considered for automation?
Ideally, one would prefer to automate as much test cases as possible to improve test coverage and take advantage of test automation. However, a typical real project will have limited time and resources. In that situation, the following tests should be given higher priority than others:
Repetitive Tests or regression testing: These are the most common cases for automation. The tests here are usually basic features of the product that need to be checked whenever a new build is available. These tests require much time to execute manually while they are typically easy to automate.
Detailed GUI Validation: These tests include cases such as checking for the exact size/color of a certain control or comparing two lengthy paragraphs.
Data Processing and Validation: Data-driven is a common practice in automation testing which helps users to evaluate whether the system processes data correctly or not. The ability to iterate a test multiple times thru a set of data or commit massive data transaction will help testers save a lot of regression time.
Cross Platforms Tests: Automation projects that cover these kinds of tests will improve test coverage and reduce manual test effort significantly. Automation tools nowadays usually support cross-platform execution where test scripts are written once and can be executed on different platforms later.
Impractical Manual Tests: These extremely-hard-to-execute test cases are also good candidates for automation. An example would be a test routine where the tester needs to interact with a certain control either in split seconds or after an exact timeout.
It’s understandable that most of the time, the testing team cannot convert every single case of their manual test into automation. However, an optimal rate as much as 70% test coverage would remarkably improve the general quality of the product.
There are a lot of automation tools that do not require the user to be an expert in order to make use of them. So, next time, when you have a new test project with many test cases that fall into the mentioned situations above, take sometimes to consider an automation plan to make your release cycle faster and boost the overall productivity of the team.
About the Author
Abhishek Kumar, Product Manager
Abhishek Kumar is the Product Manager at KMS Technology. He combines strategic and tactical management expertise with strong qualifications in software development, software testing, test automation, product management, new venture initiation, business development, project management, and general operations. He earned a Master of Computer Applications (Software Engineering) from Guru Gobind Singh Indraprastha University Delhi.