Lecture 10 - Test Execution: Regression Testing -> Quick Intro -> How to Choose Test Suites for Regression Testing -> How To Resolve The Contradiction Between Our Limited Resources And The Ever-Growing Number Of Test Suites -> Automation Of Regression Testing: Do It Right OR Forget About It -> When Regression Testing StopsIn general, test automation includes a myriad of different tools and techniques for a myriad of different purposes: code analysis, link checking, load/performance testing, code coverage, unit testing… this list goes on and on. There are tons of books written on this subject.
Today, we’ll take a look at one specific kind of test automation: automation of regression testing. Let’s call it test automation or “TA“.
My purpose here is to:
– Warn you about the dangers of incorrectly done test automation
– Share some thoughts on how to do test automation right
Please note that we’ll be talking about test automation in the start-up environment.
Here is a typical Silicon Valley story:
A Story about the Merciless Automator, Benny M.
In our beloved ShareLane.com we’ve accumulated 78 test suites. It hurts to think about it, but somebody HAS to execute them. It hurts even more to know that after each major release, the number of test suites keeps growing.
Somebody comes up with an idea to hire a test automation engineer. In a couple of weeks, this test automation engineer is hired. His name is Benny M.
The QA manager calls a special meeting, during which Benny is presented very much like the savior of humanity. When he is asked to speak, Benny gives a simple yet powerful one-liner: “I’m going to automate it all!” How in the world can somebody not fall in love with a person like that!
There is a little problem, though: the company will have to spend thousands of dollars to buy a special automation program. However, we are so exhausted by manual regression testing that we are willing to spend any kind of money to relieve ourselves of this hassle! Anyway, we decide to buy a special program called SilkTest (the original creator was Segue; as of September 2008, the owner is Borland) so Mr. Benny can feel comfortable – SilkTest is his tool of choice, you know.
After one month, Benny makes a presentation. And, my friends, what a presentation it is! He pushes a button, and in some magical way the SilkTest agent opens a Web browser window, enters a URL in the address bar, and clicks “Go”. Next, on the Web page it types an email and a password, clicks “Log in”, makes a purchase – and at the end, it compares the expected and actual results!
Witnesses of the miracle give Benny a standing ovation. Benny nods like a person who humbly accepts well deserved admiration and retires to his quarters, while we, mere mortals, gather at the cappuccino machine to make our prediction: it looks like in a couple of months we’ll have all our 78 test suites automated! No more sleepless nights – just press the button and go to the bar. Oneword: MAGIC!
But when we enter the regression stage of test execution and ask Mr. M. to run his automation, he says that the UI has changed and so his automation needs to be updated too. For example, when Benny was writing his test script for the registration flow, we had a button for log in called “Log In”, but the developers made some changes and that button is now named “Log On”, so the automation scripts must be updated to comply with that change.
“Come on!” says Benny. “I’ll fix it in no time.” He spends another two weeks making fixes, and on the day before the Go/No-Go meeting he finally runs 2 out of 10 automated test suites. With the next release, we have the same story, and the QA manager decides to fire Benny and hire two black box testers who will be much cheaper and much more effective. SilkTest stays on Benny’s abandoned machine as a silent reminder about wasted money and gained wisdom.
The inconvenient truth is that if a software company ends up wasting $100K (the cost of a commercial automation tool plus the services of an automation engineer) to gain wisdom about test automation, it’s a cheap price. There are many cases when millions of dollars are flushed down the toilet until management realizes that something really isn’t working here.
What exactly was the problem with ShareLane TA?
Was it SilkTest? No.
SilkTest is a just a tool. Each tool is effective as long as it’s properly applied for a certain purpose under certain conditions. If you use a hammer instead of a screwdriver to open the back of your Rolex, it’s not the hammer’s problem if your Rolex is smashed to pieces.
BTW, we’ll talk about SilkTest and other similar tools in a minute.
Was it Benny? Yes and no.
Benny is definitely an asshole. For example, he should never have promised to automate 100% of the regression tests, and he should never have attempted to automate flows that are prone to frequent modifications. On the other hand, Benny was a good programmer, but we didn’t give the correct direction to his boiling energy. More precisely, we didn’t give him any direction at all.
So…
Was it us? Yes.
Once we made a decision that test automation was necessary, we had to develop the correct approach towards it. We didn’t do this. Instead, we chose the easy way: hire a dude who would come and solve our problems. That didn’t work out, and we can only blame ourselves. But during the process of blaming, we can learn a lesson or two to help us avoid the same mistakes in the future.
Let me share with you some thoughts and examples regarding TA that I’ve learned. Please pay attention.
TA of regression testing usually comes in 2 forms:
1. Helper tools (let’s call them “helpers”)
2. Programs for automated regression testing (let’s call them “automation scripts”)
The purpose of helpers is to:
– Automate concrete repetitive manual tasks (Account Creator)
OR
– Give testers the essential means to perform testing (Credit Card Generator)
The purpose of automation scripts is to execute regression test cases and report the results.
On the high level, test automation consists of 3 components:
Component 1: A certain task– e.g., execute test case #1 and report results
Component 2: A certain automation program used to automate that task– e.g., Benny’s SilkTest script that executes test case #1 and reports results
Component 3: A certain piece of software tested with the help of or by an automation program – e.g., the checkout.py file is tested by Benny’s script that executes test case #1 and reports results
The question, “What to automate?” is equivalent to the question, “What task shall we automate?”
For example, we execute test case #1 manually from release to release. How can we determine whether or not it makes sense to automate its execution?
My friends, I’m here to tell you that in the majority of cases, it’s harder to select a good candidate for automation (in other words, some task worthy to be automated) than to perform the automation itself. You can pick up Python skills within days, but it takes years of experience to be able to predict whether automating a certain task will result in saving or wasting a company’s money. We at ShareLane invited a fiasco with TA because we didn’t ask Benny how he would select candidates for automation, or, in other words, what criteria he would use to single out test cases that were worthy to be automated.
The most important criterion for TA is the stability (maturity) of the software to be tested (Component 3). That criteria should be applied every time when we evaluate some part of our application for automation eligibility.
Take a quick look at the test case below (don’t execute it):
1. Go to https://main.sharelane.com.
2. Click link “Test Portal”.
3. Click link “Account Creator”.
4. Press button “Create new user account”.
5. Copy email to the clipboard.
6. Go to https://main.sharelane.com.
7. Paste user email into textbox “Email”.
8. Enter “1111” into textbox “Password”.
9. Press button “Login”.
10. Enter “expectations” into textbox “Search”.
11. Press button “Search”.
12. Press button “Add to Cart”.
13. Click link “Test Portal”.
14. Click link “Credit Card Generator”.
15. Select “Visa” from drop-down menu.
16. Press button “Generate Credit Card”.
17. Copy card number to the clipboard.
18. Go to https://main.sharelane.com.
19. Click link “Shopping cart”.
20. Press button “Proceed to Checkout”.
21. Select “Visa” from drop down menu “Card Type”.
22. Paste card number into text box “Card Number”.
23. Press button “Make Payment”.
24. Write down order id: ___.
25. Click link “Test Portal”.
26. Click link “DB Connect Utility”.
27. Make database query:
select result from cc_transactions where id = <order id>;
Expected result: “10”
As you remember, there were several problems with this, one of which was that the maintainability of that test case might become a headache. For example, we have to modify the test case if the “Make Payment” button is renamed “Order”. Therefore, we made it modular by doing several things – one of which was to appeal to the common sense of the test case executor; i.e., we assumed that a human could FIGURE OUT what to do if we just wrote “Login” as an instruction.
Problem number 1: Whatever tool we use to write TA, we cannot rely on its common sense, because the program only runs if you provide it with concrete, precise instructions.Therefore, every time when instruction must be changed (e.g., button “Log in” is renamed into “Log on”), TA must be changed too. So, we return to our original problem: maintainability, but in this case, it’s maintainability of TA.
Problem number 2: It’s a hassle to modify the steps inside a test case, but it’s much more painful and time consuming to modify/rewrite TA! When you modify your test case, you just retype the steps inside that test case, but when you modify your TA, you have to do the programming!
Thus, maintainability is the number one issue when we create TA. But why do we have an issue of maintainability in the first place? Simple: because the software to be tested was subject to changes. Change means maintenance; no change means no maintenance.
So, it really is simple: the best test case to be automated is a test case that tests software that is not subject to change.
Problem number 3: In a start-up environment, the modifications made to software are frequent and dramatic – in many cases, we might simply decide to totally redesign the UI or even replace our old product (e.g., photo sharing) with an absolutely different one (e.g., video sharing).
Test engineers cannot stop software development just to make that software more suitable for test automation. What are the chances for further employment of the test automation engineer who approaches a product manager and says, “You know, all your modifications really screw up my test automation. Can’t you just calm down with your creativity and let me do my job?”
Frequent modifications to software are INEVITABLE in a start-up environment. This means that frequent maintenance of the TA that tests that software is also INEVITABLE.
BTW
Here is a good analogy about the relationship between software maturity and TA: Parents usually don’t buy $200 shoes for 3-year-old children. Why? Because a 3-year-old’s feet will grow in several months and that $200 will be a waste of money.
By way of analogy, it doesn’t make sense to invest seriously in TA for “young,” changing software.
Now, after you have an idea about the TA killer called MAINTENANCE, let’s have deeper look into the common types of TA:
1. Helpers
2. Automation scripts:
a. Tools for component automation
b. Scripts for end-to-end automation
1. HELPERS are tools that assist the manual execution of test cases. Reading the previous chapters, you’ve had many chances to use ShareLane helpers – e.g., Account Creator and Credit Card Generator. No wonder helpers are loved by everyone! As far as I’ve seen in the majority of cases, helpers are the most effective, useful, and economical type of TA.
Helpers are usually the first pieces of TA. Why? Because manual testers usually have a very good idea about small things that can make their life easier, so if there is a person who can help with TA, it’s natural that he is asked to automate those little things. Let’s look at account creation. There are special test cases that we use to test account creation flow, but in all other cases, we just need a new account. So why not automate the repetitive, boring task of generating a new account? There are also situations where testing is not possible without helpers – e.g., when we need to use test credit cards.
But why are helpers the cheapest type of TA? Mainly, because there is usually no – or very little – maintenance required. The reason is simple – helpers don’t test software. Therefore, if the software changes, the helpers don’t necessarily need to be changed.
Example
If a programmer completely rewrites the checkout flow, the helper used to generate credit cards doesn’t need to be changed, because whatever the checkout flow is, the credit card attributes – e.g., 16 digits in number – stay the same!
Of course, there ARE cases when the helper’s code needs some maintenance.
Example
Sometimes we have to change the helper’s code – for instance, when there is an expansion of the tested software. For example, we must update the Credit Card Generator if we plan to accept another credit card (e.g., Discover).
In my experience, even if a helper needs to be changed, it doesn’t usually require much time and effort to do it. The trick is to create helpers for appropriate tasks and in correct ways (we’ll talk about this a minute).
BTW
There are also cases when the helper’s code relies on the tested software’s code – e.g., Account Creator uses lib.py (create_account.py imports qalib.py that imports lib.py). In general, I don’t recommend doing it, because TA engineer will not have a full control over that helper and that helper usually requires more maintenance compared to helper that doesn’t depend on tested software.
Once again, in the majority of cases helpers are the most effective, useful, and economical type of TA. My TA engineer career started many years ago with helpers, and many of them are still saving time and effort for my fellow testers.
Let me give you some simple arithmetic about helpers. Below is the initial data regarding Account Creator.
– It took me about 30 minutes to write and test Account Creator.
– It takes 4 seconds to create a new user account and do an auto log in with Account Creator.
– It takes about 1 minute to create a new user account and log in manually.
Let’s assume that:
– ShareLane has 10 testers
– On average, each tester creates 20 new accounts daily
This means that 52,000 minutes (20 x10 x 260 working days) or 866 hours are needed for account creation annually. With automation it takes only 1/15th of that time – i.e., 58 hours (866/15) – and note that this requires only a TINY EFFORT to create an account with Account Creator! Therefore, the delta is 808 hours (866-58). If testers make $45 an hour on average, that’s $36,360 in savings per annum. So – were those 30 minutes well spent?
BTW
Some helpers automate tasks that require hours of hard, concentrated, manual labor and/or special expertise – e.g., creating financial files for an interbank exchange.
BTW
Another benefit of using helpers is that boring tasks like the repetitive creation of a new user account can exhaust a person faster than a creative task like generating a new test case. Helpers not only save time, but they also enable manual testers to be less tired and thus more productive.
As you can see, a TA engineer who writes helpers can save millions of dollars for his company. I think that monetary savings is the primary reason for TA in the first place, right?
2. AUTOMATION SCRIPTS.
a. Tools for component automation
ShareLane -> One example of component automation is a script that performs an isolated testing of the discount rate depending on the number of books inside the shopping cart (Test Portal>Automation Scripts>Test Discounts).
Previously we came up with a set of tests for discount rates using equivalent classes. Equivalent classes (EC) is a good, but not 100% reliable, technique, because the code can contain some weird bug that won’t necessarily be caught by the EC approach.
ShareLane -> With component TA for discount rates we can generate thousands of inputs, directly apply them to the function get_discount() (Test Portal>Application>Source Code>shopping_cart.py), and get more reliable results than in case of manual testing.
The Test Discounts tool implements a short, isolated test flow – we just keep feeding various inputs to the get_discounts() function.
The words “short” and “isolated” are major points when we talk about component automation because:
– If the test flow is short, there is less probability that changes to software will get in the way of test execution.
Think about two roads with the same traffic intensity: the length of the first road is 50 miles; the length of the second road is 1 mile. As a rule, roadwork is less likely to be performed on the 1-mile road than on the road that is 50 miles long. Many factors can determine roadwork, but in general a shorter road means a lesser possibility of it.
– If the test flow is isolated, then changes to other parts of the software will not affect the tested component.
Brain Positioning
If one piece of software changes, another piece of software integrated with it can be seriously affected by that change.
ShareLane -> For example, if we remove the get_html() (Test Portal>Application>Source Code>lib.py) function, ShareLane will stop functioning, because each of the scripts inside the application core uses that function to generate the html code for the page!
Can the attractiveness of the TA component be affected by MAINTENANCE? Of course! But if we find a good candidate for the TA component, then we’ll save plenty of money for our company and, as with the Test Discounts tool, provide much better coverage.
ShareLane -> It took me about an hour to create test_discounts.py (Test Portal>Automation Scripts>Test Automation Source Code>test_discounts.py), and it takes 4 seconds to test the discounts now, so if we test the discounts once a week and it takes 15 minutes to run manual tests, then my efforts will be justified in a month. Also note that test_discounts.py provides comprehensive coverage for this very important feature (any feature dealing with a user’s money is important!).
Brain Positioning
Here is the logic: if you have 25 roads where the length of each road equals 1 mile, there’s a greater probability of roadwork when compared to a single 1-mile road. So, the more components you automate, the higher the risk of maintenance. What is the logical consequence of extensive TA? A situation known as: “I come to work to maintain TA.”
So think twice before you start automating something. It’s better to have 10 robust pieces of TA and have time to write new TA, do black/grey box testing, and maintain the existing TA than to have 30 pieces of TA and spend all week on maintenance.
b. Scripts for end-to-end automation (E2E TA)
This is the most desired (by those who have NO IDEA) and most dangerous kind of TA. Stories about wasted money usually happen when folks in power who know nothing about the subject adopt the motto “Let’s automate it all” to implement TA projects.
What is the major problem with E2E TA? The length of the road … remember our example about the two roads. E2E is about the road that is 50 miles long, and hence there is much more chance for roadwork compared to the 1-mile road. We know that software in start-ups changes rapidly, so:
– If you test one component, you are taking a risk, but that risk can be justified if the component is chosen correctly and the automation is implemented correctly.
BUT
– If your test flow goes through several integrated components, your risk increases enormously.
What is that risk? It’s the risk of MAINTENANCE. If maintaining the TA is more expensive than manual testing, your TA has become a money eater instead of a money saver.
Besides, E2E TA is usually linked to the UI, and that adds another instability factor, because the UI is usually more prone to changes than the back end.
Brain Positioning
In a majority of the cases, E2E TA automation in a start-up environment is a BAD idea. Management often doesn’t understand what they are doing when they press the TA engineer to “automate it all.” It IS possible to automate a lot, but tested software will surely go through many changes, so in the blink of an eye that smoothly working TA automation can turn into total junk or a time-consuming monster.
ShareLane -> Let’s look at the TA at Test Portal>Automation Scripts>Test Search. Here is what the test_search.py script does:
– First, it retrieves the HTML page with search results by submitting the following URL:
https://www.sharelane.com/cgi-bin/search.py?keyword=moon
– Next, it gets the expected keywords from the file:
https://www.sharelane.com/test_search_ER.txt
– Finally, it checks to see if the expected keywords are present within the HTML code of the page with the search results.
This is an extremely simple case, but even here we have a number of possible issues. Look at the URL:
https://www.sharelane.com/cgi-bin/search.py?keyword=moon
Brain Positioning
– IF the name of the script (search.py) changes, the TA will not work.
– IF the name of the variable “keyword” changes, the TA will not work.
– IF no books with “moon” in the title are present in the DB, the TA will return the message “FAIL” even though it doesn’t mean there is something wrong with search!
– and most importantly, IF some of the keywords from the expected results (e.g., “Comments”) are legitimately removed from the HTML code (e.g., because the feature containing those keywords was removed) the TA will return the “FAIL” message even thought it doesn’t mean there is something wrong with search!
Please note that we have just seen the potential problems in the case of an extremely simple scenario. What if the scenario is complex?
BTW, we’ve just learned about another side of TA – when the TA produces an error, the TA engineer needs to spend time to determine whether that error is about:
– A bug
– A problem with the TA
– A change in the tested software
– A temporary issue with the tested software, e.g., the Web server was down during the TA run
Brain Positioning
In majority cases, an error produced by the TA does NOT mean a bug in the tested software.If the TA fails by any reason except a bug in the tested software, all our efforts to fix the problem are called MAINTENANCE.
– If we are trying to determine what the heck is wrong with the TA, it’s MAINTENANCE.
– If we ask the PM if there was a change in software, it’s MAINTENANCE.
– If we even think about TA errors, it’s MAINTENANCE.
Let’s get back to the example with test_search.py. Now you are aware of at least some (!) risks for change even during that short, simple test flow. Does it make sense to automate it? After all, search IS a single component! I would personally give some thought as to whether it makes sense to automate even this piece. But the important point here is that E2E test flows are usually made up of several components integrated with each other, so it’s about the length of the road that your TA must walk through to get the actual result. If the TA of even one component (“do search”) looks like a questionable idea, think about a test flow with 2, 3, 4, 5, or more components! If we test using the UI, each component may have 1 or more Web forms to submit, so the length of the road may increase dramatically!
The only good approach I have discovered for E2E TA in start-ups is to create a short set of simple test cases for the basic flows, then automate that set using a similar approach (as in the case of Test Search), plug it into the test framework, and run that E2E TA as a smoke and acceptance test. The sad thing is that even this basic E2E TA will require your constant attention. How constant? Probably during every release…
BTW
As a rule, more bugs are found while writing automation scripts than while running them – i.e., during the several days it takes you to write an automation script, you’ll probably find more bugs than your script will find during the coming months, or maybe even years.
BTW
Here is another note about MAINTENANCE I want to mention: MAINTENANCE is very demoralizing:
A. For TA engineer
B. For those who depend upon the proper functioning of TA – e.g. the black/grey box testers who delegated running some scripts to TA.
A. It feels like a real hassle when you constantly have to go back to what seemed to be finished. Think about how you would feel if you had to go back to high school and retake every examination each time a new set of exam questions is sent to your high school by the state.
Every TA engineer will inevitably encounter situations where he has automated a task that was not suitable for TA. Life is life, and TA is a very subtle subject, so mistakes do happen. My advice is this: It’s better to acknowledge your mistake ASAP than to waste your time (and thus your company’s money) for the MAINTENANCE of poorly planned/designed TA. A TA engineer who doesn’t acknowledge his mistake and continues to do MAINTENANCE is like a person who has committed a crime and then has to constantly commit consequent crimes to cover the tracks of the original one.
B. Think how would you feel if someone who is more qualified than you and making much more money than you is, in fact, less productive than you. The failures of TA engineers are most evident to black box testers, because black/grey box testers are those who often get their test cases back with this remark: “Sorry, guys, our TA doesn’t seem to handle your test cases, so start executing them manually like in the good old days.” (The “good old days” were most likely when there was no “Let’s automated it all!” nonsense in our company.)
BTW
Another aspect of maintenance is that parts of your TA can depend on code specifically created for your TA by programmers. Therefore, when tested code is changed, you might need to ask the programmers to fix that code. This way MAINTENANCE might take an extra toll by punishing the developers for their kindness.
So now you have a basic understanding about helpers, scripts for component automation, and scripts for end-to-end automation.
Now let me share some points on how to do TA right.
Let’s look at the questions you must ask yourself BEFORE starting to write TA.
a. How stable is the tested piece of software?
b. What are the frequency and length of manual execution of the candidate for automation?
c. What is the priority of the test suite/case to be automated?
d. How much longer will that task be around?
e. How hard will it be to write TA?
a. How stable is the tested piece of software?
We’ve already seen that TA can be written only against stable piece of tested software. Otherwise maintenance will bite us in the butt every single release.
b. What are the frequency and length of manual execution of the candidate for automation?
Even if a piece of software is stable, it doesn’t always make sense to write TA for it. For example,
IF
– We don’t need to execute some task (e.g., test case #1) frequently (e.g., every release) AND
– The manual execution takes a short time (e.g., 10 minutes)
THEN
– It doesn’t make sense to spend 1 week of expensive TA labor to write TA to automate that task.
c. What is the priority of the test suite/case to be automated?
TA is expensive. It doesn’t make much sense to invest in the automation of a test suite/case that has a low priority (i.e., low meaning in terms of company business).
d. How much longer will that task be around?
If some feature is going to be removed from your software (or deprecated), it makes no sense to write TA for it.
e. How hard will it be to write TA?
Sometimes even easy trivial tasks (if they are performed manually) can turn into a TA engineer’s nightmare. This is especially the case when UI is overloaded with scripting (e.g., JavaScript or Flash).
OK, let’s proceed.
After the candidate for TA is carefully chosen, the next steps are to carefully design and carefully write TA. Again, entire books have been written about TA, and that is not the subject of this Course, but I want to give you some ideas to help you have good brain positioning on the subject.
There are two important factors about TA programming:
1. The architecture of TA
2. The technologies/tools used for TA
1. The architecture of TA
The scale of TA projects can vary substantially: in some cases, we can spend 30 minutes to create an automation helper (e.g., script create_account.py); in other cases, we can spend months writing test automation infrastructure. But in any case, TA programmers should always keep in mind the three “Es”:
A. Ease of maintenance
B. Ease of expansion
C. Ease of code reuse
A. Ease of maintenance refers to how much time should be spent updating the TA in the case of changes in the tested software. Always remember that even the most stable component can be modified dramatically.
Let’s look at an example of a bad TA practice.
Example
Let’s assume that we have 3 helper scripts:
– automate_task_one.py was written by Joe.
– automate_task_two.py was written by John.
– automate_task_three.py was written by Jason.
Each of these three tools has account creation as the first thing to do. So Joe, John, and Jason each wrote his own function to create an account. On one happy Friday ShareLane programmer Billy adds a new column to the DB table users: “age”. The “age” column must be populated during account creation, because otherwise the account will be considered nonactive. As a consequence, ALL three helper scripts must be modified to comply with that change in the tested software. Does it look like Joe, John, and Jason have been thinking about maintenance of the first place? Probably not.
BTW
ShareLane -> The solution to this particular problem is to create the function create_account() and place it into the module qalib.py (Test Portal>Automation Scripts>Test Automation Source Code>qalib.py). In order to create a new account, Joe, John, and Jason will simply need to import the function create_account() from qalib.py into their scripts. This way, in case of a change in the tested software (e.g., a new column in the DB, the TA should be changed in only one place: qalib.py.
You have to build your TA around the idea that tested software is going to change.
While you are doing TA programming, consistently ask yourself how easy will it be to modify your TA if the tested software is modified. As we have already discussed, maintenance is the biggest concern with TA.
Example
Here is a good analogy about the relationship between tested software and TA:
In elementary school we used to make little sculptures of humans. The process was simple: first we would use metal wire to create a carcass, and then we would wrap something around it. So if I used clay as a wrapper, after the clay hardened I couldn’t change the carcass position (e.g., make a sculpture of a standing person instead of a sitting person) without breaking the clay. But if I used Play-Doh that doesn’t harden, that Play-Doh would easily and naturally change its shape as I changed the carcass position.
The moral here is that your TA must be like Play-Doh: The TA must be designed to allow for easy and painless modification if the tested software is modified.
BTW
In case of Test Discounts, it’s really easy to change the TA if the product manager decides to change the logic between the number of books and discounts. We just have to spend 1 minute to modify the file with the expected results (test_discounts_ER.txt). Note that we don’t need to modify the Python code (test_discounts.py) at all!
Another interesting fact here is that while designing test_discounts.py, we expected that the most probable change that could happen to function get_discount() would be a change in the logic between the number of books and the discount rates. So we designed test_discounts.py to make it easy to update it in case of a change in the tested software.
B. Here is another aspect of this TA business: How easily can you expand your TA?
Let’s say that your TA does regression testing, and one day you identified several extra test cases that can be automated. Let’s assume that the necessary functions like create_user() are already in place.
The question is this: how much time should you spend adding new automated test cases to your existing TA? In other words, how expandable is your TA?
– If your TA is poorly constructed, then adding new test cases can become a real hassle, because every time you might need to investigate… how your TA really works.
– On the other hand, if adding new automated test cases is turned into a routine, simple task, you just add new stuff on the fly without wasting your time constantly decrypting the TA code.
In the ideal situation, you just copy existing automated test cases, change some data in them, and welcome brand new members into your TA family.
Another aspect of expandability is whether or not it’s easy to teach others to expand your automation. If your TA is written properly, it should not be a problem to create a tutorial/quick course to teach others about expanding your TA.
C. Code reuse is one of the cornerstones of good programming practices. Here are two aspects:
1. If you write your own code, you should always think about how reusable it will be for you and your colleagues.
2. In many cases, there is already code created by other people that you can reuse for your TA.
1. Code reuse means that existing code is used again and again to solve similar tasks.
For example, the create_user() function which we put into the module qalib.py can be reused every time the TA needs to create a new user account.
I can recommend the following things for better code reuse inside TA:
– It’s better to create several simple functions than to pack code inside one big complex function.
– COMMENT YOUR CODE.
– Create coding standards and adhere to them.
– Don’t try to get fancy; simplicity is the key.
– Always think about those who might want to reuse your code.
Code reusability is closely connected to both the maintainability and expandability of TA.
– Reusable code is usually well documented, well designed, and clean, so it’s easier to maintain.
– Reusable code is a set of building blocks that can be used for future TA expansion.
2. For most people it’s more fun to program than to search for code that has already been written and can be reused. But we should always remember that for everything we do, there is a chance that somebody has encountered the same (or a similar) task before and has already written solution for it. A very good example is the FREE test automation framework called DejaGnu. DejaGnu has been successfully used at PayPal QA for many years. The beauty of this framework is that you can write test cases in any programming language and just plug them in!
2. The technologies/tools used for TA
There is a huge array of technologies and tools that you can use for TA. I prefer to use the FREE and popular scripting languages (mostly Python and Ruby) under the Linux platform to do all my automation.
Let’s compare two things:
– TA using free programming languages (FL) like Python, Ruby, Perl, and PHP
and
– TA using commercial tools (CT) like SilkTest and WinRunner
I’ll list only the most important points here.
8 Reasons Not To Use Commercial Tools
Reason number | Reason to use FL | Reason not to use CT |
1 | FL interpreters are free software. | CT can cost tens and hundreds of thousands of dollars. |
2 | When you join a new company the tools for creating TA (e.g., Python interpreter and Linux utilities) are already installed on your machine (or can be installed in a matter of minutes). The company doesn’t need to buy you any special software. Thus, you can get started with your TA right away. | As a rule, automation engineers working with CT specialize in one particular CT. So, if your company doesn’t have the particular CT preferred by the automation engineer, it must be purchased. Without that particular CT, the automation engineer is like a guitar player without a guitar. |
3 | Working with FL, you expand your programming skills with popular and powerful programming languages. Those skills will surely be appreciated by all potential employers during your next job hunt. Why? Because: – Everybody in technology is familiar with Python, Ruby, Perl, and PHP. – 99% of all Web companies use at least one of those languages to develop their software. | Working with CT, you expand your skills in that particular CT because each CT: – Usually has its own programming language and – Contains lots of nuances SPECIFIC to that particular CT Your skills will be appreciated only by those employers that have already purchased (or plan to purchase) that particular CT. BTW Ask any software developer what “4Test” means (4Test is SilkTest’s own language – no other software uses it). He will probably reply: “What the heck are you talking about, my friend?” |
4 | If you want to transfer your TA knowledge, you just create documentation on the Wiki. There’s no need to explain how to write code in the FL that you’ve used (e.g., Python). | If you want to transfer your TA knowledge, you have to specifically teach another person the programming language used by that concrete CT AND all the nuances relating to that concrete CT. That’s just the first step. On top of this, you’ll also have to create documentation on Wiki, just like with FL. |
5 | When you leave the company, your TA can be expanded and maintained by anyone who knows the FL that you’ve used for TA (e.g., Python). | When a CT engineer leaves the company, there’s not much hiring flexibility – a particular TA has already been purchased. The company will need to hire a TA engineer who specializes in that particular CT. |
6 | On the Web, there are thousands of free, well documented programs written in a FL you can use for TA. | On the Web, you will not find much code for CT. |
7 | The Web has an enormous number of forums dedicated to FL, because millions of people use them. This means that the majority of the issues/questions that you’ll encounter using FL have already been discussed somewhere. In 99% of the cases, you can find your answers within seconds using Google search. You can also get help from the developers inside your company because they use the same (or similar) programming languages. | CT are expensive, thus not many people are knowledgeable about them (compared to FL). If you get stuck on something during CT usage, good luck finding an answer on the Web. Most likely, your developers will not be able to help you either; knowledge about CT usage is very specialized. In addition, you might need to pay the producer of CT for support. But even if CT support is free, you’ll probably have to wait until some support dude is available to answer your questions. BTW Note that once that CT support dude becomes available, there’s no guarantee that he is qualified to answer your question. I remember a situation where my manager had to go all the way to the CTO of one famous CT producer to get adequate support for $50K(!) CT used for performance testing. |
8 | You company doesn’t need to pay for extra copies of Python interpreter if you want others to expand your TA. | Extra CT users mean extra CT licenses; extra CT licenses mean extra money; extra money means an increase in cost – an increase in thousands or even tens of thousands of dollars. |
I hope that my reasons make sense to you (and will make sense to your employers!)
To be fair, I must admit that there ARE success stories about creating test automation with CT; I just haven’t heard very many of them. And when I hear admiration about some concrete CT, usually it’s not about how reliable and maintainable the CT scripts are in the long run, but about cool tricks that can be written under the CT environment. But here we are not talking about TA as a way to impress our little sisters with wonder tools that magically fill up Web forms inside Web browsers. We are talking about TA as a way to save money for our companies, and I feel that my anti-CT reasons make sense in most cases.
BTW
There is a huge temptation among beginner testers to specialize in SilkTest or WinRunner or some other CT. My recommendation here is this: If the company where you work has one of these tools, use the opportunity to learn that CT. Otherwise, don’t waste your time and money. It’s better to install Python interpreter and Cygwin on your Windows machine and start learning Python scripting and Linux commands.
Job market for the test engineers with Python (or other FL) + Linux skills is MUCH bigger then for those who specialize in CT.
In the future, I’m planning to write a QA course on TA with Python. But for now, I can give you one simple – but very valuable – idea on how to do UI automation with Python. BTW, we used that idea when we created test_search.py.
In a nutshell, the idea is:
1. Use the Linux command ‘wget’ to submit Web forms and retrieve HTML pages.
2. Use Python as a wrapper around wget to drive wget and parse the HTML code of the retrieved pages.
Look inside the code for test_search.py and you’ll get it. Tool wget can be installed under the Cygwin environment on your Windows machine.
Lecture 10 - Test Execution: Regression Testing -> Quick Intro -> How to Choose Test Suites for Regression Testing -> How To Resolve The Contradiction Between Our Limited Resources And The Ever-Growing Number Of Test Suites -> Automation Of Regression Testing: Do It Right OR Forget About It -> When Regression Testing Stops
