Lecture 10 - Test Execution: Regression Testing -> Quick Intro -> How to Choose Test Suites for Regression Testing -> How To Resolve The Contradiction Between Our Limited Resources And The Ever-Growing Number Of Test SuitesThe first question is this: What parts of the software can be broken?
On the one hand:
a. It’s often extremely difficult for a programmer to know how a change in one part of the software will echo in other parts of the software.
AND what’s even worse:
b. Programmers sometimes change the software without even trying to figure out if their changes might break something.
Let’s illustrate item b.
Example
Let’s assume that a programmer writes some new functionality for the shopping cart. In the course of programming, he decides to drop (i.e., remove) a certain column of a certain DB table, because in that programmer’s opinion it seems to be useless. A problem might occur if he doesn’t check to see if that column is still used by some other part of software – e.g., by register.py. The important thing to understand here is this: that other part of the software can be completely unrelated to the “Shopping cart” feature, but it can still be broken in the course of programming the shopping cart.
Another popular scenario is this: A programmer changes a certain function in the example_one.py file without realizing that the function is imported into and used by the example_two.py file. So, the functionality delivered by the example_two.py file can be completely or partially broken because of the change in the example_one.py file.
On the other hand, testers usually know (or can make an educated guess) which features will be directly affected by a change.
Example
If the programmer changes how the shopping cart works, we know that the “Shopping cart” feature is being affected.
Here is my approach is to single out 3 groups of test suites for RT.
The 1st group must contain test suites that check legacy (i.e., existing) features that are directly affected by a change in software.
Rationale: A programmer is most likely to mess up the code of the feature that he is changing.
Example
If the programmer changes how the shopping cart works, we know that the “Shopping cart” feature will be affected. So, for the 1st group we shall use the test suite dedicated to testing the shopping cart.
The 2nd group must contain test suites that check legacy features that may somehow depend on the changed features.
Rationale: While changing feature A, a developer can break a piece of code that feature B depends on.
Example
The Checkout functionality depends on the Shopping cart functionality. When a user presses the “Proceed to Checkout” button, 3 key-value pairs are passed on to checkout.py.
Example of 3 key-value pairs:
book_id=5
q=1
total=1070
Example of a URL containing those key-value pairs:
https://www.sharelane.com/cgi-bin/checkout.py?book_id=5&q=1&total=1070
ShareLane ->Try it yourself by adding book “The Power of Positive Thinking” to the shopping cart and pressing button “Proceed to Checkout”.
If the programmer breaks the code inside shopping_cart.py so that
– the key and/or key’s value is not generated at all (e.g., “book_id=” instead of “book_id=5”)
– the incorrect key name is generated (e.g., “id” instead of “book_id”)
– the incorrect key’s value is generated (e.g., “book_id=4” while book_id must (in our example) equal 5)
– the Web form cannot be submitted at all or is submitted to the wrong application core file (e.g., “register.py” instead of “checkout.py”)
then the checkout functionality will be broken. Thus, if the Shopping cart functionality was changed, for the 2nd group of RT we must use test cases that test Checkout.
There is also a 3rd group that we’ll discuss in a minute.
Look at the table below where you can see an example of the 1st group and the 2nd group with the associated test suites.
Group | Test Suite ID |
1st | TS1111 |
TS2222 | |
TS3333 | |
2nd | TS4444 |
TS5555 | |
TS6666 | |
TS7777 | |
TS8888 |
Executing each of those test suites takes time. Now let me ask you this: How do we know if we have enough time to execute those test suites within the time frame given for RT?
Let’s assume that we have only 1 tester (George) and 2 weeks for RT. If George works 8 hours a day, then 2 weeks equals 80 official business hours + 32 hours (if he works both weekends). It also can convert into 336 hours (24 x 14) if he is willing to forget about everything else but RT.
Question: Can George execute all 8 test suites within 80 hours?
Answer: To answer, we should know how much time the execution of each test suite takes.
Question: How can we know that?
Answer: Each company computes this in its own way. Some companies have special software to track the time required for test suite (or sometimes even test case) execution; in other companies, the tester who did the last execution puts the amount of time spent somewhere in the header of the test suite. In any case, it’s very important to know approximately how much time it takes to execute each test suite. I say “approximately” because that time can vary depending on a number of factors – for example, if that tester has had prior experience executing that particular test suite or not.
Let’s assume that we know approximately those execution times.
Group | Test Suite ID | Time for Execution (hours) |
1st | TS1111 | 10 |
TS2222 | 15 | |
TS3333 | 17 | |
2nd | TS4444 | 18 |
TS5555 | 12 | |
TS6666 | 14 | |
TS7777 | 26 | |
TS8888 | 19 | |
Total | 131 | |
Here, even if George worked both weekends (112 hours total), he would still need 19 (131-112) hours to finish the test suite execution. Those 19 hours can be distributed over 14 days. He’ll need to work approximately 1 hour and 20 minutes (19*60 / 14) on top of 8 hours a day for 2 weeks. This is how things often work in start-ups.
BTW
131 hours of work over two weeks doesn’t hurt at all if you work as consultant: you’ll make $4585 if your hourly rate is $35.
If you work as a permanent employee, you’ll get nothing for your overtime beyond your usual compensation. So make sure to work for a promising start-up where your sacrifice will be compensated when your stock options are converted into shares of a successful company.
We’ll talk about this soon.
Let’s assume that we are humanitarians, and we don’t want George to work more than 40 hours a week. In that case, there are 51 hours (131-80) that we need but don’t have. What can we do? Among other things – like outsourcing testing to other companies or borrowing folks from marketing – we can use test suite priority to single out the most important test suites.
Group | Test Suite ID | Time for Execution (hours) | Test Suite Priority |
1st | TS1111 | 10 | 1 |
TS2222 | 15 | 3 | |
TS3333 | 17 | 4 | |
2nd | TS4444 | 18 | 4 |
TS5555 | 12 | 2 | |
TS6666 | 14 | 1 | |
TS7777 | 26 | 3 | |
TS8888 | 19 | 2 |
If we execute test suites with
Priority 1 – we’ll need 24 hours (10+14)
Priority 1 and 2 – we’ll need 55 hours (24+12+19)
Priority 1, 2 and 3 – we’ll need 96 hours (55+15+26) – and that doesn’t work for us because it’s more than 80 hours.
Therefore, we can execute Priority 1 and 2 test suites (55 hours), and we can spend the remaining 25 hours to execute:
– P1 and P2 test cases from those test suites with Priority 3 and 4
and/or
– Test suites from the 3rd group
Now, let’s talk about the 3rd group. As a rule, the majority of test suites don’t fall into the 1st and 2nd groups. But that majority surely needs to be tested too, because, as we know, a change in one part of the software can often break other seemingly unrelated part(s) of the software. It’s usually out of question to execute ALL of those test suites during RT for a single release, so what we can do is to set up a plan to execute all the test suites within a defined period of time – e.g., 6 months.
Example
If we have 21 test suites for RT, and we do a major release once a month, we can decide to execute 4 test suites for the first 5 releases and 1 remaining test suite for 6th release. This way, we’ll execute each test suite at least once every 6 months.
Please note that there are nuances here – e.g., test suites that we plan for the 3rd group can fall into the 1st or 2nd group; there are also situations where we have to execute the same test suite more than once for RT during those 6 months, etc. Next ->
Lecture 10 - Test Execution: Regression Testing -> Quick Intro -> How to Choose Test Suites for Regression Testing -> How To Resolve The Contradiction Between Our Limited Resources And The Ever-Growing Number Of Test Suites
