Lecture 3 - Test Cases and Test Suites -> Quick Intro -> Test Case Structure -> Results Of The Test Case Execution -> Useful Attributes Of The Test Case -> Data-Driven Test Cases -> Maintainability Of Test Cases -> The Number Of Expected Results Inside One Test Case -> Bad Test Case Practices -> Test Suites -> States Of A Test CaseAs we already know, a test case helps to check only one concrete thing (which is explained in the IDEA section of the test case). Each spec might be a source for a huge number of ideas for testing, so to test the code that is written according to the spec, we need many test cases.
A combination of test cases that check
– the concrete part of our project (e.g., “Shopping Cart”) and/or
– the concrete spec (e.g., Spec #1455 “Privacy Rules”)
is called a test suite. As a rule, all test cases that belong to the same test suite are placed in the same document; for instance, an MS Word file.
Here is a real life scenario:
- A product manager gives me a new spec.
- I create a new MS Word file into which I copy and paste the test case templates (check the Downloads section of qatutor.com).
- I fill up those templates with content.
- I execute those test cases, making necessary modifications if necessary.
- If it makes sense, I integrate this suite into the main test suite where I store other test cases that test other aspects of the same functional area: e.g., “Purchase with Credit Cards.”
Let’s look at an example:
Example
Our Web site, www.sharelane.com, accepts Visa and MasterCard. We have a test suite called “Checkout with Credit Cards” that we execute before each release to check if new bugs were introduced to old functionalities (i.e., checkout with Visa and MasterCard). BTW, this type of testing is called regression testing.
This test suite was based on spec #1211 and has test cases to test Checkout with Visa and MasterCard.
For the new release, the product manager created spec #1422. This spec introduces another credit card that we are going to accept: American Express (AmEx).
First we create a new test suite, “Checkout with AmEx,” and fill it up with new test cases. Then we execute those test cases and modify them if needed.
After that, because the test cases from “Checkout with AmEx”
– are cleaned up and
– check the same functional area as the main test suite (i.e., “Checkout with Credit Cards”)
it is logical for us to integrate the test suite “Checkout with AmEx” into the test suite “Checkout with Credit Cards.”
Brain positioning
Nobody expects your test cases to be perfect right after you’ve written them. Here are some of the reasons why:
1. A spec can have bugs.
2. A spec can have some omissions and other spec maladies.
3. A spec can be changed without anyone notifying you.
Also, if you are writing test cases for a code that is not available (e.g., you generate your test cases while the programmer is writing the code) your knowledge of functionalities is THEORETICAL. So, even if a spec is perfect, there is always a chance that you have made a mistake or an omission when writing your test cases just because you didn’t have a chance to have PRACTICAL interaction with the code. This part of a tester’s job-creating test cases for code that doesn’t exist-can be really confusing for beginners, but don’t worry, your experience will heal your confusion.
So, there are many reasons why freshly baked test cases are not perfect. The best situation is when a new test case is first executed by its author: person who totally understands whatever was typed into the test case. During first execution the author usually:
– adds more test cases to check more things;
– changes the test case content, such as the ER; e.g.,if he didn’t understand spec correctly while creating that test case;
– deletes duplicate test cases (test cases that check exactly the same thing);
– makes test cases easier to maintain;
– polishes test cases to make their content more precise and crystal clear.
Here is a header that we can insert at the beginning of our test suite:
Author | Spec ID | PM | Developer | Priority |
|
|
|
|
|
OVERVIEW | ||||
| ||||
GLOBAL SETUP AND ADDITIONAL INFO | ||||
REVISION HISTORY | ||||
| ||||
Author – the author (-s) of test cases.
Spec ID – the unique ID of the spec that describes tested functionalities; this ID should be an HTML link to the spec on our Intranet.
PM (Product Manager) – person (-s) who wrote the spec.
Developer – the programmer (-s) responsible for writing a code according to the spec.
Priority – the priority of the test suite (usually from 1 to 4, with 1 being most important); usually the priority of the test suite matches the priority of the spec.
In the OVERVIEW, we usually give the general idea about what we’re testing with this test suite: e.g., “Here we test the various scenarios when a customer uses Visa and MasterCard during Checkout.” The OVERVIEW represents the IDEA of the test suite.
The purpose of the GLOBAL SETUP AND ADDITIONAL INFO section is to educate the person executing the test suite about technicalities that will be helpful during the execution of the test cases included into that test suite. It’s just like the corresponding section of test cases, but in the test case, we put helpful info about that particular test case, while in a test suite we use this section to put information that can be applied towards the execution of two or more test cases.
Here is the content of the file, credit_card_payments.doc, which includes the test suite “Checkout with Credit Cards”:
Checkout with Credit Cards (TS7122)*
*(TS7122) – each test suite must have a unique ID.
Author | Spec ID | PM | Developer | Priority |
O. Ferguson, I. Newsome | 2011 | R. Gupta | G. Snow | 1 |
OVERVIEW | ||||
This test suite checks whether users can pay with Visa and MasterCard. | ||||
GLOBAL SETUP AND ADDITIONAL INFO | ||||
Go to Test Portal>More Stuff>QA KnowledgeBase to “See QA KB“ In order to run SQL1 go to Test Portal>Helpers>DB Connect Utility. SQL1: select result from cc_transactions where order_id = <order id>; How to get credit card balance: Test Portal>Helpers>Credit Card Balance Viewer | ||||
REVISION HISTORY | ||||
11/17/2004 – Added ССPG0001 and ССPG0002 – O.Ferguson 02/03/2005 – Revised ССPG0001 and ССPG0002 – I.Newsome 02/15/2005 – Revised ССPG0001 and ССPG0002 – I.Newsome
| ||||
IDEA: Payment can be made by Visa | TC ID | ССPG0001 | |
Priority | 1 | ||
SETUP and ADDITIONAL INFO | |||
Created (date/name):11/17/2004/O.Ferguson | Reason: new way to verify that credit card transaction was successful. | ||
Modified (date/name): 02/03/2005/I.Newsome | Reason: modified steps to improve test case maintainability | ||
Modified (date/name): 02/15/2005/I.Newsome | Reason: modified steps and added second expected result to verify that card balance was reduced. | ||
1. Generate Credit Card (See QA KB) and copy card number into the temporary text file. 2. Get credit card balance and write it down: ____ 3. Create New Account (See QA KB) 4. Login 5. Find Search Keyword (See QA KB) 6. Do search 7. Add found book to the Shopping cart. 8. Copy card number from the temporary text file and Do Checkout (See QA KB) !!! While doing it, write down amount of payment: ____ 9. Write down order id: ____ 10. Make DB query: SQL1 | 10 | ||
11. Get credit card balance and write it down: ____ | Step 2 – Step 8 | ||
IDEA: Payment can be made by MasterCard | TC ID | ССPG0002 | |
Priority | 1 | ||
SETUP and ADDITIONAL INFO | |||
Created (date/name):11/17/2004/O.Ferguson | Reason: new way to verify that credit card transaction was successful. | ||
Modified (date/name): 02/03/2005/I.Newsome | Reason: modified steps to improve test case maintainability | ||
Modified (date/name): 02/15/2005/I.Newsome | Reason: modified steps and added second expected result to verify that card balance was reduced. | ||
1. Generate Credit Card (See QA KB) and copy card number into the temporary text file. 2. Get credit card balance and write it down: ____ 3. Create New Account (See QA KB) 4. Login 5. Find Search Keyword (See QA KB) 6. Do search 7. Add found book to the Shopping cart. 8. Copy card number from the temporary text file and Do Checkout (See QA KB) !!! While doing it, write down amount of payment: ____ 9. Write down order id: ____ 10. Make DB query: SQL1 | 20 | ||
11. Get credit card balance and write it down: ____ | Step 2 – Step 8 | ||
If you’ve executed that test case, you’ve seen that SQL1 still returns “10”. This happens because of BUG #1 in checkout.py (Click that link see a source file and a bug in it). Whatever card you use (Visa, MasterCard, AmEx), the program erroneously assigns 1 as card_type_id. Hence, each card is treated as Visa, because its internal id is 1. See Test Portal>DB>Data>cc_types.
Please note the following:
1. Setup/additional info that can be applied to more than one test case (e.g., SQL1) has been moved to the GLOBAL SETUP and ADDITIONAL INFO section of the test suite.
2. Data that is different between test cases ССPG0001 and ССPG0002 (e.g., expected result) has been typed in a bold and italic font. This was done to attract attention to the differences between similar looking test cases.
It’s a great idea to use the formatting FUncctionalities of your text editor
– to stress things that deserve special attention and
– to make it easier for reader to follow you.
Let’s move along.
Our manager gives us spec #1422 “Checkout with AmEx” written by product manager Y.Wang. We create a new file, amex_payments.doc, and after we’ve created and improved our test cases (or only 1 test case like in this test suite), we have:
Checkout with AmEx (TS7131)
Author | Spec ID | PM | Developer | Priority |
I. Newsome | 1422 | Y. Wang | G. Snow | 1 |
OVERVIEW | ||||
This test suite checks whether users can pay with AmEx card. | ||||
GLOBAL SETUP AND ADDITIONAL INFO | ||||
Go to Test Portal>More Stuff>QA KnowledgeBase to “See QA KB“ In order to run SQL1 go to Test Portal>Helpers>DB Connect Utility. SQL1: select result from cc_transactions where order_id = <order id>; How to get credit card balance: Test Portal>Helpers>Credit Card Balance Viewer | ||||
REVISION HISTORY | ||||
05/15/2005 – Created AEPL0001 – I.Newsome | ||||
IDEA: Payment can be made by AmEx | TC ID | AEPL0001 | |
Priority | 1 | ||
SETUP and ADDITIONAL INFO | |||
Created (date/name):05/15/2005/I.Newsome | Reason: new way to verify that credit card transaction was successful. | ||
1. Generate Credit Card (See QA KB) and copy card number into the temporary text file. 2. Get credit card balance and write it down: ____ 3. Create New Account (See QA KB) 4. Login 5. Find Search Keyword (See QA KB) 6. Do search 7. Add found book to the Shopping cart. 8. Copy card number from the temporary text file and Do Checkout (See QA KB) !!! While doing it, write down amount of payment: ____ 9. Write down order id: ____ 10. Make DB query: SQL1 | 30 | ||
11. Get credit card balance and write it down: ____ | Step 2 – Step 8 | ||
Now, after we’ve executed Test Suite #TS7131 (and, if needed, modified its test case), we can marry TS7122 and TS7131 and create an all new credit_card_payments.doc. Let’s see what this looks like:
Checkout with Credit Cards (TS7122)
PART I: tests with Visa and MasterCard. PART II: tests with AmEx. PART I <Insert header, CCPG0001 and CCPG0002 from old file credit_card_payments.doc>
PART II <Insert header and AEPL0001 from old amex_payments.doc> |
ShareLane -> See TS7122 here: Test Portal>More Stuff>TS7122
Please note: we haven’t changed:
– either the content (header and AEPL0001) from the file amex_payments.doc
– or the content (header, CCPG0001, and CCPG0002) of the file credit_card_payments.doc.
Theoretically, we could have created a combined header and/or changed the test case ID from “AEPL0001” to “ССPG0003″ (to have continuous numbering in the same test suite), but we are not going to do that and below are the reasons why:
– The content of amex_payments.doc and the content of credit_card_payments.doc represent two independent modules in the sense of test case execution
– A unique ID is given to a test case once and for all; it’s like a SSN (Social Security Number).
BTW
A unique ID for a test case can be
– generated automatically (you can do it yourself, or ask your developer to write a program which generates sequential numbers)
– generated manually. In order to do this, we have to create a simple convention (rule) inside the QA Department
Example
We can agree that an ID consists of two parts:
– The first part consists of letters, an abbreviation for the name of tested functionality
– The second part consists of numbers: e.g., from 0001 to 9999
The original test case ID is assigned by the author of the first test case in the test suite, and if a new test case (without an ID) is added to that test suite, the ID for this new test case is generated like this:
– The first part (letters) is taken from the old test cases
– The second part equals the maximum number of the second part among existing IDs plus 1
For example, the third test case to test Visa or MasterCard will have the ID CCPG0003.
BTW
CCPG stands for “Credit Cards Payments Global”
AEPL stands for “AmEx Payments Local”
Why have I chosen those names for the tested functionalities? Because those names sounded logical to me. Each company has its own standards in this regard. The main point here is to make sure that each test case has its own unique ID.
Example
Here is how to assign IDs to new test cases:
1. First, generate some test cases. No IDs are assigned.
2. Second, execute these new test cases. During the execution,
– delete those test cases not worthy of testing (e.g., duplicate test cases);
– add new test cases that would improve the testing.
3. Then, assign IDs to all survived test cases. Next ->
Lecture 3 - Test Cases and Test Suites -> Quick Intro -> Test Case Structure -> Results Of The Test Case Execution -> Useful Attributes Of The Test Case -> Data-Driven Test Cases -> Maintainability Of Test Cases -> The Number Of Expected Results Inside One Test Case -> Bad Test Case Practices -> Test Suites -> States Of A Test Case
