Lecture 8 - Bug Tracking -> Quick Intro -> The Bug Tracking System -> Bug Tracking ProcedureLet’s forget about software for a minute. Let’s say that you are test-driving a car. You check out how fast the car gains speed, how it handles turns, if controls like the audio volume are reachable, etc. After the test drive is finished, you write down on a notepad whatever didn’t meet your expectations.
Example
Line 1: “navigation system sucks”
Line 2: “acceleration delay after gas pedal is pressed”
Line 3: “trunk is too small”
That notepad is the simplest version of the BTS.
Here is a definition:
The BTS is the infrastructure that enables to
– create,
– store,
– view and
– modify
information about bugs.
As a rule, software companies use professional BTS software. Some BTS software is free, like the most popular BTS, Bugzilla; some can cost tens of thousands of dollars (for example, Test Director by HP). I have used many of them, and although I personally prefer Test Director, I must say that Bugzilla is an ideal bug tracking solution for most software start-ups.
BTW
In this course, we will consider a BTS structure and a BTP that are not directly linked to any existing BTS software. Why? Because I want to give you a universal knowledge that will allow you to easily deal with any concrete BTS and BTP.
BTW
From now on, depending on the context, the term “bug” will mean:
– a mismatch between the actual and the expected (our classic definition)
and/or
– a BTS record of that mismatch
Example
– “I found a bug in Checkout.”
– “There are 25 open bugs in Bugzilla.”
As we know, the fact that a bug has been discovered has no practical value until we share information about that bug. How can we share the information? We can make a phone call, send an instant message or email, talk personally, etc. The standard way to report bugs in software companies is to file a bug into the BTS. So, another important function of the BTS is the communication medium between SDLC participants. Of course, bugs are communicated in other ways, too (e.g., we can talk about bugs at the Go/No-Go meeting), but the rule remains: Each and every bug found must be filed into the BTS.
Question: How can we get information about the bug itself and the activities surrounding it?
Answer: We must check the bug Attributes in the BTS.
Example
One of the bug Attributes is Status. Status has 3 possible values:
– Open
– Closed
– Re-open
At any given moment, a concrete bug can have only 1 of those 3 values.
Once a new bug is filed into the BTS, its status is automatically set to “Open”. After the bug is fixed and successfully regressed, the tester changes its status to “Closed”; if the same bug appears again, the tester reopens the bug by changing the bug status to “Re-open”.
BTW, transitions from Open to Closed and from Closed to Re-open are made according to the rules set in the BTP.
ShareLane -> Now, do this quick exercise:
1. Create an account on ShareLane and login.
2. Go to Test Portal>Bug Tracking>Training BTS>Submit new bug.
3. File a bug about anything: e.g., “Today the weather is bad” (fill up only Summary).
4. View that bug and change its status to Closed.
5. View that bug and change its status to Re-open.
Bug Attributes are needed to:
1. Describe the problem with the software/spec
2. Reflect activities made towards solving the problem
3. Provide other useful info about the bug
Let’s learn about those Attributes! I urge you to actively use Test Portal to file/modify your own bugs during this lecture.
Here is the list:
The bug ID is the unique identifier of each bug in the BTS. The BTS automatically assigns next available ID after a new bug is filed. The bug ID is not changeable. Among other usages, the bug ID is the universal way to refer to a concrete bug: “Hey Billy, how is the situation with 998?”
The Summary is a short, informative message about the gist of the bug. As a rule, the text field for the Summary doesn’t exceed 80 characters with spaces. The way a tester writes a Summary is a very good indication of his professionalism.
Example of a good Summary:
“Spec2345: 2% discount is given for 12-19 books inclusively”
Example of a bad Summary:
“problem with discount”
BTW
ShareLane -> I recommend that you look into the official ShareLane BTS called the Bug Vault (Test Portal>Bug Tracking>Bug Vault) for examples of Summaries. Click the bug ID to open a Web page with full details about that particular bug.
The purpose of the Summary is to give the BTS user a quick, powerful description of the bug. A bug Summary is to a bug as a title is to an article. Professional journalists are specifically trained in the art of creating good titles. I recommend that you start practicing right now. It’s not a big deal, really. Just start writing short Summaries of all the bugs you see in real life;
“I wanted a beer, but my wife bought me an ice cream.”
“I wanted an ice cream, but my husband bought me a beer.”
(BTW, bad Summary for 2 cases above would be, “I didn’t get what I wanted.”)
Here are some guidelines about Summaries:
1. Depending on the situation, start a Summary with one of the following:
– The spec ID; e.g., Spec2345
– The name of the feature (or component); e.g., Registration
– The name of the file with bug; e.g., shopping_cart.py
There are at least two reasons to do this:
a. The ability to unite related bugs by the same first word(s) in summaries. Bugs can be related because they were found while testing the same spec (e.g., Spec #1111 “New User Registration”) or the same functional area (e.g., “Checkout”), etc.
– On the one hand, the benefit here is that the BTS user can sort a list of bugs by the first word in the Summary and thus see all related records in one group.
– On the other hand, if certain keyword (e.g., spec ID) is presented in the Summary, then the BTS functionality “Search” can be used to find and list in one group all the bugs with that keyword in the Summary.
BTW
It’s a good idea to create a convention (inside your company) or at least some guidelines about how to start bug Summaries. For example, we can agree that if we begin a Summary with the spec ID, we should do it a certain way:
“Spec2345:” (no space)
“Spec 2345:” (space)
“#2345:” (pound, no word “Spec”), etc.
b. The ability to provide instant info regarding a buggy area. For example, if I put “shopping_cart.py:” as at the beginning of the Summary, then Billy, who is responsible for the shopping cart, will immediately know where to start looking for the problem.
2. While creating a Summary, think about your target audience – i.e., PMs, programmers, and testers. For example, if a certain component has an internal nickname like “cc” for “credit cards” you shall use it only if you are 100% sure that people from your target audience will instantly understand it. Always remember that bug filing is the process of sharing of information, so be clear. Concerning the level of abstraction, I prefer to pack as many details as possible into the Summary.
3. Sometimes it’s really hard to create a good Summary. In this case, end the Summary with the phrase “(see description)” or “(see attachment)” – whichever is applicable. We’ll talk about bug descriptions and attachments in a minute. An example of a Summary might be: “Homepage: wrong layout of the logo (see attachment).”
The Description is a text area where the full details about the bug must be given.
ShareLane -> I suggest the following format for bug Descriptions (check out the Description of Bug #1 in the Bug Vault):
Description: <provide information about what happens>
Steps to reproduce: <provide steps to reproduce the bug>
Bug: <state what’s wrong>
Expected: <state what’s expected>
Always remember that nothing is set in stone. In some cases, the bug Description can be just simple repetition of the bug Summary, or just the phrase, “see attachment” (when the attachment is self-explanatory); in other cases, you’ll really need to make an effort and provide great detail.
An important thing to remember about steps to reproduce: some things, like professional slang or abbreviations adopted in the QA Department and widely used in test cases, may be absolutely incomprehensible to the PMs and programmers; always remember the purpose of the Description is to share your knowledge about the bug in a way that is understandable to others. Yes, sometimes it can be boring, and good Descriptions do take time and effort, but bug reporting is the main way to address bugs, so each tester must make sure to provide crystal clear info about bugs.
BTW
A very good approach is to copy and paste the interpreter error or log file extract or any other info that might be useful for debugging* into the Description.
*Debugging is the activity a programmer engages in when he is trying to identify a buggy piece of code and fix it.
ShareLane -> For example, you can copy interpreter error from here: Test Portal>More Stuff>Python Errors>register_with_error.py, and put it under the Bug: section of the Description.
ShareLane -> Please, go ahead and file a bug against register_with_error.py using interpreter error in your Description.
Below is a table describing the most common Web page elements. This table will:
– Help you provide better Summaries, Descriptions, and test case steps
– Deepen your technical knowledge
– Give you an idea of the typical problems with each element
ShareLane -> You can play with these elements here: Test Portal>More Stuff>Basic Web page elements.
Name of the element | Example of the element |
Example of HTML code of the element | |
Description of the element | |
Text | Price: $10.00 |
<p>Price: $10.00</p> | |
Text is the most common element of the Web page. Problems with text include: 1. Wrong text (e.g., wrong text of the error message)* 2. No text (e.g., the code doesn’t display the title of the book)* 3. Wrong text property: font, color, size* Of course, the problems of “wrong content”, “absence”, or “wrong property” can happen with other elements too, e.g.: 1. Wrong image of the book 2. No image at all 3. Image with wrong size parameters. | |
Link | Sign up |
<a href=’./register.py’>Sign up</a> | |
Link (or hyperlink) is a navigation element. Link can point to – URL – Position on the Web page. Read on for examples. Here are typical targets of links: 1. File, e.g., HTML file (Web page), PDF file, TXT file, Python file, etc. 2. Anchor on the same Web page 3. Anchor on a different Web page 4. “mailto” value So, when a user clicks a link: 1. A certain file is opened (e.g., HTML, PDF, TXT files) or launched (e.g., the Python file shopping_cart.py dynamically generates the Web page “Shopping Cart”). 2. The Web browser scrolls down to a certain anchor on the same Web page. 3. The Web browser opens a new Web page and scrolls down to a certain anchor. 4. The Web browser invokes a new message instance of the default email client. Examples 1. Click the link Link to another page on Test Portal>More Stuff>Basic Web page elements. 2. Click the link Bug #3 under the section “View bugs” on the Test Portal>Application>Source Code>shopping_cart.py. 3. Click the link Link to another page with anchor on this page: Test Portal>More Stuff>Basic Web page elements. 4. Click the link Mailto link on the Test Portal>More Stuff>Basic Web page elements. Below are some typical problems with links: A. Misleading link: The link points to the wrong file and/or anchor, or mailto value. B. Broken link: This term is usually used for problems with linked files. Here we have 2 variants: – Web server responded that linked file doesn’t exist at specified path. In that case we get the “404 – File Not Found” error. – DNS cannot find IP address associated with that domain name. In this case we get “DNS error – Cannot Find Server”. BTW, the DNS (Domain Name System) is the translator from a domain name to the IP address of a physical computer on the Internet. C. Misspelled mailto value: e.g., ‘mailto:contactsharelane.com’ instead of ‘mailto:contact@sharelane.com’ (note missed “@” sign). See examples on Test Portal>More Stuff>Basic Web page elements inside the table, “Bad Links”. Check out the Downloads section of qatutor.com for automation tools that check for broken links. BTW, linking the URL to a file can be presented in two ways: – Relative URL: i.e., we assign a path to the file related to the directory where we are right now. For example, if we are on the page https://main.sharelane.com/cgi-bin/main.py it means that we are inside the Web server directory of sharelane.com called ‘cgi-bin’. The directory cgi-bin contains the file register.py, so we can put a link like that to register.py: <a href=’./register.py’>Sign up</a>on the HTML page generated by https://main.sharelane.com/cgi-bin/main.py. – Absolute URL: e.g., we give the full path. For example, if we are on the page http://www.google.com and want to reach register.py on sharelane.com, we should do this: <a href=’ https://main.sharelane.com/cgi-bin/register.py ‘>Sign up</a> | |
Image |
|
<img src=’../images/product_2_large.jpg’> | |
Images are graphic files (as a rule, in a GIF, JPG or PNG format) that are referenced from the HTML code of Web pages. When a Web browser loads HTML code and “sees” a special tag: <img>, it tries to download the referred image to the hard disk of the user. If the file exists, the user can see the picture. If the image is corrupted or the path to the image is wrong, then we have a situation called “broken image” and we can see only an image placeholder on the Web page that looks like this: | |
Linked image |
|
<a href=’./show_book.py?book_id=8′> <img src=’../images/product_8_small.jpg’ height=’150′ border=0> </a> | |
This is a link represented by an image instead of text. The linked image can have maladies of both images and links. In the example of HTML code above, bolded is the code for a link and not bolded is the code for an image. If we put text The Moon and Sixpence instead of <img src=’../images/product_8_small.jpg’ height=’150′ border=0> then we would get a standard textual link. | |
Text box |
|
<input type=’text’ name=’keyword’ size=’50’> | |
The text box is a one-line field for text input. The text box (as well as other elements that we’ll discuss further) serves as an input mechanism for a Web form. A Web form provides a way to submit one or a set of key-value pairs to a concrete file within the application core. Here is an example of HTML code for a Web form with a text box and a submit button (we’ll learn about this button in a minute). The opening and closing tags for the form are bolded. Underscored is the code for the button. <form action=’./search.py’> <input type=’text’ name=’keyword’ size=’50’> <input type=’submit’ value=’Search’> </form> Here is what happens: after a user types expectations into the text box keyword and presses the “Search” button, key-value pair “keyword=expectations” is sent via the Web server to be processed by the application core file search.py. BTW – If you submit that form by using the Search functionality on main.sharelane.com or – If you just type in this URL: https://main.sharelane.com/cgi-bin/search.py?keyword=expectations You’ll get the same Web page as a result. Let’s get back to the text boxes. Besides standard things like the ZIP code or email fields, text boxes are also used to enter the value for captcha. Captcha is an image with some alpha, numeric, or alphanumeric characters on it. Captchas are used to prevent bots (automated scripts) from submitting Web forms: the premise is that humans can distinguish text on an image, while bots cannot (some of them already can, BTW). Here is an example of a captcha from the Yahoo site. This captcha is displayed to prevent automatic registration of email accounts:
Regarding bugs connected with text boxes, we usually have 1. The wrong value for the text box attribute name. For example, if a programmer types name=’keywor’(note: no “d” letter) instead of name=’keyword’ then no search would take place, because search.py expects the variable called “keyword”. Go to this URL to check it out: https://main.sharelane.com/cgi-bin/search.py?keywor=expectations 2. The wrong value for the attribute maxlength or no such attribute at all. Maxlength is needed to limit the number of characters (including spaces) that can be typed into the text box. For example, Willy wanted to type this to limit search keywords to 25 characters: <input type=’text’ name=’keyword’ maxlength=25> but erroneously he typed: <input type=’text’ name=’keyword’ maxlength=5> As a result, a user can enter only a keyword with equal or less than 5 characters:
The first problem is discovered during functional testing; the second problem is discovered during UI testing. BTW, very often a bug occurs because the DB column is able to store fewer characters than the user is able to submit. The grey box approach can help us find that bug. Do this: 1. Go to Test Portal>DB>Schema 2. Scroll down to table below this line: “mysql>describe users;” 3. Note the value of Type for Field first_name. It’s varchar(50). Varchar(50) means that the column first_name can store up to 50 characters, including spaces. So, if user enters 51 characters or more as his/her First name, 51st and following characters will be truncated (i.e., cut off), because the first_name column has a limit of 50 characters (for security purposes this exercise cannot be done on ShareLane). The easiest fix for this problem is to introduce a maxlength parameter to limit the number of characters that a user can type into the text box. | |
Text area |
|
<textarea columns=10 rows=5 name=’message_body’> I’m some text inside test area. I can take 1 or more lines </textarea> | |
The text area is a multiline field for text input. An example of usage is the body of a message on an Internet forum. Usual text area problems include the wrong value for an attribute name or the wrong values for the columns and rows attributes. | |
Password input box |
|
<input type=’password’ name=’password’> | |
The password field is a special one-line text box where the input is masked by asterisks or dots as the user types text. Character masking is required for security to prevent some friendly fellow from seeing your password and cleaning out your bank account. The usual problems with password fields are just like those with text boxes. In some cases, the programmer makes a mistake and uses a text box instead of the password field to submit a password. See Bug #1 in the Bug Vault. Below are two additional things that I want to mention here: Professional ethics: When somebody types a password in your presence you should explicitly turn your head (or your whole body) away from the person. It’s extremely unprofessional to stare at the monitor/keyboard while somebody types a password. Your colleagues will surely appreciate the fact that you respect their privacy. Your privacy: Many software companies install software to capture keystrokes and screenshots on their employees’ computers. This means that whatever you type or see on your monitor can be viewed by your employer. Just FYI… | |
Drop-down menu |
|
<select name=’card_type_id’> <option value=’1′>Visa</option> <option value=’1′>MasterCard</option> <option value=’1′>AmEx</option> </select> | |
A drop-down menu is a list of values to choose from. BTW, there are 2 bugs in the code above! Here is what happened: Each credit card has an internal index at Sharelane: 1 for Visa 2 for MasterCard 3 for AmEx (See Test Portal>DB>Data>cc_types) The developer made 2 mistakes. He put 1 for both MasterCard and AmEx. When we click “Make payment” button to complete the checkout, the content of the Web form is sent to checkout.py for processing. Part of the content is the key-value pair card_type_id=<internal card index, e.g., 1 for Visa>. Here is how these 2 bugs echo in other parts of the system: – The first digit of the value in the column result of table cc_transactions will always be equal 1 (Test Portal>DB>Data>cc_transactions) and – The second value on each line in the log file cc_transaction_log will always be equal to “V” (Test Portal>Logs>cc_transaction_log) In this case, I would file 2 bugs with summaries like these: “checkout.py: value 1 is given for MasterCard in drop down menu card_type_id” “checkout.py: value 1 is given for Amex in drop down menu card_type_id” Why don’t I recommend doing something like “checkout.py: value 1 is given for all cards in drop down menu card_type_id” in our case? Because the MasterCard bug is not related to the AmEx bug – if you fix the problem with MasterCard, the bug with AmEx will not be fixed. There is more to this situation, and we’ll talk about it in several minutes. I recommend that you to go to the Training BTS right now (Test Portal>Bug Tracking>Training BTS) and file those 2 bugs yourself (fill in only the Summaries and Descriptions). Back to the drop-down menus…. Now we know that the wrong value of the value attribute can give birth to a bug. In addition, we can also have problems with the wrong value of the name attribute, or with the completeness of the text parameters (e.g., if we accepted “Discover” then it should have been on the list, too) or their correctness (e.g., “Viva” instead of “Visa”). | |
Radio button |
|
<input type=”radio” name=”morning” value=”fishing”> Go fishing <input type=”radio” name=”morning” value=”kids” checked> Play with kids <input type=”radio” name=”morning” value=”dishwasher”> Repair dishwasher | |
The radio button is an element of a Web form that allows the selection of one, and only one, value from the group with the same name (within the same Web form). The group must have 2 or more values. In the code above, the group’s name is “morning”. In other words, logically the elements of the same group are mutually exclusive; e.g., in the above example you can select EITHER Go fishing OR Play with kids OR Repair dishwasher. Typical problems with radio buttons include: – Incorrect value for name – Parameter checked specified for a wrong value – Incorrect label (“Go hiking” instead of “Go fishing”) The radio button took its name from those old huge radio receivers with the big buttons: each button was associated with one frequency, so you could only listen to one radio station at any given moment. It’s bad programming style to use a radio button if a group consists of only 1 element. In that case, if you select the button, you cannot clear it. | |
Checkbox |
|
<input type=”checkbox” name=”option1″ value=”bread” checked>Bread <input type=”checkbox” name=”option2″ value=”lettuce”>Lettuce <input type=”checkbox” name=”option3″ value=”cheese” checked>Cheese | |
Checkboxes are used for situations where there is a need to select 1 or more options. There are no relationships between text boxes, so in the above example you can check/clear any checkbox. In some cases, programmers add a JavaScript that can introduce logic when the checkboxes are dependent on each other (e.g., if I check Lettuce, Cheese might be automatically cleared. Checkboxes have the same maladies as radio buttons. | |
Submit button |
|
<input type=’submit’ value=’Search’> | |
The Submit button is needed to submit Web form content to the Web server. Usually it’s referred to as just a “button.” For example, when you press the “Search” button on the ShareLane home page, you send whatever was typed into the text box on the left of the Search button. A common problem with the Submit button is entering the wrong value of the attribute value. | |
Reset button |
|
<input type=’reset’ value=’Reset’> | |
The Reset button returns the values inside the Web form back to their defaults. Example 1: Here is a UI once a particular Web page is opened:
As you can see, the option “Play with kids” is checked by default. If we select the button “Go fishing” and then press the “Reset” button, the selection will automatically shift back to the “Play with kids” option. Example 2: Here is another Web page:
As you see, the default setting here is that none of the radio buttons are selected. If we select “Play with kids” and press the Reset button, then all the radio buttons will automatically become unselected (another term is cleared); i.e., they’ll return to their default state. | |
(IE) or like this: 
(Firefox).
I hope that this little intro to Web elements was useful! Now, let’s get back to the BTS attributes.
In some cases, it may take a long time to write a good textual Description of the bug (especially when the tester has to give details about what happens on the UI). It all depends on concrete situation, but as a rule, one of the best solutions is to illustrate the bug with an attachment.
Do this exercise (instructions are for Windows OS):
1. Go to Test Portal>More Stuff>Basic Web page elements.
2. Press the keyboard button PrtScrn (or whatever the abbreviation is on your keyboard for “Print Screen”); this button is located on the upper right side of a standard keyboard.
3. Now, go to Start/Programs/Accessories/Paint.
4. Do Ctrl+v on your keyboard; i.e., press the “v” key while holding down Ctrl.
5. Save the file with the extension .jpg (.jpeg).
This is how you create an image that can become an attachment to the bugs in the BTS.
BTW
After you copied the clipboard content into the Paint program, you can
– Select->cut->save in the file precisely the area where bug is
and/or
– Put an arrow to (or a circle around) a certain area to attract attention.
ShareLane -> See Attachment for Bug #1 in the Bug Vault.
BTW
If the attachment completely explains the bug, then it’s okay if your Description consists of only one phrase: “See attachment.”
I’d also recommend ALWAYS putting the phrase “see attachment” into the Description if there is an attachment. Why? Just to attract attention, because whoever views the BTS page with the bug Description can simply miss the fact that an attachment is available.
Please note that attachments are not restricted to images. You can also attach other types of files (e.g., text files with logs, etc.) . The idea is to provide the best possible illustration of the bug.
The BTS automatically fills in this attribute with an alias of the person who filed the bug. The bug can be filed by any person who has an account in the BTS and the bug filing privilege – i.e., the tester, developer, PM, etc. This attribute’s value is not editable.
The BTS automatically fills in the value of this attribute with the precise time the bug was filed. This attribute’s value is not editable.
Here we come to the very important concept of the bug owner. The bug owner is the concrete person responsible for the next step towards bug closure.
The “Assigned to” drop-down menu is needed to specify an alias of the bug owner. When filing a bug, the tester should always select from Assigned to:
A. The alias of the appropriate developer (e.g., “Billy”)
or
B. The tester’s own alias
A. How do you know who the “appropriate developer” is? In the majority of cases, it’s the developer who wrote the buggy code. But what if he has left the company (or the world), or moved to another team inside Development, so he is not responsible anymore for fixing that particular bug? In that case, we can “take our business” to
– Either the person currently responsible for that functional area (e.g., “Search”)
– Or to the person inside Development who would reassign the bug to an appropriate developer. That person is usually the dev manager.
The logical question is this: How do we get information about current responsibilities? You can ask questions and get answers, but it’s not the best solution here. The best solution is to create and maintain two very important documents: Who Does What and Who Owns What. Needless to say, these documents should be posted on the Wiki.
Who Does What relates to current projects (a separate Who Does What can be created for each major release.)
Who Owns What relates to areas of expertise.
Example: Who Does What
Spec ID | Spec name | Product Manager | Developer | QA |
1298 | “Security Improvements to sign-up flow” | Lisa Wu | Roger Staunton | James King |
Example: Who Owns What
Area | Product Manager | Developer | QA |
Sign-up | Lisa Wu | Roger Staunton Victor Brown | Alex Young James King |
You can get templates from Downloads on QATutor.com.
It’s critical to keep these two documents up-to-date and fill them up with the correct info.
If a company consists of 3 persons, it’s more or less clear who does what and who owns what, but once a start-up begins to grow, there are more people, more projects, more communication (and more miscommunication), more things to do, more information to digest, etc. It becomes very important to have centralized sources of information about current projects and areas of expertise. It’s not only about the Assigned to attribute. It’s about keeping track of things and knowing where to go for help. I strongly recommend both of those documents.
B. In some cases (especially in a start-up mess), the tester has no idea who will be fixing the bug. But that “no idea” should not prevent the tester from filing the bug report! The tester just needs to select his name from the Assigned to list, file the bug and then find out who is “Da Man” or “Da Woman” responsible for fixing the bug. I know of a situation where on a daily basis the dev manager would simply go through the list of freshly filed bugs assigned to testers and change the Assigned to values to the names of the appropriate developers.
Below are several BTWs related to Assigned to.
BTW
In many cases, programmers file bugs against themselves. This is not masochism. Programmers do this because:
– The BTS is a convenient way to keep track of bugs
– They need to justify time spent for bug fixes with an argument like: “I spent all Wednesday fixing bugs #232, #234, and #237!”
BTW
In some cases, programmers play ping-pong with bugs by reassigning bugs to each other with comments like, “This is not my bug” -> “Not, it’s your bug”->”No, it’s your bug”->”No, you should fix it”, etc. The result of this game is always a delay in bug fixes.
BTW
People work for companies, and they leave companies. It’s life. However, after they leave, their BTS accounts should be disabled but NEVER deleted. Deleting an account from the BTS can screw up the data integrity and create a real mess.
The value of Assigned by is the alias of the person who assigned the bug to its current owner. As is the case with Submitted by, this is an automatically populated attribute. This attribute’s value is not editable.
The Verifier is a drop-down menu with the same list of aliases as Assigned to. The Verifier is the person who must regress the bug once it is fixed. By default, the value of the Verifier is the alias of the person who filed the bug (Submitted by).
Remember, a bug can be filed by anyone who has an account in the BTS and a bug filing privilege; e.g., by the PM. Of course, it’s not the responsibility of the PM to regress the bug. So, if the PM (or another non-tester) files a bug, then that person simply selects the tester’s alias from the Verifier menu and lives happily ever after without thinking about bug fix verification. BTW, that “happily ever after” can be disturbed, because sometimes a bug is returned to the bug filer – e.g., if the assigned developer cannot reproduce the bug. Read on!
BTW
Every account in a professional BTS usually belongs to a concrete group. As a rule, there are at least 3 groups:
– Testers
– Programmers
– Others
Each group is associated with a set of privileges. For example, as a rule, only the person who belongs to the Testers group can close bugs.
BTW
Professional BTSs allow for neat things like making a bug filer select a tester’s name from the Verifier menu if the bug filer doesn’t belong to the Testers group (otherwise a bug cannot be submitted).
There are many nuances about BTS privileges that we will not cover here. We have much more important things to learn and to do.
As a rule, this is a drop-down menu with a list of the functional areas of the Web site. For example, at ShareLane we have this list of components:
– Registration
– Search
– Shopping Cart
– Checkout
– Other
The benefit of having this attribute is that we can sort bugs by their Component values.
This is drop-down menu with a list of the environments where the bug was found. At ShareLane, we also added Spec to the list for spec bugs:
– Production
– main.sharelane.com
– old.sharelane.com
– Other environment
– Spec
Of course, the worst thing is when a bug is found in production.
Here are the benefits of having this attribute:
– It’s clear where to reproduce the bug.
– Bugs can be sorted by Found on values.
This is the release version of the environment where the bug was found; e.g., if the application version is 1.0-23/34, we put 1.0.
This refers to the build number of the environment where the bug was found; e.g., if the application version is 1.0-23/34, we put 23.
The DB schema version of the environment where the bug was found: e.g., if the application version is 1.0-23/34, we put 34.
This is multiuse text area. This attribute has 2 main usages:
– Comments about the bug itself; e.g., give more info about the steps to reproduce.
– Comments about bug-related actions; e.g., when a developer reassigns a bug to another developer.
As a rule, Comments is an optional field, but in some cases it makes sense to make it required – e.g., when the developer cannot reproduce the bug and reassigns it back to whoever submitted the bug.
This is a drop-down menu with values S1-S4 inclusively. Severity is the magnitude of the bug’s impact on the software. Severity is a technical characteristic. In order to determine Severity, we must find a match between
– The Severity category
and
– The type of bug’s impact
The document we use for this is Bug Severity Definitions (see QATutor.com Downloads for the template).
Here is an example of the document (explanations will follow):
Severity | Type of impact |
S1 | – crash – data loss – security issue |
S2 | – site hangs – blocking issue |
S3 | – functional issue |
S4 | – other |
Details:
Severity | Details and examples |
S1 | CRASH This is situation when a Web site is completely nonfunctional because of a software problem. DATA LOSS usually happens in the following cases: 1. Data is lost halfway to the DB 2. DB is populated with partial data 3. Data is deleted from the DB Example 1: A user submits his data via the registration form register.py, but the module db_lib.py has a bug in the function update() and the data is not inserted into DB. Example 2: A good example is the earlier situation when the first_name column in the DB table users couldn’t hold 51st+ character while the first name text field allowed the user to type in 51+characters. Of course, this was an extreme example; not many people have a first name with 51+ characters. In reality, data loss of this type usually happens when a programmer allows an unreasonably small capacity for the DB column; e.g., Varchar(5) – only 5 characters for the first_name. Example 3: This may happen when the application core file mistakenly deletes/updates data. For example, in 2.0 we introduced the “Update user info” functionality. Willy wrote buggy code that would update all the records in the DB table users instead of updating only one record: precisely, the record of the user who is changing his info. So a bug resulted in this situation: if a user changes his last name to “Lee”, the last name of every ShareLane user becomes “Lee”. BTW, bugs of this kind are another reason to set up daily DB backups in both the testing and production environments! Please note that data loss might not be related only to the DB; e.g., because of the bug the content in the log file can be overwritten with new lines instead of being appended with new lines. SECURITY ISSUE Example: Because the bug site that accepts credit cards stopped using a secure connection between the Web server and the Web browser, unencrypted credit card data can be discovered by special software and stolen. |
S2 | SITE HANGS This is usually a performance issue; e.g., it takes a long time to load the Web page. BLOCKING ISSUE The term “blocking issue” has a wide meaning. It refers to a bug that blocks: 1. Development 2. Testing 3. Site usage Example 1: Willy is supposed to write functionality A. Functionality A should use the Python classes from Billy’s code. But as it appears, those classes have a number of serious design bugs. Therefore, it just doesn’t make sense to start working on functionality A until Billy has fixed those bugs. So Willy is blocked from coding. Example 2: Willy wrote functionality A. Billy wrote functionality B. B depends on A, but A doesn’t work. Thus, tester Rajeev cannot test functionality B. In other words, Rajeev’s testing is blocked. Example 3: Users are blocked from buying books on ShareLane because Checkout doesn’t work. A very important term that is associated with a blocking issue situation is workaround. For example, if the Account Creator (Test Portal>Helpers>Account Creator) used by testers for automated account generation doesn’t work, it’s not an issue that blocks testing because new user accounts can be created manually. So, in that situation, manual account creation is a workaround. If there is a reasonable workaround, then an issue cannot be considered as blocking. |
S3 | FUNCTIONAL ISSUE This category is for all functional bugs that don’t fall under S1 or S2. It’s no wonder that majority of bugs will be S3. That’s why S3 is, as a rule, the default Severity when testers file a bug. |
S4 | OTHER This includes all bugs that don’t fall under S1, S2, or S3. As a rule, here you’ll find UI bugs (e.g., layout problems, spelling, etc.). |
This is a drop-down menu with values P1-P4 inclusively. Priority is the magnitude of a bug’s impact on the company’s business. There is often confusion between Priority and Severity, so let’s point out the differences:
Severity refers to the technical aspect of a bug.
Priority refers to the business aspect of the bug.
In other words, Severity uses technical criteria to grade the bug, while Priority uses business criteria. That’s why it’s almost always clear which severity to assign to the bug, while the Priority of a concrete bug is often the subject of arguments and political games.
Example
If we have ANY kind of data loss, the severity must ALWAYS be S1, because we firmly associate S1 and data loss. Let’s recall the bug we discussed when a global update happened to DB table users during every change in the user profile (our recent example when the last name of every user became “Lee”). The Severity of the bug is S1 – no argument about that. Let’s find out about Priority.
Tester Anitha assigned a P1 (major issue) to this bug – we are talking about a HUGE data loss. But once the bug was filed, Willy spent about 10 seconds fixing it, and then he set its Priority as P4. When Anitha asked Willy what was going on, Willy explained that the bug existed only under 1 condition: when a certain constant in the DB was set to a certain weird value – exactly the value that was erroneously set by a tester, George, when he hacked into the DB of main.sharelane.com while performing component testing on the discount functionality for big spenders. Production would never have that problem.
Does Anitha’s logic about P1 make sense? Yes, it’s no joke when THAT kind of thing happens, even in the test environment…even under some stupid, unreal condition.
Does Willy’s logic about P4 make sense? Yes, because production will never have this issue.
The moral of the story:
– There is usually a unanimous consensus about Severity.
– There can be opposite, yet legitimate, arguments about Priority.
Do you remember the quote from the great physicist Niels Bohr (during our lecture about test cases)? “The opposite of a correct statement is a false statement. The opposite of a profound truth may well be another profound truth.” This quote is a key to understanding why bug Priority is often a subject of controversy.
The best tool that helps to reduce any controversy associated with assigning Priority is called the Bug Priority Definitions
Please look at the example of Bug Priority Definitions below (you can get the template from Downloads on QATutor.com).
Bug Priority Definitions:
Priority | Types of issues | Examples | |
P1 (Critical) | 1. Critical reliability/performance issue* 2. Critical legal issue* 3. Critical business issue* 4. Critical security issue* 5. Core functionality** is not working; no workaround | For 1: Site is dead For 2: Legal suit is filed (or about to be filed) against the company For 3: Integration with partner’s site is broken For 4: Intruder’s back door is discovered For 5: 500 – Internal Server Error when user clicks link "Shopping Cart" |
|
P2 (Major) | 1. Major reliability/performance problem* 2. Major legal issue* 3. Major business issue* 4. Major security issue* 5. Core functionality is not working; workaround exists | For 1: Unreasonably long time to load a Web page, e.g. 10 seconds. For 2: Owner has demanded to stop sales of the book. For 3: Inaccuracies in banner rotation For 4: Under certain condition, name and address of another user is displayed in Checkout page. For 5: Cannot complete purchase if card expiration date is this year. |
|
P3 (moderate) | 1. Unpleasant but bearable reliability/performance issue* 2. Legal issue of moderate importance* 3. Business issue of moderate importance* 4. Security issue of moderate importance* 5. Functionality doesn’t work (cases that don’t fall into P1, P2, P4 categories) | For 1: Intermittent delays loading a Web page. For 2: Age verification during user registration was not implemented. For 3: "Show Book" page is missing info about the publisher. For 4: Log out button doesn’t work in some browsers For 5: Cannot search for keywords that consist of 3 characters or less |
|
P4 (Low) | Cosmetic issue* of low importance that doesn’t and won’t affect reliability/performance, or legal, business, security sides of company operation. Low importance/probability functional problems. | Typos Problems with layout | |
* Issue can be present or potential ** Core functionality is functionality that is essential for the book sale, e.g., Shopping Cart or Checkout. | |||
There are 2 more very important documents that are closely associated with Bug Priority Definitions.
1. The Bug Resolution Times document defines concrete deadlines for the programmer to fix the bug. Deadlines depend on Priority.
Example
Bug Priority | Production | Test Environment |
P1 | ASAP. Drop whatever you’re doing. | 24 hours |
P2 | 24 hours | 48 hours |
P3 | n/a* | 72 hours |
P4 | n/a** | Fix if possible |
* P3s are to be fixed in the next minor or major release
** P4s are fixed if possible and thus are not suitable for patch releases.
2. The Go/No-Go Criteria document is a set of conditions under which the release goes to production. As a rule, P1s and P2s must be fixed before releasing code to production. So, in some cases P2 bugs are changed to P3s to meet the criteria (talking about political games!).
You can find both Bug Resolution Times and Go/No Go criteria under Downloads on QATutor.com
BTW
Before the Go/No-Go meeting, the tester must print out several copies of the BTS report with any open bugs (for the release that is going to be discussed at the meeting) and refresh his memory about the situation surrounding each of those bugs.
At beginning of the meeting, the tester distributes these reports among the participants.
During the meeting, the tester must be able to communicate intelligently about each of the bugs. So come prepared!
BTW
If you, as a tester, have a firm position regarding one Priority and a programmer has a firm position regarding another Priority for the same bug, take the argument to your manager. This will be more productive than changing the Priority back and forth.
Always remember that Priority is a powerful tool that affects the developer’s schedule, so do your best to assign a Priority as close to the Bug Priority Definitions as possible.
To conclude our talk about Priority let me give you an important BTW.
BTW
In one of his great presentations, QA patriarch Cem Kaner describes how testers can “sell” bugs to developers to make them fix those bugs (you can find this presentation on www.kaner.com). Regretfully, this is often a reality in a tester’s work, but this is only a reality IF appropriate processes are not in place, or IF appropriate processes are introduced but not enforced.
Example
If, according to the Bug Resolution Times document, a P1 bug must be fixed within 24 hours, it MUST be fixed within 24 hours. If it’s not fixed within 24 hours, the dev manager MUST have a one-on-one meeting with the responsible developer. If that developer doesn’t care that a P1 bug blocks testers, his ass should be fired. Of course, life is life and sometimes it’s really hard to follow the process, but “sometimes” happens only occasionally, and in the majority of cases, it’s the direct fault of the developer if a bug is not fixed in time. If the developer doesn’t have enough time to fix the bug (for whatever reason, like too much other coding to do), he must raise a red flag telling his manager, “No can do,” (rather than hope that no one notices that a bug is not fixed.)
Nobody needs bureaucracy: it’s harmful for the team morale and for the business, and a company must be really careful which processes and standards to adopt. BUT once they are adopted, there shouldn’t be any “nobody has noticed it” situations; i.e., situations where the person who screwed up isn’t asked the simple question, “Why didn’t you follow the rules?”
Apart from processes being introduced and enforced, there is the great concept of VALUE that test engineers bring to developers (and not only to developers!). The idea is simple: If you are valuable to developers, they respect you and try to help you.
What do YOU value in your colleagues? Honesty, a positive attitude, a desire to help, politeness, hard work, care for the company, and experience…surely, others will appreciate those things too if they see them in you.
The things that I just listed do include experience, but experience (although it’s a really important part) is in reality just a fraction of what it takes to be a real professional. Even if you are a beginner tester, you can create value for yourself by showing quality aspects of your personality, and in place of experience you will exhibit a “passion to gain experience.” In the majority of cases, your personality means more than your experience.
During an interview, you might be asked the question: “A certain programmer didn’t fix his/her bug. What would you do?” Now you know the answer. You should say: “First, we should introduce the proper procedures to prevent that situation from happening, because then the developer would know the exact deadline for the bug fix. Second, I’ll do my best to create value for myself, and the developers will want to assist me.”
This is usually a text box where you can put the aliases of people who, in your opinion, would be interested to get emails containing:
– Notification that the bug was filed
– Updates about changes that happened to the bug
BTW
As a rule, by default, when a bug is filed, an email is sent to the bug filer (Submitted by) and to the person who will fix the bug (Assigned to).
While testing code that has been written according to a concrete spec, I usually put the PM’s alias into the Also Notify list. PMs appreciate this gesture.
This is an automatically populated field in case any events happen to the bug:
– The fact of bug filing
– Any change to the bug
The Change History usually includes:
– Date and exact time (to the second) of the bug filing/change
– Alias of the person who filed a bug/made a change
– Fact that the bug was filed, or what was changed and how it was changed
Therefore, whatever someone changes about the bug is automatically logged, and that log (called Change History) is available to everyone who has a privilege to view bugs in BTS.
This is drop-down menu with two values:
– Bug
– Feature Request
We select “Bug” if it’s about a bug. We select “Feature Request” if we want to suggest and/or keep track of some kind of non-bug-related improvement. Please, note that EFR (Emergency Feature Request) is usually filed with type “Bug”.
This is drop-down menu with 3 values:
– Open
– Closed
– Re-open
– Open is automatically assigned after a bug is filed.
– Closed must be selected by the tester when he closes the bug.
– Re-open must be selected once the bug is back. (Read on…)
When do we have a situation where we need to re-open a closed bug?
Example
Situation 1: A programmer changed one piece of code and broke another piece of code that was fixed earlier, so the bug was reintroduced.
Situation 2: A programmer changed the code in the production branch but forgot to commit the changed code into the trunk, so after the new branch was cut off from the trunk, the bug showed up in the new branch.
BTW
ALL found bugs must be filed into the BTS – no excuses and no exclusions. It is your job to file bugs. Yes, bugs can be communicated verbally and/or via email, IM, or whatever – communication is a great thing, and you have to do your best to assist a programmer to understand the bug. But the main thing is that YOU MUST ALWAYS FILE A BUG. You can file the bug right after you find it or you can go to the developer first, but the rule remains: ALL FOUND BUGS MUST BE FILED INTO THE BTS.
If the programmer or PM doesn’t understand that bug filing is a part of your profession, try to explain it to them, or send them to your manager. No sane programmer or PM would ever take bug filing personally. They might make reasonable claims that the bug Description sucks or that the Priority is wrong, but I’ve never seen a real professional who would get offended by the FACT that a bug was filed.
BTW
Bugs should never be removed from the BTS. Period.
This is a drop-down menu with the following list:
>Reported
>Assigned
>Fix in progress
>Fixed
>Fix is verified
>Verification failed
>Cannot reproduce
>Duplicate
>Not a bug
>3rd party bug
>No longer applicable
Resolution is one of the most important attributes. If Status is about global things like “was born,” “died,” and “got reincarnated,” Resolution provides details like “graduated from college”, “got married”, “bought a condo” etc. Resolution details the stages of a bug’s life.
Question: What drives a bug from one life stage to another?
Answer: Certain activities geared towards bug resolving.
Question: Where can we find information about those “certain activities” and the associations between them and the concrete values for resolution?
Answer: In the BTP.
Let’s learn about each Resolution. BTW, because the majority of bugs are found in the software (not in the specs or other documentation), we’ll be talking about situations where the person responsible for the bug fix is the programmer.
Reported
This Resolution must be chosen when the person who files a bug doesn’t know who will be fixing the bug. This is the situation when a tester assigns a bug to himself.
BTW
In case of sophisticated BTS software, we can program our BTP into the BTS; e.g., depending on the situation, the required Resolution value would automatically be selected. For this course, I assume that the people who use the BTS voluntarily follow the BTP.
ShareLane -> Bug Vault>Bug #2
Assigned
This Resolution means that the programmer in Assigned to should investigate the bug.
Example 1: The person who files a bug knows who is going to fix the bug, so that person selects name of the programmer from Assigned to and selects the Assigned for Resolution.
ShareLane -> Bug Vault>Bug #3 – please note that this bug was invented to illustrate the point. ShareLane app. doesn’t really have a bug like this.
Example 2: The person filed a bug with Reported status, but changed Reported to Assigned (and put programmer’s alias into Assigned to) once he found out who was going to fix the bug.
ShareLane -> Bug Vault>Bug #4 (see Change History)
Fix in progress
The programmer selects this Resolution as soon as he starts working on the bug fix.
ShareLane -> Bug Vault>Bug #5 (see Change History)
Fixed
The programmer selects this resolution as soon as he has checked the bug fix into the CVS. Along with that, the programmer must select the same alias as in Verifier from the drop-down menu Assigned to.
ShareLane -> Bug Vault>Bug #6 (see Change History)
BTW
Testers should remember that it takes time before the build script picks up the CVS content and populates it onto a particular environment. So, if a programmer changed the Resolution to Fixed at 8:00 am, and the next build starts at 9:00 am and takes 15 minutes to complete, then the bug fix will be available on the target environment no earlier than 9:15 am.
The Moral of the Story: Before you start regressing the bug, make sure that the build with the bug fix has already been pushed to the target environment and that the build was successful.
ShareLane -> In other words, check out 2 things before you begin bug fix verification:
– Build status (see example at Test Portal>Release Engineering>Build Status)
– Application version on the target environment (you can see it if you view the HTML source of any Web page on ShareLane.com: <!– application version 1.0-23/34 –>)
Fix is verified
Remember that bug regression consists of two parts:
1. Verification that the bug was really fixed
2. Checking if the bug fix hasn’t broken some other part of the software.
First of all, we just try to reproduce the bug following the instructions in the Description:
– If the bug is NOT reproducible, then it was really fixed.
– If bug IS reproducible, then it wasn’t fixed, so we send it back to the developer with the Resolution “Verification failed” – and we DO NOT execute Part 2 of the bug fix verification.
If the bug is NOT reproducible, we move to Part 2. This is where it gets interesting. In the case of more or less complex software, it’s sometimes extremely hard (even for a programmer) to confidently predict how a certain change to a certain piece of code will affect other parts of the software. So, the only way to do comprehensive regression testing is to perform 100% coverage of all possible scenarios (what’s usually impossible!).
The standard way to execute Part 2 is to perform a basic end-to-end test of the feature that contained the bug. In special cases (e.g., in case of EBF) you can:
a. Ask the developer who fixed the bug his opinion of what could have gotten messed up as a result of the bug fix and what he recommends that you check out in particular.
b. Next follow his instructions.
c. Then perform a basic end-to-end test of the feature that contained the bug.
Please note that you can:
– Change the Resolution from “Fixed” to “Fix is verified” AND
– Close the bug (change its status to Closed) as soon as Part 1 is finished and you have verified that the bug was really fixed.
Why bother about Part 2? We need Part 2 simply as a safety measure. Of course, in the case of Part 2 we’re not talking about serious testing; we just do a quick check that usually takes several minutes. What happens if we find a bug? Simple – we file new bug (or re-open closed bug).
ShareLane -> Bug Vault>Bug #7 (see Change History)
Verification failed
The verifier changes the Resolution to “Verification failed” and “Assigned to” to the alias of the responsible developer if the bug is reproducible; i.e., if Part 1 of the bug fix verification failed.
ShareLane -> Bug Vault>Bug #8 (see Change History)
Cannot Reproduce
This unpleasant situation occurs if the developer tries to reproduce the bug assigned to him, and cannot do it. The bug is usually not reproducible for 1 of 3 reasons:
1. The tester didn’t provide a comprehensive Description.
2. The tester’s environment and the developer’s environment are different.
3. There is no bug.
Let’s look at these reasons one by one.
1. The tester didn’t provide a comprehensive Description.
One of the main concepts about bugs is the idea that if a bug exists, then it is reproducible. NEVER file a bug until you reproduce it at least once after you discovered it. Read the following Brain Positioning, and remember it for the rest of your testing career!
Brain Positioning
Sometimes you can get really excited once you find a bug, especially a fat P1, and you can be tempted to run to the developer RIGHT AWAY to share your finding. Don’t do this. Whatever the problem is…
1. Try to reproduce it.
2. Try to narrow it down; i.e., isolate the problem.
For example: Let’s assume that after you clicked the “Make Payment” button using Internet Explorer 6 under Windows XP, you got a “500 – Internal Sever Error”. What you should do next is this:
1. Try to reproduce it again, and write down the exact parameters (book title/price/quantity/credit card info: type: number, cvv2, expiration date).
2. “Play” with those parameters; e.g.; try to reproduce the bug using a different credit card.
3. Try to use a different browser/OS combo; e.g. try Firefox under Windows XP or Safari under Mac OS.
4. If a feature is on production and you found a bug in the test environment, try to reproduce the bug on production.
After your 4-step investigation is done, file a bug report right away, or go talk to developer (and file the bug later).
Again, if a bug exists, then it’s reproducible. Each concrete bug is a resulting combination of a certain scenario, i.e.,
1. Actions
2. Data
3. Conditions.
Usually people are more focused on providing info about actions and data rather than about conditions. Try to avoid that pitfall! In many cases, it’s the condition that makes all the difference. In some cases, a condition can be very unusual or hard to grasp, but if you pay attention you’ll find that condition. Please read the story below to better understand my point.
4 Scientists and 1 Flask
Once upon a time, there was a pharmaceutical laboratory where 4 scientists worked to find cures for illnesses. Scientist Leo N. invented a unique chemical substance that could serve as the basis of a new, powerful medicine. The problem was that the 3 other guys could not produce the substance, even if they methodically followed Leo’s every step. Leo was happy to share all the information he possessed about the process, but the others had no luck – it seemed like Leo had some unexplainable ability. One evening, those 4 scientists with their PhDs in chemistry got together and decided that they were going to believe in miracles, but only after one last thing: during the preparation of the substance, Leo’s EACH AND EVERY action must be captured on video and analyzed afterwards.
Following the plan, after the video was ready, the 4 colleagues got together and made a thorough analysis of EACH AND EVERY one of Leo’s actions. After several hours of analysis and conducting tests they found out what was really happening: in the middle of the preparations, whoever was preparing the substance had to walk for 1 minute between the two laboratories, which were situated in different buildings. It was wintertime, BTW. Leo was a smoker, so before going outside he would put the flask under his coat to free his hands for a cigarette and matches. Therefore, the substance in the flask wasn’t exposed to the cold like it was with the 3 other scientists who didn’t smoke and who simply ran between the buildings with the flask in their hands! So the magical condition that made all the difference was: Don’t expose the flask to the cold!
The moral of the story here is that, in some cases, even a little, hard-to-notice nuance makes all the difference.
Let’s get back to our testing. In many cases, a bug is not reproducible not because the tester didn’t know the scenarios/conditions, but because he just didn’t write a comprehensive Description.
Brain Positioning
Please remember that even if you get excited about finding some buggy area and you want to continue hunting for bugs instead of filing bugs, it’s your direct responsibility to:
– File bugs
– Make sure that others can understand your bugs and be able to reproduce them
It really sucks when a developer must put off his coding, spend time trying to reproduce a bug, and is not able to do it because the tester forgot to include a little detail right in the middle of “Steps to reproduce”. You must avoid situations like that! Do your best to avoid “merchandize returns” that come with the Resolution “Cannot reproduce”!
2. The tester’s environment and the developer’s environments are different.
Again, it’s all about conditions. There are situations where some kind of underlying item is different in the dev environment (e.g., billy.sharelane.com) than in the test environment (e.g., main.sharelane.com). For example, the dev environment could have a more recent version of the Python interpreter, so Billy could use the Python libraries that exist in his environment but are missing on main.sharelane.com. The tester sees a bug and files it, only to get a “Cannot reproduce” Resolution, which leads to a waste of time going back and forth with Billy trying to understand what’s really going on!
3. There is no bug.
Here’s a potential situation: The DB on main.sharelane.com is down for maintenance, and the tester doesn’t know about it. He sees the bug, tries to reproduce it (with success), and then files it. The programmer tries to reproduce the bug when the DB outage is over, but the bug doesn’t exist anymore. So the programmer sends the bug back with the “Cannot reproduce” Resolution. BTW, a similar unpleasant situation might happen when the testing is done while a push of a new build is under way. ALWAYS check the build status page before starting testing!
In case of a “Cannot reproduce” Resolution, whoever selects that Resolution should also assign the bug back to the person who filed it; i.e., the bug owner must be a person from the “Submitted by” field.
What happens if a tester receives a bug back? He should either close it OR provide more details and assign it back to the developer with an “Assigned” Resolution.
ShareLane -> Bug Vault>Bug #9 (see Change History)
Duplicate
This Resolution is selected if another bug was already filed for the same issue. In software companies, the BTS may contain thousands of open and closed bugs, and sometimes it’s not physically possible to review each of them to avoid filing a duplicate. The best way to avoid duplicates is to keep track of:
– All new bugs OR
– New bugs that meet certain criteria. For example, bugs found in particular area, e.g., “Checkout”
A professional BTS usually allows you to modify the settings for your BTS account and sends you an email if any new bug is filed or a filed bug meets certain criteria. That way you’ll know what was filed and what wasn’t.
On the other hand, there can be not-so-obvious situations. For example, two or more filed bugs can be the result of the same root cause.
Example
ShareLane -> Look at Bug #4 in the Bug Vault: “shopping_cart.py: 2% discount if user buys 12-19 books inclusively“.
The WRONG way to do it is to file 8 bugs:
1. “shopping_cart.py: 2% discount if user buys 12 books“.
…
8. “shopping_cart.py: 2% discount if user buys 19 books “.
The root cause is the following statement in shopping_cart.py:
if q >= 12 and q <= 49:
discount = 2
So, one bug is enough to cover all 8 cases.
Black box testers don’t look into the code, and situations like this do take place. So when the developer returns your bug with a “Duplicate” Resolution, explaining (in Comments) that your bug was caused by the same code as another bug, don’t take it personally – that happens, and it’s okay.
Once a bug is marked as Duplicate, the person making that change must also put the original bug ID in Comments and assign the bug back to the person who submitted it.
ShareLane -> Bug Vault>Bug #10 (see Change History)
Not a bug
Start-ups, especially in the early stages, are usually a mess.
In some cases, it’s really hard to find out the correct expected result, and our old friends: common sense and life experience might be different from what the PM tried to communicate during his talk with the developer some time ago.
In some cases the PM communicates a product change to the developer and forgets to:
– update the spec and
– mention that product change to the tester.
In some cases, the tester is correct that it’s a bug, but the developer thinks that it’s a “Feature” and sends the bug back to the tester.
So be prepared to answer the question: “Why is that a bug?” If you were wrong about it, but you had reasons for believing that it WAS a bug, then there is no problem.
If you are not sure and it’s a good time to ask (“Hey Linda, is this a bug or not?”), then ask. If nobody is available, then file a bug report. IMHO, it’s better to file a bug (if you have reason to believe that it is a bug) and later find out that it’s not a bug, than be scared of the “Not a bug” Resolution and ignore the real problem.
In any case, when you receive your bug back with the “Not a bug” Resolution, read the Comments and try to understand what the developer tried to communicate. If you still think you are right, go talk to the developer, and then:
– If you both agree that it is a bug, send it back to the developer (Assigned to: developer’s alias; Resolution: “Assigned”) with your explanations in “Comments”.
– If you both agree that it’s not a bug, just close it.
In some cases when a spec sucks, the tester and developer cannot come to a consensus about what the heck the PM meant. In that case, reassign the bug to the PM and let him decide whether it’s a bug or not. In a situation like this, I’d also recommend filing a separate bug against the spec.
See an example of ping-pong with “Not a bug” here:
ShareLane -> Bug Vault>Bug #11 (see Change History)
3rd party bug
The code in a software start-up cannot rely upon only in-house software, e.g., code written by the start-up’s programmers. We use databases, compilers, interpreters, and Web servers that have been designed and developed by others. In many cases, we also integrate out software with the software of our partners, e.g., with credit card processors. So, here is the situation: you file a bug, the bug programmer returns this bug to you with a “3rd party bug” Resolution and the Comment that the bug is not in the software that he wrote, but instead in the software written by others. Here we might encounter two situations:
1. We CANNOT influence “others” to fix their software – e.g., if the bug is in the Python interpreter, we cannot call Python’s father, Guido van Rossum and ask him to fix the bug before our release goes out. So, the only way to fix the bug is to find some programming workaround. So, if a programmer returns your bug with a “3rd party bug” Resolution, it doesn’t solve the problem, because whether he likes it or not, help is not coming, and we have to find a solution ourselves. In a case like this, talk to the programmer and re-assign the bug back to him.
ShareLane -> Bug Vault>Bug #12 (see Change History) – please note that this bug was invented to illustrate the point. ShareLane app. doesn’t really have this bug.
2. We CAN influence “others” to fix their software.
ShareLane -> For example, in checkout.py we have the function get_ccp_result() (see source code here: Test Portal>Application>Source code>checkout.py). This function connects to the payment processor to process credit card payments (of course, in case of ShareLane there is no real processor – it’s all training software). That payment processor software belongs to another company, which is our service provider (also called “vendor”). If there is a bug in the vendor software, then we can call them and ask them to fix the problem ASAP. As a rule, the tester usually doesn’t make this contact. As a rule, a contact like this is the responsibility of the project manager (PjM). BTW, in start-ups PMs are often also PjMs. So, if a developer returns a bug to you with a “3rd party bug” Resolution in this case, you should reassign the bug to the PjM and that person would be the bug owner who is responsible for resolving the situation.
ShareLane -> Bug Vault>Bug #13 (see Change History)
No longer applicable
This is usually selected for bugs found in features that have been deprecated; i.e., features that are still available to users but no longer officially supported.
ShareLane -> Bug Vault>Bug #14 (see Change History) Next ->
Lecture 8 - Bug Tracking -> Quick Intro -> The Bug Tracking System -> Bug Tracking Procedure
