How to Become a Software Tester (QA Engineer)

Lecture 4 - The Software Development Life Cycle -> Quick Intro -> Idea -> Product Design -> Coding: Part 1 -> Coding: Part 2 -> Testing And Bug Fixes -> Release -> Maintenance

Luckily for us, in the software industry the term “release” is used in a totally different way:

– as a verb, “to release” means to transfer a piece of software to the users. For example, we can ask a release engineer to release the code to production.

– as a noun, “release” means a certain piece of software. For example, we can say: “We are testing release 5.0.”

The important thing to understand is that a release is not some kind of abstract software; it is a package of concrete files having concrete versions. Please pay attention.

The purpose of the release is to transfer one or a combination of the following things, to production:

1. New features.

2. Modification/removal of existing features.

3. Bug fixes.

There are two main types of releases:

1. A major (or milestone) release happens at the “Release” stage of the Cycle, after the “Testing and bug fixes” stage is over; i.e., “Go” decision was made at “Go/No-Go” meeting.

The version of a major release is presented as an integer: 7.0

2. A minor release takes place between major releases. Minor releases can have one of three variants:

– feature release;

– patch release;

– mixed release.

A FEATURE RELEASE takes place when there is a need to

– add new features;

– modify/remove existing features.

A PATCH RELEASE takes place when the code in production has a bug (or bugs). Here we release a fixed code.

A MIXED RELEASE is a minor release that occurs when there are both feature related changes and bug fixes.

The version of a minor release is presented as a number after a decimal point and incremented by one after each minor release: 7.1

1. The main difference is that, as a rule, major releases have tons of new features/bug fixes, while minor releases usually contain only one new feature/bug fix. For example, after the release of version 5.0, we signed a contract with the credit card processor so we can accept another credit card – Discover. In that case we can do a special feature release 5.1 to add just one functionality (“users can pay with Discover”) to our Web site.

2. As a rule, major releases have a recurring schedule; e.g., one major release per month or per quarter. Minor releases happen whenever they are required.

3. Major releases exist in the main Cycle, while minor releases exist outside of the main Cycle.

4. A major release cannot be considered as a maintenance release for the production version, while some minor releases, e.g., patch releases, have purely a maintenance nature.

5. A major release is always a planned event, while a minor release can be both planned and unplanned.

A planned minor release usually happens:

– if there is a certain feature that couldn’t be included in the major release due to resource/time constraints

For example, we really loved the features from the spec #1478 “Improvements for Shopping Cart,” but we didn’t have time to develop and test it for release 5.0 which had to be pushed to production on March 15th. So, we just release 5.0 and begin work on 5.1, which will contain features from 1478.

– if code of major release had known non-P1 bugs prior to that release, and we agreed to fix them in a patch release after the major release is out.

– if we’ve discovered several non-P1 production bugs and decide it’s time to do a patch release.

An unplanned minor release usually takes place if there is:

– an emergency bug fix

– an emergency feature request

1. We had 9 major releases.

2. We had 14 minor releases AFTER the 9th major release was pushed to production.

What is the version of the coming major release? 10.0.

Release infrastructure (CVS, etc.) and the actual push of released software to production is the responsibility of release engineers (REs).

Let’s imagine that our company, ShareLane Super Duper, Inc., was created to sell books via the Internet, and it has just received its first round of financing. Not much: only 5 million of rapidly depreciating U.S. dollars.

– two programmers: Billy and Willy;

– CEO Jean Batiste Emmanuel Zorg (further referred to as “Mr. Zorg” or “Evil Boss”);

– the ultra slim, cool-looking laptop of Mr. Zorg (OS doesn’t matter);

– one server (known as “the Star”) with Linux OS for the development/test environment

In fact, Billy wanted to name this server after his cat: “Borborygmus,” but after Willy and Mr. Zorg asked him a sobering question: “Are you crazy?” it was decided that they would have to be practical and give their servers beautiful, but completely understandable names, like “the Star.”

1. We register the domain name sharelane.com.
2. We rent the server for the production environment at the hosting provider.
3. We unite all our local computers (Billy’s machine, Willy’s machine, the Star, and Zorg’s laptop) into our Intranet.
4. The programmers start working on the code.

As you already learned, classic Web project architecture has these three components:

– Web server

– Application core

– DB

1. APACHE WEB SERVER

The name “Apache” comes from “a patchy” because of the enormous number of patch releases applied to this free software. However, those patch releases did good, because Apache is a very reliable, high-quality software package. In the Apache directories we store

– HTML and JavaScript files: JavaScript code (or reference to the file with JavaScript) is incorporated into the HTML code, and it serves many purposes, from enhancing user interface to checking Web forms. On ShareLane we have JavaScript file timer.js called during user login (see Test Portal>Application>Source Code>log_in.py).

– Images: For example, .GIF and .JPEG files; e.g., file logo.jpg: https://www.sharelane.com/images/logo.jpg.

2. APPLICATION CORE IS WRITTEN IN PYTHON

Python scripts reside in a special directory in Apache called “cgi-bin.” You can look into actual software code of the application core here: Test Portal>Application>Source Code.

3. DATABASE MYSQL

In DB we’ll store data about users, books, orders and other things. See all current data inside ShareLane DB here: Test Portal>DB>Data.

So, Billy, Willy, and Evil Boss make a historic decision to use CVS.

1. Versions of each file are stored in CVS repository.

We can retrieve the needed version of the file from the CVS repository to view/edit it (this operation is called “checkout”).

We can place a new version inside the CVS repository after creation/editing of the file (this operation is called “checkin”).

Here is how we do this:

– addition of new file to CVS: cvs add register.py

– checkout of the latest version of register.py: cvs co register.py

– checkin of the new version of register.py: cvs ci register.py

2. When we checkin a new version of the file (the file is considered to have a new version even if we added/deleted a single character, like “#”):

– CVS automatically assigns a unique version number to that particular version of the file.

– During checkin, CVS also stores the version number, comments, and name of the person who did the checkin and the time of checkin. So, not only we can see all the checked in versions of the file, but we also can see all of that information for each version. How cool is that!

Step 1. Checkout from CVS all application files for a specific release. In other words, we need to get an image (reflection) of the latest CVS content for a specific release. This image is called a build.

Step 2. Transfer those files to the corresponding directories in a certain environment (e.g., if we want to have files for the coming release in our test environment, main.sharelane.com, we must use these directories:

– /var/www/main/htdocs for HTML and JS files;

– /var/www/main/htdocs/images for images;

– /var/www/main/cgi-bin for application core (Python files).

Let’s elaborate on Step 1. Here is the definition for the term “build”: Build is a sub-version of the specific release.

After a code freeze, a build script is often added to cron (the task scheduler on a Linux system) to create and push builds in equal intervals of time, e.g., every three hours.

ShareLane -> See Test Portal>Release Engineering>Build Schedule.

The purpose of creating new builds over and over again is to make a modified application available for testers.

– There is no sense in doing testing from 12:00 to 12:15, from 15:00 to 15:15, etc, because the build is being created and pushed, and when you do testing in the middle of the process, some files can belong to the previous build and some files can belong to the new one.

– If a programmer fixed your bug and checked in the fixed file(s) into the CVS, you’ll be able to verify the fix only after the new build is pushed to the testing environment. So, if the checkin took place at 16:00, then your fix will be available for verification only at 18:15. Thus, in many cases it makes sense to launch the build script right when you need it. But if you do this, please make sure that the other testers know about it so you won’t mess up their testing. In light of this, it’s a good thing when each tester has his own testing environment.

Build numbering starts with 1 for a concrete release and increments by 1 every time a new build is created. So after we created the first build for the minor release 23.1, the unique build identifier called build id will be 23.1-1. After a new build is created, the build id will be 23.1-2, and so on.

Before you start testing or bug fix verification, make sure that you are testing the correct application version by checking:

– the build id and

– the DB version

As you already know, you should ask the release engineer to provide you with an interface to easily identify the application version.

Finally,

– the code is written

– the testing is finished, and the bug fixes are made and verified

– acceptance testing is finished

– at our Go/No-Go meeting we decided that we are ready for our first major release 1.0. Hooray!

Our first live application version will be 1.0-23/34

1. Configure the production machine, e.g. create needed directories: /var/www/prod/cgi-bin/, etc.

2. Upload the SQL procedure to the production machine and run that procedure against the DB to create the DB schema with version 34.

3. Configure the build script to create the build on the production machine.

4. Run a build script to create a build on a production machine. The build script checks out files of the application version that we are going to release from CVS and copies those files to the production machine.

As our project evolves, our production environment will turn into tens of servers, which will form our production pool, but for now:

Ladies and gentlemen, our first release of sharelane.com is LIVE!

Guess what? Users seem to like us! Our user base grows like crazy, and now we have hired two PMs, four more developers for the application core, one UI designer/developer, one DBA, one tester, and one CS (customer support) person. After another three weeks of hard work, we release version 2.0! But, once we poured the champagne to celebrate 2.0, our CS Nina bursts into the conference room and screams that she’s been getting tons of complaints from users, because 2.0 is as saturated with bugs as the U.S. Congress is with lobbyists for a military industrial complex.

– Push 1.0 to prod, i.e. revert back* to the good old version.

* another term is “rollback”

It might look like the first option is problematic, because at ShareLane nobody seems to remember the exact version of each file for the production version of 1.0. Our build script is primitive, and as we create each new build, we don’t save the association between the build id and the file versions that belong to that build. But after a couple of beers, Willy declares that first option is doable, because we can recreate 1.0 by checking out from CVS file versions dated right before 1.0 was released.

In the general case, it’s not easy to figure out if the second option (bug fix) is more or less attractive than the first one (rollback to old version), but this time it was easy for us. The reason is that we HAVE TO go for the second option. Why? Because we’ve just discovered that our DB procedures … have not been checked in into CVS and nobody remembers the exact DB schema used for 1.0. So, even if we get the files for 1.0, there is a fat chance that they will be incompatible with the DB schema for 2.0 and thus we might get lots of bugs.

– The programmers who fix the bugs for 2.0 don’t work on 3.0.

– The programmers who don’t fix the bugs for 2.0 cannot do a CVS checkin for 3.0, because it was decided to lock the CVS and allow only bug fixes for 2.0 to be checked in.

– The tester is spending his time verifying bug fixes for 2.0 instead of writing test cases for 3.0.

After the 2.1 patch release where all the nasty bugs have been fixed, we have a meeting in which Billy suggests that we have to take our version control to the next level by creating branches in CVS. He says:

“Okay, guys and gals. I have a four-year-old son Edward, and I have to send photographs of him once a month by email to my mother-in-law who lives in Rome. If a photo shows that Eddie is sick, then she calls and screams with anger, just as if she’s used our 2.0. So what I do is this: I save photos of Edward looking good in a special folder, and if my mother-in-law starts to complain after getting a photo where Edward doesn’t look healthy enough, I just say to her, “Wait a minute, that was the wrong photo,” and I email her a photo from my golden reserve of alive-and-kicking Eddies.

“Here is the story of our project from the release engineer’s point of view:

“One day we started to write our code, and as we proceeded further, we decided to use CVS to store versions of our files. At one point, we said “Stop’ and decided to call whatever we had in the CVS “version 1.0.’ Then we started to add and checkin to CVS our new files and checkin into the CVS new versions of existing files, and again, at one point, we said “Stop’ and decided that whatever is in the CVS must be called “version 2.0.’ We did everything right, except one thing: the files of 1.0 and 2.0 got mixed up because we didn’t separate them.”

“Now, imagine a tree: a trunk and branches.

Here is what we should have done from the beginning:

– The files created for and up to the 1.0 release make up the trunk of a virtual tree in the CVS. Dot at the right end of the trunk is also called HEAD – it has the most recent checkin and hence it’s the most recent version of the trunk.

– Once we say, “Stop” for 1.0, we create (or “cut”) a virtual CVS branch from the trunk, and that branch will contain our files for release 1.0 (the trunk will have those files, too).

– So now we have a CVS trunk and a CVS branch 1.0.

– The programmer who writes code for 2.0 must check in his files into the trunk (dotted line).

– Once the 2.0 code is finished, we cut another branch called 2.0.

– Now we have a trunk, a branch 1.0, and a branch 2.0.

What shall we use to add/checkin the files for 3.0? Of course, the trunk (dotted line)…

– and so on.

This way, the code of each release lives in its own branch, or exists as a continuously updated trunk.

There are a lot of nuances about branching, but for now it’s important that you grasp the concept of why branching is necessary.

What about our stuff? What’s done is done. In our messy CVS, we have:

– all code for 1.0

– all code for 2.1

– part of the code for 3.0.

Let’s call the trunk whatever we have in the CVS now. I’ll spend my time finding all the files in their versions for 2.1, and I’ll create a branch for 2.1, so if we release a buggy 3.0, we can easily go back to 2.1 by checking out the files from branch 2.1 and sending them to prod. And of course, from now on, we cut separate branches for each release.

And, last but not least, I’m going to fix our build scripts to

– enable logging associations between

1. The build number;

2. The file versions in that build;

3. The time when the build is pushed to the target environment.

– enable the build script to create any past build when the build id is provided as input.

And … I’m personally going to kick in the butt anyone who modifies the DB schema and doesn’t check in SQL procedures into the CVS.”

First, we can easily get back to any of the previous versions.

Second, the results of their work on each of the versions will be separated in CVS by the branching mechanism.

Third, we can control the state of each branch and trunk. Let’s set up our branching mechanism to be able to have three states of branches:

OPEN: we can add/delete/checkin files (to/from/into CVS) without getting approvals, meeting certain conditions, etc. The trunk is always open.

CONDITIONALLY OPEN: we can add/delete/checkin files if we meet certain conditions depending on concrete situations. For example, in some companies approval from the dev manager is needed to add/delete/checkin files if a bug is found during acceptance testing, i.e. at the end of stage “Testing and bug fixes”.

LOCKED: this applies to all branches with past/present production versions.

1. During “Coding,” the trunk is OPEN; the developers who are working on the coming release can mess up the trunk as much as they like.

2. During “Testing and bug fixes,” the branch is CONDITIONALLY OPEN; the developers can do add/delete/checkin operations in the coming release branch only if they provide a valid bug number during actions with the CVS.

3. Once the code is in production, then the corresponding release branch is LOCKED.

Sometimes Internet companies make a beta release prior to a major release. The idea behind a beta release it is this: Before we make an official major release (in other words, a major release available to ALL possible users), we make the code of that major release available to a limited group of people (beta testers) who represent our target users.

1. Beta testers will report bugs to us.

2. We’ll monitor our system and see how it works under real life usage. For example, if the DB crashes during beta testing, we can assume that it will also crash after a major release when many more users are going to use that code.

As beta testing goes on, we fix bugs and deal with other discovered problems (e.g., we might decide to add more servers to improve Web site performance). An example of a beta release is the email service Gmail: until Feb. 2007 new accounts could’ve been created by invitation only.

As a rule, companies use beta releases in two cases:

1. The very first release (1.0) of the software.
2. The release of a large, important project; e.g., Gmail by Google.

The logical question is: “If we have beta testing, then we must have done alpha testing, right?” Yes, alpha testing is the testing done BEFORE releasing the software to beta or regular users; e.g., the testing done during the stage “Testing and bug fixes” is alpha testing. Please note that alpha testing is performed by anyone inside the company who tries the new code before it’s released. For instance, the PM can ask the developer to play with the fresh code on the developer’s playground to see how the ideas from the spec are implemented in the software.

Testers in Internet companies are in a privileged position compared to testers from other industries. If we at sharelane.com accidentally release a bug on production, we can do a patch release and remove the production problem within minutes. In many cases, that patch release will have a very low cost, and users will have no idea that a bug ever existed. But what if a P1 bug is found in the braking mechanism of an automobile?

– It will cost the auto company millions of dollars to make a recall.

– It will require active user participation to drive to the dealership to fix the problem.

– A release that doesn’t have a critical urgency must be pushed to production while the majority of users are nonactive; i.e., during the night. You can define a “night” for your releases using some interval of time (for example, from 00:00 to 6:00) in the time zone where most of your target users live. This can be very difficult for Web sites with a big international exposure, like www.google.com. As a rule, U.S. companies make releases between 11:00 p.m. Pacific Standard Time (2:00 a.m. Eastern Standard Time) and 3:00 a.m. PST (6:00 a.m. EST), so they have a four-hour window when the majority of people who live in the continental U.S. are asleep.

– Right before and during the time the release to production is under way, put a polite message like this on the production homepage: “Sorry for any inconvenience. This site is under maintenance. We’ll be up at 3:00 PST.” It’s not a big deal from a technical point of view, but your users will really appreciate your consideration.

– In many cases, a coming release is not pushed to all the servers in the production pool, but rather to just one or a few of them. The logic behind it is that we don’t want to expose ALL of our users to the new code until we verify that this code works in real world conditions. So, random users hit our new code on one or several production machines and we monitor the quality in production by looking at the DB and log files. This approach is especially good for architectural releases when the front end is absolutely the same, but the back end is different.

– Depending on the specifics of the business, Internet companies usually can predict the times when users are going to be more active than usual. For companies that sell consumer goods, like Amazon, the period between December 1st and December 24th (the Christmas season) is the hottest time of year when a great deal of sales are made. If we know about that period of time beforehand, we must introduce a moratorium for any release to production, except EBF and EFR releases. The reason is simple:

– We don’t want to jeopardize our major revenues. Next ->

Lecture 4 - The Software Development Life Cycle -> Quick Intro -> Idea -> Product Design -> Coding: Part 1 -> Coding: Part 2 -> Testing And Bug Fixes -> Release -> Maintenance