Unified system for continuous integration and automated testing in software engineering

Material Information

Unified system for continuous integration and automated testing in software engineering
Parvin, Sarah
Publication Date:
Physical Description:
xi, 48 leaves : ; 28 cm


Subjects / Keywords:
Software engineering ( lcsh )
Computer programming -- Management ( lcsh )
Computer software -- Testing -- Automation ( lcsh )
Computer software -- Quality control ( lcsh )
bibliography ( marcgt )
theses ( marcgt )
non-fiction ( marcgt )


Includes bibliographical references (leaves 46-48).
General Note:
Department of Computer Science and Engineering
Statement of Responsibility:
by Sarah Parvin.

Record Information

Source Institution:
|University of Colorado Denver
Holding Location:
|Auraria Library
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
747558592 ( OCLC )
LD1193.E52 2011m P37 ( lcc )

Full Text
Sarah Parvin
B.S., Iowa State University, 2000
A thesis submitted to the
University of Colorado Denver
in partial fulfillment
of the requirements for the degree of
Master of Science
Computer Science
Spring 2011

This thesis for the Master of Science
degree by
Sarah Parvin
has been approved
Richard Osborne
Tom Altman
Bogdan Chlebus

Parvin, Sarah (M.S., Computer Science)
Unified System for Continuous Integration and Automated Testing in
Software Engineering
This thesis is directed by Asst. Professor Richard Osborne
Continuous Integration (Cl) and Automated Testing (AT) implement
continuous process of quality assurance in software engineering. Cl is the
strategy and practice of confirming the changes to a software projects
code that is successfully built, self-tested, reported, and frequently
released. The goal of Cl is to provide quick feedback to the team in case
defects are introduced into the code. Thus, defects can be identified and
corrected quickly. The AT involves various types of testing such as
regression tests, systems tests and various other types of tests that test
the software thoroughly. The goal of an AT is to improve the quality
control, thus saving engineering time and delivering a high quality product
to the customer on time. The Cl and AT also reduce the headache of
manual compilations of the code, manual packaging of the compiled code,
manual deployment of the package/software, and performing manual

This thesis examines the Cl and AT processes in software engineering.
Research has shown that integration and test processes can cost up to
40% of the project. Implementing unified Cl and AT can significantly
reduce this cost.
The Cl and AT process in software engineering were studied in this
authors organization. This includes studying how the continuous
integration process and automated test process work; thus creating the
technical basis to start using this process on future projects.
Improvements on Continuous Integration and Automated Testing based on
the current Cl and AT Process of the authors organization has also been
suggested, mainly to a unified system of Continuous Integration and
Automated Test Process.
Keywords: Continuous integration, version control System, software
configuration management, test automation, build automation.
This abstract accurately represents the content of the candidate's thesis. I
recommend its publication.

This masters thesis demonstrate the Continuous Integratiion and Automated
Test process and improve the quality of software development processes;
also how it can save money and reduce Engineering time. I am very lucky
to be given the opportunity to write my thesis on this topic.
I want to thank Assistant Professor Richard Osborne for supervising my
thesis work and giving me invaluable feedback. And I also want to thank
Associate Professor Bogdan Chlebus and Professor Tom Altman for
serving on the committee.
Finally, I wish to thank my Mother, my brother Shobhon and my love Iftekhar,
who are overseas but are always supportive. The biggest thanks goes to my
Father who is no longer living in this world, but who is always with me,
encouraging me in any work and is always happy in my success. Thank you
Dad. And, I wish you were here today.

LIST OF FIGURES....................................................viii
LIST OF TABLES.......................................................ix
1. Introduction.......................................................1
1.1 Background......................................................1
1.2 Objectives and Restrictions....................................3
1.3 Structure of the thesis........................................4
2. Practices of Continuous Integration................................5
2.1 Advantages and Disadvantages of Cl process.....................7
2.2 Automate Builds................................................9
2.3 Code in a Single Source Code Management Repository............11
2.4 Commit Source Code Changes Frequently.........................15
2.5 Keep the Build Fast...........................................17
2.6 Continuous Feedback...........................................21
2.7 Automate Deployment...........................................22
2.8 Make Builds Self-Testing.....................................223
3. Software Testing and Automating the Software Testing..............26
3.1 Types of Software Tests that can be automated.............27
4. Tools for Unifying Cl process and Automated Test..................29
4.1. Cost of estimating Cl.........................................29
4.2. Available Tools in the market.................................32
5. Analysis on a related project.....................................37
5.1 Current Cl and Automated Test Process of the project..........37

6. Recommendation for Improvement by unifying the Cl and Automated
6.1 Benefits of the Common Framework............................42
7. Conclusions....................................................44

2.1: Schematic drawing illustrating Cl process work flow............6
4.1: Schematic drawing of a flow chart of continuous integration
environment for the .NET platform...................................33
4.2: Schematic drawing of a Test Management for automated testing in a
.NET platform.......................................................36

Table 2.1: Integration builds duration improvements........19
Table 4.1: Cl process overhead.............................30

API Application Programming Interface
AT Automated Test
BPT Business Process Test
Cl Continuous Integration
GUI Graphical User Interface
HTTP Hypertext Transfer Protocol
HTTPS Secure Hypertext Transfer Protocol
IDE Integrated Development Environment
OS Operating System
SCM Software Configuration Management
VCS Version Control System
Agile Agile software development is a collection of software
development methodologies that follows iterative and
incremental development. In an agile software
development, requirements and solutions advance
through teamwork between various cross-functional
Smoke Tests Smoke tests, also called a Build Verification test is

done to test the build. Smoke test should be built in a
way so that it exercises the whole system. It doesnt
include all the functional tests; only key areas of
functional tests are done to report that the build status.
Software Artifacts
Software artifacts are generally use cases, class
diagrams, and other UML models, requirements and
design documents, business cases, project plans or
anything that helps describe the function, architecture,
and design of software.
QA QA stands for quality assurance. The purpose of
software quality is to assure that the standards,
processes, and procedures are inline with the projects
requirements and are correctly implemented.

1. Introduction
1.1 Background
The Continuous Integration (Cl) process has several aspects such as
maintaining code in a single Source Code Management (SCM) repository,
automated builds, frequent source code commits, self-testing builds
(generally unit testing and smoke testing), fast builds, and constant
feedback. Automated Test (AT) processing is the automation of
softwares testing process; basically thorough tests of the software which
can include functional tests, regression tests, system tests, user
acceptance tests, etc. The practice of Cl and Automated Tests are a
unified system that can be used in an organization to fully automate the
build, package, deployment and test processes. In other words a one click
automated process where this process can run thru a scheduler; thus all
these performing unattended. This can reduce the headache of finding
the defects late in the code or in the functionalities of the software;
decrease the release cycle time, decrease human interaction and thus
reducing engineering time to a greater extent [1, 2, 14].

The term Continuous Integration (Cl) has evolved from Agile software
development methodology. Agile methodology is based on the following
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Continuous Integration is a software engineering practice where
developers can checks-in their code frequently; thus it can result in
multiple integrations per day. The integration is verified by an automated
build which includes running unit tests. This way defects are found
earlier in the process and can be fixed quickly. Fixing bugs early are
crucial because research has shown that the cost of the fixing the bug is
proportional to the age of the bug [3]. Fixing the bug in a few hours or in
the same day is easier as developers mind are still fresh. However, fixing
numerous bugs that were introduced weeks back can be difficult and time
consuming, thus requiring more testing time and can delay the delivery of
the product [2, 7].

Software testing is an activity to find defects in the software. Testing can
be done manually, which requires Test Engineers to execute various steps
to test the software to ensure the functionalities work and match the
requirement. Manual testing can be a laborious and very time-consuming
process. Test automation attempts to automate the process of manual
testing, thus reducing engineering time significantly. Furthermore, a
human being can mistakenly pass the functionality of the software, but a
good automated test would catch that defect in the functionality.
Automated tests are also highly reusable which reduces human interaction
in testing the product; in the consequence it reduces engineering time
substantially [15, 18].
1.2 Objectives and Restrictions
This thesis focuses on Continuous Integration and Automated Test
Process in software engineering. This thesis also studies the current
Continuous Integration and Automated Test Process of the authors
organization. Finally, this thesis suggests an improvement of the Cl and
Automated Test Process based on the current Cl and AT Process of the
authors organization. The results will demonstrate in detail what tools and
processes are used, where improvements can be made using the Cl and
AT process.

The implemented software in the authors organization is written in C#
using .NET tools. .NET tools are used for Continuous Integration and BPT
- Hewlett Packards Business Process Test is used for Automated
Testing. However, any programming languages, platforms, or any Cl and
Automated tools can be used for other future projects.
Because, of a Non-Disclosure Agreement, the name of the authors
organization may not be mentioned throughout the thesis.
1.3 Structure of the thesis
Section two explains the fundamental software engineering practices
comprising the Cl process. Section three explains automated Test
Process. Section four discusses some tools available in the market.
Section five talks about the related project that has been analyzed in the
authors organization. Section six makes suggestions for an improvement
and creating a common framework for Cl and Automated Test Procedure.
Section seven concludes the thesis.

2. Practices of Continuous Integration
Code in a Single Source Code Management Repository, Automate Builds,
Commit Source Code Changes frequently, Make the Builds Self-Testing
(generally Unit Testing), Keep the Build Fast, Constant feedback, and
Automated Deployment are the key features in Cl. Each of these
processes is discussed in detailed in this chapter [1,2, 5].
Figure 2.1 presents Cl process work flow and shows various components
of Cl system. The description of the various components of Cl system is
given below:
- Developers check-in the code in the version control repository.
- Cl server, on the integration build machine, polls the repository
every few minutes to detect any change in the version control
repository. It retrieves the latest code from the repository when
there is a change. It then performs the automated build, integrates
database, runs unit tests and deploys the package.
- Cl server auto-generates the feedback by emailing build results to
the appropriate team members. Cl server continues to poll from the
repository to detect any changes in the product [1].

cmmit Changes
Cl Server
Build Machine
Compile Source
Code, Run Unit
Tests, Deploy
Software, Run
Smoke Test
Figure 2.1: Schematic drawing illustrating Cl process work flow.

2.1 Advantages and Disadvantages of Cl process
The practice of continuous integration has many benefits. Some of them
are listed below.
Because, frequent builds are done, the build issues can be
detected early.
In general, in a distributed development environment
developers do not always communicate with one another as
the development can happen in different facilities; thus,
continuous integration can assure the project members
that the product that is being built has the latest code.
Continuous integration also ensures that the code check-
in is frequent; hence keeping the build up to date.
The feedback loop is smaller. As frequent builds are done,
the respective project members are notified frequently about
the build. Therefore, if there is a problem with the build, it
can be fixed faster.
A developer does not have to wait until the end of the day or
a week to find out how the check-in affected the build, as the
frequent builds are done daily.

Frequent unit testing and sometimes smoke testing are also
done in the consequence; therefore, bugs are detected early
in the process.
Every check-in goes through the testing; thus problems are
caught earlier, resulting in a less problems with the build
The project can have a latest-and-greatest build to use in
demos, showcases, etc.
Good Cl process can help delivering the product on time
Like any process Cl also has few downsides. Some of them are listed
Frequent build can increase the maintenance overhead.
It requires a high level of discipline thus, it can cause
bottlenecks. For example, if a bad or partial code is
checked-in and a new build is done then the build will be
broken. Thus, causing some delays.
Because, there are frequent builds throughout the day, the
testing process has to be handled with concrete plan as
some testing requires few days of testing [1-3, 5, 6, 17],

2.2 Automate Builds
In todays software engineering environment, automating the build process
has become a key feature. There are many reasons to it such as larger
code base, bigger teams and teams located in multiple sites. If an
automation of the build process is neglected, it will become an endless
source of problems and nuisance; thus it can significantly reduce
productivity. On the other hand, a well maintained build keeps the
development running smoothly without constant interruptions. A well-
maintained automated build process increases efficiency and productivity.
Thus, it helps test teams performing their tests efficiently and helps
delivering the product on-time [1, 17, 21],
Through an automated build and package process, teams can go back to
previous builds in case they need to revert back to some functionality. It
serves many purposes such as speeding up the process, keeping different
versions of builds, reducing errors due to the missing steps that build
engineers may have forgotten during building of the product, saving cost,
etc. Build results can be precisely predicted by ensuring build
reproducibility and by undertaking minimal development down time due to
the build issues. Some of these systems can take many hours to
complete the build process specifically it is true for larger and complex

products. Thus, automating the software build process is also a primary
step in maturing the software development process [1, 2],
Build automation involves automating the compilation of source code into
binaries. Packaging means packaging these binaries. A software package
refers to computer software packaged in an archive format to be installed
by a package management system or a self-sufficient installer1^23]. Build
process is a critical process in a software development environment. The
main responsibility of the automated build script is to provide automated
generation of systems builds in a repeatable fashion. These scripts can be
scheduled to be run anytime. Builds can be classified into three different
types Individual (also called a private build), Team, and User. In general
the developer runs an individual build and runs the basic tests to verify
that the update of the code or newly written code is correct. Team build is
where code is checked-in to the base and a build is done after that. Unit
test, smoke tests and regression tests are run followed by the team build.
User build is the one that is deployed on the production system and all the
tests unit tests, regression tests, functional tests and performance tests
are run.

Build can be automated with scripts or other tools combining with scripts.
In a Windows .NET environment, builds can be automated via .NET
technology. In a Unix environment, build process can be automated via
scripting tools like Make, Perl, Sed, Ant, Awk, etc. [14, 19, 20],
2.3 Code in a Single Source Code Management Repository
This practice helps the use of a version control system for the project's
source codes. All artifacts required to build the software should be stored
in the central repository system because, building software requires that
all software assets are centralized. In practice, software projects involve
lots of files which can be modified by several people, and sometimes
modification can happen concurrently. This process can be difficult to
keep track of the files and their changes. Therefore, the solution would be
to use a good Software Configuration Management tool (SCM), which is
sometimes referred as a Version Control System (VCS) or a repository.
These kinds of management tools provide a single source code point and
prevent problems from losing data. It also provides versioning of the code,
meaning every time developers check-in the code, it will be labeled with a
version. Therefore, VCS contains the earlier versions of each file, so it is
easy to return to an older working version in case a file gets corrupted or
the latest changes need to be discarded. Centralizing the source code

system also makes it easier to perform a single command build as the
build tool can retrieve the source code through the script. There are many
VCS tools out there in the market such Concurrent Versions System
(CVS), Subversion, GIT, or Clear Case. Microsoft provides a very good
Version Control System called Team Foundation Server (TFS) which is
integrated with .NET [7, 8, 9].
VCS is fully capable of storing everything that is required for building the
software. It includes a workspace, a collection of all files maintained in a
local disk space which is mapped to a version control folders. The
workspace contains exact version of all the artifacts required to build the
product [3]. For building the product, the workspace may contain the
following items:
Products source code, specified in the appropriate package
Source code for unit tests
Third-party components and or library files for interfaces that
the project may depend on; such as JAR files, DLLs, Unix
shared libraries, Windows shared libraries, etc
Configuration files
Build scripts and build environment settings

Installation scripts for deployment steps
Smoke test scripts for smoke tests
Regression or System test scripts for deeper level tests
Project members decide which items to add on the project; it usually
depends on the projects need. It is worth mentioning that multiple
workspaces can also be created. When a defect in an older release
requires going back to the older version and compiling the exact versions,
the later version of a compiler or tool may have understated and
undetected differences in behaviors. Therefore, it can affect the bug fix
and can become a headache. Thus, system wide tools, such as compilers,
development environment or database systems, are not stored in the
version control. However, as an alternative VCS can include all required
initialization methods to create the database locally. Storing or publishing
build results or artifacts under version control system should be avoided. It
can take lots of space and thus creating more traffic; hence, slowing down
the VCS.
The directory structure must be consistent, coherent and logical inside the
repository. A common methodology is to structure the directories based on

development activities, such as requirement, design, implementation,
testing and release [7, 8].
Many VCSs provide a feature called branching techniques, for example
Visual Team Foundation Server by Microsoft. Branches are used to
handle several flows of development [3]. They are generally independent
of each other. However, researches show that sometimes branching can
be overused. Creating too many branches sometimes can create
confusion. Therefore, to avoid too many confusions a project should
create a master branch where current development can undertake.
Branching should be minimized as much as possible and should be
created only where needed [3, 7, 8],
Working in a separate branch is very useful and important. For example a
branch for bug fixing prior to production release, a branch for temporary
experimentation with a new feature, a branch for 1-2 weeks of Regression
Testing. This way main branch can be kept in a stable position. When
changes in the branch are agreeable and approved, they can be
merged back to the mainline [7],

Some VCS has the capability of Parallel Development; for example Team
Foundation Server(TFS). TFS also supports two and three way file
merging operations, and it also has the capability of resolving the merging
conflicts. Parallel development allows individual team members to
complete many areas and versions of a project at the same time. They
dont have to wait for each other; thus, saving lots of time [3, 19, 20],
2.4 Commit Source Code Changes Frequently
In VCS or under Cl, the term commit means checking-in to the
centralized repository. Developers should commit their changes to the
VCS frequently. Committing at least once per day or more is suggested.
By committing frequently developers can communicate their changes
regularly. This also ensures that other developers can check-out the
latest code. Committing frequently also means finding the problems early
on; hence fixing issues early as at that point much hasnt happened.
Tasks should be broken up into pieces to complete the tasks in few hours;
thus accomplishing frequent commits. It is suggested to split the work into
small tasks, write the source code and unit tests each task. Modifying
many tasks at once should be avoided. To err is human; therefore it
shouldnt be assumed that everyone knows everything and developers will

never make any mistakes. Therefore, having unit tests implemented at
the same time, when the change is checked in, is a good idea and can
prevent many problems in the build. This also saves many engineering
hours. Unit tests codes are generally built into the build scripts, thus
good build scripts can ensure good build. Other ways to protect
problems in the build is to have a process where developers run a
private build locally in their machine before committing any changes.
This way repository also can be kept healthy [1, 5].
However, if in any case builds are broken, it should be fixed at a top
priority. A broken build is not just a compilation problem of the code, it can
be anything that prevents the build reporting a success; for example, the
packaging could be bad, product functionalities can be broken, etc. Even
if the build and package were performed correctly, perhaps there is an
installation problem or some functionality in the software got broken
because, of the latest check-in. As there are more frequent commits,
there is a possibility of having more broken builds. Favorably, each error
is discovered incrementally when the build finishes, and with small
commits, it is easy to locate the defects much faster. Again, performing a
private build locally also helps in preventing problems.
Frequent builds can be done if there are frequent commits. However,

there should be a good plan in tracking all these builds. Because, some of
these builds will be used for deeper level testing and some testing can
take few days. Building frequently on developers machines is also not
suggested as it can take lots of time. Therefore, a dedicated build
machine should be used for frequent builds. Build script can be written
and manually and run on the server, but the most effective solution is to
use a Cl server software for this. The Cl server can monitor source code
repository for new changes, and launch build automatically when it
detects a new change or builds on a scheduled basis only if there is a
change. Cl servers can have various other automation features besides
unit testing the product. They can do smoke tests, provide feedback
about the build to the respective team members, such as sending an
e-mail or posting results to the web page. They can also be configured to
do deeper level testing and provide detailed testing results [1, 17],
2.5 Keep the Build Fast
The most important aspect of Continuous Integration is constant feedback
early on in the build process because, this way defects in the builds can
be caught earlier and can be fixed at a higher priority. Therefore, keeping
the build fast is crucial in Cl. If a minor change in the code base requires
a 4-hours build, a developer would wait half business day before finding

out if developer changes are compatible with the other developers within
the team [1,3],
In order to achieve a fast build, the build system must be able to do
incremental builds in addition to a clean" or full build. Incremental builds
will only build what has been changed. It will re-use the build artifacts from
the unchanged code. This can introduce dramatic speed increases in a
very complicated build process. Clean builds, which could run for hours,
can then be used for a release candidate (RC) builds, which can be
released to the testers for few days or a week regression tests. Clean
build should also be run before releasing the products to the customer.
Not to mention, this customer released build should be thoroughly tested
before it goes out to the customer [1, 15],
It is suggested that building time of the product is kept under 10 minutes.
Research has been done and found that most projects should be able to
achieve this, as the long necessary optimizations are done [3].
Table 2.1 presents a summary of improvement methods on how to
achieve faster builds, their priority and impact on a scalability,
performance and difficulty. Priority refers to the priority of improvement
method. Scalability refers to the capability of handling an increasing
amount of code to be built. Build performance refers to the duration of

building the code. The improvement method refers to various methods of
how build duration can improve and should be considered according to
the priority.
Table 2.1: Integration builds duration improvements [4],
Improvement Methods Priority Scalability Performance Difficulty
Use a dedicated integration build machine 1 High High Low
Increase hardware 2 High Medium Low
Improve test performance 3 Low High Medium
Run staged integration builds 4 Low Medium Low
Optimize infrastructure 5 Medium Medium High
Optimize build process 6 Low Medium Low
Build system components separately 7 Low Medium Medium
Improve software inspection performance 8 Low High Low
Perform distributed integration builds 9 High High High
There should be a dedicated build machine as discussed above and it
should be used in all cases and all projects. Builds process can speed
up by adding more hardware resources, optimizing network bandwidth,
using better and faster Cl tools, etc.
If the smoke tests are added as part of the build, then build process will
take longer. Therefore, for developer quick check private build can be

performed, and smoke tests could be eliminated if the change is only very
small. However, unit testing should be done no matter what. To save
time, builds can be separated to multiple parts. The first build is a
commit build compiling the latest version and running unit tests which
should be done on developers machine. This will expose any obvious
build errors and can be fixed as soon it is found and then commit the
check-in of the code if the private build shows no failure. After the private
build passes, the code should be committed and more extensive build
should be performed, which should include smoke tests. Clean build can
be scheduled as a nightly basis to save time during the day. Usually, this
build should be used for performing a thorough automated.
Build duration can be affected other ways too. Small network bandwidth,
or problems with virtual private networks, geographically distributed
systems constitute a risk for any continuous process, and should be
avoided if possible. Network latencies can impact the build duration,
some files can be larger and the check-out process can take longer time.
Therefore, developers should try to make their files smaller. If the
compilation step takes long time, an incremental build process can be
used instead of rebuilding the whole project. If building the whole
software still takes a long time, it can be split into separate sub-

projects within the Cl system or an incremental build can be done.
Building only the changed codes may increase the risk of introducing a
defect outside the changes and can be unnoticed, because other parts
are not built again. Therefore, this should also be taken into
consideration and obviously that says that if the clean build takes longer
then nightly clean build should be performed. Build execution can be
evaluated by measuring the time of each steps in the build. [1, 2, 14]
2.6 Continuous Feedback
The feedback process is usually a status of the build. Current, build status
can suggest if a developer can move to the next step. In Cl process
developers adds or makes small changes and commit frequently. To
perform a frequent build and to correct any problems found during the
build, feedback must be provided immediately to the developer with
the right information; particularly about the problems found and where the
problem occurred. Moreover feedback process should be clear and
detailed; else it wouldnt be very useful. It is important to have an easy,
accessible method for every interested party (QA, developer, manager,
CEO, etc.) to find out the current build status. This could be an email or a
website according to the projects need. This way, if the build is broken

developer can take quick action to fix the areas where the build was
broken [3].
Build status should be configured in a way so that it only includes the
portion that was run. If its a private build, and the smoke test was not
done, then it doesnt have to include smoke tests results. It should include
the smoke test results if the build is meant for testing and if the build failed
because of the functionality broken in the product. Then, the testers or
developers can determine whether the failure happened because of the
product code or automated test code. If the full automated test is part of a
particular build, then it should include the full test result. Failure can
happen for other reasons too such as slow network. Therefore, good build
report is very important to diagnose a problem of the build [1,4,15],
2.7 Automate Deployment
To perform Cl process, a project needs to automate the deployment.
Thus, product can be installed automatically followed by smoke tests.
Obviously, depending on the project size, the build might need to be
installed in various machines. Therefore, this automation of deployment
should have the capability of installing the product on more than one
machine simultaneously. A solid scripts need to be written to perform this

automation of deployment. This script should also include a feature where
feedback is sent to the respective team members if a failure occurs.
2.8 Make Builds Self-Testing
An efficient way to catch bugs early on a project is to include automated
tests in the build process; usually unit tests, created by the developers. In
some cases, the build is deployed in some client/server machine and key
areas of functional tests are done to report that the build status. These
tests are usually called Smoke Tests or Build Verification Test [14, 16],
Though, testing isnt perfect, it can catch a lot of bugs. Also, in order to
receive accurate feedback from a build, the build system must be able to
test the results automatically. This process verifies that any change to the
source code didnt break any other areas. In case a modified or new code
breaks, existing functionalities will result in a quick failure of the build.
Later failure can be investigated to fix the bugs before checking-in the
code to the central repository [10].
The easiest and the most consistent way is to write the testing code, either
for unit testing or for smoke testing, in parallel with the source code. This
also ensures that the testing of the code and build is done from the
beginning of the project. When the build is run, any failing test should

cause the whole build to fail in a Cl process. To call a build good, it is
recommended that 100% of tests must pass [5]. But, it is also important
to remember that just because, build report was good doesnt mean that
there is no bug in the software. Therefore, later full regression tests and
system tests should be performed [4, 10].
Automated testing is generally divided into unit, smoke, regression,
system and regression tests. But, in general as mentioned above under
Cl, unit tests and smoke tests are done. Some projects would go further
to do regression tests; however, that process requires good planning.
Therefore, in most cases integration tests, regression tests, system tests
or other tests are performed separately.
Unit tests are implemented first. They verify the functionality of small
components, which roughly correspond to a class in object-oriented
languages. A unit test is done to verify that code meets its design and
behaves as intended. A single unit test should depend only on the
component tested and test only its functionality. For example, if the tested
component requires database connection, the test case should be moved
to a higher-level testing category for example it can be under regression
tests. A key part in unit tests is having no dependencies on external

components for keeping the execution time as low as possible. Unit tests
should be created and run early in the development, i.e., from the first day.
A Cl and AT process can be unified to fully automate build, deploy, and do
a thorough tests of the product. But, a concrete planning is required as
some tests take longer to run, possibly few days. Thus, at every build all
the testing shouldnt be run, because, it will be hard to maintain and
difficult to track the results. Therefore, in addition to a frequent daily build,
a nightly build, bi-weekly build or weekly build can be performed and
tracked well.
Finding suitable tools for testing depends on the language and software
frameworks or platforms used in the project. In general, .NET types of
platforms tend to have a more complete set of Cl & AT tools [2].

3. Software Testing and Automating the Software
Before we discuss the practice of Automated Testing, first it is necessary
to understand what is software testing. Software testing is an investigation
and or verification of software to deliver a good quality product to the
customer. The purpose of software testing is to find defects earlier in the
product so that they can be fixed and verified, ensure that functionalities
match the customer requirement and thus provide the customer with a
high quality product. Software testing may involve the whole team and
should be done from the beginning of the project. Software testing can
require repetitive tasks, lots of hours, tedious process and high manual
effort. Thus, creating a test automation framework for this repetitive
process can save great deal of engineering time and can significantly
reduce manual process. It reduces human error as repeated process is
run over and over. It gives the flexibility of re-usability. It saves cost.
Most often it can run faster than human can run the test manually. Also
this automated framework can be integrated with Cl process to even
reduce further human interaction [13, 18].

3.1 Types of Software Tests that can be automated
Unit, Smoke Test, Integration Test, Regression Test and System Tests
can all be automated. Unit tests are implemented first, usually performed
by the developers and integrated in the build scripts. Research has been
done and it suggests that the unit tests should cover all paths of the code,
at least 80%. Unit tests should be created and run early in the
development, i.e., from the first day [13, 22],
Smoke tests, also called a Build Verification test is done to test the build.
Smoke test should be built in a way so that it exercises the whole system
end to end. It is not necessary to include any deeper level test, but should
consist of few tests from each features of the product. This way the
smoke tests can ensure that there is no problem in the build. Later more
thorough tests can be done to test each feature of the product, which can
sometimes be a weeks long of test suites. For example system test for
some products can take few days as system test involves end to end
product test [16, 18].
Integration testing, also called a component tests, verifies a portion of the
whole system. The purpose of it is to verify each feature of the product
and verify if their interaction works as expected. Some integration tests

can be very time consuming and can take hours or days. Automating of
this process can reduce the cost of the product development significantly.
Regression tests usually seek to uncover any new errors. Once, a new
feature or enhancement is implemented, a regression test is done against
that feature and also on any other related feature to ensure that the new
change didnt break the existing functionality. Regression testing can be
time consuming and sometime can take days. Therefore, it shouldnt run
on all the Cl build; nevertheless, it should run on a nightly build.
A system test is an end to end testing of the system. The suite of tests
verifies the software and its user interface, for example, a Web page or
GUI work, end to end as expected. It requires full installation of the
product since the testing is meant to be done very thoroughly. The tests
suite can take long time to run, sometimes days. Thus, the recommended
way is to run system tests on predefined intervals, such as a secondary,
nightly build when the previous tests have passed. Automating this
process save huge manual effort, number of test engineers, and of course
cost [10, 14],

4. Tools for Unifying Cl process and Automated Test
Tools should be chosen to meet the needs of the project. There are many
tools for Configuration Management or version control, such as Clearcase
from IBM, or CVS, which is free. Build tools are also available like Make,
Ant, etc. Testing Tools like Rational Robot or QTP for functional testing.
Automated build tools should be selected for automation. For testing
purpose many scripting languages can be used, but, its better to pick
automated test tools as they have many capabilities already [1,2],
4.1. Cost of estimating Cl
It is nearly impossible to estimate the cost of a Cl for a particular project
because so many variables are included in it. Ade Miller has studied the
project and cost in [26] by gathering data over a period of hundred working
days (approximately 4000 hours) with Cl. The study compares the results
to a hypothetical heavyweight check-in process of the team, and also
discusses the most common reasons which caused the build to break. The
size of this product development was relatively small, 12 people, and the
cost doing a full clean build was low as well. On average, the check-in was
done once per day. In this project, during the analysis period, there were

over 551 check-ins, resulting in 515 build and 60 failures. It gives a failure
rate of 13%. There were four major issues that broke the build.
Compilations issues, in the consequence unit tests reported more than
half of the failure. Code quality issues were reported 40% of the failure.
There were 6% server infrastructure issues. This project didnt include the
full smoke test or other automated functional tests. Table 4.1 shows Cl
process overhead.
Table 4.1: Cl process overhead [26]
Cl Server setup and maintenance time and build script authoring (est.) 50
Total time spent checking in 20 minutes (est.) 551 165.3
Total time spent fixing build breaks 45 minutes 69 51.75
Total overhead (hours) 267
Miller concluded that the effort allocated for managing the build was
minimal. He also concluded that the actual cost of using Cl process on
this particular project was 40% less than the hypothetical cost of a check-
in process that does not induce Cl process nevertheless, maintains the
same level of quality.
Providing and delivering the product is very important. Otherwise,
customers might not accept the product if its late. [6] The Standish

Group's just-released report, "CHAOS Summary 2009," "This year's
results show a marked decrease in project success rates, with 32% of all
projects succeeding which are delivered on time, on budget, with required
features and functions" says Jim Johnson, chairman of The Standish
Group, "44% were challenged which are late, over budget, and/or with less
than the required features and functions and 24% failed which are
cancelled prior to completion or delivered and never used."
"These numbers represent a downtick in the success rates from the
previous study, as well as a significant increase in the number of failures",
says Jim Crear, Standish Group CIO, "They are low point in the last five
study periods. This year's results represent the highest failure rate in over
a decade"[25]
From the Standish report it can be seen that more of the projects fail due
to limited functionality and late delivery. Therefore, good Cl and AT
process, solid budget and concrete planning can reduce the risk of
delivering the product on time.[6]

4.2. Available Tools in the market
There are many available tools for unifying Cl and full automated tests
such as Visual Studio from MSDN. It has source control, development
environment for software designers and developers, build and package
capability for build engineers, version control for configuration
management engineers, storing and designing capability for project
manager and business analyst and automating testing capability for
automation engineers [24],
Figure 4.1, shows the full capability of Visual Studio for a full featured Cl,
where automated testing can be fully integrated.

In Scope: Why and How to build a Continuous
Figure 4.1: Schematic drawing of a flow chart of continuous integration
environment for the .NET platform [17],

Example of build structure and code:-
Create the project tag named MySystem:

default-'build" states that the default task is build and that will
be executed if no task is specified on the command line when running this
NAnt script. Everything else added to the build script exists under this
Properties are used to set some of the values used later in the script. Add
the following to

Each name can be used as a handle to its value by referring it as
${property-name} for example ${buildbase}.

The task is an executable task that can do one or more things.
The build task is the default task that uses task to
compile the application along with calling other tasks to perform
other operations.



Testing with Visual Studio 2010
Test cases and test suites can be created and automated thru Microsoft
Test Manager 2010 under Visual Studio 2010. Below is a screenshot from
MSDN (Figure 4.2).
y f Contents
,i New t
a .> Beta Test Plan
Test suite: Beta Test Plan
Default configurations (II: Vista and It 8
State; Q In progess
jfoper *JfAdd jNew X 4 Assign Ccnfquratons
Drag a column header here to group by that column.
Order ID Title Priority
1 12781 Add to Shop
Figure 4.2: Schematic drawing of a Test Management for automated
testing in a .NET platform [24],

5. Analysis on a related project
Analysis has been done on a project in a large company on a big project.
This project itself consists of over 200 people at various locations. This
project uses .NET Framework for Building of the code, Installshield for
packaging and BPT process for Test Automation. This project also uses
the Continuous Integration System and Agile development process. By
studying this project my main goal was to understand the fundamentals
of what kind of practices are involved, what kind of implementations
has been done for the technical infrastructure for the Continuous
Integration and Automated Test Process. I was able to study and
understand the process deeply and able to discover a number of areas
where this projects Cl and AT process can improve.
5.1 Current Cl and Automated Test Process of the project
This project involves few products which interact with each other. All
these products use .NET technologies, Visual Studio 2010 for
Design and development
Version control
Build & package
Unit tests

Business Process Test (BPT) technologies from Hewlett Packard are used
Storing requirement
Automating smoke Tests
Creating and automating regression tests
Creating and automating system tests
Continuous Integration of this project involves automated build & package,
automated unit tests and automated smoke tests. Automated test process
includes more than 70% of the testing which includes regression and
system tests; 30% of the tests are done manually. However, regression
tests and system tests are not part of the Cl process.
Developers check in the code very frequently, essentially anytime of the
day. Thus, frequent builds are performed throughout the day, usually 3
times a day if the Cl server detects a change in the repository.
Developers do a private build on their machine before they commit their
code changes which include unit tests. This private build only builds the
portion of the product to save time. There are number of dedicated build
machine. Daily nightly builds are done. If the nightly build status has no
failure then it is released for regression tests. Each build, except for the
developer private build, consist of smoke tests. This build usually takes

about an hour to complete; and when the build report results 100% passed
only then it is released. A weekly build is done after the regression tests
are run and show 90% passing rate. This build is used for system tests
which are run for a week, for end to end testing.
The installation part of the products is still manual; hence both the server
side and client side of the products are installed manually. The server
side installation is done by the build engineers and takes over an hour.
The client side of the product is installed by the respective team members
such as project managers, developers, tests engineers and takes about
ten minutes.
Though, the regression test and system tests are automated, there is no
automated process of reporting the results. The test manager manually
queries the result from BPT and compiles the test result.

6. Recommendation for Improvement by unifying
the Cl and Automated Tests
In a real world, most projects dont integrate the full-featured automated
testing with Cl. Mainly, because it requires lots concrete planning and
better tracking. However, Unifying the Continuous Integration and
Automated Test can reduce the cost of the project significantly. In the
current project that I have done analysis on is very much capable of
Unifying Continuous Integration and Automated Testing. Not to mention,
the deployment of this product can be automated and can be a part of this
Unified process as well.
To implement Unified Cl and AT in this project, it will require good
planning and few engineers to work on it for 1-2 months to automate and
test the new process. There will be some cost in the short term. However,
project will see enormous benefits in the long run. This project has been
in development for last 3 years and will continue for at least 5-7 years
more years.
The following can be done to unify the current Cl and AT process.
Write windows script for installation of the server and the client.
Write windows script to start BPT and start the tests

Thus, the new process of Unified Cl and AT will perform the following for a
nightly build, if implemented:
1) Perform the build which also includes unit tests
2) If the build result is good, then perform the deployment of the server
and client and run smoke tests
3) If all smoke tests pass then perform the installation of the product to
various machines.
3) If the installation steps passed, then start BPT client and start the tests.
As there are numerous regression tests; these should be divided to run on
various machines. Nightly build can be started early night and tests
should be run on various machines. This will help completing the tests by
morning. Test engineers can look at the results and can write defects
accordingly. This whole automated process can be run periodically or can
be run on a scheduled basis essentially for the system tests.
The unified system should also have a flexibility of running each of these
pieces separately, for example, developers might just be interested in
building and packaging the code and run only few essential tests to test
his/her piece. Testers might be just interested in installing the latest build
in some client or server machine and run a particular test or do defect

verification only. Microsoft Team Foundation Server already has a
capability of doing al of these. However, in this project automated tests
have already been created with BPT tools; therefore as suggested,
windows scripts can be written to automate the deployment process and
then kick of the tests from BPT.
6.1 Benefits of the Common Framework
A project can benefit and reduce the manual work by integrating the
current Cl and AT Process. Currently human interaction is needed to
deploy the Server and the Client, which are 2-3 hours of work, and when
the smoke tests are run after build is done, monitoring the smoke tests is
required; which could be another hour. When the build is successful, each
team members installs the client; and it takes about 10 minutes. There
are about fifty test engineers in the project, which means at least 40-50
installations by testers per day. Sixty developers developing the product,
meaning at least another 50-60 installations per day. There are other
teams members also install and verify the product. Thus, this would be in
average 14-16 hours of installation time; about 2 working days just for an
installation. Thereafter, test engineers kick-off the tests and monitor them
often. Many hours are required here. This time can be significantly
reduced by utilizing a Unified Cl and AT process, thus reducing 4-6 daily

engineering times. There are other benefits to it too. The team members
can enjoy a one-click process. The project managers, business analysts
and other team members, who are not technically involved with the
product, can run this automated process full or in partial. This way, they
have a better understanding on the implemented feature of the product.
The full automated process can be run on a customer environment; thus
defects can be found and fixed early; hence ensuring a good quality

7. Conclusions
The benefits of Cl and AT compared to a non Cl and AT process are
remarkable and can be still significantly improved. In most organizations
Cl processes only include Build & Package, Unit Test and Smoke tests.
Unifying deployment and full automated test can save many hours of
manual work and headache. The Cl process also can apply fixes to the
software immediately, integrate and release without any additional cost. It
also reduces repetitive process to a greater extent and build, deployment;
featured testing can be done anytime by anyone in the project. Most up-
to-date and solid product can then be presented to the upper level or the
customer in need. Project visibility can be improved by continuous
feedback. Most importantly the Unified process of Cl and AT can reduce
cost and greatly augment the likelihood of on time delivery.
This thesis work has been researched in detail to learn Cl and AT process;
and to find out how the described benefits can be achieved. Results also
have been observed from studying the project in authors organization.
The Cl process is built on top of the fundamental software engineering
practices are considered as best practices by the industry and
should be followed as a guideline. It is advisable to establish a major
part of the Cl components and to develop guidelines prior to introducing

the process to the project. Few issues that was discovered and learned
during the analysis of the related project were 1) to keep the build time
down 2) and not to checking-in partial code or bad code. These can have
significant effect on the team behavior and can delay testing and the
The main objective was to study the technical infrastructure and benefits
unifying Cl and AT process. During the evaluation, various build tools, Cl
servers and AT processes were studied. They all have trade-off; there is a
saying you get what you pay for; using freeware tools can require many
engineering hours as process have to be written from scratch; whereas
using tools that already provide automated procedures can reduce many
engineering hours or days and the benefits are obvious in the long run.
Therefore, the tools should be carefully chosen and Unified Cl and AT
process should be architected well ahead of time to get the most benefits
out of it.

1. P. Duvall, S. Matyas, and A. Glover. Continuous Integration:
Improving SoftwareQuality and Reducing Risk. Addison-Wssley
Professional, 383 pages. 2007. ISBN:9780321336385
2. Software Engineering Best Practices Continuous Integration
[ resources /articles /continuous-
integration.aspx], 2007.
3. Team Foundation Team Foundation Build
4. Introducing Continuous Integration for C and C++ Software
Development Projects on Linux Platform.
5. M. Fowler. Continuous Integration.
6. Jez Humble, and David Farley. Continuous Delivery: Reliable
Software Releases Through Build, Test, and Depoloyment
Automation. Addison-V\fesley Professional, Page 80-101. 2007.
7. Stephen P. Berczuk and B. Appleton. Software Configuration
Management Patterns: Effective Teamwork, Practical Integration.
Addison-V\fesley Longman Publishing Co., Inc., pages 82-84. 2002.
ISBN: 0201741172.
8. Introducting Visual Source Safe
[ (VS.80).aspx]
9. Team Foundation Server Version Control.
Magazine, 2007
10. Continuous Web Application Build Testing as a Means of Quality
[ 2009.pdf]

11. Continuous Integration Theory
12. Characterization Schema for Selecting Software Testing
13. Testing and Development process.
14. How to Configure and Run Scheduled Tests After Building Your
[]. Visual
Studio 2010.
15. Test Automation Framework for Implementing Continuous
Integration. 2009, Sixth International Conference on Information
Technology: New Generations.
16. Guidelines for smoke testing
17. Why and Howto Build a Continuous Integration for .NET platform
18. Elfriede Dustin, Jeff Rasil, and John Paul Automated Software
Testing: Introduction, Management and Performance, Page 342 -
397. 2007. ISBN:0201432870
19. Extend Team Foundation Server to Enable Continuous Integration.
20. Build andn Deploy Continuously.
[ .aspx]
21. Build Automation
22. Test Automation
23. Software Package

24. Whats New for Application Life Cycle Management in Visual
Studio. 2010,
25. The Standish Group
[http://www1]. April
23rd 2009
26. A. Miller. A Hundred Days of Continuous Integration. Proceedings
of the Agile 2008, EEE Computer Society Washington, DC, USA,
pages: 289-293. 2008. ISBN: 987-0-7695-3321-6.