Ivar Jacobson International (IJI) major contributor to Use Cases, modern business engineering and the Unified Process:
UML (unified modeling language),
Objectory (object-oriented development method)
RUP (Rational Unified Process) and
Aspect-oriented software development
Check out his website now
Techniques and ideas for automating software
Vendors, Give Us Real Scripting Languages,
- Why you'd want a standard scripting language
with your test tool, instead of a vendorscript.
a Late Start on Test Automation, Bret
- What can you do if you don't start test
automation early? (January 2001)
Keys to Test Automation, Bret Pettichord
- How can you get your test automation project
off on the right foot? (December 2000)
UML Testing Framework, Martin Fowler,
Software Development, April 1999
- Uses UML to describe the JUnit test harness.
Steps to Test Automation Success, by
- Important steps for planning and running
a test automation project. Cites pitfalls
of GUI test automation.
Testing, by Brian Marick
- Describes the evolution of test tools.
Test Automation and the Product Life Cycle,
by Dave Kelly
- Advice on test automation focusing on
when to start automating.
in Test Automation, by Elfrieda Dustin
- Test managers often can find themselves
surprised at the difficulty of test automation.
This article collects lessons learned on
various projects after they started using
automated tools. Corrective actions are
Should a Test be Automated?, by Brian
- Considers relevant factors in deciding
which tests to automate.
- Integrated, Effective
Test Design and Automation, by Ed Kit
- Describes "third generation" table-driven
test automation. Includes spreadsheet templates.
to Automate Testing of Graphical User Interfaces,
by Tilo Linz and Matthias Daigl
- Describes important considerations in
the development of test procedure libraries.
Data-Driven Automated Testing, by Keith
- Evaluates two useful architectures for
to Automate Testing: A Cost-Benefit Analysis,
by James Hancock
- Common sense about what tests should be
for Making Test Automation Work, by
- Coding standards and techniques for test
the Maintainability of Automated Test Suites,
by Cem Kaner
- Descriptions of the pitfalls and challenges
of automated testing.
Testing Mistakes, by Brian Marick
- A litany of mistakes made in testing,
including a section on test automation mistakes.
Automation Snake Oil, by James Bach
- An argument against some of the naive
claims made for test automation by someone
who has been burned many times. A newer
version of article is available in PDF.
with Test Automation, by Bret Pettichord
- Description of the development of test
automation that used a table-driven approach.
the Maintainability of Automated Test Suites,
- Overview of the many pitfalls in automated
regression testing. (1997)
Driven Test Automation Framworks, Carl
- Explaining keyword & data driven drivers
and hybrid test automation engines.
Automated Software Testing - A Perspective,
by Kerry Zallar
Other listings of articles on test automation.
on Automated Testing, from Automated
for Automated Testing, from the Chinese
Association for Software Quality
Comparing and selecting various test tools.
a Test Tool, Mark Fewster and Dorothy
Short sharp advice about how to choose a testing
the Right Choice: The Features you Need
in a GUI Test Automation Tool, by Elisabeth
HendricksonAn excellent guide to selecting
a GUI test tool. (PDF format)
Tools, Elizabeth Hendrickson. A tool
selection process to help with selecting
a tool for your team.
Tutorial - Notes on the WinRunner automated
software functionality test
tool from Mercury Interactive -Insights not
obvious from available documentation.
and WinRunner Feature Descriptions, by
Horwath, Green & Lawler, 2000
Report describing the differences between
these two test tools by independent test automation
consultants. Focuses on detailed comparisons
of specific features; avoids any kind of general
Cost Analysis: Benefits and Risks, Cem
- Does testing save your company money or
is it just another expense? Quality cost
analysis describes the metrics you need
to find out.
Depends: Deciding on the Correct Ratio of
Developers to Testers, by Johanna Rothman.
- It would be nice if we could calculate
the size of the required testing staff based
on the number of developers on the team.
This paper discusses the relevant factors
to be considered by discussing several case
Is Software Testing? And Why Is It So Hard?
by James A. Whittaker
- An excellent overview of software testing
with solid examples.
Software Fails, James Whittaker and
- Provides an novel classification for software
Testing Team's Motto, by Brian Marick
- A testing team's mission is to reduce
the uncertainty regarding the quality of
- Software Testing
Best Practices, by Ram Chillarege
- This consise report describes 28 best
practices for software testing listed in
order of implementation.
Testing Resources: A Collaborative Approach,
by Cem Kaner
Test Pattern Language, by David DeLano
and Linda Rising
Testing, James Bach
- Describes several methods for using information
about risk to help direct testing.
Testing, Keith McMahon
and Tracking Defects
Improvement, by Brian Marick
- A list of things to look for when analyzing
a software failure and writing up a defect
Defect Isolation, by Prathibha Tammana
and Danny Faught
- Discusses techniques for isolating, reproducing
and reporting defects.
- Gauging Software
Readiness with Defect Tracking, by Steve
- A brief list of techniques for estimating
to Misuse Code Coverage, by Brian Marick
- Common ways to misuse code coverage tools,
by the author of the GCT coverage tool.
Negligence and Testing Coverage, by
- How many different ways can you assess
testing coverage? What are you legally required
Coverage Analysis, by Steve Cornett
- Good description of the different categories
of code coverage measurements (statement,
- Incomplete Tests
are Worse Than None At All, by Edward
- Essay argues for the necessity of module
and call-pair coverage.
with the Cost of Different Coverage Goals
for Testing, by Brian Marick
- Describes experiences at Motorola of using
code coverage measures of unit tests.
Is Exploratory Testing?, by James Bach
- A nice description of a test approach
that high-performing teams have been using
Functionality and Stability Test Procedure,
by James Bach
- An exploratory testing methodology used
for Windows 2000 application certification.
- Exploratory Modeling,
- Describes the purpose and context for
Exploratory Testing Workshop Report,
by Bret Pettichord.
- Report describing exploratory techniques
reviewed at a workshop.
Want My Test API, Linda Hayes
- Suggests that a bit of added code to software
can greatly facilitate automated testing;
argues against feeble objections to doing
this. (Datamation, Dec 2000)
Tool Criteria Tie Tester's Hands?, Joel
Maslyn & Bret Pettichord
- Notices that a purportedly comprehensive
article on requirements for GUI development
tools made absolutely no mention of testability
concerns. (IEEE Software, Nov 1997)
Test Support Code, by Brian Marick
- A case study for making small changes
to a program to improve its testability.
Software Testability, by Joy Shafer
- Indentifies factors affecting the ability
to test software
Five (Wrong) Reasons You Don't Have Testers,
- A developer argues for hiring more testers,
revealing some common (wrong) attitudes
in the process.
and Developers, by Bret Pettichord and
- Radio interview discussion of the different
ways that testers think. (February 2000)
the Heat, by Bret Pettichord.
- Is it the testers' job to take some of
the heat off of developers? (Stickyminds.com,
- Testers and
Developers Think Differently, by Bret
- Describes different perspectives that
testers and developers bring to the team.
(STQE Magazine, January 2000)
Expect is a Unix tool for testing command
there still major computer system failures
caused by software bugs?
- A long list of software failures from
the Software QA/Test Resource Center.
Great Bugs of History, by Cnet.
- Part of a special series on
Ariane 5, June 1996. The maiden launch
of the Ariane 5 rocket blew up 40 seconds
from liftoff. The rocket and it's four satellites
were uninsured and worth $500 million. The
proximate cause of the crash was an overflow
error due to an attempt to convert a 64 bit
floating point value into a 16 bit integer.
This error occurred in code that was non-functional
after liftoff, when the error occurred. The
Board report provides details regarding
the software failure and the design policies
that lead to it. A note in the Risks Digest
indicates that a complete system test would
have found the problem but was vetoed for
Therac-25, June 1985 - January 1987.
The Therac-25 medical linear accelerator was
responsible for six accidents involving massive
overdoses of radiation, three of which lead
to deaths. The proximate causes of the accidents
were eventually attributed to two separate
and hard-to-reproduce faults caused by race
conditions in the data entry system. IEEE
Computer published a summary report
by Nancy Leveson and Clark Turner. Nancy Leveson
also published a detailed
account as a chapter in her book Safeware:
System Safety and Computers.
Boris Beizer's List. Boris Beizer
has said, "Remember that the most notorious
bugs in the history of software development
were all unit bugs -- bugs that would have
been found by proper unit testing." (Letter
to swtest-discuss, 10 June 1997). Here is
- The Voyager bug (sent the probe into the
- The AT&T bug that took out 1/3 of
- The DCS bug that took out the other 1/3
a few months later.
- The Intel Pentium chip bug (it was software,
- The Ariane V bug.
In a later letter, he agreed that the Therac
25 bug was also notorious, but not really
a unit bug.
John Shore collected
a number of reports, often apocryphal, of
In the News
Software defects and testing discussed
in the news.
tech's missionaries of sloppiness, by
Cheryll Aimee Barron, Salon.com, Dec 2000.
- Discusses how buggy software is the consequence
of business strategies that focus on innovation
over reliability. Good quotes.
Call is Important to Us, Joe Salkowski,
- Software nowadays is so unreliable that
internet companies can avoid potentially
embarrassing consequences of call avoidance
strategies by claiming to be victims of
"insignificant" software bugs.
Testing Gets New Respect, Billie Shea,
Information Week, July 3, 2000
- Suggests that internet companies have
recently been more successful at justifying
the expense of automated testing.
and Quality Assurance
Notes on the difference between software
testing and software quality assurance.
is Software Quality Assurance and What is
Software Testing?, from the Software
QA/Test Resource Center
in Software Testing
a Career Path for Software Testers,
by Elaine J. Weyuker et al.
- Defines steps AT&T is taking to define
a career path for testers and encourage
certification. (requires subscription)
Tester Salaries, by the Software Testing
Ideas on how to organize software development.
Immaturity of the CMM, by James Bach
- This article criticizes the Capability
Maturity Model as a useful methodology for
organizing software development.
Discussing testing and bugs.
- Lively discussions of automated testing
and other testing topics.
- This newsletter tracks bugs in commercial
PC software. Can you find problems like
these before your product ships?
Testing and Quality Engineering, formerly
Software QA Magazine.
- A magazine for the professional tester.
The articles are written by experienced
professionals. I get all kinds of ideas
The moderated newsgroup comp.risks
contains concise and technically precise
descriptions of software failures, especially
when loss of life is involved. This is
an excellent place to learn about the
kinds of systemic problems that plague
The newsgroup comp.software.testing has lively debates
on topics such as the proper relationship
between testers and developers. This is
a place where testers and testing managers
go to for advice when they get squeezed
particularly hard. Many experienced and
helpful software testers frequent this
newsgroup. The group's list of frequently asked questions contains
much useful information.
- SQA Suite
Users Mailing List
Sites with essays, articles or lists of
- A portal for information about software
engineering and quality assurance
- QA City
- Hung Nguyen's collection of links and
articles. Like the Software Testing Hotlist,
only easier on the eyes.
- Herb Isenberg's site contains several
papers on automated testing.
- This website is a case study in web software
testing. It contains a sample test plan
and requirements document.
- A weblog on software quality assurance
by Stan Taylor.
- Website features weekly columns, archives
of papers, and other current information
on software testing and quality assurance.
Kinetics Software Testing Conference
- Online conference featuring online presentations
and links to resources.
- Several papers on testing and test management.
- A listing of software testing papers recently
published on the web. A primary reference
used by this hotlist.
- A quarterly newsletter covering software
testing and other software development topics.
- Eugene Vinsky's grab bag of information
about software testing
- Includes index
to articles on automated testing and other
- Features papers on test automation and
other testing topics.
- Includes FAQ's on software testing as
well as articles on testing, Unix and Perl.
James Bach's consulting company
- Includes many of Bach's articles on software testing
Research, Inc., a test tool vendor.
- Includes Quality Techniques
Newsletter (formerly Testing Techniques
Newsletter) and Software Quality HotList.
Software, a test tool vendor.
a test tool vendor that gobbled up Pure,
Atria and MS test.
- Includes dozens of technical papers, including
on Time, by Aki Fujimura, an interesting
article on scheduling development and testing.
Dimensions, consulting and contracting
- Includes a collections of articles from
their newsletter, The Testers Network
- Includes many of his papers and essays.
- RBSC Corporation.,
consulting, training and contracting.
- Includes lots of information about object-oriented
testing: articles by Bob Binder, a
and a list of resources.
Inc. (formerly Reliable Software Technologies),
consulting and research in software risk
- Includes a few hundred papers
documenting original research and the Software Risk Management Hotlist.
- Cem Kaner,
Testing Consultant, Attorney at Law, and
Professor of Computer Science
- Includes papers on software
testing and the
law of software quality.
Testing Stuff, by Kerry Zallar
- This web site also includes links to other
- Includes papers on software testing and
- Includes a couple articles.
- Includes testing resources, available
to registered visitors.
catalog of software quality books, David
also known as SVP.
- Includes sample
test code for QA Partner.
- Includes a salary
Thoughts, by Howard Fear
- Short essays on software engineering and
and short essays ("ruminations") on
software quality and testing by Elisabeth
- The Center
for Software Engineering Research at Florida
Robinson's Model-Based Testing home page
- Contains a comprehensive index to papers
on using state models to generate software
- Includes papers on model-based testing
and international testing.
Association for Software Quality (CASQ)
- A comprehensive web site. Includes excellent
linked bibliographies. Automated Software Testing.
providing test automation services
- Includes Software
providing test automation services
- Includes forums and downloads for popular
test tools. Also contains a job board.
Management Services, software process
improvement and quality assurance consulting.
- Includes a hotlist
Other Commercial Sites
Note: The Software Testing Hotlist no
longer lists commercial websites unless they
contain articles or other information of interest
to the industry.
I most frequently find myself recommending
the following books to people who are new
to software testing.
Testing Computer Software, by Kaner, Falk
If you read only one book on software testing,
read this book. Many books will tell you how
to test when you have enough time and cooperation.
This book tells you what to do when the schedule
is tight, the specification is missing, and
the developers are tired of your focus on
problems. It has sound advice and is a pleasure
to read. I keep coming back to it. Feel like
you have an impossible job? Read this book.
FastCounter sponsored by Microsoft
I would like to thank Bret
Pettichord for some of the above great links.
Bret Pettichord is a speaker, writer and consultant
with Pettichord Consulting
, specializing in software
testing and test automation. Familiar
with many test technologies, he develops test
automation strategies, tools and test suites.
He specializes in GUI test automation, scripting
languages and test methodology.
Please send updates and comments to firstname.lastname@example.org
All rights reserved | www.testinghotlist.com
| Software Testing Hotlist, Copyright ©
1995 - 2001 Bret Pettichord