Applied Logic Engineering, Inc.

Technology and Management Solutions

The Beta Test

Kelly Nehowig

Copyright 2010 Applied Logic Engineering, Inc.

 

Over the course of my career in software and systems development, a common item in most of the organizations I’ve worked in has been the beta test.  It’s common in the fact that most development organizations have one – but what’s not common is the definition of the beta test and how to successfully plan, execute, and review the results from this type of testing to fully benefit the product.

 

The Classic Definition

 

If you were to look up the definition of a beta test (or beta testing), you would typically find something to the effect of “the testing phase in a software development project that immediately precedes the product release”.  The definition would probably also include the fact that beta tests are typically conducted by outside users as opposed to internal quality assurance professionals.

 

While at first glance, this seems to be a solid definition of beta testing, if you study it for a bit you should begin to see that definitions such as this are mostly vague and definitely subject to interpretation.  It is this vagueness that causes problems for the unsuspecting development organization.

 

Development vs. Product Management

 

By nature, different groups involved with the software development process will view the concept of a beta test differently.  For example, the software developers will generally describe the beta test as an opportunity to learn about the quality of the product from a customer’s perspective.   They typically also will view the beta test as an opportunity to find and fix bugs that escape typical bench (internal) testing as the customer uses the product in their “real world” environments.

 

Contrast this with the typical view of beta testing by a Product Manager.  Generally, it has been my experience that these folks view the beta test as a “pre-launch”.  Additionally it is sometimes used as a chance to impress key customers by providing the software to them before it is generally available.  I have also seen beta tests promoted as a “free trial” of the software.

 

The Clash

 

Clearly, with the vastly different points of view and expectations for what the beta test will provide, an organization is headed for problems.  Does this sound familiar:

 

Product Manager : “Why does this software have bugs?  I thought we were ready to ship?”

Product Developer : “Of course we have bugs.  If we were ready to ship, we would be in production”

Product Manager: “I gave this code to my best customer.  Now we have a black eye.”

Product Developer : “ We were supposed to learn how the customer was going to use this code.”

Product Manager : “I have to change my whole roll out strategy because this code needs more testing.”

Product Developer: “It’s too late to react to anything we learn now during this beta test.”

 

And so on.  I can’t tell you how many times I have heard this same discussion within the various organizations I’ve worked in.  Without a plan and a complete understanding of what the team is expecting from the beta testing effort, there is no way that the beta testing can be conducted successfully and little chance that the organization will achieve any value from their beta testing efforts..

 

So what is right?

 

In my opinion, there is no “correct” answer to the question of “How do I conduct a Beta Test?”  The organization needs to spend some quality time discussing the beta test – I recommend the following list of items for consideration:

 

· Decide as an organization what you want from the beta test.  Is it to learn about how the customer uses the product to improve it?  Is it to garner interest in your new product just prior to launch?  What are you hoping to accomplish?

 

· Choose your beta sites wisely.  Only work with customers that understand the concept of a beta and that understand what they are getting.  Set the expectation level before the code is given to them.

 

· Publish a list of known issues with your beta release.  Let your customers know that you are aware that your product has issues – this instills some degree of confidence in case your customer runs into problems.

 

· Follow up with your customer during the course of the beta test.  I have seen many instances where beta software is given out without any subsequent conversations after the customer deploys it.  If you are not going to find out about what the customer thinks about your product, why bother with the beta test?

 

· Listen to what your customers are telling you during your beta test.  Are you ready to release your product?  Did you learn anything new about how your customer uses your product?   What did you hear that you can use to improve your product in the future?

 

We’ve just scratched the surface regarding beta testing – there are a lot of other considerations.  Proper beta testing can add tremendous value to your product and ultimately to your organization.  By spending some time to properly set expectations within your development teams, you can turn this investment in time and effort into a tangible asset.

 

————————————————————————————————————————-- 

Kelly Nehowig is President of Applied Logic Engineering, Inc., a Minneapolis-based consulting company specializing in software and embedded system design.  A 28-year veteran of the software industry, he has worked with companies from startups to large corporate clients.  Kelly holds a MS degree in Software Engineering from the University of St. Thomas and a BS degree in Electronic Engineering from Minnesota State University – Mankato.  He can be reached at kellyn@appliedlogiceng.com.