Applied Logic Engineering, Inc.

Technology and Management Solutions

Improving embedded system stability by fixing fewer bugs

Copyright 2006 Kelly Nehowig, Applied Logic Engineering, Inc.

All rights reserved

 

As embedded systems get more and more complicated, the firmware that drives the system also grows in size and complexity.  Coupled together with a test philosophy of testing to “break the product” rather than testing to “qualify the product”, more and more code gets added to an already complex system to fix problems that a customer may never encounter.  The net effect in this scenario is that system stability actually decreases as this additional code creates more paths of execution that may not be accounted for and/or tested during the qualification cycle, creating additional problems within the product.

 

Typical project

 

In many embedded system projects that deal with typical consumer electronics and many commercial industrial systems, more and more complexity if designed into the system to account for an endless array of features that customers ask for in the system.  Most of this complexity is laid at the feet of the people developing firmware (the embedded software) for the system.  It is usually cheaper to develop these features in firmware rather than change the hardware configuration of the product and in some cases the only way to add the feature is through the firmware.

 

In these types of highly complex firmware systems, defects are inevitable.  There can be literally millions of possible code paths that could be encountered in a complex, real-time system depending on the various inputs that come into play.  Testing all of these possibilities may not be a practical proposition when there is pressure to get the product qualified and on the market.  In addition, the philosophy used to qualify the product may be a problem in and of itself.

 

QA philosophy – “Test to Break” vs. “Test to Qualify”

 

There are some organizations that believe the best way to qualify a firmware-based system is to throw every possible scenario that one can think of at it to see if you can get the system to fail in any manner possible.  This is what is often called the “Test to Break” philosophy.  Not only is this a time consuming effort, but it often results in an inconsistent approach to qualification and a lot of wasted effort in non-value-added work.  This is due to the fact that while a lot of defects may be found, it is not always clear what the impact would be to the customer should he/she encounter the issue.  All defects tend to be lumped into the same category and treated the same – which is that they must be corrected.

 

In contrast, a test philosophy that is designed to qualify the product based on an established set of customer requirements can lead to much more effective test planning and execution.  By focusing on what the customer deems as important, the qualification effort can be focused in the areas that will have high visibility and high payoff.   Defect correction can then be focused in these areas.

 

The Endless Enigma – Break, Fix, Break, Fix

 

As strange as it sounds, requiring the developers to fix problems that are not necessarily high-payoff in the eyes of the customer may result in lowering the overall quality of the firmware product.   Here’s why – as these low-payoff issues are reported, the developer creates a fix that solves the problem.  Typically, the fix can result in the addition of other code paths that now increase the complexity of the overall code, change the timing of the system, or have other undesirable effects in other areas.  In some instances, creating a solution for a low-payoff issue can result in a highly complex solution that affects large portions of the base code that is already in place, increasing the overall risk that major problems may be created elsewhere.

 

You can imagine that this scenario can begin to feed on itself, with more and more issues requiring more and more patches, creating more and more problems that require more and more solutions.  The end result can be a system that is unstable, causing missed deadlines, blown budgets, and potentially failed projects.

 

A Different Approach

 

Qualifying the system based on stated customer requirements avoids the ugly scenario above and almost always results in a better, more stable system.  Using a test approach that focuses on finding out if the product meets (or exceeds) the customer expectations channels the testing resources into high payoff activity – finding and fixing defects that the customer would have most likely experienced provides the high quality feel that the customer will expect.

 

Each defect needs to be evaluated based on its impact to the customer – high visibility items will need to be corrected while low visibility items that have little customer consequence should be carefully evaluated in terms of how they should be handled.  Is there any alternative to adding more code to fix the issue?  Is there an acceptable work-around that does not require a code change?  Is the possibility that the customer would encounter this defect so small that it is dwarfed by the effort and risk we would assume in fixing it?

 

Obviously, in a device that is used in a life-support or life-sustaining situation, the bar may need to be set quite high in terms of allowing any defects at all – in these situations, the organization will need to bear the expense and time it will take to have “zero-defect” software.

 

But I believe that in most other situations, a practical and pragmatic approach that focuses on the customer and that targets correction of the defects that would be objectionable to the end user can yield great results.  Doing so might actually improve the stability of your system and provide an easy to maintain code base well into the future.

————————————————————————————————————————-- 

Kelly Nehowig is President of Applied Logic Engineering, Inc., a Minneapolis-based consulting company specializing in software development products and services.  A 26-year veteran of the software industry, he has worked with companies from startups to large corporate clients.  Kelly holds a MS degree in Software Engineering from the University of St. Thomas and a BS degree in Electronic Engineering from Minnesota State University – Mankato.  He can be reached at kellyn@appliedlogiceng.com.