Introduction

The paper being reviewed was written to support the thesis that the Software Engineering Institute's Capability Maturity Model (SEI CMM) is a collection of software engineering practices that are organized according to a simple model based on process evolution that are not completely effective in every software organization. The author makes his case by describing six areas in which he has general problems with the CMM. This is followed by a section outlining the author’s claim that a “level 1” organization is completely misunderstood by the SEI and that effective software can be (and actually is) created by many level 1 organizations. Finally, the author briefly describes an alternative to CMM that can be used as a framework for process improvement.

For the most part, I believe that the author has accurately critiqued the CMM and, from my experience, I would agree with the problems he discusses. In my mind, the CMM is a good theoretical guideline for establishing a basic understanding of the characteristics of a good software development organization, but by stringently following its processes and procedures to the letter, an organization is not guaranteed to be successful. The CMM does not deal effectively with innovation issues and people issues. It also does not reconcile the fact that many successful software organizations can claim various attributes associated with four (or sometimes all five) of the CMM levels but, due to the rules established by the CMM, would officially be designated a Level 1 organization, which unfairly describes the organization’s capabilities.

Summary of the Reviewed Article

The article is broken into several sections that describe the CMM in general, the problems that the author has with the CMM, an alternative to the CMM, and a postscript that was added to the original paper in February of 1999.

Brief description of the CMM

The author describes the CMM as a group of key practices that are divided into five levels representing various maturity levels that organizations should go through on their way to becoming “mature”.

The author lists the CMM levels as follows:

1. Initial (chaotic, ad hoc, heroic)

2. Repeatable (project management, process discipline)

3. Defined (institutionalized)

4. Managed (quantified)

5. Optimizing (process improvement)

The author states that the original intent of the CMM was that of a tool to evaluate the ability of government contractors to perform a contracted software project. His primary concern is that many tout the CMM as a general model for process improvement and he believes that in this area, it has many weaknesses.

General Problems with the CMM

The author describes six basic problem areas that he has identified with the CMM:

· The CMM has no formal theoretical basis and in fact is based on the experience “of very knowledgeable people”. Because of this lack of theoretical proof, any other model based on experiences of other experts would have equal veracity.

· The CMM does not have good empirical support and this same empirical support could also be construed to support other models. Without a comparison of alternative process models under a controlled study, an empirical case cannot be built to substantiate the SEI’s claims regarding the CMM. Primarily, the model is based on the experiences of large government contractors and of Watts Humprey’s own experience in the mainframe world. It does not represent the successful experiences of many shrink-wrap companies that are judged to be a “level 1” organization by the CMM.

· The CMM ignores the importance of people involved with the software process by assuming that processes can somehow render individual excellence less important. In order for this to be the case, problem-solving tasks would somehow have to be included in the process itself, which the CMM does not begin to address.

· The CMM reveres the institutionalization of process for its own sake. This guarantees nothing and in some cases, the institutionalization of processes may lead to oversimplified public processes, ignoring the actual successful practice of the organization.

· The CMM does not effectively describe any information on process dynamics, which confuses the study of the relationships between practices and levels within the CMM. The CMM does not perceive or adapt to the conditions of the client organization. Arguably, most and perhaps all of the key practices of the CMM at its various levels could be performed usefully at level 1, depending on the particular dynamics of an organization. Instead of modeling these process dynamics, the CMM merely stratifies them.

· The CMM encourages the achievement of a higher maturity level in some cases by displacing the true mission, which is improving the process and overall software quality. This may effectively “blind” an organization to the most effective use of its resources.

CMM’s misunderstanding of “level 1” organizations

The author’s most compelling argument against the CMM is the many successful software companies that, according to the CMM, should not exist. Many software companies that provide “shrink wrap” software such as Microsoft, Symantec, and Lotus would definitely be classified by the CMM as level 1 companies. In these companies, innovation reigns supreme, and it is from the perspective of the innovator that the CMM seems lost.

The author claims that innovation per se does not appear in the CMM at all, and is only suggested by level 5. Preoccupied with predictability, the CMM is ignorant of the dynamics of innovation. In fact, where innovators advise companies to be flexible, to push authority down into the organization, and to recommend constant constructive innovation, the CMM mistakes all of these attributes to the chaos that it represents in level 1 companies. Because the CMM is distrustful of personal contributions, ignorant of the environment needed to nurture innovative thinking, and content to bury organizations under an ineffective superstructure, achieving level 2 on the CMM scale may actually destroy the very thing that caused the company to be successful in the first place.

The author discusses the issue of “heroism”, defined as the individual effort beyond the call of duty to make a project successful. The SEI regards heroism as a negative and as an unsustainable sacrifice on people that have special gifts. It considers heroism the sole reason that Level 1 companies can survive. The author claims a different definition for heroism – taking initiative to solve ambiguous problems. He claims that this is a definable and teachable set of behaviors that enhance creativity, which leads to personal mastery of the subject matter. In his opinion, it is not a negative, but it is a requirement of most successful organizations.

An Alternative to CMM

As an alternative to the CMM, the author introduces the idea of a framework based on heuristics for conducting successful projects. The key to this model is that it is an aid for judgement, not a prescription for institutional formalisms. In this model, maturity means recognizing problems through the analysis of experience and the use of metrics and to solve them through selective definition and deployment of processes.

This process model consists of a seven-dimensional framework for analyzing problems and identifying the correct processes. These dimensions include: business factors, market factors, project deliverables, four primary processes (commitment, planning, implementation, and convergence), teams, project infrastructure, and milestones. This framework connects to a set of processes that are repeatable for performing certain common tasks.

Postscript 02/99

In an addendum to the original thesis, the author comments that not much has changed in his opinion on the CMM in the five years since originally writing his paper. Some software companies are successfully using the CMM in their organizations, but many, including most of the newer Internet-based software companies, are not using the CMM.

The author does comment on one shift in his thinking – that is the fact that he has become more comfortable with the idea of using CMM as a basic philosophy and not as an issues list. He now believes that using CMM to identify a list of issues worth addressing in the course of overall software improvement may be useful, but that it should not be adapted as a philosophy for good software engineering.

Critique

For the most part, I agree with the author’s assessment of the CMM. Some of his arguments seem weaker than others, but I believe they are valid.

Due to the fact that the CMM has no theoretical basis and that it has no empirical proof, it loses value from an academic point of view. Although this (in my opinion) is one of the author’s weaker arguments, it is important from the aspect of substantiation of the claims made by SEI. Without the theoretical proof and the lack of empirical support based on comparison of alternative models under a controlled study, the SEI’s case for promoting CMM as the optimal model for software development is weakened.

The author’s implication that the CMM institutionalizes process for its own sake without regard to current practices is an accurate assessment in my view. I have seen organizations that implement policies without regard to current organizational practices (some of which are quite successful). The result is a confused development group that gets a series of mixed messages from “management” which do not necessarily improve the development process.

Another key fault in the CMM described by the author is the overriding pressure to move to the next maturity level, sometimes by ignoring the true mission, which is the quality of the software product. The factors important in moving up to the next level may or may not necessarily benefit the organization and its products because all subjectivity is removed. What benefits one organization may not have the same effect in another organization.

The author’s claims about heroism are interesting, but I differ slightly with the conclusions that he draws. I agree that heroism, as defined by taking the initiative to solve ambiguous problems, is critical to the success of an organization. However, in my experience, heroism is a trait that is difficult to teach. I believe it is inherent within the individual for the most part and, without a good process model, can be abused by the organization in order to accomplish its goals.

Probably one of the more important points that the author touches on is the CMM’s implied claim of the importance of process over people. It has been my experience that processes are not direct substitutes for the quality of the development team personnel. In other words, the right organizational processes can improve the output of a group of talented software developers, but they do not create one. By ignoring this critical item, the CMM loses credibility with anyone experienced with a wide range of development teams.

A striking example that is prevalent throughout the author’s thesis is the number of software companies that probably exist at CMM Level 1, but that are incredibly successful. Microsoft is a prime example – although they do not model their organization in a manner that the SEI considers to be “mature”, their products mostly meet or exceed the customer’s needs and this creates a very successful company.

In considering the alternatives to the CMM, I believe that the author is correct in his assertion that a model based on past experience and the use of metrics is probably more effective in practice as compared to the CMM. The implementation of such a model is based on selective definition of problems and the selective deployment of specific processes. However, I also believe that such a model is very difficult to define and is very much dependant on individual organizations and the particular environment that each exists within. I believe that it would be a difficult proposition to create such a model that would apply to all organizations.

I agree with the author’s postscript in terms of the fact that I am comfortable with using the CMM as a basic philosophy and to study the various relationships of the characteristics established by the various functional levels. Most of these characteristics describe good traits that can benefit any software organization. However, adapting the stringent practices of the CMM and focusing on the goals of achieving each maturity level may benefit large corporate software organizations, but I believe they would not necessarily add value to small and medium sized organizations due to the cost/benefit ratio that the organization would incur to implement the model.

Many software companies (especially startups) are successful because of their ability to innovate and because of the entrepreneurial spirit that they possess. As the author mentions, these factors are not considered in the CMM. The danger, therefore, is that by implementing the CMM process within such a company, the company may not benefit and, in some cases, may be harmed by adapting its stringent polices.

In summary, I believe that Mr. Bach’s thesis defines many shortcomings of the CMM, but offers few solutions for a better model or set of practices to use instead of the CMM. Ultimately, I believe that an organization can benefit by studying the CMM and selectively implementing the “best practices” that will benefit their respective organization.