THIS FILE IS MADE AVAILABLE TH RDLIGH THE DEGLASSIFIG ATIGN AND RES EARGH THE BLAEH IS THE LARGEST FREEDGM DF AGTIGGVERNHENT REGDRD ELEARING IN THE THE RESEARCH HERE ARE FDR THE DEGLASSIFIGATIDN GF THDUSANDS DGGUMENTS THRGUGHDUT THE U FDRWARD THIS DDGUMENT TDUR FRIENDS EILJT PLEASE THIS IDENTIFTING IMAGE AT THE TDP IDF THE SD GTHERS MIDRE1 US Army Information Systems Engineering Command Fort Huachuca AZ 85 61 3-5300 V21 Faye 02 U S ARMY INSTITUTE FOR RESEARCH IN MANAGEMENT INFORMATION COMMUNICATIONS AND COMPUTER SCIENCES Software Quality and Testing What Can Learn from Commercial Practices 12 31 August 1992 AIRMICS 11s O Keefe Building 0W Georgia Institute of Technology Atlanta GA 30332 0800 Software Quality and Testing What Can Learn from Commercial Practices Prepared for the Director of Defense Information and the Deputy Undersecretary of the Afmy Operations Research 31 August 1992 by LTC Mark R Kindl Army Institute for Research in Management Information Communications and Computer Sciences AIRMICS DISCLAIMER This research was performed by the Army Institute for Research in Management Information Communications and Computer Sciences AIRMICS the research organization of the US Anny Information Systems Engineering Command USAISEC The ndingsof this technical report are not to be construed as an of cial Department of Defense or Department of the Army position unless so designated by other authorized documents The use of trade names in this document does not constitute an endorsement or approval for the use of such commercial hardware or software This document may not be cited for the purpose of advertisement ACKNOWLEDGMENTS I wish to express my appreciation to the following individuals for their valuable suggestions and assistance during the research for and preparation of this report Mr John Mitchell Director AIRMICS Mr Glenn Racine Chief CISD AIRMICS LTC David S Stevens Computer Scientist CISD AIRMICS Mr Earl Lee Senior Test Engineer IBM Federal Systems Company Houston Mr Andrew Chruscicki Air Force Rome Laboratory Mr Ray Paul Army Operational 'Ibst and Evaluation Command EXECUTIVE SUMMARY Historically software testing was the process of exercising a computer program to verify that it performed as required and expected The strategic goal of software testing was to demonstrate correctness and quality We now know that this view of testing is not correct Tbsting cannot produce quality software nor can it con rm correctness Tbsting can only verify the presence not the absence of software defects Yet the dif culty of testing and the impracticality of correctness proof have often driven us to the dangerous perception that if testing does not nd defects then the software is correct In the early 19805 software testing concepts were neither well-developed nor well-understood p 39 While testing techniques were many supporting theories were few Even worse little or no guidance existed for making intelligent choices of technique s vol 1 p 24 During the 1980s Department of Defense DOD and industry gathered much empirical evidence to justify many software quality and software development techniques As a result the scope of software testing has evolved into an integrated set of software quality activities that cover the entire life cycle Software tests now take different forms and apply to all software products including requirements design documentation 35 Plans and code Each test contributes to a total quality assurance plan Quality assurance focuses on the front of the development process and emphasizes defect prevention over detection A cost effective prevention program rst requires accurate error detection and analysis to understand where MW and why defects are inserted Though testing cannot prevent errors it is the most important method for producing error data necessary to guide process improvement However the following extract from the 1992 Software Maintenance Technology Reference Guide summarizes the dif culty of testing Software implementation is a cozy bon re wann bright a bustle of comforting concrete activity But beyond the ames is an immense zone of Testing is the exploration of this darkness The conclusions of this report are not revolutionary but they may be surprising knows how to produce quality software There are a few contractors who produce quality software though not necessarily for using many of the policies published in Standards These documents describe the need to focus on quality activities early in the software life cycle Developers and veri ers should identify and remove errors during requirements definition and design so that they do not enter the code where nding and fixing defects is extremely expensive For management information and command control systems this is a particularly dif cult task because most requirements for these systems are based upon human demands which are highly subjective easily in uenced and thus very dynamic and dif cult to state precisely Although not in common practice yet for software development quality control methods adapted from the factory paradigm may have the greatest potential to move software production from an art to a true engineering discipline 7 8 Both the products and the development process should be subjected to these procedures To engineer quality into the software products requires that we inspect test and remove defects from requirements design documentation code test plans and tests Quality control of the development process requires that we establish standard procedures to measure defects determine their root causes and take action to prevent future insertion Such a process is self correcting and future measurements will provide convincing evidence of cost effective improvement In summary software quality improvement is evolutionary and requires that we control coordinate and feedback into three concurrent processes the software development process the error detection process testing life qrcle and the quality improvement process Figure 1 depicts the relationships between the processes in the software life cycle Design 4 Code D E Y E L O E M E Design 1 Developer Test Plan Tests Testing Independent 1 TESTING Testing Forms Formal Fennel Inspections inspections Inspections Ieilvere - Measurement and Causal Aneiysls of Defects and Correction of Processes i i PROCESS IMPROVEMENT Operation aintenane TESTING PROCESS IMPROVEMENT Flgure 1 Software Quallty Control ii 5 A few corporate organizations have successfully implemented these procedures 10 11 12 13 14 15 16 17 The common key element in these successes is organization-wide commitment to a quality attitude and disciplined life cycle procedures However within the perception persists that such practices are not cost effective Simply mandating their use has not been adequate Even if enforced the techniques can be undermined and neither software quality nor the perceptions will change DOB must ump start these procedures with an active campaign to establish and nurture a quality attitude both internally and in its contractors IBM Federal Systems Company FSC Houston took 15 years to re ne their processes into producing high quality software But it also believes that other organizations can learn from their procedures without investing such time What can make this possible is the fact that their procedures already correlate well with written policies the policies of other corporate software developers and the recommendations of academia The difference is that IBM has disciplined itself to practice them should take advantage of this knowledge and experience now and adapt its own practices accordingly In order to initialize the production of higher quality software within DOD we recommend the following actions 1 Actively motivate a software quality attitude in and government contractors through management commitment incentives for process improvement and quality and technical training Make quality as visible as the software product its cost and its schedule For every change to software product cost or schedule project managers must give equal consideration to the corresponding cost of and effect on quality 2 Motivate and make standard the use of formal inspections for all software products requirements documentation design code test plans tests 3 Users developers and veri ers should jointly analyze requirements to ensure they are clearly documented implementable and testable The formal analysis of quality objectives should be an integral part of this effort A joint relationship should continue throughout the software life cycle Eventually this effort should result in documentation or data that directly cross references test cases to requirements and code At the same time both developer and veri er should independently plan design develop inspect execute and analyze the results of software tests 4 Measure and document errors throughout the life cycle Establish a formal defect prevention program which empowers developers and veri ers to analyze the causes of error and enact improvements to their own local development processes that will prevent future error insertion and enhance detection processes 5 Evolve Computer-Assisted Software Engineering CASE tools to support all aspects of software develoPment testing and maintenance should permit organizations to introduce standard CASE tools gradually in piece meal fashion An organization should purchase train and employ only those tools for which its sub-processes are de ned in writing Start small and allow adequate time to learn and gain experience Purchase and integrate a new tool only when users understand the manual procedure the tool will automate and the bene t of automating it With regard to software testing in we can summarize our conclusions in two fundamental ideas First knows how to produce quality software at low cost This is because organizations such as STEP Army STEP and Software Engineering Institute have already researched and documented policies for A few commercial software developers practice many of the policies and directives now and produce quality software for example IBM FSC Houston Second quality cannot be tested into software Only a well-de ned well disciplined process with a continuous improvement cycle can ensure software quality However testing cannotbe underestimated Systematic testing activities that detect error earliest in the life cycle are necessary to drive process improvement and optimize the development of quality software Such testing methods as formal inspection nd defects early This enables cost effective error resolution identi cation and removal of defect causes and thus prevention of future defect insertion If practiced with discipline such methods can evolve a self correcting software development process that is stable modeled measured and therefore predictable This development process engineers quality software faster at reduced cost This report discusses software testing practices and more speci cally why and how IBM's practices achieve high quality Along the way we will relate policies instructions and guidance to practices We will also discuss current initiatives within which will impact software testing and quality Finally we present our speci c recommendations for software testing and quality within We believe that these recommendations have the potential for immediate value to iv Administration NASA currently approaches 01 errors per thousand lines of source code This gure is well below the US industry average Surprisingly enough there is nothing new or revolutionary about the way that IBM FSC Houston develops or tests its software Many of the same methods are used at IBM FSD Rockville as well as at other large software development corporations IBM Houston practices basic software life cycle processes most of which have been known for at least a decade These include requirements analysis formal inspections con guration control quality control developmental testing and independent veri cation and validation So why does IBM FSC Houston produce such high quality software The difference results from a strong attitude toward quality the disciplined practice of its basic processes and a commitment to process improvement From manager to programmer the entire organization strives to achieve zero defects through prevention To the classical waterfall model of software development this organization applies basic testing processes designed to identify errors as early as possible Once identi ed defects in the software products are corrected However continuous measurement causal analysis and subsequent cause removal improves the development process and prevents future error insertion Their techniques are very closely related to concepts of Total Quality Management TQM 22 and the softwarerfactory paradigm Recent empirical evidence in other organizations 10 11 12 13 14 15 16 17 con rms the effectiveness of the software quality techniques practiced by IBM FSC Houston Later we will discuss the techniques in more detail and relate them to the environment Critics maintain that because of fundamental differences software techniques used to develop and test weapons systems cannot be used efficiently or effectively to produce information systems and the reverse IBM also believed this until the late 19803 However on the basis of its own success in developing high quality ight control software IBM FSC Houston began to develop its ground sistem software essentially MIS using the same methods The empirical evidence speaks for itself Error rates in delivered ground software decreased dramatically to the same levels achieved in ight software Furthermore this similarity in quality occurs in spite of the more extensive testing that safety critical ight software undergoes The quality achieved with early error detection and prevention techniques is largely independent of the type of software being developed The development of large Management Information Systems MIS and Command Control Systems C2 software is costly and time-consuming Much of this cost and time can be attributed to the identi cation and repair of errors '8 Closely related defect repair is maintenance re work necessitated by changing requirements or latent defects In such cases software in operation must be modi ed to re ect new requirements or requirements that were initially ill-de ned To reduce the cost and time to produce and maintain software must avoid passing immature software to the testing phase or worse to the customer Testing is one of the most important quality tools Properly applied testing helps to identify one of the greatest impediments to quality -- error However if quality software is the ultimate goal then any discussion of effective software testing must address the entire software life cycle This is because testing alone can neither produce nor guarantee software quality Tbsting only nds faults it cannot demonstrate in a practical sense that faults do not exist What we have - traditionally thought of as software testing tends to be labornintensive costly and ineffective This view of testing is a paradox 'Ibsting is a process that instills con dence in software by cleverly plotting to undermine that con dence Nevertheless there is empirical evidence to suggest that old concepts of quality control can counter this view By expanding the concepts and practices ofsoftware testing to all areas of the life cycle we can optimize test efforts increase its effectiveness and signi cantly reduce its cost The result will be the delivery of higher quality software on schedule for less money One reason for general dif culty in testing software appears to stem from differences of testing models conceived in the minds of users managers developers and testers Without common accepted concepts all vital communication in large software development projects will amount to assumptions and guesswork in the best case Therefore in order to clarify further discussion we summarize several fundamental de nitions from the GIossary of Software Engineering Terminologr considered industry standards error a discrepancy in implementing requirements or design speci cations An error may manifest itself as incorrect or undesired results fault a defect in code that has the potential to cause possibly visible incorrect or unexpected results Faults are also known as bugs Faults in code usually result from errors debuging - the process of locating analyzing and correcting suspected faults failure - the execution of software fault or defect that manifests itself as incorrect or undesired results testing - the-process of exercising or evaluating a system or system components by manual or automated means to verify that it satis es speci ed requirements or to identify differences between expected and actual results dynamic analysis testing by executing code static analysis - the process of evaluating a computer program without executing it e g review desk check inspection walla-through correctness - use of this term usually means the composite extent to which 1 design and code are free from faults 2 software meets speci ed requirements 3 software meets user expectations veri cation 1 the process of determining whether or not the products of a given phase of the software development cycle ful ll the requirements established during the previous phase 2 formal proof of program correctness 3 the act of reviewing inspecting testing checking auditing or otherwise establishing and documenting whether or not items processes services or documents conform to speci ed requirements validation the process of evaluating software at the end of the software development process to ensure compliance with software requirements Several of these terms have subtle relationships and differences in meaning It is important to recognize that errors relate to early phases of the life cycle requirements de nition and design speci cation An error in requirements or design causes the insertion of a fault into code However a fault may not be visible during code execution whether during testing or operation If a fault is executed then it may result in a visible failure but not necessarily Programmers debug code to correct faults by using visible failures as a guide However the lack of failures cannot guarantee the absence of faults Even if the fault executes it may not be visible as output Furthermore fault correction does not necessarily imply that the error s that induced the fault has been corrected From the above discussions one should conclude that effective software testing cannot be limited to code It must address all products of the software life cycle The de nition implies that testing demonstrates 1 that the code satis es a speci c requirement 2 whether faults exist in the code However these are only the ideal goals of testing In practice they cannot be achieved in the absolute sense Furthermore these goals are not necessarily mutually exclusive Code can often satisfy user requirements as de ned and still contain faults The de nition can easily convey the erroneous perception that testing can verify correctness Correctness is a major factor in software quality and by de nition relates to code requirements and user expectations But testing code only veri es the presence not absence of faults in code and cannot verify correctness or ensure quality Tbsting code can verify the presence of requirements only if they are de ned precisely as test cases Developers and users do not normally view requirements in this manner E ective testing identi es errors before they become code faults and therefore must apply to the entire life cycle Since the 1980s the scope of software testing has expanded to cover the entire life cycle Empirical data from software projects in the last decade provides convincing evidence that testing in this context can signi cantly improve software quality In its current model software testing has a variety of forms that apply to a range of products including requirements design speci cations documentation test plans as well as code These techniques must be coordinated disciplined and integrated throughout the entire life cycle to effectively impact on quality We will make the case that to have maximum positive effect on a large software project testers must participate in develoPment and gain a broad understanding of the software requirements and design Therefore in the remainder of this report we will refer to software testing professionals as veri ers to highlight their expanded roles consistent with the de nitions above 3 Involve Veri ers in the Entire Development Life Cycle STEP reports 25 vol 3 indicate that the most successful software projects established independent test and evaluation organizations Sometimes these organizations were separate independent contractors Other times they were sub-organizations under the prime contractor but having an independent chain of command This is an effective strategy which is in common practice to help ensure objective impartial and unbiased testing directives provide for such independent testing activities Each military service has its own independent test and evaluation organization General IBM testing policies also de ne the need for such Both IBM FSD Rockville and IBM FSC Houston have independent veri cation organizations within their respective projects The advantages of independent testing should not overshadow the need for communication and coordination between veri er user and developer While veri ers should plan design implement and analyze software tests independently they should not do so in isolation Veri ers who must design and perform operational tests cannot gain adequate understanding of the requirements of a large software system by studying the system documentation after development They must take an active role in the requirements de nition and system design phases The biggest mistakes in software are almost always made early during requirements de nition and design Empiriml evidence indicates that the cost of xing errors versus time in development is an exponentially rising curve IBM FSC Houston data shows that average error repair costs increase 10 times in each successive phase of the life cycle As a result of such data in the mid 19805 IBM FSC Houston decided to move 30% of its resources used in testing of code to assist in the requirements de nition and design phases This decision resulted in a signi cant increase in software quality Furthermore this shift resulted in a net decrease in total cost Shell Research reported similar results The conclusion is obvious veri ers should participate in requirements analysis de nition and design It is far cheaper to nd and x errors before they become faults in the code STEP identi ed the need for early test and evaluation activities in software development It also identi ed the need for integration of independent veri cation organizations One result of the STEP recommendations is that Instruction 5000 2 states Both developmental and operational testers shall be involved Army STEP has further de ned procedures for close coordination between veri ers users and developers The new DA Pam 73-1 Volume 6 So ware Test and Evaluation Guidelines 27 describes how software testing and evaluation activities relate to each phase of the software life cycle The adoption of all or portions of DA Pam 73-1 into D'oD_instructions and directives could reinforce and more precisely de ne the communications that should occur among veri ers developers and the customer The Air Force Standard Systems Center SSC at Gunter Air Force Base in Alabama takes customer involvement seriously Some of their standard information systems development work is contracted to local software rms However as dictated by the terms of the contracts Government personnel are participating members of contractor developer and veri er teams While this has caused a few unusual and dif cult situations the overall strategy appears to work SSC anticipates that the result of these contracts will be well de ned requirements and design better quality software and systems that are more easily maintained after acceptance This SSC practice could be a model for contracted software development Standard 2167A 19 clearly requires traceable and testable requirements 'Itaceable requirements are de ned and formulated such that direct cross-referencing exists among requirements design speci cations code and test cases 'Il'aceability also implies that each requirement can be implemented in both design and code A requirement is testable if and only if it is written so that developers and veri ers can prepare Speci c test cases that am clearly con rm satisfaction of the speci c requirement At both IBM FSD Rockville and IBM FSC Houston developers and veri ers work together to ensure that requirements are both traceable and testable when de ned In fact test engineers for the Advanced Automation System AAS project built and use a software tool which automatically maintains the relationships between requirements and test cases This tool is essentially a specialized database management wstem that assists developers and veri ers in test management and con guration control While such tools help to manage the relationships and maintain the consistency of the software products once developed they cannot replace the dif th work required beforehand to ensure traceability and testability As practiced by IBM success in this work is a direct result of close communications and coordination among the users developers and veri ers IBM describes the relationship between its developers and veri ers as friendlradversarial This means that both groups work together with the customer toward a mutual understanding of the product requirements and design and the early identi cation of errors Finding and preventing errors are considered primary job responsibilities for both developers and veri ers At the same time each group independently designs its reSpective test plans and cases for later veri cation and validation 4 Formally Inspect All Software Products Empirical evidence indicates that dynamic software testing execution of code alone cannot ensure quality and is not cost effective Yet dynamic testing is essential to con rm software quality Dynamic testing should be planned at the same time that requirements are analyzed and de ned and then executed systematically as planned However if quality is the objective then veri cation cannot wait for code Early detection techniques must be applied extensively to all software products so that dynamic testing can be a cost-ef cient and graceful confirmation of functionality and quality The analyses necessary to de ne implementable traceable and testable requirements helps to avoid errors However one of the most effective early detection methods is the formal inspection 29 30 also referred to as the Pagan Inspection Developed by Dr Michael Fagan in 1976 the formal inspection Ila is a general purpose veri cation method It is product independent and can be employed to identify errors in requirements design documentation test plans or code Thus it has the potential to identify and permit removal of errors very early during the software life cycle In fact IBM FSC Houston reports that their application of formal inspections accounts for the identi cation of 80% of the errors in the US Space Shuttle ight software Even so the acceptance of formal inspection into general practical use has been slow for several possible reasons The technique has a reputation for being low-tech It requires a fair amount of intensive detailed work although it does appear that automated tools could enhance some of its procedures The availability of good empirical data verifying its cost-effectiveness has not been available until the last several years Even now published results are not prevalent At least one software corporation considers its use of formal inspection procedures as a competitive advantage and thus declined to divulge their procedures A formal inSpection is essentially a testing technique in which a software product is formally examined by a team of experts These experts include the author of the product and several of his her peers Depending upon the product the team may also include a customer representative and a veri er The primary objective of the team is to nd as many errors as possible In such a situation nding errors must be considered in a positive sense i e the team intensively scrutinizes the product code or documentation not the author s abilities The team s responsibility is to help the author s by identifying mistakes thus preventing their entry into the next phase of the life cycle This is done by paraphrasing lines or portions of the software product at a higher level of abstraction or from a different perspective such as from the veri er s view The error detection ef ciency of this process results from its formality and intensity The procedures are de ned and repeatable Standard checklists ensure that common mistakes are not overlooked As practiced by IBM FSC Houston the formal inspection is the cornerstone of software veri cation and process improvement All software products must submit to and pass a formal inspection prior to acceptance into con guration control or submission for execution testing Each product is examined by an inspection team tailored to that product For example the inspection team for a requirements de nition document will include the customer a requirements analyst a veri er a programmer as well as the author The inspection team for the independent veri er s test plan will include the customer 3 requirements analyst and several veri ers The inSpection team for the developer s test plans will include several requirements and programmer s These particular examples illustrate how the tailoring of inspection teams establishes a cooperative yet independent relationship between developer and veri er Each inspection 10 team includes a senior peer who acts as the moderator He must foster cooperation and focus on the objective to nd errors Management strongly supports formal inspections but does not participate in them This ensures that inspection results are used to rate the effectiveness of the technique and not the performance of individuals Inspection teams record errors identi ed and subsequently require authors to correct them The requirement for re inspection depends upon the severity and number of errors recorded Error statistics from inspections of all products and phases of the software life cycle are collected to measure process effectiveness The advantages of formal inspections can be signi cant Since formal inspections are reported to detect 80% of all errors subsequent dynamic testing of code becomes more ef cient Fewer execution failures cause fewer interruptions This translates to additional time for more thorough testing and possibly less time required for regression testing Formal inspections and dynamic testing techniques compliment each other Each can detect aws that the other cannot 14 15 Execution testing detects faults and failures the manifestation of errors On the other hand formal inspections detect the errors which potentially cause faults and failures - Besides enabling early and effective error detection for a range of software products there are several indirect advantages of formal inspections At IBM they encourage the iendly adversan d relationship between developers and veri ers through teamwork cooperation distributed risk and consensus Developers veri ers and the customer tend to focus effort on the most important aspects of software development requirements and design Time and cost required for testing and repair are diminished Formal inspections foster understandability and standardization in all software products They provide excellent on the-job training for all participants since they teach technical standards and organizational culture Furthermore formal inspections proliferate good ideas and eliminate bad approaches Formal inspections can require from 15% to 25% of total development time so developers may be reluctant to expend limited resources to support them However the resources necessary to implement them are not as great as those necessary to nd and x errors later 10 11 12 13 14 15 A cost-bene t analysis at Shell Research 12 reported an average 30 hours of repair and maintenance time saved for every hour of inspection time invested Bell-Northern Research 15 reported a 33 1 retum Other organizations have reported more conservative returns of 2 1 6 1 and 10 1 Note that these estimates are based entirely on direct costs They do not include other possible 11 pi savings from indirect costs related to customer con dence and the avoidance of the consequences of operational failure Instruction 5000 2 speci cally states that contractors should practice walk-throughs inspections or reviews of requirements documents design and code Of these the formal inspection is the most painstaking and work-intensive technique Reviews and walk throughs are also useful but they have other goals so they are less effective for detecting errors While use of formal inspections has demonstrated the production of high software quality at overall reduced cost and time the earlier investment in cost and time can easily drive a decision not to employ them This is apparently because the consequences of quality are not as visible as those of cost and schedule in the early phases of the life cycle We will discuss more about this later The fact is that the resources expended to implement formal inspections can pay for themselves in a short time by removing more expensive testing and maintenance costs Of the techniques we discuss in this report formal inspection appears to be the basis for the others This technique stimulates coordinates and checks the developer veri er coordinated requirements de nition process It does this by promoting teamwork and shared responsibility for quality It also produces early defect data necessary to measure feed and guide process improvement In addition to IBM several other companies have-described their experiences with the successful introduction of formal inspections A few offer tips for overcoming the dif culties of instituting them 12 15 We summarize these tips as follows 1 There must exist a belief that formal inspections will be effective Dynamic code testing will always seem to be faster and more effective but this is not true To change this mind-set will require an active campaign to sell the ef ciency of formal inspections to all levels of the organization Circulating reports of success and training programs can accomplish this 2 Everyone must clearly understand formal inspection procedures They are not informal They are not cursory reviews audits or walk-throughs Formal inspections are manual intensive detailed and painstaking Education and training are the best ways to prepare 3 It is essential to have management support Management must be decisive and committed to the belief that formal inspections will pay off This requires that the cost of formal inspections be quanti ed and resources be allocated to accommodate them into the schedule Also organizations must anticipate adjustments to the procedures as they adapt inspections to their own local environments 12 IF 4 Early successes are critical but also dif cult to achieve Start by inspecting only one or two types of documents for example requirements de nition The rst products inspected may be riddled with defects Early inspections can easily become muddled in details until problems with standards and procedures are resolved Therefore good moderators who can maintain group momentum are essential in the early stages 5 Keep detailed statistics on defect identi cation and associated actions This data feeds process improvement and provides clear evidence of effectiveness It will con rm belief in the process and strengthen commitment to it 6 The best training for inspections is on the job training However continued formal training of inspection team moderators is particularly important Otherwise as the effectiveness of the process becomes apparent to all the amount of materials and the number of required inspections can overwhelm the best-planned schedules 7 The local devel0pment process must be well de ned and understood by the participants Otherwise formal inSpections will be ineffective 31 5 Use Error Data to Guide Defect Prevention and Process Improvement Early identi cation and correction of errors is critical to software product correctness and quality Correcting errors in software is a but not a solution Software errors are often the of a more fundamental process defect Typical process defects might be failure to follow a standard practice misunderstanding of a critical process step or lack of adequate training In the software factory paradigm the software development process is a special manufacturing system to which many traditional quality control principles apply The developers and veri ers themselves owners of the process use error data to measure and improve the process until it reaches a repeatable predictable steady state Based on principles of Total Quality Management TQM formal process improvement implements error prevention by removing the causes of errors within the development process and the causes of not nding these errors earlier in the detection processes IBM FSC Houston practices process improvement Developers and veri ers form small process evaluation teams in TQM terms process action teams to analyze defects and identify their causes These teams also determine how to remove defect cause and subsequently implement required process changes The 13 effectiveness of these teams is rooted in TQM Team members are the programmers and veri ers whose primary daily responsibilities are software development Therefore those who execute the development process also execute process improvement The key to success is total management support and encouragement The responsibility to analyze and execute rests with the developers and veri ers The responsibility to allocate resources and make decisions that support process improvement rests with the managers The practice of process improvement has a number of positive outcomes A process that is partially or totally unde ned will have to be de ned in writing in order to subject it to process improvement This further stabilizes the process and tends to make it repeatable The continued practice of improvement de nes clear procedures for change and enables gradual technology insertion There is less resistance to new technology because the implementors of change are the same pebple who suggest it At the very least there will be a willingness to try new ideas Another important advantage of process improvement is its built-in on the-job training environment Membership on a process evaluation team is an excellent rst assignment for new personnel This responsibility encourages immediate participation and teamwork teaches the process de nition and its change procedures and stimulates creative thinking in the form of improvements New personnel are generally enthusiastic about contributing and bring fresh ideas into the organization The long-term bene ts of process improvement are also signi cant The procedures make the development process self-correcting Therefore over time the number of errors inserted during each phase of software development decreases This translates to a decrease in re work and greater ef cienqr for all sub processes For example veri ers may experience fewer problems during dynamic testing because fewer if any serious errors exist that could interrupt delay or prevent test completion Process improvement techniques are not new As previously mentioned they are essentially TQM techniques applied to the software development process The SEI Capability Maturity Model CMM for the software development process contains procedures for process improvement Furthermore the SEI Quality Subgroup of the Software Metrics De nition Working Group and the Software Process Measurement Project 'Ibarn have developed a draft framework for documenting software problems Such a standard collection mechanism can ultimately measure progress enable estimations and guide process improvement Dr Michael Fagan who originally developed formal inspection procedures now trains developers and managers to improve their software productivity and quality with a three step process formal process de nition 14 formal inspection of all software products and continuous process improvement through defect cause removal DOB could achieve signi cant ef ciencies in both its software development process and its software quality by applying process improvement Current emphasis on TQM will help to encourage the incorporation of software process improvement techniques Past and current practices rely on developers to learn from their mistakes vol However as this learning is lost through personnel and project turnover organizations are doomed to repeat their mistakes Formal process improvement eliminates the causes of error documents learning and hardens solutions against erosion bytime Process improvement has been proven effective in several software development environments It can move a software development process from one that is highly reactive and ad hoc to one that is statistically stable predictable repeatable and ef cient 6 Actively Motivate a Quality Attitude IBM FSC Houston s empirical evidence is convincing and its recipe is simple Quality software results from basic disciplined processes supported by a genuine quality attitude But the greatest impediment to implementation facing does not care about producing quality software Rather cost schedule and the product take greater priority because they have the highest visibility during the early phases of software development Until the testing phase quality is essentially an unknown or invisible However by this time cost and schedule drive the software through some ad hoc forms of testing because if the software is riddled with defects dynamic testing will be extremely time-consuming costly and dif cult Once the software is in customer hands poor quality will be highly visible because the cost to x it is high on the exponential scale independent of the cost of damage to customer The implementation of early defect identi cation defect prevention and process improvement in large software development environments is apparently dif cult 12 15 They are perceived as labor intensive work with little short-term payo Investment returns are not realized until late in the software life cycle -- during system testing operation and maintenance Mthin this is a particularly critical problem program managers are generally not rewarded for delivering systems with good operation and maintenance records These managers usually control the development phases of the program Thus costs schedules and product functionality drive their decisions But the success of defect identi cation defect preventions and process improvement demands an 15 unlimitedly 35 While not a software contract this example does illustrate that a quality guarantee based on consequence can create high visibility This may be one way to help stimulate the use of proven quality techniques Unfortunately most if not all US software rms take quite the Opposite approach Rather than life cycle Included are evaluation checklists that are similar to those used by IBM in their formal inspections The ideas in this framework have appeared in other manuals but only as guidelines For example they are contained in Draft Army Technical Bulletin 18402 2 1985 and US Army lnfonnation Systems Software Center Pamphlet 25-1 1990 Softmm Quality Engineering Handbook In contrast to well-published results of formal inspections Rome Labs Software Quality Framework has received less visibility However the adoption and extensive use of very similar quality methods by NEC Corporation 38 and Metriqs Inc is evidence of its potential value appears to have a more positive pro-active attitude toward software quality We believe that the establishment and support of the Army Software Test and Evaluation Panel Army STEP is very signi cant The mandate to employ the Army STEP Metrics may be the rst high-level action taken to implement software quality practices in the military The Army s serious attention to metrics - represents a signi cant shift by Army management toward software development as an engineering discipline This also indicates management willingness to expend the resources for early measurements to gain control of quality We believe that should take this opportunity to encourage support and motivate these efforts Management supported metrics are a positive rst step However these should not be collected for the sake of project management alone Quality requirements should be formulated during the requirements de nition phase This could be accomplished using the Rome Labs Software Quality Framework Once established quality requirements should be measured through standard metrics and checked in detail thrbugh formal inSpections using the checklists associated with the requirements 7 Introduce CASE Tools to Support Well-De ned Sub Processes Because Computer Assisted Software Engineering CASE tools apply to all aspects of software development including testing and because we have expanded the view of testing to encompass the entire life cycle we discuss here the potential impact of CASE technology on and its relationship to the techniques presented thus far The activities of software engineering which most impact software quality coordinated planning formal inspection con guration control verification testing and process improvement should be repeatable and yet adjustable if they are to be effective Often the most ef cient means of standardizing a process and making it repeatable is by automating it Computer Assisted Software Engineering CASE tools do this for software development and testing processes However automating any process rst requires that its procedures be 17 well de ned well understood and practiced CASE technology mnnot impose a methodology on an ad hoc software development environment Applying CASE technology in order to structure a manual process that is not working simply exacerbates a poor process For example without a well de ned well-understood manual procedure for con guration management an organization should not expect to effectively control software con guration using 3 CASE tool Automating a bad process only escalates a bad situation Initially organizations should use CASE tools only for those processes which are well de ned and practiced There are many examples of organizations which have adopted CASE tools only to abandon them because the anticipated improvements never materialized One reason for this was described above Another reason is an underestimation of the CASE tool training requirements Because CASE tools support methods but do not impose them an organization must recognize the difference between learning the tool and learning the method the tool supports especially if the supported method is not currently practicedl There are leaming curves associated with each If the method is understood and practiced then only the tool presents a learning shortfall However if the tool supports a new method unfamiliar to the developers then two shortfalls exist The training and time to overcome such may add signi cantly to CASE tool investment The compound training requirement for both method and tool will also extend the time required to show a return on the investment should sensitize management to the large investment required to train personnel in both the new tools and as necessary the associated new methods The adoption of CASE technology within should be an evolutionary process that begins small and grows gradually Controlled institution of CASE tools has a greater potential for immediate success and visible investment return than a massive overwhelming introduction The adoption of standard software process metrics gives the means to establish the value of CASE tools should not force its managers to overreact Rather it should make the time and nancial resources available for managers to adequately train and methodically insert standard CASE tools into practical use Both CASE tool users and management must restrain expectations of immediate results They must anticipate the learning curve s measure progress and continue with process improvement to insert additional CASE technology Our observations of and discussions with IBM FSC Houston strongly support this approach to introduction of CASE tools DeveIOpers at IBM FSC Houston have produced and continue to produce high quality software using manual processes supported by non-integrated databases and word processing tools 18 CASE tools are just now being considered However deployment of CASE tools will be closely monitored and controlled through formal process improvement This will ensure that the adoption of tools will properly enhance their own methods CASE technology will probably cause changes improvements in their current methods but only through the formal improvement process We highly recommend that consider process improvement as the technology insertion mechanism for instituting CASE tools 8 Conclusions and Recommendations knows how to produce quality software The Army has recognized the critical relationship between measurement and quality control and is now implementing mechanisms that can decrease the cost and increase the quality of software production Software testing is related to these mechanisms as a key data-gathering technique However quality cannot be tested into software Quality must be designed through - engineering Engineering requires an expanded view of testing to maximize the effectiveness and'reduce the cost of veri cation and acceptance testing testing activities should begin during requirements de nition and should in uence the entire software develOpment life cycle The general principles of engineering applied by IBM and other commercial companies to produce quality MIS C2 or MSCR software are the same modeling standardization measurement and process control However the empirical evidence has come from environments in which the developing testing maintaining organization the software and the customer have been relatively constant for a long period of time For example IBM FSC Houston has developed and tested the Space Shuttle ight and ground software for NASA for over 15 years adequate time to stabilize their development process However real cost savings have been reported even in the case these procedures were initiated during the veri cation phase of a project Nonetheless our recommendations should be viewed in the context of the environment There exists a variety of contractors customers software and relationships among them Detailed standardization of software engineering activities would be too restrictive and probably counter productive Instead guidelines should be standardized locally through detailed written procedures On the basis of our discussions and conclusions we recommend the following actions 1 Actively motivate a software quality attitude in and government contractors Implement by adopting and training a process such as the Air Force 19 unlity is never an accident It is always the result of high intentions sincere effort intelligent direction and sla ll d execution Quality cannot be inspected in quality must be designed in This passage is as true today of software as it was then of hardware Quality cannot be-tested into software Furthermore inspection alone is ineffective Quality results from belief in and commitment to quality objectives well de ned processes continuous measurement supported by formal inspection and process improvement 21 10 11 12 13 REFERENCES Buckley E1 and Poston R Software Quality Assurance Transactions on So ware Engineering vol no 1 Jan 84 pp 36-41 DOD Software and Evaluation Project STEP Volumes 1-6 1983 1 Final Report and Recommendations 2 Software 'Ibst and Evaluation State of-the Art Overview 3 Software 'Ibst and Evaluation Current Defense Practices Overview 4 'Il'anscript of STEP Workshop Mar 82 5 Report of Expert Panel on Software and Evaluation 6 Tactical Computer System Applicability Study Gelperin D and Hetzel B The Growth of Software Testing Communications of the ACM vol 31 no 6 Jun 88 pp 687 695 Software Maintenance Echnology Reference Guide Software Maintenance News Inc ' 1992 Humphrey W S Software and the Factory Paradigm So ware Engineering Journal Sep 91 pp 370-376 C Kjell-Hakan N and Ohlsson L Software Factory Principles Architecture and Experiments Software Mar 92 pp 36-44 Basili V R and Musa The Future Engineering of Software A Management Perspective Computer Sep 91 pp 90-96 Dunham J R in the Next Decade Software May 89 pp 47 53 Martin R J The Application of the Principles of Total Quality Control to Mission Critical Software ISYE 6301 28 Nov 84 IBM FSC Houston Software Process Showcase IBM Federal Systems Company Houston TX 2-3 Apr 92 Dichter C R Two Sets of Eyes How Code Inspections Improve Software Quality and Save Money Unix Review vol 10 no 1 Jan 92 pp 19 23 Doolan E P Shell Research Experience with Fagan s InSpection Method So ware Practice and Experience vol 22 2 Feb 92 pp 173 182 Royce W Pragmatic Quality Metrics for Evolutionary Software Development Models TRW Systems Engineering Development Division Jan 91 - 22 14 15 16 1'7 18 19 20 21 22 23 24 25 26 27 28 Ackerman A F Software Inspections An Effective Veri eation Process Software May 89 pp 31-36 Russell G W Bell Northern Research Experiences with Inspections in Ultralarge Scale Developments So ware Jan 91 pp 25 31 Blakely EW and Boles M E Case Study of Code Inspections Hewlett-Packard Journal vol 42 Oct 91 pp 58-63 Cavano J P and LaMonim F S Quality Assurances In Future Development Environments Software Sep 87 pp 26 33 Defense Acquisition Management Policies and Procedures Department of Defense Instruction 5000 2 23 Feb 91 Defense System So ware Development 29 Feb 88 Defense System Software Quality Management Program MIL-STD-2168A Draft 6 Aug 91 Final Report Draft of the Software and Evaluation Panel STEP by the Standards and Regulations Implementation Team SRIT 29 Jan 92 Demming W E Out of the Crisis MIT Center for Advanced Engineering Study Cambridge MA 1982 Graham D R Test is a Four Letter Word The of Defects and Detection CrossTalk The Journal of Defense Software Engineering No 35 Aug 92 pP 2 7 ANSI Standard Glossary of So ware Engineering Erminology 729 1983 1983 Software 'Ibst and Evaluation Project STEP Software Test and Evaluation Manual Volumes 1- 1 Guidelines for the Tl'eatment of Software Test and Evaluation Master Plans Oct 85 2 Guidelines for Software Test and Evaluation in 25 Feb 87 3 Good Examples of Software Testing in 1 Oct 86 Williamson M Beyond Rube Goldberg Mar 91 pp 54 59 Software Test and Evaluation Guidelines DA Pamhlet 73-1 Draft vol 6 15 Jun 92 Informal discussions with Air Force Standard Systems Center Gunter Air Force Base AL Mar 92 - 29 30 31 32 33 34 35 36 37 38 39 40 41 Pagan M B Design and Code Inspections to Reduce Errors in Program Development IBM Systems Journal vol 15 no 3 1976 pp 182-211 Pagan M B Advances in Software Inspections Transactions on So ware Engineering Jul 86 pp 744-751 Pagan M Productivity Improvement through Defect Free Software Development - An Executive Overview Michael Pagan Associates slide presentation by Dr Fagan to Army Strategic Defense Command Huntsville AL 31 Aug 92 B Software Inspections SDIOISDA So ware Engineering Newsletter vol 1 no 2 May 92 Paulk M C Curtis B and Chrissis M B Capability Maturity Model for Software CMUISEI-Q and Software Engineering Institute Camegie-Mellon University Aug 91 Florac W A Software Quality Measurement A Framework for Counting Problems Failures and Faults Draft The Quality Subgroup of the Software Metrics De nition Working Group and the Software Process Measurement Project 'Ibam Software Engineering Institute Carnegie-Mellon University May 92 Ochi Hiroyuki Japanese Software Quality Brie ng notes delivered at 4th Annual Software 'Ibchnology Conference Salt Lake City UT 13-17 Apr 92 Bowen T P Wigle G B and Thai J T Speci cation of Software Quality Attributes Software Quality Evaluation Guidebook Volume 3 of 3 Rome Air Development Center Feb 85 Software Quality Engineering Handbook Draft U S Army Bulletin TB 18-102-2 Anny Automation 25 Mar 85 also U S Army Information Systems Software Center Pamphlet 25 1 Feb 90 Sunazuka T Azurna M and Yamagishi N Software Quality Assessment Technology Proceedings of the 8th International Conference on So ware Engineering 1985 Murine G B Integrating Software Quality Metrics with Software Quality Progress Nov 88 Jones C Missing Elements Spectrum Jun 92 pp 38 41 Kemerer C F How the Learning Curve Affects CASE Tool Adoption Software May 92 pp 23 28 24 I'm 42 Lee Earl Senior En informal discussions in Jun 92 43 Dumont AB Quali Engineers Nov 50 ty in Engineering 'neer for IBM Federal Systems Company Houston Proceedings of the Institute of Radio ll 10 11 12 BIBLIOGRAPHY Bergman M The Evolution of Software 'Ibsting Automation Proceedings of the 8th International Conference on Testing Computer Software International and Evaluation Association 17-20 Jun 91 pp 1-10 Deimel LE Scenes of Software Inspections Software Engineering Institute Carnegie-Mellon University May 91 DeMillo RA McCracken W M Martin R J and Passa ume J F Software Testing and Evaluation Benjamin Cummings Publishing Co 1987 Dyer M The Cleanroom Approach to Quality Software Development Wiley and Sons Inc 1992 Fear H S Creating A Test Process Proceedings of the 8th International Conference on Testing'Computer Software International and Evaluation Association 17-20 Jun 91 pp 35 43 Fewster M A The Managing Director Wants 100% Automated Testing A Case History Proceedings of the 8th International Conference on Testing Computer Software International and Evaluation Association 17-20 Jun 91 pp 59 70 Kelly C Sherif S and Hops An Analysis Of Defect Densities Found During Software Inspections Journal of Systems and Software vol 17 Feb 92 pp 111 117 Martin R J The Challenge of Software Engineering Project Management Ten Years Later Software Engineering Research Center Purdue University May 90 Mays R G Jones C L Holloway G J and Studinski D P Experiences with Defect Prevention IBM Systems Journal vol 29 no 1 1990 pp 4-32 O Neill D What Is the Standard of Excellence New Views of Mature Ideas on Software Quality Productivity Software May 91 pp 109-111 Parnas D L van Schouwen AL and Kwan S P Evaluation of Safety-Critical Software Communications of theA CM vol 33 no 6 Jun 90 pp 636 648 Schneck P A Virtually Defect Free Code as a Direct Result of a Well-De ned Comprehensive Testing Method Proceedings of 8th International Conference on Testing Computer Software International Test and Evaluation Association 17 20 Jun 91 pp 133 141 26 t 13 14 15 16 17 18 19 20 21 Thebaut S M and Martin R J Af liate Current Practices Software Engineering Research Center Purdue University 1989 Draft DOD So ware Rehnology Strategy Software Ilechnology Working Group Dec 91 Presentations of the 1990 Multiservice Software 'Ibst and Evaluation Symposium Reston VA 30 Oct - 1 Nov 90 Quaiity Program U S Army Bulletin TB 18 102 Army Automation Mar 84 Report of Task Force on Military Software Defense Science Board 1 Jul 87 Software Test Tools Report U S Air Force Software Technology Support Center Hill AFB Utah 1991 Testing of Computer So ware Systems U S Army Technical Bulletin TB 18-104 Army Automation 20 Aug 82 U S Army Software Standards and Regulations Reference Guide Draft Using Test Data Generators to Reduce Software-Development Costs U S Army Technical Bulletin TB 18 22 Management Information Systems 10 May 74 27 AIRMICS ANSI Army STEP C31 CASE CMM STEP FSC FSD IBM MIS MSCR NASA OASD SEI SSC TQM APPENDIX - List of Acronyms Advanced Automation System Army Institute for Research in Management Information Communications and Computer Sciences American National Standards Institute Army Software 'Ibst and Evaluation Panel Command and Control Command Control Communications and Intelligence Computer Assisted Software Engineering Capability Maturity Model Department of Defense Software 'Ibst and Evaluation Project Federal Systems Company Federal SEctor Division A International Business Machines Corporation Institute of Electrical and Electronics Engineers Independent Veri cation and Validation Management Information System Materiel System Computer Resource National Aeronautic and Space Administration Of ce of the Assistant Secretary of Defense Software Engineering Institute Standard Systems Center Total Quality Management This document is from the holdings of The National Security Archive Suite 701 Gelman Library The George Washington University 2130 H Street NW Washington D C 20037 Phone 202 994-7000 Fax 202 994-7005 nsarchiv@gwu edu