411 AD-A286 i i1f El 11111 11111 IN II11111i REPORT OF THE DEFENSE SCIENCE BOARD TASK FORCE ON ACQUIRING DEFENSE SOFTWARE COMMERCIALLY JUNE 1994 1994 01 OFFICE OF THEUNDER SECRETARY OF DEFENSE FOR ACQUISITION TECHNOLOGY WASHINGTON D C 20301-3140 94-35782 1 t94 'V • This report is a product of the Defense Science Board DSB The DSB Ls a Federal Advisory Committee established to provide independent advice to the Secretary of Defense Statements opinions conclusions and recommendations in this report do not necessarily represent the official position of the Department of Defense This document is UNCLASSIFIED Security review completed 13 July 1994 by OATSD Public Affairs Dikr-ctorate for Freedom of Information and Security Review Reference # 94-S-2796 UNCLASSIFIED SECURITY CLASSIRI ATION OF THIS PAGE Form Approved REPORT DOCUMENTATION PAGE 0MB No _Exp lb RESTRICTIVE MARKINGS REPORT SECURITY CLASSIFICATION la 0704-0188 Date Jun30 1986 N A Unclassified 3 DISTRIBUTION AVAILABILITY OF REPORT 2a SECURITY CLASSIFICATION AUTHORITY N A Distribution Statement A 2b DECLASSIFICATION DOWNGRADING SCHEDULE Approved for Public Release N A is Distxibution miilimnit-C' S MONITORING ORGANIZATION REPORT NUMBER S 4 PERFORMING ORGANIZATION REPORT NUMBER S N A N A 6b OFFICE SYMBOL 6a NAME OF PERFORMING ORGANIZATION the Under Secy of Def A T DSB OUSD A T 6c ADDRESS City State and ZIP Code The Pentagon Room 3D865 Washington DC 20301-3140 N A 7b ADDRESS City State and ZIP Code N A 8b OFFICE SYMBOL if applicable 8a NAME OF FUNDING SPONSORING ORGANIZATION 7a NAME OF MONITORING ORGANIZATION if applicable Ofc of Defense Science Board Defense Science Board OUSD A • DSB OUSD A T 9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER N A 10 SOURCE OF FUNDING NUMBERS 8c ADDRESS City State and ZIP Code The Pentagon pcan 3D865 Washington DC 20301-3140 PROGRAM ELEMENT NO PROJECT NO N A N A N A WORK UNIT ACCESSION NO TASK NO- N A 11 TITLE Include Security Classification Report of the Defense Science Board Task Force on A cquring Defense Software Commercially Unclassified 12 PERSONAL AUTHOR S N A 14 DATE OF REPORT Year Month Day 13b TIME COVERED 13a TYPE OF REPORT I Final FROM 64 1994 June TO __ A _ N A 15 PAGE COUNT 16 SIJPPLEMENTARY NOTATION N A 17 FIELD COSATI CODES SUB-GROUP GROUP 18 SUBJECT TERMS Continue on reverse if necessary and identify by block number 19 ABSTRACT Continue on reverse if necessary and identify by block number DTIC QUALIVY INEPECTED 5 20 DISTRIBUTION AVAILABILITY OF ABSTRACT C1 SAME AS RPT 0 UNCLASSIFIED JNLIMITED INDIVIDUAL 22a NAME OF RESPONSIBLE Diane L H Evans DD FORM 1473 84 MAR 21 ABSTRACT SECURITY CLASSIFICATION 0l DTIC USERS 22b TELEPHONE Include Area Code I22c OFFICE SYMBOL 703 695-4157 8 83 APR edition may be used until exhausted All other editions are obsolete DSB OJSD A T SECURITY CLASSIFICATION OF THIS PAGE UZXMMSIFI D OFFICE OF THE SECRETARY OF DEFENSE WASHINGTON D C 20301-3140 1994 JUN DEFENSE SCIENCE BOARD MEMORANDUM FOR UNDER SECRETARY OF DEFENSE ACQUISITION TECHNOLOGY SUBJECT Report of the Defense Science Board Task Force on Acquiring Defense Software Commercially I am pleased to forward the report of the Defense Science Board Task Force on Acquiring Defense Software Commercially The Task Force co-chaired by Dr George H Heilmeier and Dr Larry Druffel was chartered to determine the conditions under which defense software could be procured using commercial practices and to develop a strategy for procurement that incorporates such practices In its investigation into applying commercial practice to DoD software procurement the Task Force reviewed a broad spectrum of interrelated elements from software program management policy and DoD software acquisition practices to DoD's investment in the software technology base I wholeheartedly concur with the group's key finding that DoD must develop a more coordinated approach to the oversight of its diverse software capabilities and programs The Task Force's report provides an outstanding point from which to begin addressing revamping DoD's software procurement practices Paul G Kaminski Chairman SAv i an o--Aviaiit D Bt Spca oe WisLiuionFo -A - - OFFICE OF THE SECRETARY OF DEFENSE WASHINGTON D C 20301 -3140 3 0 JUN 1994 DEFENSE SCIENCE BOARD Dr Paul Kaminski Chairman Defense Science Board Office of the Undersecretary for Acquisition and Technology The Pentagon Washington D C Dear Dr Kaminski Enclosed is the final report of the Defense Science Board Task Force on Acquiring Defense Software Commercially We were tasked to determine the conditions under which procurement of defense software can use commercial practices and to define needed changes to permit such use In the attached report we make specific recommendations with regard to DoD process credibility software program management required expertise of DOD personnel using modern software practices use and integration of commercial off-the-shelf software DoD software acquisition practices use of software architectures by DoD as a management tool and DoD's investment in the software technology base Based on our review we concluded that issues associated with defense software are applicable across the wide spectrum of DoD software intensive systems However to reap maximum benefit from improvements related to the myriad aspects associated with software the DoD requires a more coordinated approach to the oversight of its diverse software capabilities and programs To provide uch Department-wide oversight and to facilitate implementation of the other spetific recommendations contained within this repo t the Task Force recommends that the Secretary of Defense assign to the Under Secretary of Defetise Acquisition and Technology the responsibility for DoD-wide software technology policy practices and acquisition This centralized approach will best serve the Department for all software intensive systems and programs To support the implementation of this recommendation and to ensure that the key stakeholders are participants in the process the Task Force further recommends the formation of an Executive Council consisting of the appropriate principals from OSD the Services and the Defense Agencies We thank all of the members and government advisors of this Task Force for their dedicated efforts and significant contributions to this study ruffel Co-Chairman ge 1 Heilrneier Co-Chai man _ Tfable of Contents Executive Summary 1 0 Task 1 1 1 2 1 3 1 4 Force Overview 1 Terms of Reference Caveats Topics Addressed Membershiýp 1 1 2 3 4 2 0 Current DoD and Commercial Software Acquisition Practices 5 2 1 Background 5 2 2 Comparison of Defense and Commercial Software 10 3 0 Major 3 1 3 2 3 3 3 4 3 5 3 6 3 7 4 0 Summary 4 1 Atta Persons 4 2 Overarching Recommendation A Terms of Reference A-1 B Briefings Provided to the Task Force B- 1 C Comparison of Software Acquisition Methods C- D Proposed Action Plan froin Service Software Executive Officials D-l Findings and Recommendations Process Credibility iDoD Program Maniagemnent DefD Personnel Use and Integration of Commercial Off-the-Shelf Software Acquisition Software Architecture Software Technology Base 20 20 23 25 27 30 34 38 40 40 41 1 Executive Summary The Defense Science Board Task Force on Acquiring Defense Software Commercially recognizes that DoD systems are becoming increasingly dependent on the use of software as the mechanism for implementing operational capabilities To adapt to changing military and national security situations DoD is more dependent than ever on its ability to modify mission software rapidly often in near real-time However software remains the schedule and cost driver for the development and maintenance of many important defense systems In its review of current DoD and commercial software acquisition practices the Task Force found notable differences as evidenced in Appendix C There are however many similarities between the various categories of DoD and commercial software systems Although there are indications that commercial development efforts have achieved better predictability and lower costs the Task Force noted a significant lack of credible quantitative data to substantiate this assessment In general the Task Force concluded that DoD's investment in software requires greater DoD-wide management control and oversight in the coming years if the Department is to exploit the use of commercial software acquisition practices fully as well as rapid advances in software technology The following is a summary of selected findings and recommendations toward that end Process Credibility Current DoD practice is not compatible with commercial business practices DoD should work to make necessary changes to acquisition regulations such as Having program managers manage 3 of 3 price schedule functionality but only constrain 2 of 3 0 Defining successful performance on contracts as delivering a solution with predictable price schedule and functionality not adherence to government processes procedures and specifications Not requiring c-level specifications for software projects developed in Ada Establishing mechanisms to allow both current ability to perform as well as past performance as key factors in source selection Encouraging offerors to demonstrate as much functionality as possible as part of bid without eliminating domain knowledgeable competition DoD Program Management DoD program management approaches discourage the use of' commercial practices Program managers lack incentives to tailor procedures to fit individual program needs or to develop corporate solutions e g employ common architecture or common software components DoD should establish and implement overarching F ftware life cycle guidelines more conducive to the use of commercial practices and products such as Defining software architectures to enable rapid changes and reuse • Facilitating early system engineering and iterative development • Participating in development of commercial and international standards • Allowing the fielding of software directly from test beds with user consent Requiring program managers to stay with programs at least through beta testing to maintain continuity of understanding of original nuances in requirements DOD Personnel There is currently a shortage of sufficiently qualified software personnel at all levels within the Department DoD should establish a Department-wide software program management education and training initiative that includes changing courses for PMs to reflect best commercial practices and other recommendations of this task force and providing for changes to reflect the dynamics of the software industry rotating government and contractor personnel between PM and developer organizations to build understanding and trust encouraging use of IPA's from industry and integrating software-qualified personnel into senior DoD acquisition staff Use and Integration of Commercial Off-the-Shelf COTS Software DoD has not fully identified the pros and cons associated with the use of COTS software and as a result has not determined when and how best to use COTS software To facilitate this process DoD should require trade studies and analysis of the use of COTS software in DoD's software acquisition process where appropriate Further DoD should establish customer friendly application-specific information technology component stores to enable program managers to assLtuble systems rather than develop them through use of reusable prequalified components DoD should also increase technology base funding for security audit tools for systems employing COTS software and should capitalize on innovative costeffective techniques for acquiring and using COTS software products such as the use of enterprise licenses Software Architecture Software architecture was emphc zed by the Task Force as a means for achieving important ends There is currently little emphasis on architecture in DoD software progranms or regulations As a result DoD is not benefiting from architecture as a key tool for evolutionary development and for early and frequent involvement of users with functional capability and facilitating reuse requirements changes with minimum cost and schedule and product line management DoD should require vendors to propose manage and control the architecture and should establish an early architecture deliverable in all developments Software Technology Base The current DoD software technology base investment does not adequately take advantage of commnercial R D Further software technology transfer both internal and external is not receiving adequate emphasis within DoD DoD should provide for the evolution of the DoD Software Technology Strategy to align with emerging commercial technology and practices Overarching Recommendations To facilitate implementation of the many recommendations contained in this report the Task Force concluded that DoD's investments in software require greater management control and DoD-wide oversight To this end the Task Force recommends that the Secretary of Defense SECDEF assign the Under Secretary of Defense Acquisition and Technology USD A T the responsibility for DoD-wide software technology policy practices and acquisition In carrying out this responsibility the Task Force recommends that the USD A T consider forming an executive council with the Director Defense Research and Engineering DDR E the Assistant Secretary of Defense Command Control Communications and Intelligence ASD C31 and appropriate representatives from the Services and Defense Agencies as members Further USD A T should provide for a supporting process action team to assist in implementation of the recommendations of this Task Force study Defense Science Board Task Force on Acquiring Defense Software Commercially 1 0 TASK FORCE OVERVIEW 1 1 TERMS OF REFERENCE Terms of Reference Objectives Determine - Conditions Under which Procurement of Defense Software Can Use Commercial Practices - Changes Required to Permit Such Use Develop Strategy that Incorporates Such Practices - Not Constrained by Existing DoD Standa-ds - Viewed as Coexisting Alternative Strategy - Includes DoD Use of Commercial Software Products Compare Proposed Strategy with Current DoD Strategy Indicating Circumstances Where Each is Most Beneficial Scop e All Software Intensive Systems All Stages of the Software Life-Cycle Appendix A provides the Terms of Reference by which the Task Force was established The objectives and scope of this Task Force are outlined abbove At the fiust meeting of the Task Force the sponsor of the study the Director Defense Research and Engineering requested that the Task Force provide a strategy that was sensible pragmatic and unconstrained by current DoD acquisition practices was based on evaluation of mechanisms for integrating defense software efforts with commercial software efforts did not require legislative relief and addressed the full spectrum of DoD software applications 1 2 CAVEATS Caveats Relied on Inputs from DoD and Industry Experts e Provided No Detailed Quantitative Assessment or Evaluation of Individual Topics e Did Not Address - Success or Failure of Ada - Recommended Software Development Approach for Specific Programs - Specific COTS Products for DoD to Exploit - Trade-Offs Between Hardware and Software The Task Force relied heavily on inputs from a variety of DoD and industry experts regarding a wide range of topics related to defense software technology policy practices and acquisition Appendix B lists the many briefings that were provided Although the Task Force chose not to provide detailed quantitative assessments or evaluation of individual topics it used the information associated with these topics in the formulation of findings and recommendations The areas not addressed by the Task Force while important in and of themselves were determined not to be directly germane to the development of an overall defense software strategy •- 2 1 3 TOPICS ADDRESSED Topics Addressed Topic DoD Software Acquistion Policies DoD Management of Commercial eelp m tPrcs Software Risk Management Techniques and Supporting Tools Minimum Effective Delivery Time Affordabiliky Ma intenain ce Aft-er Prc-c elv Post-Deplcyment Product Enhancemnent Software Process Support Tools 0 'ICAsurEd1iI Availability W-se of Development nd Maintenance Tools --ntractinr -lec-aDataNTechnical Rihtsj FInteletl Prpry ights 0 Alternative Forms of Procurement Agreements o Incentives for CreatioV se of Reusable Software Components 0 Economic Incentives Importance of Software Architecture State-of-the-Art and Blest Commercial Practices 19 Dvelome n t Tool euabl Toitivare C omponents 6 1 eclnique't rools for T ailoring Commnercial Components tor Defense Use In its efforts to assess the appropriateness of DoD use of commercial practices the Task Force addressed software management contracting and technical issues The above viewgraph lists the primary topics considered within each of these categories 3 1 4 MEMBERSHIP Members Co-Chairmen Dr George H Heilmeier Dr Larry Druffel Bellcore Software Engineerin Institute Members Dr Jacques Gansler Mr Jack Hancock Mr Patrick Hillier Mr Arthur E Johnson Dr Bruce Johnson o Mr Alan McLaughlin Dr Alvin F Nashman Mr John Stenbit Dr Terry Straeter TASC Rietired Exec VP Pacific Bell EDS Loral Federal Systems Andersen Consulting MIT Lincoln Labo-atory Computer Sciences Corporation TRW GDE Systems Inc Indep endent DS13 ReviewerLs Mrs Joan Habermann Mr Philip A Ode en Logistics Management Institute BDM International Inc Executive Secretaty Ms Virginia L Castor M DSR Secretariat Representative CDR Robert C Hardee ODDR EIAT IJSB Government Advisors e o e @ Ms Mr Mr Mr Ms Mr Deborah Castleman Frank Kendall Gene Porter William Mounts Linda Brown ChriG Dipetto Toint Staff Mr Joseph Torna Services o LTG Peter Kind o RADM John G Hekman Mr Lloyd K Moseznann 11ArIoc Agencies 9 Ms Belkis Leong-Hong Dr Ed Thompson OASD C3I OUSD A T OUSD A T ODUSD AR ODASD IM OU SD A TI J6A Am Nv DS AT Task Force mnembers represented a valued cross section of software expertise within both the DoD and commercial sectors The government advisors represented senior enecutives including the three Service Software Executive Officials from the major software management organizations within the Departmnent 2 0 CURRENT DOD AND COMMERCIAL SOFTWARE ACQUISITION PRACTICES 2 1 BACKGROUND Background Previous Studies F DSB Summer Study on Technology Base 1981 Joint Service Task Force on Software Problems 1982 AF SAB High Cost and Risk of Mission Critical Software 1983 CODSIA Report on DoD Management of Mission-Critical Computer Resources 1984 DSB Task Force on Military Software 1987 Ada Board Response to DSB Task Force 1988 Summer Report on Defense-Wide Audit of Support for Tacticai Software 1988 Workshop on Executive Software Issues 1988 Workshop on Military Software 1988 Army Materiel Command Study '989 Software Technology Development and Deployment Plan for DoD Technology Base 1989 AF SAB Adapting Software Development Policies to Modem Technolcgy 1989 Draft DoD Software Master Plan 1990 Draft DoD Software Technology Strategy 1991 AF SAB Study on Information Architecture 1993 Study on Military Standards Impacts on the Acquisition Process 1993 Draft Software Action Plan Working Group Report 1993 Evolutionary Acquisition Study AFCEA June 1993 As a point of departure the Task Force noted a number of previous studies addressing issues related to the defense software technology policy practices and acquisition Despite the increased emphasis given to software issues by the DoD as evidenced by the above list the majority of the recommendations resulting from these studies have not been implemented 5 F Background Impediments to Change e DoD Software Management No Single Operative Mechanism Exists to Implement Change Three Separate Domains MIS C31 Embedded Two Separate Acquisition Management Structures Bureaucratic Turf Inhibits Real Reform e Acquisition Process 5000 and 8000-Series Developments Typically Employ Waterfall Approach Not Incremental Spiral Approach e Culture Systems Are Stove Pipe -- My System My Program PM is King DoD Acquisition Training Reinforces the Wrong Approaches Procurement The Contracting Process Inhibits Creativity and Investment by Contractors Limits Options Interpretation of Competition in Contracting Act ro ensure that its recornmendations could be readily implemented by the DoD the Task Force identified the primary reasons why recommendations from previous studies had not been acted upon The Task Force then formulated its recommendations to appropriately address these impediments 6 Why Is This Study Important S DoD Military Capability Increasingly Software Dependent Software One of Few DoD Budget Areas That is Projected to Grow - Software Has Become the Overall Defense System Schedule and Cost Driver 1 Declining DoD Budget - Affordability a Major Concern Escalating Costs of Software Development and Life Cycle Significant Level of DoD Resources To Be 3pent on Information Technology FY92- FY94 -$10 BillionlYear EIA Forecast for FY95 - FY98 -$10 BillionlYear In-House vs Contracted Out 30% In 70% Out Will Require Greater Management Control Rich Commercial Base to Tap Many Opportunities - o a Custom Software Off-the-Shelf Products - Acquisition Methodologies - Approach to Requirements Determination Functionality and Flexibility More Embedded in Software than Hardware Study is Timely Because DoD Leadership is Focused on Acquisition Reform Something May Actually Get Done Source EAM Given its increasing reliance upon software as the mechanism for implementing system capabilities coupled with the rising costs associated with software development and maintenance DoD must take action now to address the issues associated with software acquisition The commercial sector provides numerous opportunities upon which the DoD could readily capitalize The time is ripe for assertive DoD action particularly since the current leadership is so strongly focused on acquisition reform 7 The Software Domain is Very Large ILARGE DODEMBEDDED REAL SI T I II T I COMMERCIAL I I I MM 0° I I COMPLEXITY --- 12 TIME SYSTEMS SYSTEMS BUSINESS SYSTEMS f _T I OPLEX ICOMMERCIAL • • COTS PRODUCTS RELEAE i DODI EMBEDDED REAL DOD BUSINESSESYSTEMS CUSTOM COMMER-CIA-L SMALL I SYSTEMSCUSTOM •' D1 El Iv l II I COM MERCIAL DO D0I llENGINEERING I iE I ENGINEERING T REAL TIME SYSTEMS COST TO DEVELOP The software domain reviewed by the Task Force encompasses a wide variety of DoD systems ranging from software tools to large embedded real-time systems The associated cost complexity and time required to develop these systems vary widely both for commnercial and military applications Background DoD Software Acquisition Management-__ Two Different Sets of Software Policies Rules Managed by Two Different Organizations - USD A T for Software Embedded in Weapons Systems - ASD C3H for C3H Software and MIS Applications Majority of Software Issues are Applicable to All DoD Systems Growing Similarity Between DoD and Commercial Software Developments Across the Various Types of DoD Software - Modem Software Tending to Blur Distinction Between DoD and Commercial Applications Central DoD Leadership Needed More than Ever - Major Revisions Needed in DoD Software Policies Rules and Management Across All DoD Applications in Order to Meet SECDEF's Acquisition Reform Goals - Interpretation and Implementation of DoD-Wide Policies Rules by Management is a Central Issue Today the management of DoD software acquisitions is quite complex There are two sets of software policies rules managed by two different organizations • USD A T for software embedded in weapons systems ASD C31 for C31 software and MIS applications There currently exists within the DoD a dichotomous organizational structure for the management of DoD software intensive systems The Under Secretary of Defense Acquisition and Technology is responsible for weapons systems the Assistant Secretary of Defense Command Control Communicatiois and Intelligence C31 is responsible for C31 systems and Management Information Systems MIS However as technology has evolved over the years the issues associated with the use of software in all of these systems have become essetlally the same The issuance of different software related policies and regulations that are oriented according to the type of system is no longer appropriate If the DoD is to adequately address this fundamental issue central focused management is required 9 2 2 COMPARISON OF DEFENSE AND COMMERCIAL SOFTWARE Comparisonof Defense and Commercial Software Needs e Defense and Commercial Software Applications Needs Are Merging Breadth of Application Typical size Unique 400 000 SLO Upgrade Sensitivity to Change Errors Complex High R-1e Time App- ca nn - DoD Real Time Software Inflexible - 'omoircid C ustom rottwate Integrated Into Real-Time Operaitions Cormmandand Contrl A1pliealinn C31 s2lom oware T Into Business Operations 40t 000 Complex Inflexible T1Th Moving Toward Wide Moving oward Wide 2K00O Complex Flexible Moderate 20 000 Lomplex Moving Toward Flexible Moderate Very Wide 200 O0 Low Very Wide 200 000 Moving Toward Comotercial Exploits Object S ounmercial DoD Automated Information Systems - Unique Cormercial Automated Information Systems Low Oriented Technology _ BRuabl• CoponentjUn'dueft - DoD Component Stores VWry Wide 50 110 Tied to Weapvn •_ - Commercial Shrink Wrapped Very Wide 50 000 System Cycle Tied to Commercial Low to High Deiending on Use L-w cle DoD and commercial applications for software are different in some ways but very similar in others The above table highlights these similarities and differences for real-time applications command and control applications MIS systems and reusable components and products This apparent disparity in the classification of software between DoD and commercial vendors increases the complexity of DoD's management task Companies people methods and organizations are usually specialized to one or more commercial type systems not to DoD type systems Defense and commercial software applications needs are merging providing the potential that DoD can exploit commercial capabilities more effectively over time Major revisions are needed in DoD software management across all DoD applications in order for DoD to capitalize on the evolving commercial base Process Comparison Summary Clv lianI Custom Custom Commercial DOD Frocess Attribute Orlflulo Osawi byr Definition Problemn By Speu RIld Flexiblity Vay 1 011011 M teborreettshort L40FE h Hav Deoweeg OuelonForeity Gullorerlubar InVlvernerit in Devlopnmn -1 9by PfoceesMonitorinf Customer Aceetaten Prooees - ni5 iniernalfl waivin NotConabalned 8 Informal opfi k iea Vovlrtaul rVe ol Adancedo Eme'Ebob Tochnologlos WWA at Wam Reduced Cost Schodule Docurnwvt us S w r Defvh end Aom ao esCurig User CotmelewJducdbl SOS S5 Zostf NmReduced Haysome Reduced Copif ipeSchedlute Drie4Product Spec ISO P05elb45 Reduced Costl prom Ipowin rI 51114e hoorullit Sboac DW More Practtcal Req uiramote Ont iude MSdeW SerweXWWO18de Proces DetirbiUflo Commercial Result Commercial Product Bud road so PoNobA I Quaimty tpea -W as Lass all Source IBM Federal Systems As is evident from the process comparison summary above there are not only differences between commercial and defense applications but also in the proccss used by each to develop acquire and support complex software systems Defense and other federal acquisition regulatioiis and detailed specifications require a much more complex set of delivera olcs within the context of DoD's contracting process a Software Acquisition Practices Re quiremnentsr Definition Commercial - More flexible and open between users and supplier - Based on strategic plan And usually ccost schedule driven - More willing to a djust requiremntrit based on availability of off-the-shelf products - Evolves capability - o Vendor Seleciion C ommnercial - Much more fle xible no requirement for fairness or te maintain the public trust vendais to offer besi solution Pat meet 100% of requirement s Accomm odate teaming and long-term relationships -Eancourzge s - Development Process - Commercial - More flexible product inaprovamentL anticipated -Multi-year acquisitions riot re-justifled each year Yn essence the'rask Force found major differences between DoD and commercial softwvare acquisitkio practices as outlined above and ort thz next page 12 Comparison of DoD and Commercial Software Acquisition Practices Cont Business Practices - Commercial practice more flexible with greater incentives Integration Testing and Delivery Commercial provides integration and functional testing according to need - DoD uses separate test agency with added time and complexity - Absence of beta testing within DOD increases costs - Rights in Data - Commercial more flexible especially regarding resales Maintenance - Commercial Maintenance considered and integrated with development DoD Maintenance not major factor in development process Appendix C piovides a more complete comparison of DoD and commercial software acquisition practices as developed by this Task Force 13 Assessment of CurrentDoD Software Acquisition Approach Strengths - Highly Structured Process Tied to Individual System Developments - Tightly Defined Requirements - Produces High Quality Product for Mission Critical Systems That Demand Extremely Low Failure Rates e g Flight Controls for Man-Rated Platforms The strengths of the current DoD approach to software development acquisition and operation are surmmarized above The Task Force found that the highly structured DoD process has in fact provided a high quality software product in most cases 14 q __________ I Assessment of CurrentDoD Software Acquisition Approach Weaknesses - High Life Cycle Cost in Time and Dollars - Software Development Cycle Tied to Weapon System Developments Incredibly Long Typically 13 to 15 Years from Concept to Fielding - Encourages Excessive Acquisition Agent Involvement in Design Detail and Process - Based on Mistrust vs Mutual Trust Excessive Documentation • Excessive Formal Review Excessive Testing of Non-Critical Systems Poor Communication Between Vendor Acquisition Agent and User - No Requirements Addressing Cost and Schedule - Traditional Approach Used Design it All and Then Build it - Little or No Requirements Relaxation for High Cost Items - Inadequate Beta Testing in Early Phase - Little Focus on Designing in Reusability However there are a number of weaknesses associated with the current defense approach as summarized above Many of these weaknesses derive from the need for a fair and open procurement process and the necessity to prove that public dollars are wisely spent p 15 II r Assessment of Current Commercial Software Acquisition Approach o Strengths - Open Architecture Compatible with Usage of COTS Software - Much Less Formal Competitive Procurement Process Includes Prior Performance as Major Selection Criterion - Trend Toward Joint Assumption of Risks Between Buyers and Suppliers Across Development Operation Maintenance and Modernization Process is More Flexible Shorter Cycle Product Release Times - Tailorable Level of Documentation and Oversight Emphasis on Reuse and Tailoring Requirements to Existing Products Beta Testing Widely Used Weaknesses - Less Responsive to Continual Changes in Requirements - Less Assurance that Software Will Function Properly Under All Situations - Potentially Locked into One Vendor's Proprietary Application L_-- The strengths and weaknesses of the current commercial approach to software development acquisition and operation are summarized above LIt I _L PrincipalReasons DoD Software ProgramsGet Into Trouble Poor Requirements Definition - Lack of User Involvement in Development Process Inability of Users to Foresee Benefits of Automation Without Incremental Capability Inadequate Software Process Management and Control by Contractor Lack of Integratcd Product Teams - Failure to Establish Team With Vendors and Users - Little Participation of Functional Area Experts Ineffective Subcontractor Management Lack of Consistent Attention to Software Process Too Little Attention to Software Architecture Poorly Defined and Inadequately Controlled Interfaces Between Computer Hardware Communications and Software a Assumption That Software Upgrades Can Fix Hardware Deficiencies Without Assessment of Cost and Schedule Risks o Focus on Innovation Rather than Cost and Risk Limited or No Tailoring of Military Specifications Based on Continuing Cost-Benefit Evaluations Over the last two decades a number of DoD software development efforts have gotten into trouble particularly in terms of actual costs and schedule vs expected or predicted costs and schedule The Task Force heard briefings from a number of DoD program managers where this was the case Based on the specific programs discussed and or other inputs the Task Force prepared a listing of the principal reasons DoD software programs have gotten into trouble as shown above Certain of these reasons are a reflection of DoD practice Others are tied to the software engineering level available at the time such programs were initiated The Task Force believes that today's software technology and practices can directly address many of the root causes for such past problems 17 Resource Allocation Where Does the Effort Go Military Commercial Engineering 30% 50% Evaluation 20% 20% Management 15% 10% Meeting Support 15% 5% Documentation 15% 7% • Customer User Support 5% 8% Soujrce Magnavox and CapersJones There are some indications that commercial development efforts have achieved better predictability and lower costs than DoD counterparts The Task Force solicited quantitative data on this issue from both government and commercial software experts The figure above summarizes one type of indicator of the difference between commercial and government projects in terms of the percentage of the effort expended for different aspects of a typical development 181 _ _ _ _ _ Comparison of Commercial and Government Projects Application Type N kboProec Commercial Government Percent Difference 14917 75 Commercial Government Perc_ nt Difference 259 29 Commercial Government Percent Difference Commercial Government Percent Difference Summary of Average Percent Difference Schediile rime 26 000 26 000 12 52 1 ý Effort MoStbs St-iff Months 15 1 17 09% 44 24% 26 000 26 000 17 4 20 9 16 75% 65 7 117 3 43 99% 295 21 212 000 212 000 22 256 14 06% 150 2 242•• 38 16% 56 7 442 000 442 000 I_ Small sample may be statisticallyinvalid Average Size SLOCNew and Modified -_ • _ _ 45 52 13 46% 15 34% f i _ 2736 2740 36 64% 40 76% I Source QSM Inc This figures summarizes other quantitative indicators of the difference between commercial and government projects in terms of the typical size of the code time to develop the application and overall level of effort It should be noted that the Task Force was unable to find reliable quantitative data supporting the notion that commercial practices are more cost-effective than DoD piactices This lack of reliable indicators was a major concern of the Task Force 19 3 0 MAJOR FINDINGS AND RECOMMENDATIONS 3 1 PROCESS CREDIBILITY Process Credibility Findings Attempts to achieve absolute frirness in competition in contracting fair and reasonable pricing have led to a lack of trust between government and individuals contractors - Current DoD practice Is not compatible with commercial business practices is focused on contractor managementlaudit activities and costs Hinders flexibilit managers take no pe - onal risk Is not well suited to procuring complex knowledge-intense first of a kind systems Is too costly Does not prevent malfeasance or incou petence Leads tt adversarial relationships Reduces reuse and contractor incentive to invest because of stringent data rights interpretation - Contractors must make profit on single contract no long term relationship DoD business practices do not view profit as a legitimate cost of doing business - No individual fully understands or owns total process The Task Force spent considerable effort on how the requirement for DoD to ensure public trust in its acquisition process influences DoD's ability to employ commercial best practices The Task Force found that attempts to achieve absolute fairness in competition in contracting have in fact led to a lack of trust between the government and individuals contractors DoD acquisition processes are focused c i contractor audit activities to an extent that hinders the flexibility of Program Managers and contracto i DoD PMs are not incentivized to assume any personal risk associated with allowing for flexibility comparable to commercial practice Criminal sanctions are a significant disincentive for such efforts DoD's acquisition system still does not prevent malfeasance or incompetence and leads to adversarial relationships rather than partnerships which are the norm within commercial industry software developments One problem highlighted by the Task Force was that the current DoD system does not allow one individual or manager to contr Ithe total process even for a specific project This lack of control leads to a diffusion of accountability and hinders DoD's ability to oversee complex first of a kind software developments 2W ii1111U I EII i ilrIiiilii rrfiu ll 1E II U iri nE P Process Credibility Findingi s Requirements and Source Selection Inflexibility - Vendors attempt to meet every requirement Ease of protest Requirements focus vendor on particular solution Requirements have become alternative to profesoional judgment Departures from requirements have caused protests and bad publicity Price Schedule Functionality - In commercial sector managers constrain 2 out of 3 e g cost and schedule In DoD managers constrain 3 out of 3 Constrained Communication During Solicitation - Questions are provided to competitors can give away proprietary concepts Questions are often misinterpreted and answered incorrectly Guarded way of asking questions limits substantive feedback Complicated Regulations - Restrict variety of proposals Restrict competition and limit government options Consume time and resources Drive government employees to follow conservative procedures as safest path inhibits best value Government-Unique Accounting Procedures and Audits - Audit requirements limit available vendors Add substantial cost Create need for separate accounting systems - Drive vendor to lowest labor cost solution rather than best value solution - Lead to rejection of good solutions based on value pricing vs cost-based pricing The viewgraph above lists important hindrances that the Task Force sees with regard to the adoption of commercial software practices Many of the Task Force recommendations address these hindrances Process Credibility m en d ation s SRecom Make necessary changes to acquisition regulations - Have Program Managers Manage 3 of 3 Price Schedule Functionality But Only Constrain 2 of 3 Define Successful Performance on Contracts as Delivering Solution with Predictable Price Schedule and Functionality Not Adherence to Government Processes Procedures and Specifications Do Not Require C-Level Specifications for Software Projects Developed in Ada Prior to RFP Government Should Perform Independent Market Analyses of Off-theShelf and Contractor Products to Assure Best Value Solution Establish Mechanisms to Allow Both Current Ability to Perform as Well as Past Performance as Key Factors in Source Selection Require Source Selection Evaluation of Development Contractots Through a Formal Software Process Capability Evaluation Encourage Offerors to Demonstrate as Much Functionality as Possible as Part of Bid ucinlt mnorg r Knowledgeable Without Eliminating Domain Competition Executable Architecture as a Minimum Weight Heavily in Selection The Task Force makes the above recommendations with regard to process credibility 22 3 2 DOD PROGRAM MANAGEMENT DoD ProgramManagement Findings Program management does not encourage 80% solution for 20% cost Users often not significantly involved in process Little emphasis on life-cycle issues including maintenance and support Existing policies methodologies and procedures and their implementation are inadequate - Little evidence that policies are influenced by actual experience and vice versa - Little effort to measure effectiveness and costs of policy directives Some indication that DoD is migrating toward development and use of standards-based architectures • Program Managers lack incentives to allow tailoring of procedures to fit individual program needs or to develop corporate solution e g employ common architecture or common software components The Task Force found that the DoD program management system itself discourages the use of commercial best practices These findings are not unique to software but are particularly important for software given the significant cost savings associated with software reuse There are few incentives for Program Managers PMs to develop or even employ corporate solutions common architectures software components particularly if they are more expensive to acquire Rather each PM will tend to optimize his her development for its own purpose Further it is difficult for users to be significantly involved until late in a software development process unless some sort of prototype can be constructed DoD PMs place little emphasis on life-cycle issues such as software maintenance and support Existing DoD-wide software policies methodologies and procedures and their implementation by PMs are inadequate There is little evidence that policies are influenced by actual experience and vice versa and there is little effort to measure the effectiveness and costs of policy directives 23 ___ - - DoD ProgramManagement cont fRec ompmendations 0 Establish Overarching Software Life Cycle Guidelines - Tools Methods Define Software Architectures to Enable Rapid Changes and Reuse To Achieve the Benefits of Using Standards-Based Atchitecture DoD Must Manage Programs Using iteratived irlopma 1 of Ithe Man dards eProactiver participatione i development Promote Development Use of Community-Wide Metrics and Models e g SEI's Capability Makturity Model - Acquisition Revise o the Milestones for Software-Intensive Development peeled Asqle Ilti in eedIllar d t pleiminn obiaeld w rp i hardwaeand lerinfe llding baeadon the ban n sewý 1 thespecuifi Require Early Interaction Between User Acquisition Agent and Developer Idatetify and Get Early User Involvement Apply Evolutionary Development with Rapid Deployment of Initial Functional Capability Encourage Competition of Technrical Approach vs Coat Provide Incentives and Guidelines to Encourage Software Reuse Architecture-Based Reuse l Reduce Documentation and Review Requirements for 'Mature' Companies i e Companies Determinted to Be Matur Through Evaluation Mechanisms Tailor operational testing to develop DoD 'Bets Teat' philosophy 9 Allow fielding ofasotware divedlfrom Iwi bedswilth seeirconsent Have Program Manager Stay with Programs at Least Through Beta Testing to Maintain Continuity of Understanding of Origienal Nuances 'in Requirements The Task Force makes the recommendation that DoD establish overarching software lifecycle guidelines directed at facilitating program manager employment of commercial practices and software and that a DoD-wide effort be made to oversee implementation of these guidelines The tools methods and acquisition approaches recommended by the Task Force are listed above 3 3 DOD PERSONNEL DoD Personnel Findings J shortage of sufficiently qualified software personnel A currently exists at all levels within the DoD - Expertise for software acquisitions software evaluations and software maintenance support - Expertise to represent DoD customer interests with commercial sector - Expertise in domain software design and applications - Expertise in software technology to develop policies standards and guidelines - Expertise in software program management L Based on the inputs provided the Task Force found that a shortage of sufficiently qualified software personnel exists at all levels within the DoD Without personnel who are highly qualified in modem software practices DoD will not be as capable of effectively exploiting complex software within its systems This personnel shortage has been a major contributor to the problems that have arisen in past DoD software development programs F DoD Personnel Cont Recommendations Establish DoD-wide software program management education and training initiative - - - Change DSMC and IRMC courses for PMs to reflect best commercial practices and other recommendations of this Task Force and Provide for changes to reflect the dynamics of the software industry Develop and provide interactive training tools for senior managers to perfect software management skills Rotate government and contractor personnel between PM and developer organization to build understanding and trust encourage use of IPA's from industry Incorporate software management principles in senior management education and seminars including senior services colleges Provide mechanisms for keeping software expertise current in the workplace Develop Acquisition Managers with software program management expertise - Integrate software-qualified personnel into senior acquisition staff Establish Norms for the Number of Software Experts in Program Offices Upgrade Educational Requirement for Personnel Assigned to Acquisition Management Development and Oversight of Software Intensive Programs Develop Expertise in Analysis of Domain Software Design - Promote Software Reuse in the Design The Task Force makes the above recommendations with regard to DoD software expertise of its personnel The Task Force strongly recommends an emphasis on increasing the capability of its personnel in modern software practices and techniques 3 4 USE AND INTEGRATION OF COMMERCIAL OFF-THE-SHELF SOFTWARE Use and Integrationof Commercial Off-the-Shelf Software Findings DoD Does Not Normally Perform COTS Market Analyses Incident to Requirements Definition DoD is Beginning to Exploit Evolutionary Development Approaches Tools Methodologies Are Evolving to Facilitate Use of COTS - Computer Aided Software Engineering CASE Object oriented methods Reuse repositories Integrated product teams - Integrated risk management DoD does not routinely perform COTS software market analyses during the requirements definition phase of an acquisition program Nor does it employ prototypes or simulations of new capability in a way that could influence the requirement process Rather requirements are evolved in a manner that is disconnected from the availability of commercial applications and are not typically influenced by such available capability This situation is true for both hardware and software Today's technology can facilitate early interaction between users and available capability particularly in software Further DoD is beginning to exploit evolutionary software development approaches that could provide for the inclusion of existing commercial software functionality into early prototypes and allow users to test such capabilities Modem software tools and metho -dolo -ies have evolved that •fa•cilitau use of COTS such as those listed above 27 use and Integration of ommercial Off-the-Shelf Software co Findings Cont 0 DoD has not fully identified the pros and cons associated with the use of COTS - Pros - Saves money in dsulgn and development of those components Can Significantly Reduce Development Risk Support If Comoem S Selection of COTS products early in project cycle will enable requirements to be driven by comnmercial software capabilities Very useful in rapid prototyping - QnL Commercial products must still be integrated into system and qualified with system Difficulty in Configurement Management and Support for Older Releases Additional testing may be required to qualify system widi commercial comipenents Subsequent releases determined by vendor not DoD Security Aspects of Use of COTS Not Well Understood Dol not addreuslng multiple aapects OD vlpement ssvilmeme1t - - Tdkactl crmpler prsgrp VI e prolet -C ftia l UPI-P Cau l•fled software System problem for commrcil Companies Iin addition DoD has not determined ihJnIto use COTS - Commercial Off-the-Shelf Software Products May Not Apply to All DoD Systems Should be used as is- avoid tailoring or special features Most weapon system real-time application software for DoD will not exclusively be custom but will involve some COTS In general DoD has not identified the pros and cons associated with the use of COTS DoD must learn how to balance the cost savings associated with design and development of commercial software products and the significantly reduced development risk with the concern for longer term support and system security Commercial products must still be integrated into and qualified within each defense system and there is difficulty in configuration management and support for older releases The latter point is important since many DoD software systems are not deployed before a commercial software product is retired or replaced DoD is not addressing many aspects of the use of COTS software - Development environments - Virus protection Commercial espionage In essence DoD has not developed a corporate way to decide when to use COTS software Commercial off-the-shelf software products do not apply to all DoD systems Most weapon system realtime application software for DoD will not exclusively be off-the-shelf Integration and configuration control for COTS software then become important concerns Further DoD must learn to use COTS so that it avoids tailoring or special features 28 Use and Integrationo Commercial Off-the-Shelf Software Recommendations ti Require Trade Studies and Analysis of tt-e Use of COTS in DoD's Software Acquisition Process Where Effective - - Performed by Acquiring Organ Uation as Essential Part of Defining Requirementa and in Rapid Prototyping Situations Employ broad Agency Announcements or Similar Contractual Approach to Facilitate Such Sludies Use of COTS Appropriate When De•lining Requirements Rapid Plrotyplng Sluatlov - Not Required to Tailor CO0S Source Code to Application - Not Required to be Erfnr-Free i COTS Soltware is Cloe Enough' to Tailor Requirements Establish Customer Friendly Application-Specific Information Technology Component Stores - Generic Arnhiteclure for Specific Domains Rapid Requirements Definition Proce n Protolyping Reusable Prequalifled Components Assemble Systems Rather than Develop Thervm Reduce Lead Time Security is not Paramount Increase tech base funding for security audit tools for systems employing COTS Capitalize on Innovative Cost-Effective Techniques for Acquiring and Using CCTS Software Products -Such as Use of Enterprise Licenses Given these findings the Task Force makes the above recommendations with regard to DoD use and integration of commercial off-the-shelf software The Task Force sees great benefit to be gained through exploitation of COTS software however DoD must develop corporate approaches to the use and integration of COTS software if it is to gain this benefit 2 3 5 ACQUISITION Acquisition F Findings - Acquisition practices have led to Distance between user and developer ý Limited participation by commercial software companies - Adherence to DoD regulations for reviews and documentation is increasing software costs DoD software costs are estimated to be increased by at least 20 % over commercial best practice ýCommercial best practice requires much less documentation than DoD - Funding for maintenance planning execution starts late - Maintenance assumed organic inhibits teamirig partnerships - Acquisition focus is on mandatory how to specifications and standards rather than the product what ' Lengthens process and adds costs Discourages harmonization with commercial practice Creates adversarial relationship •- Acquisition process does not reward development of reusable software j The Task Force was concerned that the specific acquisition and contracting approaches used by DoD inhibit use of commercial practices and software The Task Force findings in this area are shown above Strict adherence to DoD regulations for reviews and documentation is increasing software costs DoD software costs are estimated to be increased by at least 20% over commercial best practices The DoD focus on detailed technical specifications has lengthened the process added costs discouraged harmonization with commercial practice and created a highly adversarial relationship between the Government and industry There is a strong belief that certain commercial companies or divisions of companies have opted to stay away from government contracts due to the complexity of the acquisition rules and regulations 3J Recommendations-Acquisition Recommendations - Implement the following acquisition approach Establish acquisition focus on functionality and consistency with commercial best practice Revise procedures encouraging interaction between user and developer and achieving early functionality Minimize DoD regulations for review and documentation that are different than commercial best practice Require planning for maintenance at beginning of development process Provide government funded vehicle in contracts to incentivize development of reusable SW - Review all existing military standards and military specifications pertaining to software development and documentation for continued applicability such as DoD-STD 2167 coni j To this end the Task Force makes the above recommendations with regard to DoD software acquisition practices In esscnce these recommendations can move DoD toward an acquisition approach that is more consistent with commercial approaches while not requiring changes in statutes 31 Proposed Software Life Cycle Pre-Prososal Demos Proposal Demo Simulation Capability Market Analysis valuations USER •Evaluations •Site Version 1 0 Basic Functions Release to Beta then Users Version 2 0 Improved Functions rchltecture S Version 3 0 Improved Functions To implement this recommendation the Task Force proposed that DoD evolve a more appropriate software life cycle approach as depicted above This approach provides for early capability even in the initial bidding process and for a gradual enhancement in capability over time 32 Acquisition Approach Developing the RFP Contract Proposal RFP Response SQualified Software Organization -aag Go CDevelopment -Proc Integrated Top Level Specification Perforancen Environmnent 0 Domain Experience Environment Process Control Peer Reviews Metrics and Milestone Plan Reuse Plan o Program Managment Plan Proposal Demonstration Test Validation Selection e Acquiring Agency - Evaluate Technical Solution and Management Plan - Evaluate SW Process Maturity and Past Performance Users - Define Requirements - Evaluate Demmnntxations - H-eavy Participation in Selection The Task Force proposed approach to competition is shown in the above figure The core government role in such an approach would be to Develop the RFP no How To statement of work no management CDRLs no government inprocess approvals Provide an integrated Top Level specification architecture COTS reuse software engineering environment and test validation approach Employ software metrics as a key determinant Evaluate proposed technical solutions and the proposed management plan The contractor would then Provide an execution plan manageme nt controls and progress - ilestones etrics SDescribe an in-place mature software development organization and relevant domain experience Provide a skills matrix describing personnel to be employed Identify a robust development environment and describe applicable prior experience Describe automated process control software 0 Describe the extent to which peer inspections will be used Provide a metrics usage plan and purposes for which they will be used • Provide specific reuse and program management plans Propose specific architecture s in executable code Fýr 33 3 6 SOFTWARE ARCHITECTURE Software System Architecture The Missing Link What is software architecture Software architecture consists of - Software system components - The relationships among those components - Rules for their composition constraints Defining document would address - System functionality - Software system components - Interfaces standards protocols - Execution model - Data flows - Control flow - Critical timing throughput aspects - Error handling Software architecture was emphasized by the Task Force as a means for achieving important ends Software architecture consists of software system components the relationships among these components and the rules for their composition constraints To use architecture as a management tool DoD needs to define system functionality along with software system components to be employed interfaces standards and protocols to be employed and the execution model to include data flows control flow critical timing throughput aspects and error handling approach 31 Software System Architecture Cont Findings Why is it important - Essential for effective management over the lifecycle Software system lifecycle costs - 65% for maintenance 65% of maintenance costs are due to changes modifications upgrades - Software architecture is a prime enabler of flexibility and reuse - Well-formulated architecture might reduce costs of changes upgrades by 3050% $4-$7B year assuming software expense - $30B year Why doesn't it play a larger role - Focus is usually on initial cost schedule functionality - not lifecycle - 2167A reinforces this approach - requires proof that design satisfies functionality - Difficult to specify test etc - Not well understood Software architecture is a prime enabler of flexibility and reuse and a well-formulated architecture might reduce costs of changes upgrades by 30-50% $4-$7B year assuming software expense -$30B year Software architecture has not been emphasized because PM focus has usually been on initial cost schedule and functionality and not on the life cycle DoD-STD-2167A reinforces this approach by requiring only proof that a design satisfies the required functionality S Im Software System Architecture Cont r Findings Cont Little emphasis on architecture in DoD software programs or regulations Impact of software architecture issue - DoD not benefitting from architecture as a key tool for Evolutionary development Early and often involvement of users with functional capability Ability to include changing commercial technology Reuse Facilitating requirements change with minimum cost and schedule Facilitating product line management Insufficient Progress in - Developing models standards for domain specific software architectures - Open system architectures work There is currently little emphasis on architecture in DoD directives or regulations As a result DoD is not benefiting from architecture as a management tool Further the Task Force sees insufficient progress in developing models and standards for domain specific software architectures -- - 35 Software System Architecture Cont Recommendations Emphasize Use of Software Architecture - Establish model and context for architecture selection • Standards-based with emphasis on implementable Require vendors to propose manage and control the architecture - Require delivwry of software architecture definition as first step in any software acquisition - Foster migration strategies at architecture level Assign responsibility within Government for domain analysis and product line developments e Provide expertise and resources to ensure coordinated DoD participation in commercial international standards bodies and users groups The Task Force makes the above recommendations in order to facilitate greater use of software architecture as a management tool for DoD software programs and activities In particular the Task Force sees great value in requiring the delivery of software architecture definition as a first step in any software acquisition Where possible such software architecture definition should be operational i e executable -- I 3 7 SOFTWARE TECHNOLOGY BASE Software Technology Base Findings e The current DoD software technology base does not adequately take advantage of commercial R D Software technology transfer internal and external is not receiving adequate emphasis within DoD There is a paucity of data to support prediction of cost schedule and performance The software technology base both defense and commercial provides DoD with ample opportunity for significantly improving the defense software acquisition life-cycle II Software Technology Base Cont C Recommendations Provide for the evolution of the DoD Software Technology Strategy to alig i with emerging commercial technology and practices Emphasize Technology Transfer External and Internal - Fund technology transfer programs in such topics as Architectural principles Architecture description languages Standard interfaces and Integration technologies - Initiate demonstration programs e g ATDs to facilitate software technology insertion into systems Examples of candidate criteria 'Open standards Use of COTS and GOTS Frequent releases to include a number of users Multiple platforms Satisfies commercial standards and interoperability standards across DoD organizations a Initiate formal data collection and analysis The Task Force makes the above recommendations with regard to DoD software technology base investments The Task Force supports a DoD technology base program that is more closely aligned with the wide range of similar efforts ongoing in commercial organizations __ __ 3 4 0 SUMMARY 4 1 ATTA PERSONS 'Atta Persons Electronic Systems Command Software by Assembly of Modules - Software Component Store A-109 Acquisition Methodology Use of Commercial Practices and Products in Reserve Component Automation System RCAS Software Reuse Prototypes and User Involvement of Global Command and Control System GCCS Initiative Peiry-Paige Initiative Toward Migration Systems The Task Force identified several ongoing efforts worthy of note The Air Force Electronic Systems Command ESC has defined an approach for software system development through the assembly of pre-qualified software modules PRISM ESC is pursuing the development of such software modules The Task Force was veiy supportive of this program Office of Management and Budget OMB Circular A-109 outlines an approach for major Federal system acquisitions that encourages definition of top level needs vs detailed specifications exploitation of innovative private sector contributions and use of early competitive demonstrations of competing approaches These all are acquisition attributes recommended by the Task Force The Reserve Component Automation System RCAS is an ongoing MIS development to support the reserves The program has successfully employed the A-109 acquisition approach and extensively used commercial acquisition practices and products The Task Force commends this approach The Global Command and Control System GCC S is an initiative of the Joint Staff J3 and J6 to provide vertical and horizontal interoperability of combat information systems across Services Combatant Commands and Agencies It was highlighted for its software reuse approach its use of prototypes and its emphasis on operational user involvement The Perry-Paige migration systems initiative has established a focus on selecting a set of target computing systems including MIS C31 and embedded towards which DoD will aim This migration strategy will enable a more cost-effective DoD investment in software across the lifecycle 4 2 OVERARCHING RECOMMENDATION OverarchingRecommendation I SECDEF Assign USD A T the Responsibility for DoD-Wide Software Technology Policy Practices and Acquisition This Responsibility Includes Allocating Resources and Assigning Responsibility for - Drafting and Institutionalizing a New Software Acquisition Policy That Includes the Recommendations of this DSB Task Force - Creating Incentives and Ensuring Compliance to the Policy - Creating Terms and Conditions and Financial Rewards to Maximize the Ability of DoD to Realize the Benefits of Commercial Best Practices - Requiring Source Selection Evaluation of Development Contractors Through a Formal Software Process Capability Evaluation - Advising the Acquisition Executive on Matters Concerning Software Technology Acquisition and Architecture for Major Programs - Maintaining a Digest of Lessons Learned and Best Practices and Communicating the Same to Program Managers and Contractors Throughout its deliberations the Task Force acknowledged that the issues associated with defense software were applicable across the spectrum of DoD software intensive systems The Task Force also frequently learned of obstacles based in part on the current dichotomous DoD structure associated with software technology policy practices and acquisition In order to ensure that the DoD reap maximum benefit from its recommendations the Task Force formulated the above overarching recommendation F 41 • Ia OverarchingRecommendation Cont In Carrying Out This Responsibility Consider Forming an Executive Council - Including the DDR E the ASD C31 and Appropriate Representatives from the Services and Defense Agencies - Provide Supporting Process Action Team to Assist in Implementation The Task Force also identified the mechanism by which its overarching recommendation could be readily implemented APPENDIX A TERMS OF REFERENCE THE UNDER SECRETARY OF DEFENSE WASHINGTON DC 20301-3000 06 lDG1993 ACQUISITION MEMORANDUM FOR CHAIRMAN SUBJECT DEFENSE SCIENCE BOARD Terms of Reference--Defense Science Board Task Force on Acquiring Defense Software Commercially You are requested to form a Defense Science Board Task Force on Acquiring Defense Software Commercially Determine the conditions under which the procurement of defense software i e operational software support software and software tools can appropriately use commercial practices and develop a strategy for defense software procurement that substantially incorporates such practices Specifically address within this strategy DoD use of commercial Loftware products The scope of this effort should include all DoD systems that are software intensive It should address all stages in the life cycle of a software component from initial procurement to evolutionary upgrade of software or of software hardware combinations This commercially-based strategy should not be constrained by existing DoD standards it should be viewed as a coexisting alternative to rather than replacement of the current DoD procurement strategy Accordingly you should compare your proposed strategy to the current DoD strategy to indicate circumstances in which each strategy is most beneficial To assess the appropriateness of DoD use of commercial practices the Task Force should identify and apply objective measures such as elapsed development time software life-cycle cost management risk as well as measures of software product quality The Task Force should consider at least the following topics • Technical state-of-the-art and best commercial practices development tools reusable software components and techniques and tools for tailoring commercial software components for use in defense systems Management DoD management of the commercial development process software risk management techniques and supporting tools minimum delivery time affordability maintenance after product delivery post-deployment product enhancement software process support tools quality and assured availability and use of development and maintenance tools FN Contracting technical data rights property rights liability alternative agreements and incentives for creation components as well as their subsequent intellectual forms of procurement of reusable software reuse The Director Defense Research and Engineering will sponsor this study Dr George H Heilmeier and Dr Larry E Druffel will serve as Co-Chairmen The Office of the Director Defense Research and Engineering will provide the necessary funding and support contractor arrangements The Executive Secretary will be Ms Virginia L Castor Commander Robert C Hardee will be the Defense Science Board Secretariat representative It is not anticipated that this Task Force will need to go into any particular matters within the meaning of Section 208 of Title 18 U S Code nor will it cause any member to be placed in the position of acting as a procurement official jJohn Mi Deutch 92 ng zir tl u - i I fruit WW APPENDIX BRIEFINGS PROVIDED TO THE TASK FORCE Briefings Provided to the Task Force Joint Staff Views on How DoD Can Adopt Commercial Software Practices Army Views on How DoD Can Adopt Commercial Software Practice MG David Kelley Vice Director J6 Naval Views on How DoD Can Adopt Commercial Software Practice RADM John Hekman Commander NISMC Air Force Views on How DoD Can Adopt Commerciai Software Practices DISA Views on How DoD Can Adopt Commercial Software Practices Mr Lloyd Mosemann Deputy Assistant Secretary of Air Force Communication Computers Support Systems Dr Mark Scher Director of Infrastructures Defense Information Systems Agency Mr Gene Porter Director Acquisition Program Integration Mr Harry Pontius Director C3 1 Policy DoD 5000 Series Regulations DoD 8000 Series Regulations 4 LTG Peter Kind Director Information Systems for C Data Modeling vs Data Standards Dr Singh Senior Manager Space Naval Warfare Systems Command Dr Jack Ferguson Program Manager Software Engineering Institute LTG Peter Kind Director Information Systems for C4 Case Study B2 Case Study Ensemble of Real-time Software Systems Mr Fred Schwartz Director of Engineering MajGen Israel Director Defense Airborne Reconnaissancce DoD-STD-2167AIMIL-STD-498 Draft White Paper on Software Acquisition Office Case Study Reserve Component Automation System RCAS Case Study AEGIS Case Study PRISM Case Study BMC3 Boeing Views on DoD vs Commercial Software Practices MG Gary Stemley Program Manager for RCAS CAPT Richard Cassidy AEGIS Technical Director Mr Robert Kent ESC ENS Lt Col Robert Phelps BMDO Mr John Hanson Director Systems Software Engineering Boeing McDonnell Douglas Views on DoD vs Commercial Software Practices IBM Views on DoD vs Commercial Software Practices Object Techno ogy and a COTS Product Called SNAP Procurement Regulations Issues ntegrated Computer Aided Software Engineering ICASE Program Computer Sciences Corporation Case Study on Commercial Software Development Commercial Software Acquisition Practices Experiences in DoD Software Maintenance MITRE Corporation Experiences in Exploiting Commercial Practices FAA Study on the Use of Commercial Software NSIA Study on the Use of COTS Software DoD Software Reuse Initiative MITRE Study on Feasibility of Using a Software Acquisition Maturity Model in DoD Software Management Metrics and Reliability Innovative rechniques in DoD SW Acquisition F-22 Integrated Mr L George Hite Senior Manager McDonnell Douglas Mr Dave Coyer Program Manager IBM Federal Systems Company Mr Joe Fox Chairman Template Software Inc Mrs Eleanor Spector Director Defense Procurement OUSD A T Col Gary Case ICASE System Program Director Mr Daniel Kemp Senior Partner Computer Sciences Corp ___ Mr Alex Monow Geaeral Manager Lotus Development Corp Mr Bobby McDonald Deputy Director Electronic Warfare Directorate Warner Robins ALC Mr Steven Crisp Dept Head MITRE Corp Mr John Stenbit Vice President General Manager TRW 3Systems Integration Group Gen William Richardson USA Ret Executive Vice President Army Programs Burdeshaw Associates Lttd and Ms Linda Connor Program Manager Rockwell International Ms Linda Brown ODASD IM lnformation Technology Ms Judith Clapp Division Assitant The MITRE Corp Mr Douglas Putnam and Mr Lawrence Putnam Jr Quantitative Software Management Inc Colonel Robert Kayuha Product Team Approach Legal Impediments to Acquiring Defense Software Commercially Mr Robert Gorman OSD General Counsel B-i APPENDIX COMPARISON OF SOFTWARE ACQUISITION METHODS Comparison of Software Acquisition Methodsl Requirements Definition Best Commercial Practice Requirements based on strategic plan and market analysis Requirements based on life-cycle resource constraints Detailed requirements generated by interdisciplinary team including users domain experts and system engineers Buyer user and vendor are a team Attitude cf partnership trust and cooperation Presumption of trustworthiness for reputable commercial organizations Functional specification is modified by knowledge of availability of existing products Vendors involved early in study analysis and prototyping with emphasis on reuse and evolution of existing systems Need is based on business case and decisions are based on return on investment time to make Efficient decision processes Level of documentation is negotiable based on individual user needs and complexity of system being developed More detailed analysis of cost versus feature Dropping lower value higher cost options or reducing requirements is practiced More requirements trade-off decisions involving complexity and schedule for reduced time to field Selected vendors may assist in preparing specifications Tools used to create system models for use in requirements definition e g GUI Building Flexibility allowed in choice of programming lang uage Evolutionary and incremental approach favored Current DoD Practice Requirements based on using command Mission Need Statement master plans and top-level certification Requirements based largely on annual budget resource constraints Detailed requirements generated by buyer in collaboration with user Team generally includes domain experts and acquisition personnel Us vs them mentality about contractors Government thinks in terms of control accountability detailed auditing and double checking Presumption that contractors cannot be trusted Functional and or performance specification little to no regard for existing products May contract for prototypes but contractor involvement in pre-award discussions is discouraged Need based on Mission Need Statement decisions based on need economics and politics time to field Decision processes formal and time-consuming Extensive often redundant or unnecessary documentation required under 2167A Tailoring of documentation requirements is often minimal or discouraged Little or no requirements reductions on high cost items -__ Very little flexibility to trade-off requirements creep versus complexity and schedule Vendors not involved in preparing specifications Requirements definition based on Mission Need Statement Specific requirements regarding use of programming language e g CMS-2 etc Requirements defined up fi'ont with little flexibility for modifications Summary Commercial is more flexible and open between users and suppliers and requirements are based on a strategic plan In the commercial world there is more willingness to adjust requirements based on availability of products and thus to filed a system sooner and evolve it to include more capability at significant cost savings 1 This appendix was initially derived from a White Paper on software acquisition methods prepared by the Software Engineering Institute The resulting content represents the consensus of this Task Force 02 C- I Vendor Selection Best Commercial Practice Solicit multiple but not all qualified vendors -a selected few Fricourage teaming with a view to attaining a long term relationship that covers the entire life cycle and fosters ttade-offs in cost and schedule Compare vendor history and experience Maintain long-term relationships The organization that will be responsible for a system over its full life cycle is heavily involved from the beginning Use site visits and demonstrations to gain knowledge of vendor capabilities Negotiate for best values based on 1 trade-offs of costs and requirements licensing and 2 consideration of both vendor and buyer best interests Overall goals 1 obtain product at reasonable cost as soon as possible and 2 achieve the business case for the system Relatively few review and approval steps once vendor is selected Past performance weighted heavily sometimes primary factor in selection process More flexibility in vendor selection based on metrics and overall assessment Modifications made as procurement proceeds in order to get best results Current DoD Practice Solicit all possible vendors Vendor proposals must meet 100% of requirements Teaming seldom encouraged development and maintenance usually separate entities Can compare previous performance but normally can't have lon-term relationships Maintenance organization not usually involved in vendor selection process Site visit only by capability evaluation team or other expert teams Visits are very structured Negotiation based on lowest cost and shortest schedules endangering maturity of finished product Overall goal Obtain lowest cost product that rigorously meets all requirements but be fair Review and approval process more structured and complex once vendor selected Past performance considered but only as a minor factor Selection of vendor forced by use of pre-defined metrics for proposal evaluation Change difficult once process begins Summary Very different processes with commercial much more flexible but with no requirement for fairness or to maintain the public trust Commercial encourages vendors to offer best solution but solution may not meet 100% of the requirements Teaming and long-term relationships are more easily accommodated by RAM VE C-2 Development Process Best Commercial Practice Vendor often tailors existing systems and uses COTS System designed to fit in a defined product or product line architecture Current DoD Practice Varies with application Some systems use COTS However usually a new system that doesn't reuse legacy software Unique systems are built with little regard for architecture Buyer may have heavy involvement in design and Formal structured spiral or waterfall model Buyer development as part of the team Integrated Product oversees but team approach is not usually Development team emphasized Reviews typically informal and stress progress Reviews usually very formal and include technical against goals design details in addition to progress metrics Buyer actively involved as co-participant in Micro management of technical details management of technical details Heavy user involvement Limited user involvement Heavy buyer involvement Vendor embraces one or more industry standards Government and industry standards called out Not which improves interface and integration with COTS all government standards enhanced by COTS products products Buyer requirements may be translated to more Tailored sysieni little if any focus on designing in general purpose requirement for potential reusable code software reuse Management reviews and degree of oversight are Notably more detailed reviews and oversight commensurate with size and risk of program performed Prototyping common with joint applicatioDs Prototyping seldom used development teams user and developer working to clarify requirements and incorporate new requirements that do not affect cost or schedule Use of flexible architectures allows insertion and Ue of MIL standard computers and legacy plug and play of COTS products iustruction set architectures restricts new _____________________________________________devolopnient _____________ Summary Commercial more flexible with likelihood of a team approach and iG biases towa'd reuse and tailoring of existing systems Multi-year acquisitions not re-justified each year Product in xrovements are anticipated C- 0-3- Business Practices Best Commercial Practice Informal contracting joint ventures partnerships with mutual economic benefit and vested interest in success Oversight built on established relationships Can hire and fire vendors and managers Multi-year effort and funding Current DoD Practice Difficult to write contract to motivate contractors to reduce cost expensive to terminate contractors Burdensome cost accounting procedures required extensive oversight reporting and documentation requirements Government personnel regulations policies and practices determine qualifications of its managers rotations of assignment and training Multi-year effort Yearly unpredictable funding Summary Commercial practice more flexible with greater incentives Integration Teoring and Delivery Best Commercial Practice Unless system is for a new plant then there are major cut-over issues Sometimes difficult to assemble complete system in laboratory environment due to cost Current DoD Practice Similar cut-over or transition issues Usually integrate system in laboratory prior to operational testing Development testing vs operational testing via statutory mandate Beta testing widely used to quickly find errors Little beta testing Ultimate acceptance authority rests with buyer not a Structured specified operational testing conducted separate organization by separate authority Acceptance authority rests with buyer Buyer user vendor are a team Emphasis on DoD as oversight and approval authority Summary Integration and functional testing seem appropriate to the need DoD use of separate test agency adds time and complexity Absence of beta testing increases cost to DoD G-4 Rights in Data Best Commercial Practice Current DoD Practice If custom development buyer gets all rights but Specified by contract vendor may retain right to subsequent sales Government usually demands all rights for government use due to organic support and maintenance needs and competition via statutory mandate If tailored version of standard system buyer only Same as commercial gets rights to tailored parts proprietary material May have exceptions for Summary Similar but commercial is a little more flexible especially regarding resales Maintenance Beat Commercial Practice Current DoD Practice Organic support shifting to outsourcing or vendor Organic support with reluctance to be dependent on vendor Use of depot maintenance makes interoperability issues more manageable Also must be responsive to user for critical systems Level of discrepancy reporting required is based on user needs Problem resolution usually delayed until next major release of software Buyer has limited power to implement fast resolution of problems Bureaucratic and paper-intensive discrepancy repolting and change control board often imposed Award fees may be tied to problem workoff resulting in fast resolution of problems COTS solutions take advantage of economics of Unique software requires custom maintenance all scale since maintenance costs are leveraged across borne by the single buyer multiple buyers _ Summary The DoD and industry currently have different requirements and trends are moving apart However DoD is currently reevaluating its practices for hardware systems and perhaps should also reevaluate for software on APPENDIX D PROPOSED ACTION PLAN FROM SERVICE SOFTWARE EXECUTIVE OFFICIALS S--- I_ _ -- - - --- ----- -- r - DEPARTMENT OF THE ARMY OFFICE OF THE SECRETARY OF THE ARMY 107 ARMY PENTAGON WASHINGTON DC 20310-0107 Office Direcor ol Information Systems for Command Control Communications Comrputer MEMORANDUM FOR THE EXECUTIVE SECRETARY DEFENSE SCIENCE BOARD TASK FORCE ON DOD ACQUISITION OF COMMERCIAL SOFTWARE SUBJECT Proposed Action Plan with Regard to Common Changes Proposed by DoD Speakers to Revise the Software Acquisition Process The DSB Task Force asked us to provide a list of common recommended changes needed to revise DoD's software acquisition process Our recommended list of proposed changes is provided in the enclosure to this memorandum As we worked together to develop this list we found it useful to categorize the recommended changes into five major categories Several of our recommendations did not receive adequate emphasis in our presentations yet are important to the process we are studying We feel that if OSD is serious about addressing Software Acquisition issues a single senior office with primary responsibility for effecting the reforms proposed by the DSB will be essential Further such an official or office must receive from the heart sustainment and backing from the highest levels within OSD The enclosed list addresses our concerns with regards to the questions before the DSB We hope that this will be beneficial to the DSB Task Force and we all look forward to continued active participation in this effo N il XS Deputy Assistant Secretary Communications Computer and Support Systems PE ER A KIND Lieutenant General GS Director of Information Systems for Command Control Communications and Computers Date Date 4 Feb 94 C _ ___________ A OH G HEKMAN RearOAdmiral SC USN Commander Naval Information Systems Center 4 Feb 94 Date Print i oni Fiumy Papov JOINT RECOMMENDED PROPOSED ACTIONS I DESIGNATE SINGLE COMMON DOD SOFTWARE MANAGEMENr OFFICIAL ACTION OFFICE USD A ESTABLISH DEPUTY ASSISTANT SECRETARY OF DEFENSE SOFTVARE B PROVIDE SUFFICIENT RESOURCES FOR MISSION II SOFTWARE ACQUISITION AND LIFE-CYCLE MANAGEMENT ACTION OFFICE NEW DASD A PROMOTE REUSE BASED ON DOMAIN MANAGEMENT 1 Define domains of interest or areas of business 2 Assign respunsibilities to manage the domains 3 Establish and manage commoni architectures within the domains B REDUCE MONITORING OF MATURE COMPANIES 1 Identify criteria for assessing determine process maturity 2 Consider maturity in source selection 3 Allow sole source extension for high quality vendors C PROMOTE GOVERNMENT INDUSTRY TEAMING 1 Reduce Documentation Requirements to a minimum 2 Provide for electronic delivery evaluation exchange D MANAGE RISK 1 Mandate the use of market research 2 Use metrics effectively for management 3 Develop Risk Management Disciplines 4 Stick with a winner punish poor perforners ITPOLICY AND STANDArDS A ADOPT COMMERCIAL STANDARDS ACTION OFFICE ASD C31 l DoD invests in commercial standards developments 2 Change FAR DFAR and SD-I as appropriate B REVISE THE MILESTONES FOR SOF-TWARE-INTENSIVE DEVELOPMENTS ACTION OFFICE USD A T ASD C31 1 Address the need for a software-first philosophy 2 Provide for a layered software hardware standards based architecture VI_ ' DEVELOP DOD BETA TEST PHILOSOPHY ACTION OFFICE OSD T E 1 Team with Universities other appropriate activities 2 Allow fielding cf software direct fozom test beds with user consent IV PERSONNEL A DEVELOP SOFTWARE ACQUISITION MANAGERS ACTION OFFICE USD A T 1 Provide a career path for software Mnanagers 2 Integrate software personnel into senior acquisition staff 13 PROVIDE SOFTWARE EDUCATION FOR SENIOR M kNAGERS ACTION OFFICE USD A T 1 Develop DoD Senior Software Managers Course 2 Develop and provide interactive trjaiiing tools fur senior managers to pei fect software management skills C PUBLISH AND PROMOTE BEST PRACTICES HANDBOOK ACTION OFFICE OSD T E D ENSURE DOMAIN EXPERTISE ACTION OFFICE MII-TARY DEPARTMvIENTS AGENCIES _ V SOFTWARE TECHNOLOGY BASE AN D TRANSITION ACTION OFFICE DDR E A PROVIDE FOR SOFTWARE TECHINOLOGY INSERTION INTO SYSTEM ACQUISITION B INVEST IN THE DOD SOFTWARE TECHNOLOGY STRATEGY C PROVIDE FOR THE EVOLUTION OF THE DOD SOFTWARE TECHNOLOGY STRATEGY TO CAPTURE EMERGING COMMiERCIAL PRACTICES mom