United States General Accounting Office GAO Report to Congressional Requesters March 1999 YEAR 2000 COMPUTING CRISIS Customs Has Established Effective Year 2000 Program Controls GAO AIMD-99-37 GAO United States General Accounting Office Washington D C 20548 Leter Accounting and Information Management Division B-281520 Letter March 29 1999 The Honorable Amo Houghton Chairman Subcommittee on Oversight Committee on Ways and Means United States House of Representatives The Honorable Philip M Crane Chairman Subcommittee on Trade Committee on Ways and Means United States House of Representatives This report responds to your requests that we evaluate the United States Customs Service's efforts to address its Year 2000 computing problem 1 If key automated systems affecting trade between the United States and other countries valued at over a trillion dollars a year malfunction trade processing could be delayed trade revenue lost and illegal activities such as narcotics smuggling money laundering and commercial fraud could increase The objective of our review was to determine whether Customs has established effective management structures and processes for managing and reporting on key aspects of its Year 2000 program We performed our work at Customs headquarters in Washington D C and the Newington Data Center in Newington Virginia from July 1998 through January 1999 in accordance with generally accepted government auditing standards In brief we analyzed Customs' Year 2000 program management against our Year 2000 guidance2 to determine whether key management controls were in place and functioning and we traced the reported status of selected system components back to Customs' documentation to verify the status information's accuracy Appendix I provides details on our objective 1The problem is rooted in the way dates are recorded and computed in automated information systems For the past several decades systems have typically used two digits to represent the year such as 97 representing 1997 in order to conserve electronic data storage and reduce operating costs With this two-digit format however the year 2000 is indistinguishable from 1900 or 2001 from 1901 As a result of this ambiguity system and application programs that use dates to perform calculations comparisons or sorting may generate incorrect results 2 Year 2000 Computing Crisis An Assessment Guide GAO AIMD-10 1 14 issued as an exposure draft in February 1997 issued final in September 1997 Year 2000 Computing Crisis Business Continuity and Contingency Planning GAO AIMD-10 1 19 issued as an exposure draft in March 1998 issued final in August 1998 and Year 2000 Computing Crisis A Testing Guide GAO AIMD-10 1 21 issued as an exposure draft in June 1998 issued final in November 1998 Page 1 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 scope and methodology We requested comments on a draft of this report from the Commissioner of Customs or his designee The Commissioner provided us with written comments which are discussed in the “Agency Comments” section of this report Results in Brief Customs has established effective Year 2000 program management controls including structures and processes for Year 2000 testing contingency planning and Year 2000 status reporting As a result the agency's latest status reports to the Department of the Treasury show good progress in converting its systems and mitigating century date change risks to its core business operations Specifically as of January 1999 Customs has met milestones recommended by the Office of Management and Budget OMB for renovating and validating most of its mission-critical systems 3 Also Customs has actions underway and plans and management controls in place to help ensure that it completes remaining validation and implementation activities for all its mission-critical systems by June 1999 Very important tasks remain to be accomplished such as completing endto-end tests and validating contingency plans for ensuring continuity of core business functions and serious risks outside of Customs' control remain such as Year 2000-induced failures of both public infrastructure and business partner systems Customs has plans in place for completing key tasks and addressing external risks and it has the management controls in place to ensure that they are accomplished While these controls do not guarantee that Year 2000-induced system failures will not occur if Customs follows through on its plans and continues to implement its management controls as it has to date its risk of Year 2000-induced business failures will be effectively reduced Background Customs' mission is to ensure that all goods and persons entering and exiting the United States do so in compliance with all U S laws and regulations It does this by 1 enforcing the laws governing the flow of goods and persons across the borders of the United States and 2 assessing and collecting duties taxes and fees on imported merchandise To accomplish these goals Customs has identified and organized its 3 OMB requires that agencies complete renovation of their systems by September 1998 validation by January 1999 and implementation by March 1999 Page 2 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 operations around six core business missions--trade compliance passenger outbound finance human resources and enforcement investigations See appendix II for a description of these core business missions To carry out its responsibilities Customs relies extensively on information technology For example Customs depends upon the Automated Commercial System as its primary vehicle for tracking controlling and processing all commercial goods imported into the United States In addition Customs uses cargo inspection systems and x-ray systems to screen and inspect cargo for narcotics and other contraband Since every computer-controlled system may be vulnerable to Year 2000 failure Customs' Year 2000 problem extends to all of its systems Customs' Mission-Critical Systems A Brief Description In managing its Year 2000 program Customs divided its mission-critical systems into information technology IT and non-IT systems e g office equipment facilities security systems and vehicles Customs has five IT systems that run mission-critical applications the Automated Commercial System ACS Treasury Enforcement Communications System TECS Administrative System ADMIN Automated Export System AES and Seized Asset and Case Tracking System SEACATS These five systems are mainframe-based and are accessed by users around the country from terminals or personal computers PC emulating terminals 4 Customs’ IT systems also include its telecommunication systems See appendix III for a description of Customs' IT and non-IT systems Customs Reports That Y2K Remediation of Most Mission-Critical Systems Is on Schedule OMB's guidance as amended in January 1998 requires that agencies complete renovation of their systems by September 1998 validation by January 1999 and implementation by March 1999 Customs' reported status of its mission-critical IT and non-IT systems are described below 5 As of January 1999 Customs reported that it had renovated and validated its mission-critical application software components and was running the application software components in a production environment Customs 4A program that enables a microcomputer to appear to be a mainframe terminal by using procedures and codes compatible with the mainframe computer 5 We did not independently validate all of Customs' reported system status information Page 3 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 also reported that it has completed systems acceptance testing of its five mission-critical systems i e the full complement of application software running on the target hardware and systems software infrastructure Endto-end testing for these systems and associated telecommunications systems is scheduled to be completed by March 1999 See sections on system acceptance testing and end-to-end testing for further explanation Customs' telecommunications systems include 1 equipment at Customs' national data center where its mainframe-based applications are operated and maintained 6 2 local equipment in Customs' field offices that supports intra-office communications and connects these offices to the data center and 3 headquarters and field office voice communications including telephone and voice mail systems As of January 1999 Customs reported that • National data center-related telecommunication systems were 100 percent assessed and 4 percent of this inventory required renovation or replacement Of this 4 percent Customs reports that 92 percent has been renovated validated and implemented Customs plans for the remaining 8 percent to be fully implemented by March 1999 • Field offices' telecommunications equipment was 100 percent assessed and all of the equipment required renovation or replacement Customs also reported that 68 percent of the equipment had been renovated validated and implemented and that Customs plans to complete implementation of the remaining 32 percent by May 1999 • Voice communications were 100 percent assessed and 50 percent required renovation or replacement Of this 50 percent Customs reported that 40 percent had been renovated or replaced validated and implemented Customs plans to complete implementation of the remaining 60 percent of its voice telecommunications by June 1999 Customs is still assessing some of its non-IT equipment As of January 1999 Customs reported that 82 percent of the mission-critical non-IT products had been assessed and that of this 82 percent 95 percent of the products were compliant 4 percent require renovation or replacement and 1 percent is being retired Customs plans to complete all assessment validation and implementation activities by May 1999 6 This equipment includes front-end processors external dial-in rotaries for the trade community and network connection devices such as nodes pads and switches Page 4 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 Customs Has Implemented an Effective Year 2000 Management Structure According to our Year 2000 assessment guide a successful Year 2000 program begins with establishing an effective Year 2000 program management structure Such a structure includes an empowered and accountable program office and program manager It also includes an active committee of agency executives to guide direct and facilitate program office efforts Our guidance also states that among other things Year 2000 program management organizations should develop comprehensive Year 2000 strategies and plans that 1 identify organizational roles and responsibilities and define tasks 2 establish schedules 3 establish reporting requirements 4 define performance measures and 5 estimate and allocate resources Program management organizations should also develop and implement Year 2000 guidance and standards and should establish processes within their Year 2000 management structure for ensuring that guidance is understood and for tracking progress against plans Customs Has a Well-Defined Year 2000 Program Management Structure Customs established its Year 2000 Program Office and designated a Year 2000 Program Manager in May 1997 Customs officially chartered the Program Office in October 1997 giving it authority over and responsibility for agencywide Year 2000 efforts including such functional areas as Year 2000 contracting budgeting and planning technical support to project teams quality assurance auditing and reporting Team leaders were established within the Program Office to manage these functional efforts Also the program was structured around IT mainframe systems and telecommunications systems and non-IT systems and managers for each area were designated and empowered These functional team leaders and area managers are accountable to the Year 2000 Program Manager who is accountable to the chief information officer CIO In addition to chartering the Year 2000 Program Office Customs also engaged its senior executives by charging the agency's Executive Council with approving and overseeing the implementation of the Year 2000 strategy and resolving such issues as institutional Year 2000 priorities The Council is co-chaired by the CIO and the chief financial officer and includes the Year 2000 project managers as members The Executive Council reports to Customs' Investment Review Board Page 5 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 Customs Has Defined Year 2000 Strategies Plans Guidance and Standards The Year 2000 Program Office in collaboration with the Executive Council issued Customs' Year 2000 Strategic Program Management Plan and Operational Program Management Plan in June 1998 7 Consistent with our Year 2000 guidance these plans 1 identify organizational roles and responsibilities and define tasks 2 establish schedules for completing each program phase i e awareness assessment renovation validation and implementation and describe the tasks to be completed under each phase 3 establish reporting requirements to track progress in the various phases 4 define performance measures and 5 estimate and allocate resources for the tasks and system activities within these phases In addition to the strategic and operational plans the Year 2000 Program Office has issued policies guidelines and procedures for managing and implementing the Year 2000 program in accordance with our guidance For example the Program Office has issued Year 2000 quality assurance configuration management and testing guidance to use in managing the conversion of its IT systems as well as Year 2000 business continuity and contingency planning guidelines to be used by business owners field offices and project managers To ensure that the plans policies and guidelines are being implemented the Year 2000 Program Manager is 1 holding weekly status meetings with the Year 2000 Program Office staff and the project teams 2 tracking prioritizing and managing the risks associated with the IT and non-IT system conversion efforts 3 overseeing and managing budget-related issues and 4 conducting internal audit reviews to monitor and assess the implementation of established Year 2000 procedures For example in June 1998 the Program Office assessed the implementation of configuration management and testing procedures for mission-critical IT systems identified weaknesses in the procedures recommended solutions and followed up on the findings and recommendations to ensure that configuration management and testing procedures are consistently followed across Year 2000 projects Program Office staff are also tracking the development and providing quality reviews of the contingency plans for continuity of operations The Customs' Year 2000 Program Office has also developed a central database for tracking progress against plans and for identifying issues that 7 Customs initially developed a draft management plan in July 1997 Page 6 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 may affect the strategy The database includes information on the status of systems conversion e g schedule and risks and external interface information The database provides the information required for Customs' monthly and quarterly reporting to Treasury The Year 2000 Program Office has also developed estimates of resource needs and has obtained and allocated funding to support these needs Specifically Customs' Year 2000 cost estimate from fiscal year 1997 through fiscal year 2000 is $118 million This includes the costs to renovate over 20 million lines of COBOL in mainframe-based applications telecommunications equipment personal computers commercial off-theshelf software COTS and non-IT systems It also includes Year 2000 Program Office support costs As of January 1999 Customs reported that it had spent $83 million on Year 2000 efforts Customs Has Adopted a Structured and Disciplined Approach to Managing Year 2000 Testing Activities Complete and thorough testing is essential to provide reasonable assurance that new or modified systems process dates correctly and will not jeopardize an organization's ability to perform core business operations during and after the transition to a Year 2000 computing environment Our Year 2000 test guide describes a structured and disciplined approach to Year 2000 test activities This five-phased approach begins with establishing an organizational testing infrastructure followed by designing conducting and reporting on four incremental levels of system-related testing software unit testing software integration testing system acceptance testing and end-to-end testing Customs Has Established an Effective Testing Infrastructure The purpose of establishing an effective testing infrastructure is to put in place a structured and disciplined framework for managing i e planning directing controlling overseeing and reporting each of the next four phases of testing i e software unit testing software integration testing system acceptance testing and end-to-end testing Our test guide defines 11 key processes associated with establishing an effective testing infrastructure including 1 assigning Year 2000 test management authority and responsibility 2 establishing an agencywide definition of Year 2000 compliance 3 engaging independent quality assurance and verification and validation agents 4 providing for Year 2000 compliance of vendorsupported products and 5 establishing a Year 2000 test environment Customs has satisfied each of the 11 key processes For example Customs 1 designated a Year 2000 test manager for mission-critical IT systems and Page 7 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 assigned this manager authority and responsibility for key test activities e g defining exit criteria designing and planning the tests executing the tests 2 established in its Year 2000 Application Testing Strategy and Plan an agencywide definition of Year 2000 compliance i e proper handling of system date changes correct manipulation of date-related data proper storage of dates with four-digit years and accurate inference of century values given two-digit years 3 engaged an independent verification and validation IV V agent to ensure that process standards have been followed and that software products perform as intended 8 4 provided for testing of vendor-supported IT and non-IT products e g vendor-certified mainframe operating systems and utilities and PC equipment and COTS software to ensure that they are Year 2000 compliant and 5 established a Year 2000 test environment consisting of a Year 2000 logical partition9 on Customs' mainframe computer along with Year 2000 compliant telecommunications components that replicate the organization's operations in a future-date environment By implementing these key processes Customs has established an effective framework for managing Year 2000 testing Customs Has Completed Software Unit and Integration Testing for Its Mission-Critical IT Systems The purpose of software unit and integration testing is to verify that units of software both individually and combined work as intended According to our test guide effective unit and integration testing includes among other things 1 developing unit and integration test plans 2 preparing test procedures and data 3 documenting test results 4 correcting defects and 5 ensuring that test exit criteria are met Customs has completed software unit and integration testing for its mission-critical IT systems For the system components that we reviewed Customs satisfied all of the unit and integration test phase key processes For example Customs 1 developed combined unit and integration test plans that specified the Year 2000 compliance criteria defined in the Year 2000 Application Testing Strategy and Plan 2 prepared test procedures and data that included Year 2000 date conditions specified in test guidance 8The agent is in the process of using an automated tool to for example independently analyze renovated and tested mainframe-based code for Year 2000 problems such as conflicting date formats and deviations from date windowing standards As of January 1999 the agent had analyzed 84 percent of Customs' mainframe-based code and found 7 errors per 10 000 lines of code 9 A logical partition is a distinct portion of memory that functions as though it were a physically separate unit Page 8 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 and specified among other things test results documentation requirements and test exit criteria 3 documented test results including problems 4 corrected documented problems and 5 ensured that exit criteria were met by requiring user and tester sign-off approval before software was moved into production and the Year 2000 logical partition for subsequent system acceptance testing By implementing these key processes Customs should be well positioned to begin the next phases of testing Customs Has Completed Systems Acceptance Testing The purpose of system acceptance testing is to verify in an operational environment either simulated or actual production 10 that the complete system i e the full complement of application software running on the target hardware and systems software infrastructure satisfies specified requirements e g functional performance and security and is acceptable to end users According to our test guide effective system acceptance testing includes among other things 1 developing systems acceptance test plans that specify such things as the type of tests to occur and whether users will actively participate in the test 2 confirming compliance of vendor-supported system components 3 executing the system acceptance tests and 4 ensuring that system acceptance test exit criteria are met As of January 1999 Customs reported that it had completed system acceptance testing of all its mission-critical IT systems In doing so Customs satisfied our system acceptance test phase key processes For example Customs 1 developed detailed system acceptance test plans for the five mission-critical IT systems that specify the types of acceptance tests to be performed e g functional security performance and stress testing 2 confirmed that vendor-supported components are Year 2000 compliant 3 executed acceptance tests in accordance with plans and procedures which included the participation of user representatives and configuration management staff and 4 required users to attest to the systems' Year 2000 compliance according to the exit criteria defined for the systems By implementing these key processes Customs should have reasonable assurance that individual systems perform as intended 10 Risks of testing in the production environment must be thoroughly analyzed and precautions taken to preclude damage to systems and data Page 9 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 End-to-End Testing Underway The purpose of end-to-end testing is to verify that a defined set of interrelated systems which collectively support an organizational core business area or function interoperate as intended in an operational environment 11 either actual or simulated According to our test guide effective end-to-end testing includes among other things 1 defining the system boundaries of the end-to-end tests 2 securing the commitment of key data exchange partners 3 confirming the Year 2000 compliance of vendor-supported telecommunications and other infrastructures and 4 executing end-to-end tests and documenting results Customs has either performed or has plans to perform all of our end-to-end key processes For example Customs 1 defined the system boundaries of its end-to-end tests to include field site configurations external partners and telecommunications systems 2 secured commitments from key data exchange partners and 3 confirmed the availability of compliant telecommunications products and services Also while Customs has not yet completed end-to-end tests it has developed test plans and procedures that assign responsibilities for executing tests and documenting test results Additionally to validate that it can exchange data with some of its commercial trade partners Customs conducted a preliminary end-to-end test in May 1998 The test demonstrated successful data interchange between ACS and external partners represented by four leading software vendors 12 By completing these end-to-end test key processes and activities Customs will have greater assurance that its systems and the systems of its business partners interoperate as intended 11Risks of testing in the production environment must be thoroughly analyzed and precautions taken to preclude damage to systems and data 12 These vendors provide software to 32 percent of the import community with whom Customs exchanges data Page 10 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 Customs Is Developing Contingency Plans to Ensure Continuity of Core Business Operations Despite organizations' best efforts to remediate their mission-critical systems core business processes may still be disrupted by Year 2000induced system failures and errors in internal systems business partners' systems or public infrastructure systems such as power water transportation and telecommunications systems Business continuity and contingency plans help mitigate the risks associated with unexpected internal and uncontrollable external system failures Our business continuity and contingency planning guide provides a four-phased structured approach for business continuity planning--initiation business impact analysis contingency planning and testing Customs Has Completed the Initiation Phase of Contingency Planning According to our contingency planning guide effective initiation of a contingency planning effort includes among other things 1 establishing a business continuity project work group 2 developing and documenting a high-level business continuity planning strategy 3 developing a master schedule and milestones 4 implementing a risk management process and establishing a reporting system and 5 implementing quality assurance reviews Customs has implemented all of the initiation phase key processes For example Customs 1 formed a business continuity project work group in January 1998 to lead manage and oversee the continuity planning effort for its core business processes 2 developed and documented its Contingency Management Strategy in February 1998 which defines roles and responsibilities for business process owners and systems owners defines the continuity project structure and specifies that plans are to be developed for each of Customs' six core business processes 3 defined in its Contingency Management Strategy a master schedule for the planning effort along with milestones for the delivery of interim products 4 implemented a risk management process i e steps for identifying and ranking internal and external risks and developing risk mitigation plans and defined progress reporting requirements to manage the business continuity planning tasks and assist business units in developing individual contingency plans and 5 implemented a quality review process to verify that the continuity of operations and contingency plans satisfy information requirements e g on September 8 1998 the quality review team reported its findings regarding missing and or incomplete information to the preparers for correction or clarification By satisfying these key processes Customs has established an effective structure for managing its contingency planning efforts Page 11 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 Customs Has Assessed the Impact of Mission-Critical System Failures on Core Business Processes The principal objective of this phase is to determine the effect of missioncritical information systems' failures on the viability and effectiveness of agency core business processes According to our contingency planning guide effective business impact analysis includes among other things 1 defining and documenting Year 2000 failure scenarios 2 performing risk and impact analyses of each core business process and 3 assessing and documenting infrastructure risks Customs has performed all of the business impact analysis phase key processes For example Customs has 1 defined and documented potential disruption scenarios e g the cause and nature of the disruption the duration of the disruption the business processes and supporting systems affected by the disruption 2 performed risk and impact analyses for its six major business processes that include an evaluation of business legal and regulatory impacts and 3 determined if existing disaster recovery plans address potential disruption scenarios and if not Customs is expanding the plans By performing these key processes Customs acquired the information needed to develop effective contingency plans for continuity of operations Customs Is Developing Contingency Plans The purpose of the contingency planning phase is to integrate and act on the business impact analysis results According to our contingency planning guide effective contingency planning includes among other things 1 assessing the costs and benefits of identified alternatives and selecting the best contingency strategy for each core business process 2 defining and documenting triggers for activating contingency plans and 3 developing and documenting zero-day strategy and procedures 13 Customs has performed all of the key processes of the contingency planning phase For example Customs 1 used its business impact and disaster recovery analysis14 to assess the costs and benefits of alternative contingency plans for each core business process and to select business continuity strategies and 2 defined triggers for activating contingency plans and designated responsible individuals to ensure that the plans are 13A zero-day strategy is a risk reduction strategy for the period between Thursday December 30 1999 and Monday January 3 2000 The strategy may include an agencywide shutdown of all information systems on December 31 1999 and a staged power-up on January 1 2000 14 Business Impact and Disaster Recovery Requirements Analysis DynCorp June 17 1996 Draft Page 12 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 executed if necessary Additionally Customs considered developing zeroday procedures and concluded that it was not necessary because its business continuity and contingency plans address potential Year 2000induced failures including the period between December 31 1999 and January 3 2000 Customs has completed development of most contingency plans and it has plans to complete development of all contingency planning by March 1999 Customs Is Testing Its Contingency Plans The objective of the testing phase is to verify that when implemented contingency plans provide the required levels of business performance According to our contingency planning guide effective testing includes among other things 1 developing and documenting contingency test plans 2 updating disaster recovery plans and procedures and 3 updating continuity plans based upon lessons learned from the tests and retesting the plans if necessary Customs has either implemented or has plans for implementing the key processes for this phase For example Customs 1 developed contingency test plans for each business process that specify such things as the test approach required facilities and resources and schedules and locations and conducted preliminary tests to validate certain test procedures and 2 assessed its disaster recovery plans and procedures15 and updated them to address failures of facility power telecommunications and networks Customs is currently testing contingency plans and it has plans to complete contingency plan testing including plans for non-IT systems by June 1999 In addition Customs plans to update the plans to reflect the test results and to retest these plans as necessary Customs Has Established Processes to Help Ensure the Reliability of Y2K Status Reporting To effectively manage and oversee Year 2000 programs managers and executive decisionmakers need reliable information about the nature and status of Year 2000 conversion efforts Our Year 2000 guides recognize the importance of such information Accordingly the guides provide for establishing formal reporting mechanisms early in the Year 2000 program life cycle and using the information reported to oversee and control program efforts Additionally the guides describe the need to specify the content and format of the reports and the reporting frequency and to 15 Information Technology Infrastructure Recovery Assessment Customs’ Office of Information and Technology September 23 1998 Page 13 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 establish management controls e g the use of quality assurance and IV V groups to ensure that the information being reported is reliable Customs has established formal reporting mechanisms for both its IT and non-IT conversion efforts and it has clearly defined the content and format of the reports and the reporting frequency For example Year 2000 project managers are required to report weekly on the number of system components assessed renovated validated and implemented as well as actual expenditures according to specified cost categories Project managers must also submit more detailed weekly reports on validationspecific activities and results e g the number of lines of code renovated waiting to be tested and behind or ahead of schedule and business-owner contingency planning-specific activities e g the status of specific contingency planning tasks for each mission-critical system component These activity and progress reports are entered into Customs' Year 2000 central database and are used to control and oversee the program as well as to prepare Customs' monthly and quarterly status reports to Treasury To ensure that the information reported to Customs' executives and Treasury is reliable a quality review team reviews the information submitted by project managers for 1 consistency by comparing it to previously reported information 2 completeness by comparing it to reporting standards and 3 accuracy by validating it through either observation inquiry or review of supporting documentation For example in June 1998 a quality review team compared the test data reported by project managers to the actual test results for both high-risk and randomly-selected components and found no discrepancies between the reported data and the actual test activities and results To further determine the reliability of the information included in Customs' monthly reports to Treasury we independently traced the reported status for selected components of three systems back to supporting source documentation i e project-level status reports and documented test results and found no discrepancies 16 Because we did not statistically sample Customs' IT and non-IT components and verify the reliability of the reported information in the sample we cannot conclude that the Year 2000 status information that Customs is reporting is reliable However in light of the results of our work on the reliability of Customs' reporting we decided not to perform additional tracing 16 The selected components were ACS Quota TECS Pre-clearance Alert and QIK Sun Solaris Page 14 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 Conclusions Customs reports with a few system exceptions that it has met OMB Year 2000 milestones and Customs' plans provide for completing all remaining Year 2000 efforts well in advance of January 1 2000 Customs' good progress to date is attributable to the effective Year 2000 management structures and processes that Customs has established Clearly Customs still has much to accomplish before it is ready for the century date change including completing conversion efforts for its internal systems and preparing for the possibility of external system failures However Customs has plans in place for completing key tasks and it has the management structures and processes in place to ensure that they are accomplished While these structures processes and plans do not guarantee that Year 2000-induced system failures will not occur if Customs implements its plans and follows its policies and procedures it will have effectively reduced its risk of significant Year 2000-induced business failure Agency Comments In commenting on a draft of this report Customs agreed with our analysis and conclusions In addition Customs stated that our Year 2000 guidance has been of great value to Customs in establishing a sound foundation for Customs’ Year 2000 effort We are making copies of this letter available to Representative Bill Archer Chairman and Representative Charles B Rangel Ranking Minority Member House Committee on Ways and Means Senator Robert F Bennett Chairman and Senator Christopher J Dodd Vice Chairman Senate Special Committee on the Year 2000 Technology Problem Representative Stephen Horn Chairman and Representative Jim Turner Ranking Minority Member Subcommittee on Government Management Information and Technology House Committee on Government Reform Representative Constance A Morella Chairwoman and Representative James A Borcia Ranking Minority Member Subcommittee on Technology House Committee on Science We are also sending copies to the Honorable Robert E Rubin Secretary of the Treasury Raymond W Kelly Commissioner of Customs and the Honorable Jacob J Lew Director of the Office of Management and Budget Copies will also be made available to others upon request Page 15 GAO AIMD-99-37 Customs’ Year 2000 Program Controls B-281520 If you have any questions about this report please contact me by phone at 202 512-6240 or by email at hiter aimd@gao gov Major contributors to this report are listed in appendix IV Randolph C Hite Associate Director Governmentwide and Defense Information Systems Page 16 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Page 17 Customs Year 2000 Program Controls Contents Letter 1 Appendix I Objective Scope and Methodology 20 Appendix II Customs’ Six Core Business Missions 22 Appendix III Customs’ MissionCritical Systems 24 Appendix IV Comments From the U S Customs Service 26 Appendix V Major Contributors to This Report 27 Page 18 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Contents Abbreviations ACS ADMIN AES CIO COTS IBIS IT IV V OMB PC SEACATS TECS Page 19 Automated Commercial System Administrative System Automated Export System chief information officer commercial off-the-shelf Interagency Border Inspection System information technology independent verification and validation Office of Management and Budget personal computer Seized Asset and Case Tracking System Treasury Enforcement Communications System GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix I Objective Scope and Methodology ApeInxdi Our objective was to determine whether Customs has established effective management structures and processes for managing and reporting on key aspects of its Year 2000 program We accomplished this objective by identifying Customs' Year 2000 program management controls and comparing these to the controls i e key processes described in our Year 2000 guides 1 Additionally for selected systems and systems' components we reviewed supporting documentation to verify that the management controls were functioning as intended More specifically we reviewed Customs' Year 2000 program management plans guidance procedures and organizational structures relating to Year 2000 conversion testing contingency planning reporting quality assurance independent verification and validation and risk management In particular we analyzed the following Customs’ Year 2000 Strategic and Operational Program Management Plans Customs' Year 2000 Application Testing Strategy and Plan Customs' Telecommunications Year 2000 Program Plan Customs' Year 2000 Contingency Management Strategy and Plan Customs' Year 2000 Quality Assurance Plan Customs' Year 2000 program office charter defining roles responsibilities and authority Customs’ progress reporting and tracking procedures and correspondence between Customs' Executive Council and the Year 2000 program office We then reviewed documentation associated with individual systems and systems' components to determine whether these structures and processes were being implemented For example we reviewed system component unit and integration test plans system acceptance and end-to-end test plans telecommunications criticality analyses weekly status meeting minutes progress and status reports business impact analyses for core business processes IT infrastructure risk assessments and existing disaster recovery plans We then selected software components from two mission-critical systems ACS' Quota and TECS' Pre-clearance Alert and one telecommunications system QIK Sun Solaris which consists of five components to determine whether conversion activities required in Customs' plans procedures and guidance were being executed For these systems and components we reviewed supporting documentation relating to conversion and testing 1 Year 2000 Computing Crisis An Assessment Guide GAO AIMD-10 1 14 issued as an exposure draft in February 1997 issued in final in September 1997 Year 2000 Computing Crisis Business Continuity and Contingency Planning GAO AIMD-10 1 19 issued as an exposure draft in March 1998 issued in final in August 1998 and Year 2000 Computing Crisis A Testing Guide GAO AIMD-10 1 21 issued as an exposure draft June 1998 issued in final in November 1998 Page 20 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix I Objective Scope and Methodology such as project schedule risk mitigation strategies print-outs of before-and-after screen changes and database field changes print-outs showing data entry of new date formats and correct processing of dates configuration management documentation showing the movement of programs into source control libraries and subsequently into production sign-off sheets showing quality assurance and user approval to move changed programs into production results of quality assurance reviews of reporting accuracy test results for Customs' mainframe operating system utilities and environmental software along with vendor information regarding the compliancy of Customs' lab environment and plans for testing contingency plans For the selected systems and system components we also traced the status information that Customs was reporting back to supporting source documentation e g project level status reports and test results We did not statistically sample Customs IT and non-IT components To supplement our analysis of documentation we interviewed key Year 2000 program officials such as the Year 2000 Program Director the Year 2000 Test Manager individual project managers quality assurance officials business process owners the Contingency Planning Team's members and support contractor representatives We conducted our work in collaboration with the Treasury Inspector General and in accordance with generally accepted government auditing standards from July 1998 through January 1999 Page 21 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix II Customs’ Six Core Business Missions ApIenxdi To accomplish its mission Customs is organized into six business areas--trade compliance outbound passenger finance human resources and investigations Each business area is described below • Trade compliance includes enforcement of laws and regulations associated with the importation of goods into the United States To accomplish its trade compliance mission Customs 1 works with the trade community to promote understanding of applicable laws and regulations 2 selectively examines cargo to ensure that only eligible goods enter the country 3 reviews documentation associated with cargo entries to ensure that they are properly valued and classified 4 collects billions of dollars annually in duties taxes and fees associated with imported cargo 5 assesses fines and penalties for noncompliance with trade laws and regulation 6 seizes and accounts for illegal cargo and 7 manages the collection of these moneys to ensure that all trade-related debts due to Customs are paid and properly accounted for • Outbound includes Customs’ enforcement of laws and regulations associated with the movement of merchandise and conveyances from the United States To accomplish its mission in the outbound area Customs 1 selectively inspects cargo at U S ports to guard against the exportation of illegal goods such as protected technologies stolen vehicles and illegal currency 2 collects disseminates and uses intelligence to identify high-risk cargo and passengers 3 assesses and collects fines and penalties associated with the exportation of illegal cargo and 4 physically examines baggage and cargo at airport facilities for explosive and nuclear materials In addition the outbound business includes collecting and disseminating trade data within the federal government Accurate trade data are crucial to establishing accurate trade statistics on which to base trade policy decisions and negotiate trade agreements with other countries By the year 2000 Customs estimates that exports will be valued at $1 2 trillion as compared to a reported $696 billion in 1994 • Passenger includes processing all passengers and crew of arriving and departing air sea and land conveyances and pedestrians In fiscal year 1997 Customs reported that it processed nearly 450 million travelers and by the year 2000 expects almost 500 million passengers to arrive in the United States annually Many of Customs' passenger activities are coordinated with other federal agencies such as the Immigration and Naturalization Service and the Department of Agriculture's Animal and Plant Health Inspection Service Activities include targeting high-risk passengers which requires prompt and accurate information and Page 22 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix II Customs’ Six Core Business Missions physically inspecting selected passengers baggage and vehicles to determine compliance with laws and regulations • Finance includes asset and revenue management activities Asset management consists of activities to 1 formulate Customs' budget 2 properly allocate and distribute funds and 3 acquire manage and account for personnel goods and services Revenue management encompasses all Customs activities to identify and establish amounts owed Customs collect these amounts and accurately report the status of revenue from all sources Sources of revenue include duties fees taxes other user fees and forfeited currency and property The revenue management activities interrelate closely with the revenue collection activities in the trade compliance outbound and passenger business areas • Human resources is responsible for filling positions providing employee benefits and services training employees facilitating workforce effectiveness and processing personnel actions for Customs' 18 000 employees and managers • Investigations includes activities to detect and eliminate narcotics and money laundering operations Customs works with other agencies and foreign governments to reduce drug-related activity by interdicting seizing and destroying narcotics investigating organizations involved in drug smuggling and deterring smuggling efforts through various other methods Customs also develops and provides information to the trade and carrier communities to assist them in their efforts to prevent smuggling organizations from using cargo containers and commercial conveyances to introduce narcotics into the United States Page 23 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix III Customs’ Mission-Critical Systems AIpIenxdi To carry out its responsibilities Customs relies on a variety of information systems and processes to assist its staff in 1 documenting inspecting and accounting for the movement and disposition of imported goods and 2 collecting and accounting for related revenues In managing its Year 2000 program Customs divided its mission-critical systems into information technology IT and non-IT systems e g office equipment facilities and security systems Customs has five IT systems that run mission-critical applications Automated Commercial System ACS Treasury Enforcement Communications System TECS Administrative System ADMIN Automated Export System AES and Seized Asset and Case Tracking System SEACATS These are described below • ACS supports the trade compliance core business process by tracking controlling and processing all commercial goods imported into the United States Over 97 percent of the data filed for imported cargo entries are transmitted to Customs electronically through ACS ACS has been operational since 1984 and is accessed by over 15 000 trade and other government agency users • TECS supports the enforcement core business process and provides support to federal law enforcement missions Consequently TECS interfaces with a number of law enforcement systems including the Federal Bureau of Investigations’ National Crime Information Center system and is the major automation component of the Interagency Border Inspection System IBIS IBIS serves as the clearinghouse for law enforcement data and provides border inspection support software and communications In addition TECS supports Customs' Informed Compliance Targeting Identifying and Examining strategies as well as Investigations Narcotics and Money Laundering Approximately 27 000 users access TECS including Customs the Immigration and Naturalization Service the Internal Revenue Service the Bureau of Alcohol Tobacco and Firearms and the State Department • ADMIN provides information technology support for the financial and human resources core business processes It is comprised of 40 separate systems which interface with each other and with ACS AES and TECS ADMIN is accessed by almost 19 000 users predominantly from Customs' Office of Finance and Human Resources Management • AES supports the trade compliance and outbound core business processes It electronically gathers export-related information from exporters and carriers and is used to help Customs' target export violators AES is accessed by over 28 000 users nationwide Page 24 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix III Customs’ Mission-Critical Systems • SEACATS supports the enforcement finance and trade compliance core business processes and tracks activity associated with seizures from the initial enforcement interest in the property until its final disposition SEACATS is accessed by over 16 000 Customs’ users and interfaces with the Justice Department and Internal Revenue Service systems Customs' IT systems also include its telecommunications systems Customs' telecommunications devices and software components include 1 the interface between the mainframe systems and the Treasurymanaged TCS equipment 2 voice telecommunications systems installed throughout Customs’ locations and 3 the local area network equipment which connects Customs' field locations to the mainframe systems Customs' mission-critical non-IT assets include office equipment such as check-writers scanners and optical readers facilities such as heating and air lights and fire alarms security systems including badge readers cameras secured doors and safes planes and automobiles These assets are installed in over 900 facilities Page 25 GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix IV Comments From the U S Customs Service Page 26 ApV eInxdi GAO AIMD-99-37 Customs’ Year 2000 Program Controls Appendix V Major Contributors to This Report Accounting and Information Management Division Washington D C Jack L Brock Jr Director Governmentwide and Defense Information Systems Dr Rona B Stillman Chief Scientist for Computers and Telecommunications Deborah A Davis Assistant Director Garry D Durfey Senior Information Systems Analyst Atlanta Field Office Teresa F Tucker Senior Information Systems Analyst John M Ortiz Senior Information Systems Analyst 511127 Page 27 eL rtet ApV enxdi GAO AIMD-99-37 Customs’ Year 2000 Program Controls Ordering Information The first copy of each GAO report and testimony is free Additional copies are $2 each Orders should be sent to the following address accompanied by a check or money order made out to the Superintendent of Documents when necessary VISA and MasterCard credit cards are accepted also Orders for 100 or more copies to be mailed to a single address are discounted 25 percent Orders by mail U S General Accounting Office P O Box 37050 Washington DC 20013 or visit Room 1100 700 4th St NW corner of 4th and G Sts NW U S General Accounting Office Washington DC Orders may also be placed by calling 202 512-6000 or by using fax number 202 512-6061 or TDD 202 512-2537 Each day GAO issues a list of newly available reports and testimony To receive facsimile copies of the daily list or any list from the past 30 days please call 202 512-6000 using a touchtone phone A recorded menu will provide information on how to obtain these lists For information on how to access GAO reports on the INTERNET send an e-mail message with “info” in the body to info@www gao gov or visit GAO’s World Wide Web Home Page at http www gao gov United States General Accounting Office Washington D C 20548-0001 Official Business Penalty for Private Use $300 Address Correction Requested Bulk Rate Postage Fees Paid GAO Permit No GI00