##$%% ' ' $- %' 0'1 # %'2 3 #' 4565'7 8 ' 3'2 9 % %$ ' % ' - ' ' ' % '@ A B''7 C'% '1 # %'D 9 0$E 0' 8 ' F %$ G'6 A8 $%H'3 ' '% ' '1 %$ ' I $%H'6% 33'2 9 %'' 7 5'D ' 33 %EJ' $ # ' ##$%% ' ' $- %' 0'1 # %'2 3 #' 7 5' K' 0 C J' $ # ' 68LA ##$%% ' '1 # %' 9 %$ ' 7 5'M$GG'78 0J' $ # ' 68LA ##$%% ' 'N 3 # %$ ' A G -H' 6 9% #L 'OJ'PQ R www oversight house gov A Letter from the Chairman September 7 2016 To Federal Chief Information Officers The advent of the information age presents a paradigm shift about how our federal institutions collect store distribute and protect information The data breach at the Office of Personnel Management 0PM is a de ning moment and it is up to you the community of federal chief information officers to determine how the country will respond The effectiveness of our country s response depends on your answer to this questiontrusted with highly personal highly sensitive data on millions of Arnericans Federal C105 possess expertise and technical knowledge that support the mission- related activities of their agency As Departmental heads focus on managing the bureaucracy of the executive branch substantive challenges of their agencies mission and Congress CIOs play a critical role in keeping technology working for Americans and in furtherance of the agencies mission Federal ClOs matter In fact your work has never been more important and the margin for error has never been smaller As we continue to confront the ongoing challenges of modemiaing antiquated systems CiOs must remain constantly vigilant to protect the infon'natien of hundreds of millions of Americans in an environment where a single vulnerability is all a sophisticated actor needs to steal information identities and profoundly damage our national security The mission of our Committee is to ensure the ef ciency effectiveness and accountability of the federal government and its agencies We have a constitutional duty to provide meaningful oversight of the executive branch and to recommend reforms that are informed by our investigative ndings- Taxpayers also rely on the lCommittee to bring a measure of accountability and transparency in cases where there is evidence of misconduct That is why I am releasing this report to the American public For those whose personal information was compromised I hope this report provides some answers on the how and why Most of all however it is my hope that the findings and recommendations contained herein will inform and motivate current and future CIDs and agency heads so we as a government can be smart about the way we acquire deploy maintain and monitor our information technology- The 0PM data breach and the resulting generational national security consequences cannot happen again It is up leaders like you and Congress to ensure it does not happen again Sincerely Jason Chaffetrt Chairman ii The Damage Done This is crown jeweis materiai a goia' mine for a foreign service This is not the end of A merican human intelligence but it s a signi cant biow it Joel Brenner former NSA Senior Counsel We cannot undo this damage What is done is done and it take decades to it John Schindler former NSA of cer The SF 196 gives you any kind of information that might be a threat to the empioyee security ciearance Jeff Neal former DHS of cial My iists every piece 1 ve ever iivea' since i was 18 every foreign travei i ve ever taken ofmy tmiiy their addresses So it s not jastmy identity that s a ected I 've got sioiings i ve got ve kids oftnat is in there i James Conley Director of the FBI data remains a treasure trove of information that is to the Chinese antii the peopie represented by the information age of Titere s no xing it Michael Hayden former Director of the CIA David Perera Joseph Marks Newiy Disclosed Hack Got Crown Jeweis POLITICO lune 12 available at kground-checks-l 18954 U icer 0PM Hack is Serious Breach of Worker Trust NPR June 13 3015 available at Id Maggie Ybarra James Conley FBI Chief 3st His info was Hacked in 0PM Breach it was Enormous WASH TIMES July 9 2m 5 available at save-his-own-info-was haeked Dan Tlierton impact of FM Breach Conic Last More Titan Feats Jul r 12 2015 available at iv Executive Summary The of the United States of America has never before been more vulnerable to cyberattacks No agency appears safe In recent data breaches hackers took information from the United States Postal Service the State Department the Nuclear Regulatory Commission the lntemal Revenue Service and even the White House None of these data breaches though compare to the data breaches at the Of ce of Personnel Management 0PM In what appears to be a coordinated campaign to collect information on government employees attackers ex ltrated personnel files of 4 2 million former and en rrent government employees and security clearance background investigation information on 21 5 million individuals 1 Additionally ngerprint data of 5 6 million of these individuals was stolen The loss of personally identi able information I ll is deeply troubling and citizens deserve greater protection from their government Further the damage done by the loss of the background investigation information and ngerprint data will harm counterintelligence efforts for at least a generation to come The Signi cance of What the Attackers Stole Certain individuals appiy for a security clearance to gain access to our country s most sensitive national security secrets These individuals are required to complete Standard Form 36 or and undergo a background investigation Many applicants are obvious targets by adversaries for intelligence purposes by virtue of their holding some of the most sensitive positions in our including anyone accessing classi ed information and anyone employed in a national security sensitive position This encompasses a wide-range of federal employees and contractors at all federal agencies including the U S Department of Defense and throughout the Intelligence Community Background investigations conducted on these individuals are designed to identify the type of information that could be used to coerce an individual to betray their country Therefore applicants are required to provide a wealth of information about their past activities and lifestyle For example applicants are required to provide extensive nancial information as well as employment history and home addresses for the past ten years Applicants are also required to provide the names of any relatives including stcp siblings or half siblings and their home addresses The also requests disclosure of some of the most intimate and potentially embarrassing aspects ofa person s life including whether the applicant There is some overlap between the 4 2 million individuals impacted by the personnel records breach and the 21 5 million individuals impacted by the background investigation breach 0f the 4 2 million individuals impacted by the personnel records breach 3 5 million on these individuals also had their background investigation data stolen See Letter from Jason Levine Dir Congressional Legislative d Intergov't Affairs LLS Of ce of Personnel Mgmt to Jason Cliafl'ela Chairman H Comm on Oversight d Gov t Reform Aug 2015 The aggregate number of individuals impacted by this breach totals 22 1 million Ir consult ed with a health care professional regarding an emotional or mentai health oondition 1- illegally used any drugs or controlled substances I abused alcohol resulting in a negative impact on your work performance or personal relationships your nances or result in intervention by law enforcementipublic safety pcrsonnelf and - experienced nancial problems due to gambling In short the 313-36 asks individuals to turn over their most personal details information that in the wrong hands could be used for espionage purposes The intelligence and counterintelligence value of the stolen background investigation information for a foreign nation cannot be overstated nor will it ever be fully known The Director of the Federal Bureau of Investigation FBI James Comey described the data breach as a very big deal from a national security perspective and from a perspective It s a treasure trove of information about everybody who has worked for tried to work for or works for the United States government 2 Nor is there any way to remedy the problem now that the information is in the hands of our adversaries Foimer Central Intelligence Agency Director Michael Hayden wamed he does not think there is recovery from what was lost and it remains a treasure trove of infotmation that is available to the Chinese until the people represented by the information age off There s no xing it 3 How the Breach Happened Despite this high value information maintained by OPM the agency failed to prioritize cybersecurity and adequately secure high value data The 0PM Inspector General IS warned since at least 2005 that the information maintained by OPM was vulnerable to hackers In 2014 the IG upgraded issues surrounding information security governance at 0PM from a material weakness to a signi cant de ciency But fundamental aspects ofOPM s information security posture such as the absence of an effective managerial structure to implement reliable IT security policies remained a signi cant deficiency or worse since 2000' 4 Indeed even after the data breach as of November 2015 the 0PM IG continued to report that continues to struggle to meet many FISMA requirements and with overall lack of compliance that seems to permeate the agency s IT security program 5 3 Ellen Nakashima Incline databases compromised peopie federot' authorities any WASH POST July 9 2015 available at ffected-Z -5-mi ll ion-peopie-fcd craI antholities-sayi Dan Verton impact Breech Conic Last Mote Titan 40 Years FedScoopeom Jul 12 2015 available at comio nm losses-a-40-vear problem -for inte liaence-co mm unity 1ch of Gen US Of ce of Pers Mgmt No 0044-01-5 iiy'brmotinn Security Management Act Auriit 201 Nov 12 2014 available at manage ment-act-audit-fy-20 Of ce of Inspector Gem LLS Of ce of Pets No 1 Finoi Audit Report Federni habituation Security Modernization Act Audit 5 Nov 10 2015 available at ins pector genera lirepo rtsi20 1 5i fed oral-inf orm ation-securitv-mod ernization aet-audit-fy-20l naI-audit-report-4 a- ll-1501 l idl hereinafter Fi FISMA Audit vi The agency also failed to implement the Of ce of Management and Budget s OMB longstanding requirement to use multi factor authentication for employees and contractors who log on to the network In a 2015 OMB report on IT security 0PM was identi ed at the end of scal year 2014 as one of several agencies with the weakest authentication profile s and only having one percent of user accounts requiring personal identity verification cards for access 6 The agency also allowed key IT systems which were later compromised to operate without a security assessment and valid Authority to Operate ATO In 2014 the it called the increasing number of OPM IT systems operating without a valid ATO alarming The last state of information security left the agency s information systems exposed for any experienced hacker to in ltrate and compromise On March 20 2014 the US Department of Homeland Security s DI-IS United States Computer Emergency Response Team noti ed Computer Incident Response Team that a third party had reported data ea ltration from network In an effort to better understand the threat posed by the hacker Ui lvl monitored the adversary s nrovements over a two month period The agency s senior leadership failed to fully comprehend the extent of the compromise allowing the trackers to manuals and other sensitive materials that essentially provided a roadmap to the 0PM IT environment and key users for potential compromise While 0PM monitored the first hacker for convenience here we will refer to this actor as Hacker X1 on May 201-4 another hacker posed as an employee of an 0PM contractor performing background investigations KeyPoint which we can call Hacker X2 Hacker X2 used the contractor s 0PM credentials to log into the UPM system install malware and create a backdoor to the network As the agency monitored Hacker I s movements throughout the network it noticed Hacker was getting dangerously close to the security clearance background information 0PM in conjunction with developed a plan to kick Hacker X1 out of the system It termed this remediation the Big Bang The agency was confident the planned remediation effort in late May 2014 eliminated Hacker 's foothold on their systems But Hacker X2 who had successfully established a foothold on systems and had not been detected due to gaps in IT security posture remained in system post-Big Bang The Ertfiltration of the Security Clearance Files Could Have Been Prevented After the May 27 Big Bang Hacker X2 moved around system until they began ek ltrating data in July 2014 As Director ofiT Security Operations Jeff Wagner explained the KeyPoint credential was used for the initial attack vector and then the attacker used various tactics to obtain domain administrator credentials to ultimately perform operations and maintain persistence from rnalwarc Beginning in July through August 2014 the Hacker X2 eatiltrated the security clearance background investigation tiles Then in December 2014 personnel records were exfiltrated and in early 2015 ngerprint data was ex ltrated 6 Office if Budget Exec Of ce of the President Fl 2014 Annual Report to Coiigi-ars err'ei'rri tyrornintion Security Management Act at 23 2G Feb 27 Elli 5 available at itebousegovfsitesi de stilnaljy 4_ sina report_f 2_2 7_2 l 1 5 pd f Of ce of Personnel Mgmt Of ce of the Inspector lGeneral Federal linfornint ion Security Management rte Andi Fl Edit at 9 Nov 12 21314 available at vii Had 0PM implemented basic required security controls and more expeditioust deployed cutting edge security tools when they first learned hackers were targeting such sensitive data they could have significantly delayed potentially prevented or signi cantly mitigated the theft Testimony from DHS made clear implementation oftwo-factor authentication for remote logons in early 2015 which had long been required of federal agencies would have precluded continued access by the intruder into the OPM network Further if 0PM had fully deployed in a preventative mode available security tools and had sufficient visibility to illy monitor their network in the summer of 2014 they might have detected and stopped Hacker X2 before they had a chance to exfiltrate the security clearance background investigation les Importantly the damage also could have been mitigated if the security of the sensitive data in ClPMis critical IT systems had been prioritized and secured The exact details on how and when the attackers X1 X2 gained entry and established a persistent presence in OPlvt s network are not entirely clear This is in large part due to sloppy hygiene and inadequate security technologies that left 0PM with reduced visibility into the traf c on its systems The data breach by Hacker X1 in 2014 should have sounded a high level multi-agency national security alarm that a sophisticated persistent actor was seeking to access highest-value data It was not until April 15 2015 that 0PM identified the first indicator its systems were compromised by Hacker X2 Front April 16 2015 through May 2015 during the primary incident response period security tools from an outside contractor Cylancc lnc consistently detected key malicious code and other threats to 0PM While these types of security tools were generally available to PM the agency did not choose to deploy a preventative technology until rr er the agency was severely compromised and until n er the agency s most sensitive information was lost to nefarious actors Notably Director of Security Operations Jeff Wagner recommended deploying Cylance s preventative technology to insulate OPM's enterprise from additional attacks after the initial attack by Hacker in March 2014 The Committee obtained documents and testimony proving security posture was undermined by a woefully unsecure IT environment internal politics and bureaucracy and misplaced priorities related to the deployment of security tools that slowed vital security decisions Swifter action by 0PM to harden the defenses of its IT architecture could have prevented or mitigated the damage that systems incurred While 0PM continued its incident response efforts throughout April 2015 another outside contractor named CyTech Services provided forensic support after conducting an onsite demonstration of its technology While 0PM and CyTech provide differing accounts oftlte role onylilR in detecting unknown malwarc on systems it is clear CyTech detected malwarc and assisted for at least two week in the response to the 2015 data breaches To date CyTech has not been compensated for any of its work The Anti Dcficiency Act prohibits a federal agency from accepting voluntary services without payment and without obtaining an agreement in writing that the contractor will never seek payment In this case there was no such agreement Most concerning the agency destroyed 035 les and directories located on device prior to returning the device to its owner while a request from the Committee for this information was pending All of those les were material to the Committee's investigation responsive to the Committee s subpoena requests for information and documents and subject to a preservation order by the Committee 0PM Misled Congress and the Public to Diminish the Damage As the agency assessed the damage caused by the hackers 0PM downplayed the fallout 0PM failed to proactively announce the 2014 breach to the public and claimed the two cyberattaeks were not connected The 2014 and 2015 incidents however appear to be connected and possibly coordinated The first confirmed adversarial activity for both incidents came within a two- month span in November and December 2013 The back discovered in March 2014 by Hacker K1 appeared to move through the system looking for security clearance background investigation data and was removed when they got too close Hacker XI did however manuals and other sensitive materials which would be useful for targeting background information data systems Hacker X1 was cleared from the system in May 2014 during the Big Bang exercise Within three months Hacker X2 nished targeting and stealing background investigations data by early August 2014 Hacker X2 later stole personnel records in December 2Ul4 and fingerprint data in March 2015 The two attackers shared the same target conducted their attacks in a similarly sophisticated manner and struck with similar timing Further the manuals estiltrated by Hacker X1 likely aided Hacker X2 in navigating the OPM environment The Committee s year-long investigation to understand how the attackers perpetrated their intrusion movements and ultimately the es ltratiun of data began with hearings wherein then-0PM Chief Information Of cer 310 Donna Seymour made a series of false and misleading statements under oath regarding the agency s response to the incidents announced in 2015 Seymour testi ed that 0PM purchased CyTech licenses but 0PM did not make any purchases from CyTech She also testi ed that CyTech s tool was installed in a quarantine environment for the demonstration but this tool was running on a live environment at 0PM when it identi ed malware on April 22 2D I 5 Seymour also misled the public about the signi cance of the data stolen in the 2t 1 4 attack She testified on April 22 2015 that our antiquated technologies may have helped us a little bit 3 Two months later on one 24 2015 she testified that the stolen manuals that were a roadlnap to Ol'M s systems were merely outdated security doeuments 9 The Bottom Line The longstanding failure of leadership to implement basic cyber hygiene such as maintaining current authorities to operate and employing strong multi-factor authentication despite years of warnings from the Inspector General represents a failure of culture and leadership not technology As 0PM discovered in April 2015 tools were available that could have prevented the breaches but failed to leverage those tools to mitigate the agency s extensive vulnerabilities a Enhancing Cybersecnri'rv Contractors and Vendors Hearing Re ne the H on Oversight d Ger Reform 114th Cong Apr 22 2015 hereinafter Enhancing Hearing statement of Donna Seymour Chief Info Officer of the U S Office ofPers Mgmt 9 0PM Darn Breach For H Hearing Before the H Comm on Oversight ii Gov 't Rgforni Cong 69 June 24 21315 hereinafter Hearing on 0PM Doro Breach For statement of Donna Seymour Chief Info Of cer of the US Office ofPers Mgmt ix As a result tens of millions of federal employees and their families paid the price Indeed the damage done to the Intelligence Conununity will never be trulyr blown Due to the data breach at 0PM adversaries are in possession of some of the most intimate and embarrassing details of the lives of individuals who our country trusts to protect our national security and its secrets This report documents how the government allowed this unthinkable event to happen and makes recommendations in an attempt to ensure this never happens again The Committee remains hopeful that 0PM under the new leadership of Acting Director Beth Cohen is in the'process of remedying decades of mismanagement Table of Contents A Letter from the Chairman ii The Damage Done Executive Summary Table of Contents 1 Timeline of Key Events 5 Findings 14 Recommendations 20 Table of Names 28 Chapter I IT Security Record Preceding Breaches 30 The Rise of Advanced Persistent Threat Hacking 3c Federal Contractors Holding Sensitive Federal Employee In Formation Targeted and Attacked Federal Initiatives to Increase Information Security in Response to Increasing Attacks 0PM Failed to Recognize the Threat and Implement Effective IT Security Measures When It Mattered 35 Cybersecurity Spending Consistently Trailcd Other Federal 0PM Attempts to Balance IT Security with Competing Priorities The Katherine Archulcta and Don na Seymour Era 42 0PM Failed to Prioritize the Security of Key Data and Systems Chapter 2 The First Alarm Bell Attackers Discovered in 2014 Target Background Information Data and Esfiltrate System-Related Data 51 Discovery Incident Response for Attackers Discovered in 2014 52 Monitoring the Adversary and the May 20 4 Big Bang to Expe Attackers Discovered in NH 55 During the NIH Incident Response Period the Eit ltration ofFIPS-rclated In Formation Made Clear the Attackers Target was Background Investigation Data Held in FIPS til Tactics Techniques 3 Procedures of Attackers Discovered in 2014 Hikit Malware and 3MB Protocol Network Logging Capabilities Limited Investigating the How and How Long for Attackers Discovered in 2014 Chapter 3 0PM Attempts to Mitigate the Security Gaps Identi ed in 2014 While Iron Man and Captain America Go to Work May 2014 April 2015 75 IT Security Posture and Mitigation Efforts After the May 2014 Hi Bang Key 2614 Recommendations Highlighted 0PM IT Security Vulnerabilities 0PM Efforts to Buy Security Tools to Secure the Legacy Network and Rebuild Ver Insecure Insecurely Architected Network Missed Key Developments 31 In April 2015 0PM Realized They Were Under Attack Again 33 1 Captain America The First Indicator that Led to the 2015 Discovery orthe Background Investigation Data Breach 34 The Avengers Anatomy of the Data Breach Discovered in 1015 35 Chapter 4 The Role of Cylance Inc 91 Cyber Climate During Cylance Product Demonstrations Jverview of the Cylance Cyber Tools 93 April 15-16 2015 The First 24 Hours 34 April 2015 Con rms Plugx 38 April 17 2015 CylanceProtect Deployed 100 April 13 2015 Prelch Lights Up Like a Christmas Tree 102 April 19 2015 Severity of the Situation Becomes Clear 103 April 20-23 2015 More Key Trojans Identi ed GIG First Noti ed 4 108 April 24-25 2015 0PM Upgrades Protect to Auto-Quarantine Mode 1 10 April 26 April 30 2015 First Signs of Lost Background Materials 113 The Decision to Purchase CylanceProteet 1 115 Political Challenges on the Desktop to Counterpoint Lack of Compliance I 19 Purchases Plotect After Nearly Losing Access to It 1121 Chapter 5 The CyTech Story 125 CyTech Is a Small Business Contractor with Significant Cyber'l'ool Capabilities 126 CyTech Was Invited to Conduct a Demo at 0PM 12 Prior to the April 21 2015 Demonstration at CIPM 123 The April 21 2015 April 22 2015 Demonstration at 0PM 123 The CyTech Demo Turned into Incident Response and Forensic Support 135 CyTech Provided Onsite Incident Response and Forensic Support From April 23 to May 1 2015 135 Was Deployed on the UPM Network beginning in April 2015 and Remained on Network through August 2015 133 The Wolf Street Joni-no Reports on CyTech s Role in the OPM Incident on June 10 2015 141 Cy'l'ech Coordinated with 0PM Prior to the June 10 2015 Story 142 0PM and Respond to the Article 143 Description oFCyTech s Role Was Misleading 140 Archuleta and Seymour Provided Misleading Testimony to Committee 1415 Data on CyTech's Appliance Collected During the 2015 Incident Response Period was Deleted 148 Retained CyTech's Appliance Through August 2015 149 Before Returning the Appliance 0PM Deleted Key Data 149 0PM Sanitizod the Appliance 151 0PM Violated the Anti-De ciency Act 152 The prohibition on accepting voluntary services 152 The gratuitous services exception 152 2 The emergencies exception 153 The ADA applied to the 0PM and CyTech Situation 153 Cmih expected to be paid 154 Chapter 6 Connections Between the 2014 and 2015 Intrusions 157 One Group Several Names 153 The 2014 Data Breach The Unique Malware of the Axiom lCirroup 159 Malware Discovered during the E l Data Breach 162 21114 a 21115 Likely Connected Possibly Coordinated 163 Chapter 7 OCIO and its Federal Watchdog 173 The G's Memorandum ofConcern l'ir'4 Four Instances Where the 01 10 Failed to Cooperate Fully Seymour failed to appropriately notify the of the April 212115 intrusion detection ITT Seymour fai led to notify the DIG of the loss of background investigation data in a ti rnely manner 1311 Seymour failed to notify the BIG show the 21114 incident 182 Meetings Itvith Federal Law Enforcement Agencies 183 KcyPoint Audit 134 Noti cation Concernin New IT Infrastructure 135 Five Incorrect auditor Misleading Statements 1 First Misstatement before the Senate Committee on Appropriations Second Misstatement Before the Senate Comtnittee on Appropriations 133 Third Misstatelnent Before Senate Committee on Appropriations and House Cotnmittee on Clversight and Government Reform 133 Fourth Misstatement Before the House Committee on tilhrersight and Government Reform 1 139 Fifth Before the Senate 139 Current State of Relationship 139 Summary of ODS and relationship 193 Chapter 3 The IT Infrastructure Improvement Project Key Weaknesses in Contracting Approach 194 The 1G Issues a Flash Audit Alert and Interim Reports on the IT Infrastructure Project 196 The G s Concerns Continued through the Fall of ID I 5 198 10 Reports Progress in Responding to Concerns but Challenges Remain as ofMay 2016 198 The Story of IT infrastructure Improvement Project and the Sole Source Contract EUD Timeline IT Infrastructure Improvement Project EUI 0PM Initiates Contact with Imperatis and Awards Sole Source Contract 205 Imperatis and 0PM Buy Security Tools to Secure the Legacy IT Environment lmperatis Role in Responding to 0PM Data Breach Incidents 207 Sole Source Schedule and Cost 1G Concerns Related to IT Infrastructure Improvement Contract Validated 208 Summary of Investigation 214 3 Committee hearings on the data breaches 4214 Committee request For information regarding identity theft services 11 5 Productions related to the 0PM data breaches and CyTe eh 116 The Committee investigated the role oFCylance 219 The Committee investigated the role of SEA 221 The Committee Investigated OPM's IT Infrastructure Improvement Project and the Contract Awardee lnrperatis 22 Document productions by Department of Homeland Securityr 221 Unnecessarv delays restrictions redactions and a congressional subpoena 122 Unnecessary delays 222 Unnecessaryr redactions 222 Subpoena issued to 0PM 224 Conclusion 225 Appendix Cyber security Spending at 0PM Fiscal Years 2012-2015 227 Timeline of Key Events Jae 2012 r Attackers had access to network according to US-CERT found maiware Hikit resided on an 0PM server since 2012 3 November ZUIJ First evidence of adversarial activity by the attacker associated with the breach that USHCERT informed 0PM about in March 2014 3 December 2013 v First evidence of adversarial activity associated with the 2015 breaches including harvesting of credentials from 0PM contractors by the attacker that was not identi ed until April 2015 4 More 20 20h r US-CERT notifies 0PM ol a data exiiltration from OPM's network 5 0PM working with US-CERT determines and implements a strategy to monitor the attackers movements to gather This breach involved data that included manuals and IT system architecture information but the full extent ot'esliltrated data is unknown v The strategy remains in place Until the Big Hang on May 2014 March 25 20 v Situation report takes place with CID Donna Seymour and US-CERTF March 2014 s As 0PM monitors the hackers it develops a Plan for full shut down if needed 7 June 2014 0PM Incident Report at Production Sent l3 hereinafter June 2014 0PM Incident Report Note This Report was authored by and provided to 0PM US Dep't of Homeland SecuritnyS CERT Digital Media Analysis Report-465355 June 9 2m 5 at 154 Production Dec 22 2015 Hereina er June 9 21315 1 Hearing on 0PM Dore Brena-Ii Part II statement of Donna Seymour Chief Info Officer ofthe US Of ce of Personnel Mgmt Brie ng by to H Comm on Oversight dc Gov't Reform Stafleeh 19 2t 16 one 2014 incident Report at 1 Id 7' re April I1 20 Tactical mitigation strategies and security remediation plan developed for brie ng to Donna Sey'rrtour S April 21 2M4 1 0PM contractor SEA discovers a specific piece of malware which is brought to attentionf April 25 2014 v is registered to Steve Rogers a k a Captain The hackers later used this domain for command and control and data ea ltration May 2014 v The attacker later associated with ea ltrating background investigation data establishes their foothold into network This attacker poses as a background investigations contractor employee KeyPoint uses an 0PM credential remotely accesses network and installs Plugx malware to create a backdoor v OPM did not identify the attacker s May 7 foothold despite the fact that 0PM was monitoring and removing another attacker that US-CERT had noti ed 0PM about in March 2014 May 2914 OPM shuts down its compromised systems in the Big Bang event in an effort to remove the attacker This decision was made after OPM observed the attacker load a key logger onto several database administrators workstations and they got Id at Hooansis-eciaai '3 Id at HOGRUS I 84101242 '0 ThreatConneet Research Team 0PM Breach June 5 21115 available at H Comm on Oversight and Oov t Reform Transcribed interview of Brendan Saulebury Senior lCyher Security Engineer SKA Ex 4 Feb 2016 Hereinafter Saulshury Brie ng by US-CERT to H Comm on Oversight dc Gov't Reform Staff Feb I9 2016 at 59 '3 H Comm on Oversight dc Oovit Reform Transcribed Interview of Jeff P Wagner LLB Of ce of Personnel Mmgt Dir of Information Technology Operations at 121128 Feb 18 Z l ti hereinafter Wagner Tr Dep t of Homeland and Of ce of Pers 0PM Cybersecurity Events Timeline Aug 26 at HOORGEDB Production May 13 2016 hereina cr 0PM Cybersecurity Events Timeline Brie ng by LIB-CERT to H Comm on Oversight a Gov l Reform Staff Feb 19 EDI KeyPoint CEO testi ed that there was an individual who had an OPM account who was a KeyPoint employee and l the credentials of that individual were compromised to gain access to Hearing on 0PM Dara Breech Pent If statement of Erie Hess Chief 13sec Of cer KeyPoint The OPM Director of IT Security Operations Wagner explained that a KeyPoint user credential was utiliaed for the initial vector infection but that user did not have administrative credentials so the adversary utilized tactics in order to gain domain administrator credentials to move through the environment and conduct operations related activities Wagner Tr at as too close to getting access to the PiPs system which held the background investigation data it Meanwhile the attacker that established a foothold on May 7 2014 continues their presence on the OPM network June 5 2914 it Malware is successfully installed on a KeyPoint web server accounts differ as to whether or not administrator privileges were used to install this malware '4 June it 2014 v 0PM C10 Donna Seymour testifies before the Senate Homeland Security and Governmental Affairs Subcommittee on OPlvl s Strategic Information Tecitnoiogy Pian atsid does not disclose at this hearing the manuals breach discovered in March 2014 June I2 2014 v 0PM executes a Cyiance product evaluation agreement that allowed it to test the anctionaiity of both Cylance products V and Protect for a limited period of time June 20 2014 Attackers conduct a remote desktop protocol EDP session indicating contact with important and sensitive servers supporting background investigation processes The remote session was not discovered until spring 2015 '7 June 22 2014 r DHS issues a nal incident report for the 0PM manuals breach first discovered on March 20 2014 '3 Saulsbury Tr at 15-26 '4 Brie ng by H Comm on Oversight a new Reform Staotrch 19 2016 Letter from Kcyl oim Government Solutions to the Hon Elijah E Cummings Ranking Member H Comm on Oversight 3t Gov t Reform July 2 Note Kcyl oint maintains that No unaccounted securitv tokens were used during the time the malware was operational on KeyPoint s network The Report of the KeyPoint intrusion disagrees stating that a domain administrator account was used to instal1 the malware on the web server reported that this administrator account had full access privileges 1 A More E cient and Effective Government Examining Federai iT initiativar and the i1 Workforce Hearing Before the S on the E eienev ona E ectiveness of eti Programs d tire Fed Wot-Horse oftire S Comm on tiotnet'and Sec Gov 't Affairs 1 13th Cong June It 20 I4 H Comm- on Gversight d Gov t Reform Transcribed Interview of Stuart McClure Chief Exec Of cer President 5 liounder Cylance Ex 2 Feb 4 Mid hereina er McClure H Comm on Oversight 3L Gov t Reform l'ranscribcd interview of i3 hris Coulter Managing Dir of Incident Response and Forensics Feb 12 E lt i Ex 18 hereinafter Coulter 5 June 2014 orM Incident Report at Hoonoals-ooiass-as June 23 2M4 v US-CERTIOPM identi es this as first known adversarial access to mainframe July August 2014 v Attackers successfully es ltrate the background investigation data from systems Jul 9 2014 0PM acknowledges the March 2014 manuals breach to the New York Times ll This information had not previously been disclosed publicly v 0PM states that no was lost in the breach and does not disclose the ex ltration of the manuals ll ulf r 29 2014 opmlearningorg is registered to Tony Stark a k a iron Man 12 The attackers used this domain for command and control during their intrusion into environment August 16 20M The malware installed on KevPoint systems on one 5 2014 ceased operational capabilities 1 Dumber 201 4 9 FBI Cyber Division issocs a Cyber Flash Alert regarding a group of Chinese Government af liated cyher actors who routinely steal high value intonnation from US commercial and government networks through cyber espionage and notes l Dep t of Homeland Brie ng to Staff Feb 19 2016 Cybersecuritv Events Tilnclinc Id 2' Michael E Schmidt David E Sanger Nicole Perlroth Chinese Hackers Pursue Key Dam on LES Workers MY TIM 9 2014 available at Tlu eatConneet1 Breach Auuhuis Saulsburv Tr Ex 4 Letter from KcyPoint Government Solutions to the Hon Elijah E Cummings Ranking Member H Comm on Uversight uh Gov t Reform July 2 citing USHCERT Report Aug 30 2tl14J KeyPc-int notes that signi cantly the malware was a zero day attack it had an electronic signature that was not known by anti virusfanti malware software at that time activity associated with this group should be considered an indication ofa compromise requiring extensive v Meanwhile the attackers move through the OPM environment to the US Department of interior data center where 0PM personnel records are stored 5 November 2014' v A group oi'private-industry security companies wams about threats to the human resources components of federal and releases a report on Chinese Advanced Persistent Threat APT activity 5 December 2014 4 2 million personnel records are ex ltratecl after attackers moved around system and through the 001's database which holds 0PM personnel records March 3 2015 v is registered by attackers Attackers would use this domain for C2 and data en ltration in the final stage of the intrusion 23 More 201' 5 9' The last beaconing activity to the unknown domain opmsecurityorg occurs This domain was registered in April 20M to Steve Rogers a k a Captain America March 26 2015 v Fingerprint data appears to have been ex ltrated on or around this date 3 Cyber oiv Fed Bureau rear eyes-Foss Aim Oct 15 2014 35 0PM Cybersccurity Events Timeline 2 5 Novella Operation MN Arron ThreorAeror Group Report 9 2014 contentfuploads ltifl 1fEmcutivejurnmary FinaL -pdf The report emphasises Hikit malware stating Among the industries We observed targeted or potentially infected by Hikit included Asian and Western government agencies responsible for a variety of services such as Personnel 17 Brie ng by us eaa'r H Comm on Oversight s Gov t Reform Sta 'thb to 201 is one Events limelinc DOMAIN 2- last visited June 28 201s available at 39 Saulshury Tr at 59' mJune 9 2315 DMAR at 153 see also Dcp t of iomcland Brie ng to Staff Feb 19 E id 0PM Cybersec urity Events Timeline April 15 201' 5 1 After being alerted by an 0PM contractor SKA working on IT security 0PM noti es about suspicious network traf c related to opmsecurityorg This domain was registered to Steve Rogers a-k a Captain America in April 2014 and the test beaconing activity occurred in March 2015 April 16 201 5 1 0PM contacts Cylanee for technical support on use ot Cylance V which was an endpoint detection tool that 0PM had purchased in Siptember 2014 32 Cvlance is not intended to be an enterprise wide prevention tool April I 2015 v 0PM begins to deploy enterprise-wide on a demonstration basis and in Alert mode a Cvlance tool called CylanceProtect At this time CytaneeProteet was not in quarantine mode but the tool would later identify r and alert 0PM to the widespread presence of malware on their system 0PM brings Cylancc onsite for incident response 4 0PM does not upgrade this tool to the highest preventative setting 35 April 13- 9 ZEUS v Cylaneel roteet is deployed to over 2 000 devices as of this date makes tons of findings and as a Cvlanee engineer described the tool it lit up like a Christmas tree indicating widespread malicious activities within the OPM system- April' 2 I 2015 1 CyTech Services arrives onsite to conduct a product demonstration with their CyTeeh Forensics and Incident Response tool and remains onsite until May 1 2015 to assist with incident response April 22 2015 v Donna Seymour testi es before the Committee about eyberseeuritv and publicly discussed the discovery of the manuals breach saying the adversaries in today s environment are typically used to more modern technologies and so in this case potentially our antiquated technologies may have helped us a little bit But I 3 June 9 2015 Drums at 15s 3 Coulter Tr Ex 1 2 33 McClure Tr at 3 34 McClure Tr at 21-22 35 111' DPle upgraded from the Eviction tool to the PROTECT tool However the tool remains in Alert mode only not Quarantine mode 3 McClure Tr Ex Coulter Tr at 20-2 3 H Comm on Oversight Gov t Reform Transcribed Interview of Benjamin Cotton Services Chief Executive Of cer at NH 5 Sept 3t 2015 hereinafter Cotton it think also it comes down to culture and leadership and one of the things that we were able to do at 0PM was to recognize the problem B v UPM's Of ce of the Inspector Genera DIG learns of the breach for the first time after a staffer bumped into the 0PM Director of Security Operations in the hallway if The staffer testi ed that Director of IT Security Operations said there was no need to notify the public ofthe breach Aprif 23 201' 5 if 0PM determines there had been a major incident involving the cs ltration of personnel records which triggers a requirement to notify Congress if 0PM noti es Congress of a major incident on April 30 2015 41 April 24 205 v 0PM orders a global quarantine to address malware identi ed by CylanceF'rotect April 215 2015 i Cylance engineers identify adversarial activity related to an session to a background investigation database indicating this session took place in June 2014 May a 2015 v establishes with a high degree of certainty that personnel records datafPII had been stolen May 20 ENE it 0PM detelmines there was a major incident regarding the ex ltration of background investigation data which triggers a requirement to notify Congress v 0PM noti es Congress on May 201 5 5 3'3 Enhancing Cybersecnriry of ine-Party Contractors and Vendors Hearing Eefore the H Comm on Oversight a Gov'r Reform I 14th Cong Apr 22 2015 statement of Donna Seymour Chief Info Of cer 3 Office of Pers Mgmt testifying that 0PM was hacked and that no PM was taken The word manuals is not used at this time though it is how we have since described the 2 14 breach 3'9 H Comm on Oversight d Gov't Reform Transcribed Interview of US Of ce of Pers Mgmt Of ce of Inspector Gen Special Agent at N-IS Dot I5 EDIE hereinafter Special Agent Tn chcral Information Security Modemization act of 2am Pub L No 1 13-233 12s Stat stirs 3030 201d 4' 0PM Cybersecurity Events Timeline 3 Coulter Tr is '13 Coultcr Tr Ex 13 Briefing by to ii Comm on Oversight a Gov t Reform Staff Feb 19 0PM Cybersecurity Events Timelinc 11 v 0PM indicates to the GIG that background investigation information may also be compromised June 4 2015 v 0PM briefs the media and releases a press statement that revealed the personnel records of 4 2 million former and current federal employees have been compromised June 8 2015 v US-CERT establishes with a high degree of certainty that background investigation datar PIl has been ex lttated and stolen 43 June 16 2015 v Then 0PM Director Katherine Archuleta acknowledges that background investigation data may be compromised June 24 2015 v Donna Seymour testi lies before the Committee and minimizes the importance of data removed in 2014 Manuals breach saving those documents were some outdated security documents about our systems and some manuals about our systems 5n June 29 ENE v The American Federation of Govemment Employees AFGE les a class action suit against 0PM 51 5 Briefing by n Comm on Oversight a Gov t Reform Sermon 19 2015 om Cybersecuritv Events Timeline- 45 Special Agent at 46 U S Office of Pcrs Mgmt Press Release 0PM to Nomi Empiovees oijvbeisecm-iifv June 4 EMS vine wafrelea ses Ol 3 Brie ng by to H Comm on Oversight Gov't Reform Staff Feb 19 2016 0PM Cyberseeuritv Events Timeline 49 0PM Doro Breach Hearing lie-fore the H Comm on Oversight Gov Re rrm lath Cong Jone 16 21315 statement of Katherine Arehuleta Din US Office ofPers Hearing on 0PM Doro Breach For statement of Donna Seymour Chief Info Of cer US Office of Pers Mgmt 5' American Federation of Government Employees v US Q 'iee ofPei-s Mgnn No 1 15wev i 15 led June 29' 2015 12 June 30 2015 After i'4 days of deployment to over 10 250 devices CylanceProtect detected and blocked almost 2 000 pieces of malware including critical samples related to the breach nearly one piece of malware for every ve devices My 9 rats 0PM issues a press release continuing background investigation date for 21 5 million individuals was compromised 52 Jutjt 1'0 201' 5 0PM Director Katherine Archuleta resigns Juiy 21 2015 v The Committee sends the first of a series of document requests to 0PM August 20 3035 v 0PM returns the tool to CyTech with key information deleted The tool before it was deleted contained images from incident response of more than 11 000 les and directories September 23 2015 0PM updates its original estimate that 1 1 million ngerprint records were compromised The new estimate 5 6 million 53 February 22 2016 v Prior to testifying before the Committee 0PM C10 Donna Seymour resigns February 24' 2016 v Committee s planned hearing Data Breach Part is cancelled in the wake of OPM Donna SeymoUr s resignation 54 53 Press Release US Of ce ofPers Mgmt 0PMAmtouucer Steps to Frame Fedet'ai Workers and Fiber me Cybet- Tit-teats July 9 2015 available at 53' Press Release US Of ce ofPers Mgmt Statement by 3PM Frets Secretory Strut Scittuttucit on Background Investigations Incident Sept 23 2015 available at 923 5 0PM Date Breaches Fur Hi Hearing Before ii Comm on het sight Gov't Reform 1 14m Cong Feb 24 201 5 hearing cancelled l3 Findings Cha ter 1 later PM IT Securi Record 0PM has iong been piagneri by afaiiare to prioritize information security in practice and to retain ieariers that are committed to information security over the iong hani FINDING FINDING FINDING FINDING FINDING FINDING 0PM leadership failed to heed repeated recommendations from its Inspector General 0PM has historically maintained a fragmented lT infrastructure and still lacks a full accurate inventory of all its major IT systems As the IG noted in its FY2015 audit failure to maintain an accurate inventory undermines all attempts at securing information systems 1 Over the EGGS-2015 timeframc 0PM failed to suf ciently respond to growing threats of sophisticated cyber attackers 0PM failed to prioritize resources for cyber security In FY 2013 FY 2014 and 2015 0PM spent seven million each year on cybersecurity spending that was consistently at the bottom relative to all other agencies that are required to report such expenditures to the Office of Management and Budget Slow implementation of critical security requirements such as dual factor authentication is a true case of misplaced priorities As early as 21305 16 issued a warning in a semiannual report that given the sensitive data 0PM holds on former and current federal employees and family members any attack or breakdown could compromise efficiency and effectiveness and ultimately increase the cost to the American taxpayer Key 0PM systems including the Persoimel Investigations Processing System PIPE Enterprise Server Infrastructure E31 and the Local Area NetworWidc Area Network LANKWAN were all operating on expired Authorities to Operate at the time of the data breach Fin i elat ta Br iscov 'n 2 14 In the spring onDi 4 0PM sn ered a data breach that resuited in the toss of documents reian'ng to the most databases on 13 environment FINDING Due to security gaps in network and a failure to adequately log network activity the country will never know with complete certainty all of the documents that the attackers ca ltrated from 0PM in connection with the breach discovered in March of2014 14 FINDING FINDING FINDING The 2014 attackers used an uncommon toolkit designed for late-stage persistence and data ex ltr'ation The malware observed on systems in 2014 were two variants of Hikit maiware termed Hikit A and Hikit B During an approximately two month period 0PM watched the adversaries take sensitive data relating to high valued targets on systems the server that holds background investigation materials but was never able to determine how the adversary initially gained entry into their network The documents taken by the 2014 attackers included information about systems that would have given an adversary an advantage in hacking the background investigation database and other sensitive systems in environment 3 QPM Attempts to Mitigate the Securing aps gumjed in 2911 hjlg WW FINDING FINDING FINDING FINDING In June 2014 issued an incident report with 14 observations and recommendations to address the security gaps identified after the 2014 manuals breach LIE-CERT deemed network very insecure insecurer architeeted and found 0PM had a signi cant amount of legacy infrastructure US-CERT also said there was a gap in information technology leadership across 0PM as an agency and that it was not uncommon for existing security policies to be circumvented to execute business functions while exposing the entire agency to unnecessary risk- Had 0PM leaders rlly implemented basic required security controls including multiufactor authentication when they rst learned attackers were targeting background investigation data they could have signi cantly delayed or mitigated the data breach of background information in April 2015 an 0PM contract employee identi ed a domain that was purposely named to emulate a legitimate looking website and upon further found the domain had a randomized email address and was registered to Steve Rogers a k a Captain America This was one of the rst indicators of compromise identified by OPM in April 2015 15 er 4- Fin an Information security tool's onyionce Inc detected Critical malicious code and other threats to 0PM in Aprii 201 5 and thereofi er pioyerl' rt critters role in responding to the riots breaches in 20 35 FINDING FINDING FINDING FINDING FINDING FINDING While Cylance tools were available to 0PM as earl f as June 2014 0PM did not deploy its preventative technology until April 2015 after the agency was severely compromised and the nation's most sensitive information was lost Swi er action by OPM to deploy CvlanceProtect would have prevented or mitigated the damage that UPM's systems incurred Following the May 27 2014 Big Bang remediation 0PM decided not to purchase and deploy CylanceProteet due to as Cylanec CEO Stuart McClure put it political challenges on the desktop meaning overcoming the tensions between IT security and program functionality On April 15 2015 0PM found an indicator ofcomprornise and turned to Cvlance for assistance Cylance tools found the most critical samples of malicious code present at 0PM related to the breaches and that correspond to ndings of DIIS US-CERT As of April 3 19 2015 CvlanceProtect was deployed in Alert mode to over 2 11 th devices made tons of findings and as a Cylance engineer described the tool it lit up like a Christmas tree indicating widespread malicious activities in IT Environment Former Director Katherine Archuleta and former CIO Donna Seymour made questionable statements under oath about use of a quarantine to isolate malware and malicious process tinting the incident response 0PM eventually purchased CvlanceProtect on June Bill 2015 but only as it was about to lose access to the product as the demonstration period was ending Despite Cylanee s proven value during the 2015 incident response 0PM failed to timelyr make payments 16 Orr June i 0 201 5 the Wait Street PERU reported that y'i ecit Services inc network forensics piaiform aetrtaiiy discovered that data breach at 0PM in mid Aprii during a saies demonstration FINDING CyTech a service disabled veteran owned small business contractor did participate in several meetings with 0PM in early 2015 to discuss the capabilities of their CyTech Forensics and Incident response tool and provided a demonstration of their too on April 21 2015 at 0PM headquarters FINDING During the April 2 demonstration Ccheh did identify malware on the live 0PM IT environment related to the incident CyTech was not aware at the time that 0PM had identi ed on April 15 an unknown Secure Sockets Layer SSL certificate beaconing to a malicious domain opmsecurity org not associated with 0PM FINDING Beginning on April 22 2015 CyTech offered and began providing significant incident response and tbrensic support to 0PM related to the 2015 incident FINDING CyTeeh did not leak information about their involvement with the 0PM incident to the press FINDING The testimony given by the now former 0PM CID Donna Seymour before the Committee on June 24 2111 5 regarding the CyTech matter is inconsistent with the facts on the record FINDING Documents and testimony show CyTec n provided a service to 0PM and 0PM did not pay The Anti de ciency Act ADA prohibits a l'ederai agency from accepting voluntary services to he no eete214 Intrusions a QEM The data breaches 0PM mattered to BUM anti share relevant not only to attribution out more importaatt'y 0PM 's reaction or tacit thereof'ia the waits oftiis 20M intrusion FINDING The data breach discovered in March 2014 was likely conducted by the Axiom Group This conclusion is based on the presence of Hikit maiware and other Tactics Techniques and Procedures associated with this group which have been publicly reported FINDING The data breaches discovered in April 2015 were likely perpetrated by the group Deep Panda aka Sbe11_Crcw a k a Deputy Dog as part of a broader campaign that targeted federal workers This conclusion is based on commonalities in the 2015 adversary s attack infrastructure and TTPs common to other hacks publicly i'i attributed to Deep Panda These groups include Wellpoino nnthem VAE Inc and United Airlines However the cyber intrusion and data theft announced by Anthem in 2015 is a separate attack by a separate threat actor group unrelated to the hack against 0PM discovered in 2015 FINDING As publicly reported both the Axiom and Deep Panda groups are highly likely to be statc sponsored threat-actor group supported by the same foreign government FINDING it is highly likely that the 2014 and 2014 9015 cyber intrusions into networks were likely connected and possibly coordinated campaigns ha - Peder-at watchdogs play a crt'a'cat' role in thefcrierto government partnering with agencies to improve and safeguard programs and operations including daring and after data breaches FINDING The relationship between the UPM Office of the Inspector General 01 3 and Office of the Chief Information lElfficer became strained during the tenure of former Director Katherine and former CID Donna Seymour The relationship became so strained that on July 22 2015 then InSpectOr Genera Patrick McFarland issued a memorandum to OPM's Acting Director Beth Cobert to share serious concerns regarding the DCIO FINDING Former 0PM Director Katherine Archuleta and former 0PM CIO Donna Seymour engaged in activities that hindered the work of the DIG including when OCIO failed to timely notify the BIG of the 2014 and 2015 data breaches or the data that was compromised Director Archuleta stated that the DIG could not attend certain meetings relating to the data breaches because the 010's presence would interfere with the FBI and work 3 The OCID failed to notify and involved DIG in a major investment to develop a new ET infrastructure and 4 The DIG delayed an audit of KeyPoint Government Solutions at the request of the DCIO after an October 16 2014 meeting only to learn later 0PM knew in early September 2014 that KeyPoint had been breached and did not disclose this information to the 01G FINDING Former 0PM Director Katherine Archuleta and former 0PM CID Donna Seymour made five incorrect andtor misleading statements to Congress These statements WEFBI 1 Director Arehuleta testi ed June 23 2015 before the Senate Committee on Appropriations Subcommittee on Financial Services and General Government that 0PM completed a Major 1T Business Case formerly known as the OMB Exhibit 300 for the infrastructure improvement project contrary to the nding ofthc 0PM 18 FINDING At the same June 23 2015 hearing Director Archuieta testi ed that my CID has told me that we have indeed an inventory of systems and data contrary to the ndings of the GIG in both a ash audit alert and the FY 2014 FISMA audit Director Archuleta and CID Donna Seymour testi ed before the Senate Appropriations Conunittee and the House Committee on Oversight and Government Reform that the sole source contract with contractor Imperatis for the IT infrastructure Improvement project covered only the rst two phases of this multiphase IT Infrastructure Improvement project and contracts for the later phases migration and cleanup of the project had not been awarded However the BIG found that the sole source contract provided for work under all tour phases of the project 0PM CID Seymour testi ed before the House Committee on Oversight and Government Reform on June 16 2015 that the 11 0PM systems operating without authorization were no longer a concern because she had granted an interim authorization to these systems However the 1G found that OMB does not allow interim or extended authorizations and At a June 25 2015 hearing held by the Senate Committee on Homeland Security and Govermnental Attairs Director Archuleta stated that 0PM had received a special exemption from OMB related to System authorization because of the ongoing IT Infrastructure Improvement project however this claim could not be substantiated The relationship between the OPM DIG and 0PM leadership has improved under Acting Director Beth F Cohort Chapter 3 Findings Related to the IT Infrastructure Project in response to tire o rrtn breach at 0PM in 201' 4 and o er identrf iing serious virinerobiir'tr'es in the OPM network the agency of the recommendation initiated the IT Improvement project FINDING FINDING FINDING IT Infrastructure Improvement project is a case study illustrating why agencies need to ensure robust communications with the OIG particularly in responding to cybersccurity incidents Former 0PM CID Seymour said she was not aware ofa requirement to notify the 1G ofevery project that we take on use of a sole source contract in an emergency situation illustrates why there should be pro established contract vehicles for cyher incident response and related services There is a pressing need for federal agencies to modernize legacy IT in order to mitigate the cybersccurity threat inherent in unsupported end of life IT systems and applications 19 Recommendations In 201 5 0PM announced the iargest data breach ofpersonaiiv identit iabie information Pit of 22 i miih'on Americans ihr s rihrre ofcuiture anti cannot happen again The fetierai government must recognize and mitigate the ever increasing cyher threat and protect the information that Americans entrust to the government White there was much that went wrong for years in the federai government approach to information security this episode presents an opportunity for Congress and other aget'tcies to inject new teadershr andr a cuiture ofsecurity in federai it The recommendations iisted heiotv are aimed at taking iessons iearnedfrom the 0PM experience and charting a path of ever vi giiant i 1 security in order to secure the PH of Americans heia' hy the ferterai government Recommendation 1 Ensure Agency 3105 are Empowered Accountable and Competent Each federal agency must ensure agency CIOs are empowered accountable competent and retained for more than the current average two year tenure The C10 at federal agencies and independent executive agencies is a critical leader who should be accountable to the head of the agency Under federal laws such as the Federal Information Security Management Act FISMA and the Federal Information Technology Acquisition Reform Act CIDs are responsible for IT security and management functions within the agency In the last two years Congress revised FISMA and FITARA to reflect the new prioritization agency heads should place on IT management and security CIOs typically serve an average of two years but greater priority should be placed on retaining these leaders for at least ve years 55 This Committee and in particular the IT subcommittee has made IT management and security an oversight priority to ensure vigorous implementation of FISMA and FITARA Such oversight has included a FITARA scorecard to assess agencies implementation ofthis law This oversight will continue and agencies will be expected to ensure there is an empowered accountable and competent C10 serving in this critical role Recommendation 2 Reprioritize Federal Information Security Efforts Toward a Zero Trust Model OMB should provide guidance to agencies to promote a zero trust IT security model The 0PM data breaches discovered in 2014 and 2015 illustrate the challenge of securing large and therefore liigh-vaiue data repositories when defenses are geared toward perimeter defenses In both cases the attackers compromised user credentials to gain initial network access utilized tactics to elevate their privileges and once inside the perimeter were able to move throughout network and ultimately accessed the crewnjewei data held by 0PM The agency was unable to visualise and log network traffic which led to gaps in knowledge regarding how much data was actually en ltrated by attackers To combat the advanced persistent tin-eats seeking to compromise or exploit federal government 11 networks agencies should move toward a zero trust model of information security and Ti 55 Gev t Accountability Of ce 1-1534 Fetter-at Chief hfarnmtion O icers Opportunities Exist to improve Rate in hn hrmation chhnaiogv Management Del 20 l stating the average ClD s tenure is two years 20 architecture The zero trust model centers on the concept that users inside a network are no more trustworthy than users outside a network 5 The zero trust model requires strictly enforced user controls to ensure limited access for all users and assumes that all traffic traveling over an organization's network is threat traffic until authorized by the IT team In order to effectively implement a zero trust model organizations must implement measures to visualize and log all network traf c and implement and enforce strong access controls for federal employees and contractors who access government networks and applications Recommendation 3 Reduce Use of SSNs by Federal Agencies Federal agencies should reduce the use of Social Security Numbers in order to mitigate the risk of identity theft SSNs are key pieces of that can potentially he used to perpetrate identity theft The potential for misuse of SSNs has raised questions about how the federal government obtains uses and protects the SSNs it obtains In May 2007 OMB required all federal agencies to review their use of SSNs in agency systems and programs in order to identify opportunities to reduce such use 57 Agencies were required to establish a plan within 120 days of the memo to eliminate the unnecessary collection and use of SSNs within 18 months They were also required to participate in government-wide efforts to explore alternatives to the use of SSNs as a personal identi tier for federal employees and in the administration of federal programs In response to a 2016 request by Chairman Challeta the US General Accountability Of ce GAO is currently reviewing actions agencies have taken to reduce the use of SSNS government-wide actions OMB has taken to ensure agencies have adhered to its directive and what progress has been made in reducing the use of SSNs across the federal government Congress should carefully monitor the progress of these important actions and work with agencies to ensure steps are taken to ef ciently and effectively reduce agency use of SSNs Recommendation 4 Require Timer Justifications for Lapsed Authorities to Operate Agencies that fail to re authorize the authorities to operate or their critical federal systems should be required to provide Congress within 5 days of the system s authorization expiring a justi cation as to why the system authorisation was allowed to lapse Designated critical information systems lacking adequate justification for a lapsed ATO should he removed immediately from the production environment ATOs provide a comprehensive assessment ofthc IT system s security controls and are a vital part of ensuring federal systems operate securely FISMA requires agencies to assess the effectiveness of their information security controls the frequency of which is based on risk but no less than annually OMB Circular A-130 Appendix required agencies to assess and authorize formerly referred to as certify and accredit their systems before placing them into operational environment and whenever there is a major change to the system but no less than 5 This model was proposed by Forrester Research Inc an American owned independent research and advisory rm in response to a 2313 National Institute of Science and Technology NIST request for information entitled Developing a Framework to Improve Critical Infrastructure Cybersecurity NIST 1332G31 19 31 19-01 See T3 Fed Reg 131324 Feb 26 21313 available at commentsf040313 57 Memorandum from Office of Mgmt d Budget L s-rec Of ce of the President to the Heads ofExee Dep'ts 3 Agencies a Safeguarding Against and Responding to the Branch ofPenronm iv Ideirry'r'uhie May 22 available at 2i every three cars thereafter 53 At 0PM critical systetns were operating in FY 2014 without a valid ATO 9 Of the 21 0PM systems due for reauthorization in FY 2014 11 were not completed On time and were operating without a valid authorization and several were among the most critical containing the agency s most sensitive information Ell This led the IG to warn 0PM that t he drastic increase in the number operating without a valid Authorization is alarming and represents a systemic issue of inadequate planning by 0PM ptogt'am of ces to authorize the information systems that they A failure to maintain current ATOs negatively impacts the security of federal information systems As the OPM pointed out there are currently no consequences for 0PM systems that do not have a valid Authorization to operate nr Consequently agencies should account for lapses to Congress and be prepared to take critical systems out of production Further at 0PM the IG reconunended the adoption of administrative sanctions for the failure to meet security authorization requirements 34 Congress and the Administration should consider options including legislation or policy guidance to ensure there are appropriate consequences for lapsed ATDs Recommendation 5 Ensure Accountability and Empower DOD IT Of cials Implementing Necessary Security Improvements for NBIB Clear rules for accountability and dedicated funding should be established by the end of FY 2017 to ensure the US Department of Defense is successful in securing the background investigation materials that will now be held at the new National Background Investigations Bureau N BIB In an effort to reform the background investigation process and secure related data this function will now reside at the new NBIB and the DOD CID will be responsible for The DOD CIO has testified that he will ultimately answer to the Secretary of Defense in matters relating to were and that non will provide short-term funding rat IT at note it What a Budget Eaee omee of the President oars it tsc Management errederat information Resources Nov 28 available at alJD al3 transdl OMB Citeular lit-t 30 was recently Updated and includes new guidance for agencies on Authorization to Operate and Continuous Monitoring Of ce of 3 Budget Exec Office of the President OMB Circular alt Bil Management of Federal Infonnation Resources July 27 available at The Committee expects to continue oversight in the areas covered by the revised A-l3 5 oniee of the Inspector oen us or ee of Fcrs MgmL Raport No sat-er -i 0-l4-Dlo retreat in urination i'i'ecttriiryJ Management Act Audit FY Jill 4 Nov 12 2014 available at govlour i nsnector rna na gemen -act- audit-fy-gtl 14-4 a-ci 1 4-0 1 pdf Id at 9 E-mail front Inspector Gen Staff U5 O iee ofPers to H Comm on lit Clov't Reform Staff Epee 4 21115 on tile with the Committee '3ch of the Inspector Gen ILLS Office of Pers Mgmt Report No flat Cl U l4 Dl 6 Federal information Security Management Act Andi FY 20M at 9 Nov l2 2014 available at Hit ederal-informat i on-sec uritv- man agent ent-act-audit-fv-ZO 1-1-4a-c i as- l4 tt1 3 at at to 5 Iri at 1 t '55 White House Press Release The Way Forwartifar Federal Background lnrertigntimrs Jan 22 2016 ill 1 l l wa y forwa rd federal background investigat ions a Security Clearance Reform The Performance Accountability Council '3 Pail Forward Hearing Before the House Comm on Oversight tit Got 'i Re n-rn I 14th Cong Feb 25 Kilo testimony of Terry Halvorsen Officer U S Dep t of Defense 22 However it is not yet clear whether future IT funding will come from DOD 0PM or another source It is also unclear how disagreements between DOD and 0PM regarding IT security spending would be resolved mi To ensure that security is appropriately prioritized at NBIB 0PM and DOE should establish clear sources of funding and decision making processes for IT security and the DIG at both 0PM and DOD should work to oversee such implementation and management Recommendation ti Eliminate Information Security Roadblocks Faced by Agencies To the extent there are non-security related bureaucratic hurdles to quickly implementing l'l security policies and deploying cyber tools agencies should make every effort to streamline processes and prioritize security The federal government s most important responsibility is to protect this nation and our citizens including when it comes to protecting this nation against cyberattacks The process of deploying security tools can be cumbersome and requires navigating a bureaucratic process that may involve notifying unions and overcoming program manager oppositional Congress should enact legislation sponsored by Rep Gary Palmer in the House HR 4361 and Senator Joni Ernst S 2935 to clarify agencies authority under FISMA by Stating the heads of federal agencies are able to take timely action to secure their IT networks and without being required to rst provide unions with the opportunity to bargain Recommendation 7 Strengthen Security of Federal Websites and Breach Noti cations Congress should enact HR 451 the Safe and Secure Federal Websites Act of 2015 legislation sponsored by Rep Chuck Fleischmann that increases the certi cation requirements for public federal websites that process or contain The bill requires an agency s C10 to certify the website for security and functionality prior to making it publicly accessible The bill also increases the requirements for agencies when responding to an information security breach that involves PII The events that unfolded at 0PM in 2014 and 2015 demonstrated an unwillingness by some of cials to notify the public ofa compromise in a timely manner The bill directs OMB to develop and oversee implementation of the certification requirements which include reporting the breach to a federal cyber security center and notifying individuals affected by a P11 compromise Recommendation 8 Financial Education and Counseling Services Through Employee Assistance Programs Congress should encourage federal agencies to provide federal employees with nancial education and counseling services that are designed to help employees recognize prevent and mitigate identity theil through existing Employee Assistance Programs BAP An EAP is a voluntary work-based program that offers free and confidential assessments short-term 5 Id frf 59 in the case ofUPM s efforts to deploy a tool called Forcscout which is a tool to manage network access control for devices there were deployment delays duo in part to the need to notify unions Imperatis Weekly Report Aug 3 201 S-Aug T 2015 Attach a at lmperatis Production Sept 1 Z l l stating project sponsor is in noti cation stage with the Union and mitigation was to prepare updated project timelinc plan a memo to pilot Fore-Scout to non-union agency users 23 counseling referrals and follow-up services to employees who have personal andior work- reiated problems ml Recommendation 9 Establish Government-wide Contracting Vehicle for Cyber Incident Response Services OMB and the General Services Administration GSA should lead efforts to establish a govemment-wide contracting vehicle for Incident Response Services or Congress should establish a statutory requirement for such a vehicle After the data breach discovered in March 2014 0PM awarded a sole source contract for a multi-phased IT Infrastructure Improvement project Under this contract 0PM procured cyberseeurity tools to secure their legacy IT environment Instead of duplicative sole source contracts across various agencies the federal govemment should have pro-established contracting vehicles that have the bene t of competition and are available to provide incident response services including tools to secure IT environments post breach Agencies should not be in the process of establishing contracts or these services during the incident response period In October 2015 OMB published a Cyber Security Strategy and Implementation Plan for the federal civilian government agencies The CSIP included a nutnber of deliverables including one related to establishing contracting vehicles providing incident response services A govemment wide contracting vehicle for incident response services should be established as soon as possible and before another agency faces the same situation as 0PM This will ensure such contracting vehicles have the bene t of competition and provide a robust suite of services to assist agencies in an incident response scenario Recommendation 10 Improve and Update Cybersecuritv Requirements for Federal Acguisition OMB should refocus etforts on improving and updating the current patchwork and outdated cyberseeurity requirements in existing federal security and acquisition rules There have been a number of initiatives launched over the last few years to update and improve cybersecurity requirements in federal acquisition- To date few of these efforts have been nalized has the Committee recommends that the Administration prioritize and complete efforts to develop and implement clear eyberseeurity requirements for federal acquisition as soon as possible The importance of the partnership between agencies and federal contractors in securing sensitive data held by agencies and contractor-operated systems cannot be overstated Existing cyberseeurity rules and requirements in federal acquisition are ad hoc overlapping potentially conflict and are in need of updating In Febmary 2013 the President issued Executive Order 13636 improving Critierrt' Cybersccurity and Presidential Policy Directive RFD 21 Criticrii Security and Reiianee that directed agencies to complete a broad range of tasks to enhance national 7 What is on Empioyee Assistance Program U S OF Pass available at govtr 1' aqsiDA asos'i'tid 43 1 3st I 4949 b4TS 34039al cl 1T4 1' Meanolauduln from Shaun Donovan Dir and Tony Scott Fed Chief Info Of cer Office of Mgmt 3 Budget Exec Of ce of the President to Agency ileads M lo d vaetsecnritv Strategy and iirrpimneomrimi Woofer the Peder-oi Government Get 3t 2015 available at go visitesidc l'aululr lesiom bi tn emoranda l oim- f 24 cyberseeurity and resilience 72 lOne group of deliverables included a mandate to incorporate cybersecurity requirements into the federal acquisition process In January 2014 GSA and DOD delivered a report improving anti Reiicnce through Acquisition that made recommendations to achieve this obj ective These report recommendations have not been implemented to date- The existing framework for cybersecurity requirements in federal acquisition should be reviewed and updated immediately The January 4 report recommendations provide useful guidance to inform such an update Recommendation 11 Modernize Existing Legacy Federal Information Technology Assets Federal agencies should utilize existing tools and Congress should consider new tools to incentiviae the transition from legacy to modernised IT solutions Federal agencies spend over $89 billion annually on IT with the majority of this spending focused on maintaining and 7-4 75 operating legacy IT systems Over i5 percent of this spending is focused on legacy IT costs GAO reported legacy IT investments are becoming increasingly obsolete with outdated software languages and hardware parts that are not supported Such reliance on legacy IT can result in security vulnerabilities where old software or operating systems are no longer supported by vendors and aging IT infrastructure becomes difficult and expensive to secure OFM testi ed before the Committee there are some ofour legacy systems that may not be capable of accepting these types of Sm The solution to this legacy IT challenge must be multifaceted and should include the use of existing and new tools to incentiviae modernization FITARA provides important tools for IT management and acquisition including facilitating the transition from legacy IT to modernised solutions 8 In terms of new tools incentives for agencies to achieve savings throuugh modernization and innovative financing options to promote modernisation should be considered Recommendation 12 Agencies Should Consider Using Critical Pav for IT Securigy Specialists Agencies may request and be granted critical position pay authority Agencies may request critical position pay authority only after determining the position in question cannot be lled L sec Order No l3 3ti TE Fed Reg l'i39 Feb 19 Z lli White House Press Release Presidential Policy Directive El CriticalT fiUri'fISii iiCl'iil E Security and Reiicuce r Feb ll 2013 73 Gen Serv s Admin Dep't of Defense improving vacisecnriiy and Residence it-cogs Acquisition Nov 2 13 available at rtalimedialdi and resi Ii cnce_throngh_acquisition action 7 The annual total of $39 billion for IT understatcs the federal government's total IT investment because it does not include DOD classi ed IT systems investments by 58 independent executive branch agencies including the and IT investments by the legislative orjudicial branches Data available through the IT Dashboard and OMB Df ce ofE-Gov and Information Technology T5 Gov t Accountability Of ce GAD-145463 irrfoirirniion Tecimoiogv Federal Agencies Need to Address Aging Legacy Systems May if Id Dart Brooch Hearing Before the H Comm on Oversight d't Gov Reform June HS 2015 testimony of Donna Seymour Officer US Of ce of Pers it National Defense Authorisation net Fv 2e15 Pub L No 113-291 Title vm Subtitle u 123 Stat 3292 3433- 513 Dec 19 21314 25 with an Exceptionally well-quali ed individual through tlte use ofother available human resource exibilitics and pay authorities 0PM in consultation with OMB reviews agency requests When approving a request 0PM must determine whether the position requires an extremely high level of expertise in a scienti c technical professional or administrative field and is mission critical Authority is used to recruit andfor retain exceptional talent and is capped at 300 positions at any one time Generally critical pay may be establislted up to Cabinet Secretary pay levels $205 700 and can be increased with approval by the President but pay and bonus generally cannot exceed the vice president s salary The Committee intends to collect more information on the use of critical pay authority in order to conduct appropriate oversight and tnake adjustments to the authority and to ensure it provides agencies the necessary exibility for recruitment and retention of IT security talent 0PM should also consider establishing a pay band for information Technology Security Specialists Recommendation 13 Improve Federal Recruitment Training and Retention of Cyber Security Specialists Recruiting training and retaining cyber security specialists should be a critical national security priority Following the cyberattacks at 0PM the federal CIO and the OMB Director issued a Memorandum concerning a cybersecurity strategy and implementation plan CS IP for the federal civilian government The CSIP included several federal cyber workforce related taskings including directing I 0PM and OMB to compile special hiring authorities by agency that can he used to hire cyber and IT plofcssionals across govcr'tuncnt 2 Agencies to participate in Cyber Workforce Project an effort to code cybersecurityjobs by specialty tor the purpose of gaining knowledge about the gaps and challenges in cyber recruitment and retention 3 DHS to pilot art Automated Cybersecurity Position Description Hiring Tool to assist in implementation of the National Initiative for Cybersecurity Education NICE framework and posting analysis of the cyber workforce on the C10 Council s knowledge portal as a best practice for other agencies to follow 4 0PM DHS and OMB to map the entire cyber workforce across all agencies using the NICE National Cyberseeurity Workforce Framework 5 0PM D115 and 0MB to develop recommendations for federal workforce training and professional development The Administration and Congress must work together to complete these tasks and swiftly take the steps needed to recruit train and retain a world class cyber workforce The Committee notes 1'9 Memorandum From Shaun Donovan Din and Tony Scott Fed Chieflnfo Of cer Of ce of Mgmt Budget Exec Itilfl iee of the President to Agency Heads hide 04 Strategv and iriipiemenrntion the Fedora Civil'irm Government Jct 3D 2015 available at still 1 rm l o Udpd f 26 OMB and 0PM jointly transmitted a memorandum to agency heads on a Federal Cybersecuritv Workforce Strategy on July 12 2016 and appreciates this opportunity to continue the dialogue in this area Finally Congress and the Administration should consider non-traditional mechanisms to recruit and retain cyher talent Such mechanisms should complement private sector experience rather than compete with the private sector recognize the need to quickly hire top talent and provide an opportunity for public service to those in the private sector 27 Table of Names Office of Personnel Management Name Title Katherine L Archuleta Director May 21313 - July EDIE Morrell John Berry Director April 21309 - April 2013 Beth F Cohort Acting Director July EDI 5 present Jason K Levine Director of f ce of Congressional Legislative and Intergovernmental Affairs August 2015 present Patrick McFarland Inspector General August l99 February 21316 Lisa Schlosser Acting Chief Information Officer March - August 2016 Donna K Seymour Chief Information Of cer December 20 E February 2 016 Special Agent in Charge Of ce of Inspector General Linda M Springer Director June Clifton Clif N Senior Cyber and Information Technology Advisor Norbert Eert E Vint Acting Inapector General February 21316 - present Deputy Inspector General EU 1 I5 Jeff P 1 nir'agner Director of Information Technology Security Operations Assurance Data Inc Name Title Matthew Morrison President and Chief Executive Of cer Inc am Title Chris Coulter Managing Director of Incident Response and Forensics Stuart McClure Chief Executive Of cer President and Founder 3 rant Moersc Director of Sales Engineering Nicholas Warner 1'r'icc President of Worldwide Sales CyTech Services Name Title Juan Eonilla Sr Security Consultanl Solutions Engineering with 0PM April 23-May1 5 Ben Cotton Chief Executive Dt' eer SBA Name Title Brendan Saulsbury Senior Cyber Security Engineer March 21312 May 2016 Jonathan Tonda 0PM Branch Chief Security Engineering September 2015- present Network Security Team Lead ERA May ecu- Septernher 2t 1 5 28 Imperoris Name Title Patrick Technical Lead for CHM contract Misc Name Title Joel Brenner Former National Security Agency Senior Counsel James B Comey Jr Director of the Federal Bureau of Investigations Michael V Hayden Former Director of the Central Intelligence ngnoyrr James Andrew Lewis Senior 1'rr'ice President and Director Strategic Technologies Program Center for Strategic and International Studies Jeff'Neal Former Chief Human Capital at the US Department of Homeland Security John Schindler Former National Security Agency of cer Richard A Spires Former Chief Information Of cer at the US Department of Homeland Security r and the Internal Revenue Service 29 Chapter 1 IT Security Record Preceding Breaches The attackers who Successfully penetrated the US Of ce of Personnel Management network were sophisticated but neither their methods nor their ambition was unprecedented The federal government had been subject to attacks for years by the same or similar groups using similar variants of malware In fact 0PM had reportedly been hacked in 2012 A vast amount of publicly available information on similar hacks within the past decade was available that should have put 0PM on notice Furthermore 0PM had every incentive to prioritize information security given the volume of sensitive information and PII it holds Despite red ags that began as early as 2005 appropriated IT security funding consistently lagged behind other agencies its most sensitive data was inadequately protected and 0PM leadership failed to heed recommendations from The Rise of Advanced Persistent Threat Hacking The longstanding 0PM cyber security failures that culminated in the theft ofpersonnel records background investigation data and ngerprint data began a decade earlier when the federal government was put on notice regarding the nature ofthe threat In July 2005 the LLB Computer Emergency Response Team issued an alert regarding sophisticated multi year efforts in which hackers send targeted socially-engineered emails commonly called spear phishing emails for the purpose of having a user download a le that would eventually lead to the extiltration of sensitive information Eml Though the term would not emerge for several years the alert described what would come to be known as an advanced persistent threat attack Such attacks are focused on a particular set of high-value assets or physical systems with the explicit purpose of maintaining access and of stealing data overtime Because the attackers are sophisticated they can learn how to jump from system to system within a given network otten attempting to compromise administrator accounts in order to gain wider and higher levels of access and creating new footholds to maintain their access When a particular security precaution or obstacle prevents further compromise the attackers change tactics and maintain a presence on the network until they reach their ultimate objective The 2005 alert noted that APT attacks had already taken place and that they often used malware speci cally designed to elude anti-virus software and firewalls BI The alert specifically noted the use of McAfcc and Symantec names in connection with APT hacks foreshadowing the MeAfee name that would later be relevant in the 0PM breach Since 2005 the federal government has been repeatedly victimized by sophisticated sustained 1 attackers In 2005 an APT intrusion gathered data from Vehicle 0 Teehoieof vber Security Afar fingered ojan Email Attacks July 2005 Id 31 Id see rise Saulsbury Tr at 60 30 Assembly Building 53 Media outlets reported that Chinese involvement in the back was likely 34 In 2002' James A Lewis of the Center for Strategic and Intemational Studies testi ed before Congress that intrusions occurred at the Defense Department State Department and the Commerce Del-tartinenti'5 In late 2014 a media report catalogued a number of recent attacks against federal entities including the White House the State De artment the United States Postal Service 0PM and the Nuclear Regulatory Commission 6 Federal Contractors Holding Sensitive Federal Employee Information Targeted and Attacked In addition to the targeting of federal agencies the government contractors that provide services to these agencies and hold sensitive federal employee information increasingly have been targeted by APTs including several 0PM contractors that provide background investigation and healthcarc services The rst public reports of data breaches involving 0PM contractors surfaced in the summer of 2014 In August 2014 the largest background investigation contractor U S investigations Services LLC publicly acknowledged a data breach impacting employees of the Department ofHorneland Security BE Documents and testimony provided to the Committee indicate that USIS self detected this cyber attack in June 2014 immediatel noti lied 0PM and by early July 2014 had mitigated the attackers activity on their systems i In a June 22 2015 document provided to the Committee USIS said based on the results of an investigation conducted by a company called Stroa Friedberg it was determined that 1813 had been the tar at of an attack carried out by a state sponsored actor commonly referred to as an APT attack USIS told the that P11 for over 31 000 individuals associated with Keith Epstein a Ben Elgin Network Security Breaches Plague sass Bus WEEK Nov- 2o seas id 3 5 Hoiistic Approaches to Critter-security to Enabie Nenrorir Centric Operations Hearing ich EJ-re tire On Terrorism Unconventionai Threats and ofthe H Comm 2n Arman Ecru I I 1th Cong Apr 1 2003 statement of James Andrew Lewis 9'6 Jack Moore the Yea-refine Breach i0 Federai agency Data Branches in 2014 chroov Dec 30 2014 http iiwvviv ne slgov conu'cybersecurityi20 3 In 1990 USIS was established as a result oithe privatization ofOPM s Investigations Services and over the years was awarded a series of contracts to perform security clearance background investigations for more than 95 federal agencies There were a variety oftransition iSsues when the privatization rst occurred including questions about USIS employees access to government databases See General Accounting Office Privatisation Investigations Service Aug 22 1996 In September 2014 decided to end these contracts with USIS In early 2015 parent company filed for bankruptcy See Jill Aitoro It is G ietai USES is No More with Fianna-rt Banks-apron Feb 4 2015 33 Ellen Nakashil na DHS Contractor Stg ers Major Computer Breach D ictais say WASH POST Aug 6 2014 available at breach-of cials-sayt20 l4i03i06i8cd 13 134 1 139 1 1e4-ac54a0cfc I f9't4f8 a_story html 39 Hearing on 0PM Data Breach Part II statement of Robert Giannetta Chief Info Of cer U S Investigations Services LLC 9 Letter from Counsel for U S Investigations Serv s LLC to the Hon Elijah E Cummings Ranking Member H Comm on Oversight it Gov't Reform June 22 2015 Id Ex 12 Siren Friedberg Summary of investigation Dec 2014 31 USIS background investigation work for Customs and Border Protection the National Geospatialdnteliigence Agency Immigration and Customs Enforcement and the U S Capitol Police may have suffered compromise in the indicated this APT began in in late December 2013 and the last attacker activity was observed on July 4 2014 112 The US IS investigation also determined that this APT was focused on access to computer systems related to the background investigations business of U818 which should have made it very clear to all stakeholders that the target was background investigation dataf As a consequence of the USIS activity in the summer of 20 14 LIE-CERT visited the facilities of Keyl oint Government Solutions KeyPoint to do a network assessment which found items of concern that prompted additional review 94 In December 2014 press reports indicated that KeyPoint had been breached resulting in the possible PII exposure of over 43 000 federal employees is In June 2015 KeyPoint CEO Eric Hess testified before the Committee saying there was an individual who had an 0PM account that happened to be a KeyPoint employee and that the credentials of that individual were compromised to gain access to At the time of the 2015 data breach 0PM gave contractors a username and password and investigators would log-in with this 0PM credentiaLW In addition 0PM contractors holding sensitive healthcare information of federal employees have been the targets of APTs In February 2015 Anthem one of the largest health insurers in the country and provides coverage for 1 3 million federal employees announced a data breach involving 30 million records of current and former customers and employeesfl'S Then in March 2015 Premera another health insurance company that has an 0PM contract covering about 130 000 federal workers in Washington state and Alaska announced a data Letter from Counsel for US investigations Serv s LLC U313 to the Hon Elijah E Cummings Ranking Member It Comm on Oversight Sr Gov t Reform at 5 June 22 2015 9 Id at Sufi In describing USIS activities related to the June 2014 discovery noted that an employee of the forensic investigation rm Strez Friedherg they hired attempted to provide LIE-CERT additional forensic copies of hard drives with evidence of the attack on September 9 2014 but the LIE-CERT employee declined saying CERT was on a stand downStroz Friedberg Summary of Investigation Dee 2014 9 Hearing on 0PM Dotti Breach Pair statement of Ann Barron-DiCalnillo Director 95 See Christian Davenport Keanr'nt Network Breech Cotth Affect ofFea erni Workers WASH POST Dec 13 2014 12ft fedc'i 46c-B e1 -l 95 Hearing on 0PM Drtto Breach Part II statement ofEric Hess CEO KcyPoint Government Solutions 011 June 2'9 2015 the American Federation of Government Employees sued 0PM over the data breach and also named KeyPoint as a defendant in the lawsuit 9 Saulsbury Tr at 20 21 Wagner the UPM Director of IT Security Operations said multiple credentials were compromised during the 2015 incident but a KeyPoint credential was likely used for the initial attack vector Wagner added the adversary utilizing a hosting server in California created their own F18 Federal Investigator Service background investigator laptop virtually They built a virtual machine on the hosting server that mimicked and looked like a FIS investigator s they utilized a compromise key point user credential to enter the nettvork through the F13 contractor VPH portal Wagner Tr at 815 123 9 Reed Abelson St Matthew Mth'ons' ofAntltem Customers Int-gated in Cyberot tocft NY TIMES Feb 5 2015 available at Aliya Sternstein 0PM Monitoring Anthem Hock Feds Might be A eeted Feb 5 2015 available at neatgov 5t02iesclusi vc opin monitori ng-anthem- hack-b reach con ld impact- 13 ni fedsi 104T00i 32 breach that exposed medical data and nancial information or 11 million customersgq These attacks highlight the persistent target that federal employee data presents and the need to secure such data whether it is maintained in a federal or a contractor-operating IT system 0PM as well as other agencies faces the challenge of seeming their systems as well as overseeing the systems that government contractors operate on behalf of the government- In a 2014 report GAO found that while agencies established security requirements and planned for assessments the agencies reviewed including 0PM failed to consistently oversee the execution and review of these assessments m In response to recommendation to 0PM to develop document and implement oversight procedures for ensuring that a system test is fully executed for each contractonoperator system 0PM promised to review existing security policies and procedures to enhance their oversight According to website this recommendation remains open In the case of the OPM background investigation contractors who experienced data breaches in 2014 and 2015 0PM had approved IT security plans for both USIS and KeyPoint In April 2015 GAO repeated the message about the need to address the cybersecurity challenge of ensuring effective oversight of contractors implementation of security controls for systems contractors operate on behalf of agencies Based on testimony and documents submitted to the Committee the record indicates that 0PM had not informed USIS or KeyPoint about the March 2014 data breach before it became public '05 It is unclear whether the attack could have been mitigated if 0PM had informed their background investigation contractors but given the threat environment and the background investiggtion systems targeted it would have been prudent to alert the contractors immediately I 95' Framer-o Bitte Cross Says Doro Breech Erposed Medicoi Doro N Y TIMES Mar 2 15 Elise v'iebech P'et'ierni Wot ti'etzr Might be Victims ofPi-entero Doro Breech THF HILL Mar 19 2015 remera-breach m Gov't Accountability oirtcc onto-14cm Agencies Need to imp-tics event-grit Conit'ois Aug 21314 http t'i go viassetsi 'i i b Ed pd Gov t Accountability oi cc Need to ot-cctigst Connor s as Aug 2014 ma Open 14' U i Iii-5i Agencies Need to improve Oversight Controis last visited July 2 2915 so ovirec mmend ation order by bloc I Odinow sort seoretdesc3toa2e is Hearing on once octc Brooch For it testimony by Robert Chief Info of cer on Investigations Services Letter to the Hon Elijah F Cummings Ranking Member H Comm on and Gov't Reform from Counsel for LLS Investigations Services LLC USIS June 22 2315 Ex 8 9 It ATOs signed by 0PM and May 2014 Site Survey Assessment Form Hearing on Dotti Breech Port if statement of Eric Hess CED KeyPoint Government Solutions Email from KeyPoint Counsel to Majority Staff H Comm on Oversight dc Gov l Reform Feb 22 2 lo on le with the Committee - Enhancing Cybersecnrity ofTin'rri Forty and Vendors Hearing Before H Comm on Oversight Gov t Reform 1 14th Cong Apr 22 2015 testimony of Gregory C 1lt iilshusen Dir Info See Issues Gov t Accountability Of ce ms Hearing on 0PM Dotti Breech Port ii statement oI'Robert Giannelta Chief Info Officer U S Investigations Serv s DeSpite a contractual obligation to notify contractors immediately of a new or unanticipated threat or hazard 0PM did not notify their contractors KeyPoint and ofthe March 2014 incident id '05 Hearing on 0PM Dotti Breech Port Lt' Rep Gowdy questioning of OPM contractors and 0PM of cials on the de nition of 33 Agencies today rely on federal contractors to operate IT systems on behalf of the federal government and must access federal systems in order to perform services for the federal gevcmment The potential risk of unauthorised access to IT systems operated by federal contractors on behalf of the federal government or contractors IT systems should not have been Surprising to 0PM in the years leading up to the data breaches Federal Initiatives to Increase Information Security in Response to Increasing Attacks As the first warnings of APT attacks began in 2005 the federal government was beginning to strengthen access controls On August 5 2005 OMB issued guidance to implement a Directive requiring the development and implementation of a mandatory govemment wide standard for secure and reliable forms of identification for federal employees and contractors The guidance Implementation of Homeland Security Presidential Directive HSPD l2 Policy for a Common identi cation Standard For Federal Employees and lOontraetors advised the heads of all departments and agencies that ijnconsistent agency approaches to facility security and computer security are inef cient and costly and increase risks to the Federal government m The Administration issued implementation guidance in the immediate years after the 2005 Directive was issued In response to multiple attacks in 2003 the federal government began a major new initiative to improve the security of its systems 1 Meanwhile attacks on federal systems continued and increased in volume and sophistication Federal agencies only needed to look at attacks on government contractors and other private sector entities for a playbook about what they needed to able to counteract ln 2009 Chinese groups with ties to the People s Liberation Army reportedly carried out dozens of APT attacks against inter a'iin Notthrop Grumman Lockheed Mania and Dow Chemical 1 1' Memorandum from Joshua Bolton Dir Office of Mgmt 3 Budget Exec Of ce oftlte President to Dep t and Agency Heads vi-05424 impiententation of omeiand Securirv Presidential Directive HSPD i2 Pniiev n- or Common Identi cation Standard for ederai Employees and Contractors Aug 5 On August 2004 the President signed 2 Policy for a Common Identi cation Standard for Federal Employees and Contractors the Directive 33 Memorandum from Joshua Bolton Dir Office Budget Exec Of ce of the President to Dep t and Agency Heads impienientotian ofHomeiand Security Presidentiai Directive i2 Poiicvfor a Common identification Standard Jfor Federal Ettipioyees mtri Controetonr Aug 5 2095 Memorandum from Karen S Evans Admia'r Office efE-Gov't EL Info Tech Exec Of ce ofthe President to Chief lnfo Of cers and Senior Agency Of cials for Privacy Sangria Privacy DoctnnentsforAgenev Impiementatiaa of Homeland Secttrirv Directive i 2 Feb See aim Exec Of ce of the President Press Release Certified Precincts anti Services New AvoiiabieforAgency Acquisition July 5 20136 a National Security Presidential Directive 54 Policy Jan 3 available at 1 ayyaz Rajpari Finding the Advanced Persistent Adversary SANS Sept 29 21314 5512 34 Four years later the situation had not improved and appeared to be getting worse A 2012 white paper by FireEye stated Federal agencies are increasingly the victims of advanced persistent threats often comprised of inulti-staged coordinated attacks that feature dynamic malware and targeted spear phishing emails In fact in spite of massive investments in IT security infrastructure on a weekly basis over 95% of organizations have at least 10 malicious infections bypass existing security mechanisms and enter the network Further 80% experience more than 1110 new infections each week Every day mission-critical systems are compromised and sensitive and classi ed data is eal'iltrated from federal government and civilian networks' '2 0PM itself was also targeted in the years leading up to the breaches discovered in 2014 and 2015 In May2t112 a hacker reportedly broke into an 0PM database and stole 3 user 1135 and passwords That breach was reportedly carried out by a group called @klldetec an activist affiliated with the hacking group In 2011 the Department ot Homeland Security issued a cyberseeurity bulletin that called Anonymous script kiddies using rudimentary exploits If true Anonymous did not need advanced technical pro ciency to gain access to an 0PM database 115 0PM Failed to Recognize the Threat and Implement Effective IT Security Measures When It Mattered The threat ol'APTs was well known throughout the federal government and 0PM was a prime target given the sensitive information it held on current and former federal employees and contractors Thus 0PM should have made infomtation security a top priority In the years preceding the breaches at 0PM in 2014 and 21115 however information security was just one of several competing agency priorities and network vulnerabilities became more acute In late 2013 and early 2014 under Director Katherine Archuleta and 310 Donna Seymour 0PM attempted to re-l'oeus on improving IT security It did not work Ineffective leadership and poor decision-making plagued the agency during a critical period in 2014 leaving the agency in a weak position to prevent the breaches 1 phat-Attacks on Government How APTAtt nchs are Contprontising Federal Agencies nna How to Stop Them 21 1 12 rcyel irnagesfii rceye-cyher attac Paul Rosenzweig fherllorrning Trend onyhersecwiry Breaches and Failures in the Government Continues HERITAGE FDUND Nov 13 21312 available at 2r 1 continue citing Privacy Rights Clearinghouse Chronology of Data Breaches available at see also Plaintiff s Class Action Complaint and Demand for Jury Trial 2 Aug 14 2015 Krippendorf v US Of ce ofPersonnel Mgmt D D C No 1 15 CV 11321 at 21 available at gs reuters eoml alison frankelr lesl 2t 1 51031 kripp endorfvopm-complaint pdf Lee Johnstene LES O icc ofPemonne Management Hacked E Horn Looked by @h detec CYBER WAR NEWS May 13 2012 available at That individual also carried out an attack on the Glade County Florida Sheriff s department 5 Nat l Cybersecul lty dc Cemm n Integration Ctr Dep t ofHomeland Sec Bulletin 35 0PM consistently reported spending less than other federal agencies on cybersecnrin In 2013 FY 2014 and FY 2015 0PM spent seven million each year on cybersecurity spending that was consistently at the bottom relative to all other agencies that are required to report such expenditures to the Of ce of Management and Budget 1 '5 The previous scal year 2012 0PM also lagged behind other federal agencies 0PM sought additional funds for oybersecurity but only after US-CERT noti ed the agency about the damaging breach in 2014 On March 20 2014 Computer Incident ReSponse Team CIRT received notification from that data was being ex ltrated from network 1 In the weeks that followed 0PM leadership would become aware the intrusion led to the breach of background investigation data in 0PM systems holding the crown jewels ofthe American federal workforce and national security personnel I 5 0PM requested additional cybersecurity nding in its FY 2016 Budget Justi cation released February 2015 and only then ten years alter 0PM took over the background investigation function acknowledged it was a target rich environment In a February 2 2015 letter to the House Appropriations Subconunittee on Financial Services and General Government concerning its budget request then-Director Katherine Archuleta noted Y201 6 request is $32 million above our FY 2015 appropriation Most of these funds will be directed towards investments in IT network infrastructure and security As a puoprictor of sensitive data including personally identifiable information for 32 million federal employees and l'ctil CGS hRFM has an obligation to maintain contemporary and robust cybersocurity controls I After years of neglect the request for increased funding in February 2015 was too little too late It came more than one year after attackers stole security documents that provided a roadmap to systems '20 And the request came after hackers had already successfully ea ltrated sensitive data including background investigations data in July and August of 2014 and federal employee personnel records in December 201-11 I21 is See Mm Report Appendix Cybet security Spending at 0PM Fiscal Years 2012-2015 see also Uf ce of I'vIgmt 3 Budget Exec Df ce of the President Annual Report to Congress Federnl' fin orninn'on Security Management Act 32 Mar 13 20115 availabie at docsf nal fy 2015 sma report to congress 03 13 2015 pdf 3ee oiso Of ce of Mgmt 8 Budget Exec Of ce of the President Ainnm'nF Report to Congress Federal Information S'eem'l'tj J Management Act 33 Feb 2015 available at docsf nal_fv14__ sma report 02 27 2015 udf i June 2014 one Incident Report 3 June 2014 0PM Incident Report at HOURUSIE 001245 1 1 5 Of ce of Pete 0PM Congressional Budge Justification I erfonnanee Budge 1 20 at 2 Feb 20 15 ww oprn gov If about-us-r'bud get-performance Ir bud getsfcon gressionai budget j usti cation fyml o pdf June saw one Incident Repelt a Hoeaosls nurses 0PM ILlybersosurity Events Timeline 36 The year 2005 was a key year for both 0PM and federal cybersecurity The IG and US- CERT issued a general technical alert which should have made 0PM aware of the need to increase IT security in the face of increasing APT threats '22 and 0MB was gearing up to announce and begin implementation of HSPD-IZ 123 The 0PM 10 also issued a warning in a semiannual report that would be repeated in subsequent reports It warned 0PM relies on computer technologies and information systems to administer programs that distribute health and retirement benefits to millions of current and former federal employees and eligible family members Any breakdowns or malicious attacks hacking worms or viruses affecting these federal computer based programs could compromise ef ciency and effectiveness and ultimately increase the cost to the American tartpayer 24 Amidst efforts to tierti fy federal cybersecurity 0PM was also working in 2005 to assume responsibility for the processing and storage of federal background investigations 0PM accepted the transfer of the Personnel Security Investigations function and personnel from the Department of Defense s Defense Security Service authorised by the National Defense Authorization Act of 2004 at 108-136 '15 The transfer from use to name Federal Investigative Services F18 division brought under one roofa trait that is conducting 90 percent of background investigations for the entire Federal Government u Congress applied pressure on 0PM to process the background investigation caseload more ef ciently by tasking with meeting timeframes imposed under The intelligence Reform and Terrorism Prevention Act PL 103-458 This was an important function in the wake of '33 Cyber Seenrityriiert masters rergereer Trojan amen Arreet-r July snag '13 Memorandum from Joshua Bolton Dir Of ce 3t Budget Exec Of ce of the President to Dep t and Agency Heads impiententatimr rr-I'Hotneitmti Pt'esitientiai Directive Poticyfor t't Common ia'ent ication Standard for Feo'et'ai Empioyees and Contractors Aug 5 On August the President signed lfSl D-lE Policy for a Common Identi cation Standard for Federal Employees and Contractors the Directive 124 Of ce ofthe Inspector Gen US Office of l ers Mgmt Report to Congress Detober t Marci 3 i Etittj 11 May 1 BUGS available at Leportsisar lpdf '33 us other at Pets Mg mt arenas Congressionai trrerraeerrerr Performance Badge 9 tree 5 acct available at US Of ce ofl ers Mgml Press Release 0PM Consolidates Bttiit' ofFea erai Security Ctearance Process with Transfer- rJOver Emptoyeesfronr Defense Department Vast Majority ofFerierai Background tnrestigationr to be Centered at 0PM Nov 22 The LLS Of ce of Personnel Management and Department of Defense announced today the transfer of over 1 343 personnel security investigation staff from Don to UPM This move will consolidate the vast majority of background investigations for the Federal government with 126 US Office of Pete F'I Ett t Congressional Budget Justi cation Performance Budget 9 Feb 5 200 available at 7 Intelligence aeterm and Terrorism Prevention Act of zoo-r Pub L No res-453 so use see oirto Rebeca Lafl urc How Congress Screwed Up America '3 Security Cieoronca System FOREIGN POLICY Oct 1 2131 available at 3 the terrorist attacks in September 1 l 2001 Various federal agencies and defense contractors increased their counter-terrorism staff '13 That staf ng surge caused a backlog in processing background investigations The backlog was at least 133 000 by 2004 119 The Intelligence Reform and Terrorism Prevention Act 10841-53 required that 90 percent of clearance applications had to be resolved within 60 days by 2009 a reduction of 84 percent from the then- 375 day average wait time am Clearing the background investigation backlog was a priority but there was also a clear need for 0PM to prioritize the information security of its data Over the 2005-200 timeframc the 10 s annual auditing identi ed weaknesses in the security of the agency's information systems which would deteriorate to material weakness status in 2001' In March 2003 the HTS Seminannal Report to Congress recognized a need or the agency to focus on protecting sensitive information and over the Unfortunately in today s high tech world inappropriate access to this sensitive information can lead to adverse consequences for the American public we are sworn to protect and serve Consequently the Office of the Inspector General has identi ed and reported the protection of personally identifiable as a top management challenge for the U S Office of Personnel Management 0PM and we believe it is a challenge that will be ongoing because of the dynamic and ever-evolving nature of information security Recognizing the adverse consequences of lost or stolen Pll including substantial harm embarrassment and inconvenience to individuals as well as potential identity thell Director the Honorable Linda M Springer initiated a series of actions beginning last fall She wanted to make sure that all 0PM employees clearly understood what meant the impoitancc of protecting Pl and their responsibilities in protecting it In systemt 1 1 3 Of ce of Pers Mgmt FYEWS Congressional Budget Justi cation Pel rrmanee Budget 9 Feb 5 2013 See Rebeca La ure How Congress Screwed Up America s Security Clearance System FOREIGN POLICY Clot l 2013 available at clearance-systems '19 Id Intelligence Reform and Terrorism Prevention Act of 2004 Pub L No 103 458 5U U S C 334 2012 see also Rebeca Lallure How Congress Screwed Up America 's Seem-int Clearance System FOREIGN PULICY Dct l 1'1 Of ce of Inspector Gen LLS Of ce of Pens Mglut Seminal-moi Report to Congress Aler I 290 September 3H Elli at H Sept available at onstsar 't df Of ce of Inspector Gen 11 5 Office of Pers Mgmt Searinmmal' Report to Congress October l 2th to March Elliott at i Mar 2003 available at '3 ltilflice of inspector Gen LLS Ull'iee of Pets Sentimental Report to Congress Detoher I 2th to March 31 Edtl' t at I Mar available at 1n ow'newsi're ortsr nblicalionstsemi-annual- reportafsar pd 1When the agency made a push in EDGE to ensure all 0PM employees clearly understand what PII meant the importance of protecting Pit and their responsibilities in protecting it 0PM security stat'fthat were 38 In the fall of 2003 however the 10 reported that the material weakness from the prior year had not been fully addressed and that it had some signi cant concerns with aspects of the agency s information security program '34 The warned that major elements of policies had not been updated in ve years found signi cant de ciencies existing in the control structure of management of major system certi cation and accreditation as well as in the plan of action and milestones process and that the agency operated without a permanent IT security of cer for over six months 135 In the spring of 2009 0PM underwent a leadership transition At John Berry's Senate con rmation hearin in March 2009 Mr Berry was questioned extensively on the security clearance backlog 1 however Congress did not pose any questions to him about information security In Berry was con rmed in April 2009333 and in September 2009 he testi ed at length on the need to modemize the security clearance and to eliminate the clearance backlog 9 His prepared testimony noted that work to improve background investigation processing would include efforts to strengthen access controls Berry testi ed We are working to bring the benefits of access to the veri cation system to new user types to support agencies in Personal Identity Veri cation PW credentialing We are working with the stakeholder community to identify potential enhancement to the veri cation system to permit greater reciprocity We are developing a web-based automated tool to assist agencies in identifying the appropriate level of investigation Meanwhile in September 2009 the reported that the state of information security at 0PM was worsening The 1G stated In our FY 2007 and 2003 FISMA audit reports we reported the lack of policies and procedures as a material weal-mess While some progress was made in FY 2009 detailed guidance is still lacking This yearbreach response were already working at 0PM For example Jeff Wagner UPM's current Director of IT Security Operations began working at in June 2005 In transcribed interviews Mr Wagner also admitted that he had been on a Performance Improvement Plan in 2012 or 2013- He said believe the PIP that I was placed on was became in my aggressive nature towards IT security I had offended a few cople See Wagner Resume at 00000 Production Aug 23 2015 Wagner Tr at 141-142 3 Of ce of Inspector Gen US Office of Pers Report to Congress Aprii i 2003 September 2003 at 15 2008 available at m ovinewsi orts- ublieationsisemi-annual re o sisarEEl df til '16 Nomination qf'i-trm M Joim Berry to he Director U ee ofPers-omrei Management Hearing Before the S Sperm on tiomeiond See it Gov 'tA ot'rs 11 1th Cong Mar 26 2009 Id 3 us Of ce ofPers Mgmt rate Release John Her tjt Cory hmeti as 0PM Director Apr a zone Securith Cieormree Reform Moving fret-word on Modernization Hearing Be rre the on Dt'eteigirt of Gov 't the tied Wot-Home rll' D C oftire 3 Comm 0n itomeirma See t'E' Gov 't A oirs ll 1th Cong Sept 2009 statement of John Berry Director US Of ce of Pers Mgmt- _ tri 39 expanded the material weakness to include the agency s overall information security governance program and included our concerns about the agency s information security management structure For example in the last 18 months there has not been a permanent Senior Agency Information Security Of cial or a Privacy Program Manager resulting in a serious decline in the quality of the agency s information security and privacy programs With the recent appointment of the new SAISO and the planned Office of Chief Information Of cer reorganisation which may involve increased staf n levels we will reevaluate this issue during the FY 2010 FISMA sstait 1 In the spring of 201 U the continued to report significant concerns regarding the overall quality of the information security program at 0PM '42 The warned that the agency had not fully documented information security policies and procedures or established appropriate roles and responsibilities and that while an updated Information Security and Privacy Policy was nalized in August 2W9 it did not speci cally address IT environment and lacked detailed procedures and implementing guidance '43 The 16 also questioned in 21310 whether 0PM leadership was conunitted to information security over the long-term The stated This year we expandcd the material weakness to include the agency s overall information security gevemance program and incorporated our concerns about the agency s information security management structure The agency appointed a new SAISO in September 2009 however the individual left in January 2010 Another new SAISU was appointed in late April 2010 With a new Chief In formation Of cer also recently selected 0PM may nally be in a position to make long needed improvements to its IT security program However given this turbulent history it remains to be seen whether senior management is fully committed to strong IT security governance for the long term 44 In 2012 0PM Director Berry ordered the centralization of security duties to a team within Office ofChieflnformation Of cer DCIO In March 2012 the 16 reported that Our audit showed that the agenc continues to struggle with improving the quality of its information security program The lG also found that the agency s OCIO lacked the authority it needed to manage security matters effectively and that the agency needed to move to a more centralized system because the fundamental design of the program is flawed The 1G Office of inspector Gen US Of ce of Pcrs Mgmt Report to Congress Aprt't I Still to September 3Q 2909 at Sept 2DU9 Of ce of Inapector Gem US Of ce emes Report to Congress October 20 79 More 20m at Mar 2010 14 Id 1-H 5 Office of inspector Gen LLS Office of Pets Mgmt Sentimental Report to Congress J 2i t to March ENE at Mar 21 2 n1 ovtnewst re orts licalienstsemi-annual-re g s saral pdf '45 1 1 3 Uf ce offersonnel Mgmt IL'lf l ice General Sentient-met Report to October t to March 31' 2W3 at 8-9 Mar 2013 available at 4D pointed out that OPlvt s designated security of cers were appointed by and report to the program offices that own the systems but very few of the 3th have any background in information security and most are only managing their security responsibilities as a secondary duty to their primary job function m The It found that IT security at 0PM was limited because the OCIO has no authority to enforce security requirements and concluded IT security is a shared responsibility between the OCIU and program of ces The OCIO is responsible for overall information security governance while program of ces are responsible for the security of the systems that they own There is a balance that must be maintained between a consolidated and a distributed approach to managing IT security but it is our opinion that approach is too decentralized 0PM program of ces should continue to be responsible for maintaining security of the systems that they own but the D30 responsibility for documenting testing and monitoring system security should be centralized within the 0010' In other words there were increasing calls iior centralizing and fortifying authority and power under the OCIO by the BIG By the end of FY2013 the centralised structure for information system security of cers remained understaffed and hampered by budget restrictions t4 And in 2013 as the agency prepared to transition to new leadership the IG released two key reports First its newest FISMA audit found that the security of information systems remained a material weakness 50 Second the It also issued a warning about the information system where background investigation materials are stored In June 2013 the Ca audited OPM's Federal Investigative Services Personnel investigations Processing System PIPE The made clear the importance of this system Approximately 15 million records of investigations conducted by and for 0PM the Federal Bureau of Investigations FBI the US Department of State the 11 8 Secret Service and other customer agencies are maintained in FIPS Furthermore the system interfaces with several other FIS systems to process applications while its data ow relies on both the UPM Local Area Networio Wide Area Network and Enterprise Server Infrastructure 531 general support systems '51 area I47 HIS w Office of inspector Gen LLS Office of Pete Fedora ft'yr t m t't' tt Security Management an Audit ET 2513 at 5 Nov 2 I 2-313 man 3 4321 pdf 150 Office of Inspector Gen LLS Of ce of Pers Report to Congress October t to Mot-cit 3i rota at m ovfliews rc orts- ublicati Office of inspector Gen US Office of Pers Mgmt Semionmtot Report to Congress April I Etit j to September 30 2 33 at 1 Sept 2013 available at 41 In the case of PIPS we tiound that there were a number of controls inappropriately labeled in the system security plan as or inherited As a result these controls were never tested increasing the risk that these controls may not be functioning as intended and therefore posing a potential security threat to the system This omission is particularly concerning given the purpose of the system and the nature of the data the system contains '52 The G s warning about the weakness in and the need to protect the background investigations data was prescient The 10 s warnings were in effect when in 2013 the agency welcomed new senior leadership On May 23 2W3 Katherine Arehuleta was nominated to serve as Director of OPM 153 The Ll 8 Senate con rmed Arehuleta on October 30 2013f and she was sworn into of ce on November 4 2 113 155 Archuleta was a former teacher public administrator community leader from Colorado and the National Political Director For President Dbama s reelection campaign' Shortly thereafter in December 2013 Donna Seymour began her tenure as C10 57 During her Senate continuation hearing on July 16 2013 Archuleta made a commitment to work with her senior management team to create a plan for modemiaing IT within 100 days of assuming of ce and to identify new IT leadership using existing agency expertise and with advice from government experts '53 As Archuleta and Seymour began their tenure modemieation was a key part of the Director s early agenda Director Archuleta announced a new Strategic Information Tecnnot'ogy L53 White House Press Release President Announces His intent to Nominate Katherine Archia eta as Director of the O 'ice ofPersom-iei Management May 23 2013 officeiil 4 Lisa Rein Senate Cot irms Katherine Arct'tnieta as the New Fetter-oi Personnel Chief WASH lDiet EU 21113 available at ton ost con1t olitiestr fedgral-p ersonnel-ehief'i 3 1 55959bb0 4 a l le3 a624-41d bt th b'i'R storv html '55 US Of ce of Pers Mgml Press Release U18 O ice ofPers Katherine Sworn-in as i an Director oftite Q 'iee ofPersonnei Management Greets Enipioyees as the New Director ana Gets to Wot-tr Nov 4 2013 available at 1 Uth-direetor-of HE Cecilia Munoz Wetcoming KatherineArchateta the First Latina Director oftiie ice off ern'atme'i Management THE Nev 4 2013 4 39 pm available at segovtblogt'l 1 3 nel- management 71am Miller Continues at see one 0PM Fen NEws More Dec 20 2013 hi comttechnology l 3t 1 Etcio-shuf e-contin uesml sha d hS Dp mt U3 US Of ce of Pets Mgmt Strategic Infomiation I'eciinotogy Ptan Feb 2014 available at cet'strategi 0 pla nsistrategic- it-plan pd F 42 Pian in as working days 12 calendar days alter being sworn in on November 4 The Plan listed information Security as one of sin IT Enabling lnitiatives that is initiatives to provide the strong foundation necessary for successful operation development and management of IT that increases accountability ef ciency and The sixty-nine page report includes a brief discussion of the background investigation systems '51 but the overall discussion relatcd to background investigations focused largely on process reform and automation 52 The Plan also included two-and-a-ltalf pages on information security wherein 0PM stated it will 0 follow guidance from the Federoi Information Security Management Act NIST SUD-53 Security and Privacy Controls for Federal Information Systems and Organizations '63 I follow guidance from OMB to ensure protection of these systems that contain PII and protected health information i work with DHS to implement continuous diagnostic monitoring and use information security continuous monitoring ISCM tools I implement a three-phase plan to carry out its ISCM strategy and Ir attempt to secure additional resources to hireitrain 1T staff 54 Seymour later recounted early efforts to assemble the Strategic tight-motion Technology Pton with Archuleta In June 2014 Seymour testi ed to the Senate Conunittee on Homeland Security and Governmental Affairs its Chief Information Of cer or the Office of Personnel Management 0PM 1 am responsible for the IT and innovative solutions that support mission to recruit retain and honor a '59 Joe Davidson 0PM Utrue iis ET Finn to improve Federai Retirement Operations Recruitment WASH Post Mar 1U 2014 available at imp rove-fetleral-retirernent-on erations-recrpit 4i 3i iineerbSZ-a ii Ef i LLB Office of Pete Strategic Itf t ttmii ti Tachooiogv Finn at vii Feb ante id at 32 The Plan s reference to background investigations included one line on security 'i he initiative will also support reform in the investigative process and drawing on the enabling initiative of information security protect and secure the volume of sensitive information in the EPIC systems the automated suite ofbachground investigation systems U S Of ce of Pers Strategic Ity omtation Technoiogv Pian 32 Feb 2014 ILS Dep t of Commerce HIST Spec Publ n SUD-53 Rev 4 Security and Privacy Controls for ederai ftyr t'ttit'tiittit Systems and Drganiaotions Apr 2GB available at littp iinvl pubs nist govi nistpuhsi peciaiPublicationsiI i IST SP3 00-53r4 pdf 1 1 3 Office ot Fers- l viganL Strategic information Teehnoiogv Pia-n at Iii 19 Feb 2014 Note While 0PM worked to craft the new Plan key corresponding updates to key internal security guidance and protocols and Authority to Operation For example OpMis Incident and Response and Reporting Guide was not updated guide issued in 21309 The Guide contains protocols for responding to breaches among other things See US Office oFPers incident Response and Reporting Guide 3 July See aiso Special Agent Tr at 8 The 0PM GIG special agent testi ed on October 6 2015 that the incident Response annIT Reporting Guide issued in Bill was still the guidance in efibct at 0PM as of October 2015 43 world class workforce Director Katherine Archuleta tasked me with conducting a thorough assessment of the state of IT at 0PM including how existing systems are managed and how new projects are developed This process has led us to identify numerous opportunities for improvement in the way we manage IT Ful lling the Director s promise 0PM released a Strategic IT Plan in March 2014 We developed the Strategic IT Plan to ensure our IT supports and aligns to our agency s Strategic Plan and that OPlvi s mission is ful lled It provides a framework for the use of data throughout the human resources lifeeycle and establishes enabling suecess il practices and initiatives that de ne IT modernization efforts The plan also creates a exible and sustainable Chief information Of cer organization led by a strong senior executive with Federal experience in information technology program management and HR policy 0PM also understands that new IT implementation will be done in a way that leverages eybersecurity best practices and protects the personally identi able information 0PM is responsible for '55 MS SEY MOUR 1 Donne Saga-near testi es to the Coimm'oec on Oversight and Government Reform When Seymour testi ed before Congress in June 2014 however she did not mention that the agency learned in March 2014 of a signi cant data breach at the agency nor did 55 A More Efficient and E erttive Government Fedemf IT Initiatives and the IT Worriforce Hearing Before the Snbcomm on a ieieliev d'c c eerit-eness ofFed Programs rE Fed Wei-Home ofrhe S Con-mt on Homeland Sec cf Gov r L f'oirs 1 13th Cong June It 2014 statement of Donna Seymour Chieflnfo Of cer US Of ce of Pets Mgmt 44 she mention that the agency under her and Arcbuleta s watch had spent the previous two months monitoring attackers and remediating a signi cant incident 1'55 On July 9 2014 The New York Times broke the news previously unknown to the public that 0PM suffered a breach The Times drew attention to the severe implications of the breach for anyone who had ever applied for a security clearance The story stated The intrusion at the Oi ce of Personnel Management was particularly disturbing because it oversees a system called in which federal employees applying for security clearances enter their most personal information including nancial data Federal employees who have had security clearances for some time are often required to update their personal information through the website The agencies and the contractors use the information from to investigate the employees and ultimately determine whether they should be granted security clearances or have them updated '53 While Tire Times immediately grasped the potential implications for the country CID was trumpeting the merits of the agency s IT Modernization plan in fact downplayed the damage from the breach to the Tire Times The story stated But in this case there was no announcement about the attack The administration has never advocated that all intrusions be made public said Caitlin Hayden a spokeswoman for the Obama administration- We have advocated that businesses that have suffered an intrusion notify customers if the intruder had access to consumers personal information We have also advocated that companies and agencies voluntarily share information about intrusions Ms Hayden noted that the agency had intrusion detection systems in place and notified other federal agencies state and local governments about the attack then shared relevant threat information with some in the security industry Four months after the attack Ms Hayden said the Obama administration had no reason to believe personally identi able information for employees was compromised None of this differs from our normal response to similar threats Ms Hayden said '59 1'56 June Edi-4 DPM Incident Report see oiso A More otto' E 'ective Government Examining Fetter-oi Initiatives and tire TT Workforce Hearing Before tire Subcomm on E iciency dihfectiveness ofFeti Programs 5 Peri Workforce oftire 3 Comm or - i-iomeiood See Gov t A nirs 113th Cong June it EDI-4 statement of Donna Seymour Chief Info Of cer LLS Of ce of Pets MgrnIJ Michael B Schmidt David E Sanger tit Nicole Perlrotb Chinese Hookers Kev Data on US Workers MY July 9 2014 available at FD res id at 45 Archuleta and Seymour later testi ed in 2015 that no PII was exfiltrated during the 2014 data breach Documents and testimony show gaps in audit logging practices led DHS to conclude the country will never know with complete certainty all of the documents the attackers exiiltrated during the breach discovered in March 2014 1 It is clear however sensitive data was ea ltrated by the hackers W2 As discussed in the following chapter 0PM watched the attackers steal documents related to 0PM IT systems including PlPs contractor information and documents containing names and the last four digits of associated Social Security numbersm Archuleta and Seymour did make some progress in addressing security governance issues by continuing to centralize IT security responsibility They committed to make IT a priority with the release of their IT Modernization plan in early 2014 and arguably had more ownership of its IT security at this point than ever before l-lowever they failed to prioritize data security and implementation of basic cyber hygiene measures at a time when it became critically important to meet the increasing cyber threat Karherine Arcltaiera restyies to the Conmtf ee on Oversight and Government Reform 0PM Dara Breach Part II statement oFDonna Seymour Chief Info foicer U S Uf cc of Pcrs Mgmt _ During this hearing then-Director Katherine Archuleta and then-CID of 0PM Donna Seymour testi ed nine times in a single exchange with Chairman Jason Chaffetz that no personally identi able information was stolen June 2014 0PM Incident Report a HooncSIs no 1233 1246 1 The sensitivity ofthese documents is evidenced in part by the fact that 0PM refused to produce these documents to the Committee in unredactcd form until February 16 2016 The Committee initially requested this information on August 13 EDIE 3 June 2014 0PM Incident Report at I-IOGRUSI -001245-1246 46 0PM Failed to Prioritize the Security of Key Data and Systems OPM's failure to prioritize high-value targets like the background investigations data compounded the problems caused by inadequately investing in eybersecurity in the rst place Neither the data held by OPM nor the access to 0PM systems were adequately protected Indeed REM did not even have a complete IT inventory of servers databases and network devices Further on the system level 0PM had not implemented multi factor authentication making weak access controls a vulnerability that attackers were able to exploit OPM's failure to prioritize multifaetor authentication implementation was a key observation that US-CERT made in their analysis of the data breach discovered in 2014 mi 0PM was pressed about these and other issues during congressional hearings For example the background investigations data was not is the foundation of data-level security During a June 16 2015 hearing before the Committee Chairman Jason Chaffetz asked Director Archuleta why 0PM did not use an industry best practice and Director Archuleta said It is not feasible to implement on networks that are too old 8 Similarly C10 Seymour told Ranking Member Elijah Cummings that the agency was working to use She testi ed 0PM has procured the tools both for of its databases and we are in the process of applying these tools within our environment But there are some of our legacy systems that may not be capable of accepting those types of in the environment that they exist in today In addition key systems were also operating in FY 2014 without a valid Security Assessment and Authorisation Ia l Also called authorizations to operatefauthorities to operate provide a comprehensive assessment of the IT system s security controls The 0PM IG Of ce of Inspector General us Df ce ofPers Report No meteors-ct l Federeti Itrfonttatimt Security Managementzict Audit Fi 23M at i Nev to 2015 available at I 5- 01 hymn-ration Techno er Spending and Data Security at the O iee ufFerw-mmei Management Hearing Before the Sitbcomm 0n inseam SEW 's and Gen Gov ofthe Sen Joiner on Appropriations with Cong June 23 testimony of Richard Spires fon ner CID of the Internal Revenue Serv 1 Sen Iry ro Chapter 2 newsman Technoth Spending and Data Security at the U ice ofPersomiet Monogram-sat Hem-tag Before the 0n Fimmctat' Sew '5 and Gen Gov oftl'is Sou Comm or Appropriations 114th Cong June 23 2015 testimony of Richard Spires former 310 of the Internal Revenue Seer 3 0PM Data Hit-each Hearing Before the H Comm on Oversight ri- Gov't Reform 114 Cong June Id 21115 statement nt Kalherine Archuleta Dir LLS Office of Pers Mgmt can Dot-s Breach Hearing Before the a Comm on Oversight a Ucv't Reform 1141h Cong June is 2015 statement of Katherine r trchuleta Din LLS Of ce of Pers m Of ce of the inspector General IDfiice ofPers Mgmt Fedora ItJormotiou Security Management Act Audit i 20H Nov 12 2W4 available at information-seeurity rn it f 2 l 4-4a-ci- il considers the authorization process to be a critical step toward preventing security breaches and data Of the 21 0PM systems due for reauthorization in FY 2014 were not completed on time and were operating without a valid and several were among the most critical containing the agency s most sensitive information 133 This led the IG to warn 0PM that The drastic increase in the number of systems operating without a valid Authorization is alarming and represents a systemic issue of inadcq uatc planning by OPM program of ces to authorize the information systems that they own l 4 ISMA requires agencies to assess the effectiveness of their information security controls the frequency of which is based on risk but no less than annually '35 Appendix of OMB Circular A lJt in place at the time requires that agencies assess and authorize formerly referred to as certify and accredit their systems before placing them into operation and whenever there is a major change to the system but no less than every three years rirararr ar '35 In November 2014 the 10 s FISMA audit stated We therefore also recommend that 0PM consider shutting down systems that do not have a current and valid Authorizationmm 0PM CID Donna Seymour responded however that The IT Program Managers will work with 830s to ensure that 0PM systems maintain current ATOs and that there are no interruptions to OPt vt's mission and Of the eleven major 0PM information systems that were operating without a valid Authorization in more three ofthese systems should have been an irru'ncdiate priority for Director Archuleta and C10 Seymour to ensure were addressed Personnel Investigations Processing System PIPS Enterprise Server Infrastructure E31 and the Local Area Networltf Wide Area Network LANFWAN The security ofthese systems is critical because the ow of background investigation data through relies on both the 0PM LANHWAN and Enterprise Server Infrastructure general support systems LANMAN serves as the hardware and software infrastructure at at 11 ft'f at 9 '33 E-mail from Of ce of Pure Mgmt Inspector cert Staffto House oversight at Gov t Reform Staff Dec 4 2015 on file with the Committee Office of Personnel Of ce of the Inspector General Federal Iryfir rrirrti'on Sacra-try Management Act Audit FY2314 at 9 Nov 12 available at infonnation sec 44a-ci-DIJ- pdf Federal Information Security Management not otzcez Pub L No Wit-34 44 use a 3541 2012 Of ce of Mgmt 8 Budget Exec Of ce of the President OMB Circular r t l 3t Management of Federal Information Resources Nov 23 20% available at at3 a1 see also US Dep t of Homeland See Security Authorization Process Guide 1 Mar 16 Etll available at lesfuublicationsfs ccurity%2 Authori2ation%2 DProccss%2thuide v l le f Of ce ortrte not us otaee or Pers Mgmt Report No Federal irtroraratioa Security Management Act Audit FY 2014' at 2 4 Nov 12 2914 available at l4-4a-ci- - l4-t'i pdf Irf '39 at at s 48 environment supporting systems housed at Washington Macon Georgia and Boyers Pa facilities LaNtwan also supports the cars PTPS imaging and FTS Fingerprint Transactional System ESI is the general mainframe environment that supports PIPS OPl vl s mainframe is considered a separate infrastructure or general support system from PIPS and 1331 were all operating on expired Authorities to Operate The need to prioritize the security of these systems was well-known after the IG wanted in June 2013 that PIPS had vulnerabilities and that the system interfaces with several other FIS systems to process applications while its data ow relies on both the OPM Local Area Nemorki Wide Area Network and Enterprise Server Infrastructure general support systemamgl However the ATO for PIPS was not reauthoriacd in 2014 and the 16 s FY2015 FISMA showed that management of system Authorizations has deteriorated even further l Experts from outside 0PM also criticized choices regarding IT security following the breach On June 23 20 I 5 Richard Spires the former CIO of the Internal Revenue Service and at testi ed before a Senate Committee on Appropriations Subcommittee on Financial Services and General Government that 0PM should have set better priorities and focused on securing the data itself rather than the systems as an initial priority- Spires stated I f I had walked in there as the CIO and you know again I m speculating a bit but and I saw the kinds of lack of protections on very sensitive data the first thing we would have been working on is how do we protect that data Not even talking about necessarily the '90 OPIS was also operating with an invalid authorization to operate See Of ce ofInspeetor Gen LLB Of ce of Pet's Mgmt Report No 4A-IS-0t ls tiutl24 ity oraratiott Ieeitnoiogy Sean'in Controt's oftire wire ofi ersonnei Management 's Personnet' Investigations Processing imaging System July I I see who mail from U S Of ce of Pers Mgmt inspector Gen Staff to House Oversight 52 Gov t Reform Staffmec 4 an I 5 on le with the Committee Of ce of Inspector Gcn US Of ce of Pers Mgmt Report No 4A lS U vl 343122 Atra'it oftae Information teeitnoiogy Security Controls of tire US D iee ofPersonaet t'u-ionagetnent 's Personnei investigations Processing System WW 3 June 24 2013 available at Of ce or Inspector Gem U5 Dt cc nt'Pers Mgmt Report No ti Federaf information Securith Management Act Audit 1 29M Nov 5 21312 available at optngoviou 2012 pdf Of ce Gem 11 5 Of ce of Pet's Mgml Report No Audit oftire information Teehnoiogy Security Controis oftt're DES O ioe ofPetsonnei Monagemeotlr Loeai Area Nehvorir t Wide Area Network Generai Support Evstem FY Edi May 16 EDIE available at manageme i lpdf '91 of the Inspector General LLS Of ce of Pets Mgmt Semimorooi Report to Congress Aprii i to 3t 2013 at 'i Sept 20E 3 available at re ortsisar49 df 1 Of ce of Inspector General US lLitf ee of Pets Mgmt Report No 1 Fedora information Security Management AetAoa'it FY Edi-i Nov 10 2615 available at ge neralireportsiztl I Sifedcral in fonnation1secu ritv-mo dcrnization ae t au dit-fv-ZU 15 na audi t-reoort-4a-ci-DD l 5 UI l pdf 49 Spires also stated that management issues posed a greater obstacle than resource problems in systems How is it we get better protections and then control access to that data better solving IT security problems Spires testi ed practice and to retain leaders that are committed to information security over the long haul A focused effort on protecting the sensitive data with the right and the right access-control capabilities if you put the focus there I think most federal agencies would have the funds have the resources to be able to accomplish that iltik Because of the sparse nature of the way IT has been run in a lot of agencies there are so many let s say inefficiencies that have crept into this system that I don t believe we effectively spend the IT dollars that we receive So I believe that with the proper drive towards management you can actually derive a lot of savings from existing budgets 5 0PM has long been plagued by management s failure to prioritize information security in Years of neglect compounded by an abject failure of key leaders to make the right decisions at 0PM in 2014 led to the worst data breach the federal government has ever experienced flyr rm timt Technofegv Spending and Data Security at the ica anenrnn-ne Management Hearing Before the on Financial Saw and Genera Gov ofrhe Comm on Appropriations 114th Cong June 23 2015 testimony of Richard Spires Former Chief lnfo Officer Internal Revenue Serv W i Id 50 Chapter 2 The First Alarm Bell Attackers Discovered in 2014 Target Background Information Data and Exfiltrate System-Related Data In the March 2014 USHCERT alerted 0PM to an intrusion that laid the groundwork for the breach of OPM systems holding background investigation data the crown jewels of current and former federal employees contractors and national security personnel '95 0PM considered their reopense to the data breach which they learned about from in 2014 a success C10 Donna Seymour touted the response strategy one of the things we were able to do immediately at 0PM in 2014 was recognize the problem We were able to react to it by partnering with DHS to put mitigations in place to better protect information m However the data breach of background investigation data and personnel records rst announced in June and July of 2015193 raises serious questions about response to the data breach discovered in EDI 4 Documents and testimony obtained by the Committee Show Successes and failures but some of the most important questions were unanswerable For example while 0PM testified that no personally identi able in formation was ea ltrated during the 2014 data breach 1w gaps in audit logging practices led DHS to conclude that the country will never know with complete certainty the universe of documents the attackers ea llraled 2m Documents and testimony show the materials eafiltrated from 0PM likely would have given an adversary an advantage in hacking This evidence calls Donna Seymour s testimony into question She told the Committee the adversaries in today s environment are typically able to use more modern technologies and so in this case potentially our antiquated technologies may have helped a little bit 202 a putting forward a security through obscurity defense the C10 downplayed the reality that 0PM was facing a determined and sophisticated actor while only having minimal visibility into their environment 1% June 2Ul4 0PM Incident Report see also David Peters 5 Joseph Marks sewn Disclosed Huck Got Frown Jewefs June Ill 2015 available at security-haekground-checits-l 18954 Enhancing of Word-Forty Contractors and Vendors Hearing Before the H Comm on Oversight of Gov 't Reform 114th Cong Apr 22 2015 Question by Mr Cummings U S Of ce of Pers Press Release 0PM to Nott ' Employees ofCvaersocnrt'ty Incident June 4 it ll 5 available at vtnewst'rclcascs l US Of ce of Pers Mgmt Press Release 0PMAnnonnoes Steps to Protect en srnl Workers and letters From halter threats July 9 21315 available at 1W Hearing on 0PM Doro Branch Port statement of Donna Seymour Chief Info Of cer US Of ce of Pers During this hearing then-Director of 0PM Katherine Archaicta and then-CID of OPM Donna Seymour testi ed nine times in a single exchange with Chairman Jason Chaffeta that no personally identi able information was stolen June 2014 0PM Incident Report or HUGHES 3-0131233 124d 2m Saulsbury Tr at It'll 28 Enhancing Cyhot'seotn't'ry Contractors and Vendors Hearing Hefm o the H Conn on Oversight t i For 't Reform 114th Cong 2015 Question by Mr Cummings 51 In the aftermath of their 2t 4 response available threat intelligence about the relevant actor groups targeting federal employee information and the types of malware discovered in 2014 also raised the stakes for 0PM In the fall of 2014 Novetta and a number of supporting industry organizations produced a detailed report containing information pertinent to Chinese APT activity with an emphasis on Hikit malware This malware was found during the 2014 incident response The Novetta paper speci cally looked at the Axiom Threat Actor Group which according to public reports was responsible for the OPM data breach discovered in The analysis wamed that among the industries being targeted or infected by Hikit were Western government agencies with responsibility for personnel management The report also warned that w ithin these targets Axiom has been observed as going out of its way to ensure continued access regardless of changes to its target s network topology or security controls 21314 0PM leadership downplayed the signi cance of the 2014 breach Instead 0PM should have raised the alarm and recognized this initial attack as a serious and potentially devastating precursor given how close the early attackers got to the background investigation systems and the related data taken during this breach The following discussion describes 2014 discovery and incident response efforts and how Hikit malware was found and sensitive data related to the background investigation function was taken from OPM's systems Further this discussion highlights key observations that were made about the weaknesses and vulnerabilities ofOPM s l'i' security during this incident response period On March 20 2014 0PM's Computer Incident Response Team received noti cation from that data had been eafiltratecl from Oi ivt s network Beginning March 2014 and through May 2014 0PM in consultation with investigated the incident monitored the attacker developed and implemented a mitigation plan and removed this initial attacker from system noti ed 0PM that a third party had reported data being es ltrated from system to a known command and control server Jeffrey Wagner Director of IT Security testi ed about 0PM activities upon notice from T he initial response to the 2014 data breach is a call from DHS All right So on 3f2t'i DHS called us and let know hey we think this is bad We began pulling logs and records and things of that nature and on 3f25 is when we veri ed that it was a malicious activity 2m i Novetta Operation SMN Axiom Threat Actor Group Report Id 3-9 ittne 2c14 ornr Incident Report at Hooacs M 0PM contractor Brendan Saulsbury stated that the NH incident was rst detected by LIE-CERT via the Einstein appliances that they have on network And that was communicated to 0PM via email Saulsbury Tr at 13 The 0PM Incident Report states that a third party reported the data ex ltration to June 2cm Incident Report at IIUGRUBI 8 4301233 It is possible that both accounts are correct and that the third party referenced in the NH Incident Report is an Internet Service Provider who reported network activity collected by an Einstein sensor Wagner Tr at 13 52 Wagner also described process for analyzing and elevating information security reporting or alerts to a cybersecurity incident He stated Once we get forensic evidence that there s actual adversary activity within the environment it escalates the level of response So for instance on a regular basis we get alerts or reports of an email ying to be sent to us that has a malicious link it creates an alert We ll do initial forensics on that alert and we ll see that our current tools will stop that malicious link from being able to connect or downloading anything And it de-escalates the situation So from an incident response perspective everything rises to a critical level and then once we have forensics evidence and identify speci cally what is going on and it then escalates into the speci c response As incident reaponse activities began documents show that as of March 20 2014 the following facts were among those known to 0PM I PIS Investigator accounts had been compromised The malicious C2 server was communicating with an 0PM server I The malicious C2 servers conununications with 0PM were During the incident response period 0PM learned the C2 server was connecting with an 0PM network monitoring server between the hours of 10 pm and 10 am then the attackers were using this server and a compromised Windows domain administrator credential to search for PIPs-relatcd files on network 2m An initial examination of the network trailic between the_ server and the C2 server found that the celmnunications were utilising a four byte XOR key indicating a speci c intent to disguise themselves amongst network traffic Brendan Saulsbury an 0PM contractor working in the OPM IT Security Operation group testi ed that 0PM used the security tool NetWituess to identify what devices on OPlvl s network were actively communicating or beaconing to the C2 server 2 Using the network traf c inicnnation gathered by NetWitness Saulsbury was able to design a custom script to reverse engineer the obfuscation algorithm the attackers were using to mask their traf c so it would not be detected by sensors like DPle's security tools 213 Saulsbury s team could then 303 M June 2014 GEM Incident Report at uooacsw came M at scoliosis-001233exclusive or is a form of pri vale key that relies upon a simple binary formula to develop its ohfusoation of the underlying data 2'3 Saulsbury Tr at 3'9 3 Saulsbury Tr at 4B 53 observe the infected machines communicating with the C2 server and also the commands that were being sent down from the actual attacker sitting at the keyboard 4 Thus 0PM and their interagency team were able to identify the adversary s initial foothold in network where the attackers had established a persistent presence in the environment Once it was determined which devices on network were beaconing to the hackers C2 server 0PM was in a position to begin a Full forensic investigation and look for malware on the compromised machines 5 On or about March 25 in the words of OPM Director of Security Operations Jeff Wagner a critical level m was reached and 0PM was able to make a full determination on the who and what 217 of the data breach to know where the hackers are going what they are seeing and most importantly what the hackers are interested in 213 As a result 0PM determined the incident was malicious on March 25 2014 moved DHS onsite to assist the response and began a full monitoring phase to gather information to answer the question of howimg During the three-month incident response period 0PM undertook a number of other incident response activities For example according to 2014 Report timeline on March 26 2014 0PM searched for embedded malware on end points at its Washington DC headquarters at its Boyers data center and at a back-up data center in Macon Georgiam On March 2014 0PM took steps to remediate the 0PM Personnel Investigations Processing System Imaging System system that provides an electronic representation of case paper les to expedite the roeessing of background investigations and performed this remediation work in late i'vlarel'i 22 On March 28 2014 in recognition of the fact that 0PM did not have the ability to monitor traf c in and out of HTS the system that held background investigation data 0PM installed a fiber tap to begin to monitor such traffic- Finally during this period 0PM watched the attackers take sensitive data relating to high-valued targets on systems such as the PIPS system 222 0PM was never able to determine how the adversary initially entered their systems Then from late March through April 2014 the incident response team continued to identify additional infected workstations and malware on key systems 223 Speci cally 0PM found Hikit malware on several 0PM systems 224 Hikit is a variant of rootkit malware which is an extremely stealthy form of malware designed to hide its malicious processes and programs from the detection of commodity intrusion detection and antimvirus As 1 Saulsbury Tr at 46 3 Saulsbury Tr at 39-40 6 Wagner Tr at 13 rnnc 2014 card incidcnt Report at Hooaasi nets-ta ii 13 June 2314 orM Incident Report at Hoeitcsis caiadi Int see also Office of Pers Mgmt 0PM Personae Investigations Processing Eastern Ira-raging System DIES Primer Assessment available at olic t' i s-ima in stem df 13 June 2 14 DPM Incident Report at 23 June EUM 0PM incident Report at 241 1242 f June 2014 can Incident Report at sconce sear 234 at at Appendix c 33 June stud 0PM Incident Report at Hoortcsi s-aa 234 54 explained in the June 2014 0PM Incident Report HiKit allows the attacker to run commands and perform functions from a remote location as if they had the equivalent of a monitor and keyboard connected to the compromised 0PM Time is crucial in an incident response scenario According to NIST organizations should strive to detect and validate malware incidents rapidly because infections can spread through an organization within a matter of minutes m The agency s slow response made matters worse According to NIST minimizing the number of infected systems which will lessen the magnitude of the recovery effort 128 Once the incident was identi ed and 0PM along with their interagency partners entered into an advanced monitoring phase necessary intelligence was gathered on the adversaries tactics techniques and procedures the kind of threat infonnation necessary to harden information security not only at 0PM but at other agencies From March 25 2014 to May 27 2014 0PM upon the advice of engaged in a prolonged intelligence gathering phase The goal of this advanced monitoring phase was to carefully observe all ofthe malicious actors activities in order to gain an understanding oftheir tactics techniques and procedures as well as to identify all of their other unknown or inactive infected systems within networ 329 The advanced moniton'ng of the adversary ended in a Big Bang on May 27 2014 -an effort that commenced once the hackers got too close to the background investigation material accessible from the PIPE system 130 Saulsbury described the comprehensive monitoring strategy during a transcribed interview with Committee investigators He testified advice was to basically do an ongoing investigation and gure out do our best to nd the entire attacker foothold in the network and then remodiate them all at once to prevent the attacker from realizing that you are aware of them and then changing their tactics and techniques to irther avoid detection Wagner also described the scope of the monitoring phase He testi ed that 0PM was not just looking for TTPS but other indicators Wagner stated 31 June 2014 new Incident Report at liDGR EiE- li Peter Mell Karen Kent Joseph Husbaum Nat'l Inst of Standards at Tech Spec Publication Gm de to Motivate Incident Prevention and Handling 3 Nov 21305 available at 9 June 2014 0PM Incident Report a Hooaesis caress 11' Saulsbury Tr- at 26 23 Saulsbury at 25-26 55 You re trying to nd speci c actions they re doing to give you an indication ofwhat they re doing and what they want You re also looking for as a former pen tester usually what you try to do to try to prevent people from catching youother back doors or means in which you can create a persistent attack It s just making sure you always have a secondary way in In June 20 4 0PM Incident Report there is almost a daily catalogue of monitoring efforts As part of the monitoring effort 0PM established a series of alerts and system rules to watch the adversary employing a full packet capture glogging data tool to gather network traffic between the infected machines and the C2 server An interagency team including FBI and was involved in the incident response effort The team received automatic notifications during the monitoring phasem During this 2014 incident response period 0PM used its existing set of security tools and infrastructure to conduct their monitoring effoer in addition to monitoring 0PM was prepared to implement preventative measures For example Wagner testi ed that they were instructed to shutoff internet access if any P11 was leaving the networkm By March 2014 reported that 0PM had heightened proactive readiness and was developing plans for full shutdown 233 By April It 2014 tactical mitigation strategy and security remediation plans were being developed to eliminate the adversary s foothold on network The process of setting up alerts and tipping points identifying infected workstations and elevating monitoring continued until the Big Bang on May 27 2014 While the timeline is helpful to understand the 2014 incident response activities some entries illustrate gaps in visibility their systems and applications including the highly sensitive system which housed the sensitive background investigation data For example the March 23 2014 timeline entry states PM did not have the ability to monitor tren'te infout of PIPS Installed PIPS fiber tapi 'l Wagner responded to this entry by testifying So in that speci c instance -- a mainframe functions significantly different 1 Wagner Tr at lit June sot-t Incident Report at means 13 octane Saulsbury Tr at 43 brought the NBA Blue Team 3 Wagner Tr at 59 So ifthe adversary's activity was from It p 111 to to am but it was normally in a period cf3 to at am where they were active when they would throw something on our network or send a script to the network 1 would get a phone call I would then call DHS and FBI So it was a concerted effort It wasn t simply 0PM by 3 June 2014 0PM Incident nepen at Wagner Tr at 10 The question posed to Mr Wagner was whether or not the security staff at 0PM had the authority to make operational decisions his answer stated that guess a good example would be during the NM or 2 15 breaches the security operations group was under a standing order from the director that if we indicated that information was leaving we could shut down the Internet at any tints 1 June 2014 earn Incident Report at HUGRDSIB eaten in are 56 from a standard distributing environment say Linux or Windows or like you have at your home A mainframe is a giant cloud computer which runs on a proprietary type operating system and it communicates in a far different method than a standard distributing environment So at the time we did not have equipment installed to try to navigate between distributed and mainframe We had a project to implement these pieces and what we did is we sped up the project to get the ber taps installed to be able to set up a communication method to where we could see the traf c as it traversed between the distributing environment and the mainframe environment Saulsbury also described limited ability to monitor Internet traffic during and prior to the 2014 incident He testi ed 0PM had the ability to monitor traffic going out to the Internet at all times or at least going back prior to the 2014 incident The reason for putting a network tap on the PIPE segment is to be able to monitor what is called what we refer to as east-west traf c so intemal-to-internal traffic from the general network going in and out It was not until March 31 2014 that 0PM was able to turn on the monitoring capabilities for all PIPS and Federal Investigative Services PIS related systems 243 In other words it took almost eleven days from the time 0PM was noti ed on March 20 2014 about the data breach for 0PM to deploy the capabilities necessary to monitor one of the most high value targets on their lT environment PIPs The timelinc also highlights other gaps in information security posture that made 0PM vulnerable to attack and put sensitive data 0PM held at risk For example a March 31 2014 entry states high value targeted users only needed to authenticate with username and password which could be compromised remotely Enforced PW access for 5 high-value users Jeff Wagner testi ed about challenges related to implementing PW functionality Q Were they not being enforced prior to that A No Why was that A It was a project that was on the list and to completely change the culture and the ittctionality of some systems it takes planning 3 lIp v'agner at 19-20 3 Saulsbury Tr at 35 3 June 2c14 Incident Report at access is hoist l 1 June 2014 0PM Incident Roporl at Hooacsis smarts 57 Q When you say the culture of some systems what do you mean by that A So as users have built systems throughout years or decades they have become accustomed and there s business or operational procedures that rely on speci c methods In order to change authentication methods from like user name password to PW some of those processes have to get rede ned and republished 245 Thus the challenge of fully enforcing multi factor authentication through the use of Pl v cards arose in part from the agency s culture Wagner testi ed that maintaining the mctionality of the production environment was related challenge in deploying PW He said full deployment of PW caused certain applications and certain functionalities to break Wagner testi ed that in response to the 2014 breach remediation plan 100 percent of windows administrators began utilising Pl v' cards through an Xceedium appliance and by September 2014 all 0PM users were PW compliant 241i According to an OMB Report on Fiscal Year 2014 activities 0PM still had not fully implemented PW card access rules 0PM was identi ed in this OMB Report as one of several agencies with the weakest authentication pro lelsj meaning a majority of the agency's nnprivileged users logged on only with a user ID and password making an unauthorized access more likely While 0PM monitored the situation in 2014 to the extent their 2014 security posture allowed the next step was to develop a remediation plan to eliminate the attackers presence on the network Prior to the May 2014 Big Bang effort to eliminate the attackers from network Jth began taking other ad hoc measures to mitigate the damage In early May 0PM began setting up green zones the security team s effort to eliminate certain administrators from being on the network to be esploited m Wagner described the green zone during his testimony He stated the green zone was Wagner Tr at 33 3 16 Wagner Tr at T4 Mr 1Wagner testi ed that There is a piece of network equipment that needs to gel purchased and installed to nalize the last couple pieces at the Macon site But to clarify they re all forced to utilize PW through the Xeccdium Appliance There just happens to be a potential workaround that we have mitigation pieces in lace to prevent '43 Wagner Tr at t5 explaining that the exact date that all administrator accounts began PW compliant varied based upon the location As of April EDIS 0PM reported to OMB that ltlt percent of their privileged users were required to use PW cards and only 41 percent of their unprivileged users were required to use PW cards After a 30 day cyber sprint launched in July EDIE 0PM reported 9 percent PW card compliance as of July 5 Office of Mgmt at Budget Exec Of ce of the President Cybet-Spn'nt Results July 31 2015 On le with the Committee 9 Uf ee of Mgmt 3 Budget Exec Office of the President Report to Congress Pattern Information Management Act 23 Feb 2015 available at defaultt lestom btassetstegov doest nal fy14_fisma_rep on_32_2 _2ll 5 pdl' cards tacilitate multifactor authentication credentials to control access Such technology can at a minimum slow attackers who attempt to use unsecure credentials to move around an IT network Memorandum 'om Jacob J Low Din Of ce 3 Budget Exec Office of the President to Heads of Exec Dep ts and Agencies M l 1 1 Continued oannretrn-trl Seem-try Directive HSPD IE Pott'ey n- Con-mien Identgr ientt'en Entptovees and Conn-newer Feb 3 201 I defaultt'fi lesto In emotandat35 Wagner Tr at Bil-133 53 creation of independent machines that the database administrators utilizing that was wholly separate from the normal network so that all database access of the database that we knew the adversaries were looking for could only be accessed through this one controlled machine which was not on the network Green zone machines were configured at locations in Washington DC and Boyers Deployment and con guration of the green zone workstations continued through May 23 201 4 Between May 23 and May 27 the timeline does not provide a clear description of activities prior to the May 27 2014 Big Bang effort to eliminate the attackers nor provide the reason after two months of monitor May was the designated date 152 However testimony given before the Committee does fill in some of this gap Wagner testi ed We needed preparation to do the Big Bang The three-day weekend was coming up It was something that looked like a perfect time to prestage everything However we wanted to ensure that the users were involved and we could get full direct identity of the users when changing passwords We didn t want to just get a phone call from somebody saying hey I need my password changed We wanted to be able to physically verify that passwords were being changed by users So that date was specifically chose to prestage all the back end processes that needed to be in place in order for a ill user reset 253 Wagner stated the decision to remove the adversary from the agency s network on May 2 was made as a result of the forensic analysis process and not necessarily related to how close the adversary got to the background investigation system PIPs He testi tied Q So beyond the period of time to stage the event were the attackers moving in the network they gave you an indication that you needed to kick them out at this point Were they getting close to Were they getting close to a A It was a point of presence in which the interagency response team felt that there was nothing more to be gleaned from the presence of the adversary We weren t learning anything new They weren t searching for anything different And so the risk of kicking them out too early had come and gone and now the risk was becoming having them in too long and we didn t want to keep them around any longer than we had to 254 Wagner Tr at 131-133 June 21314 om Incident Repurt at Hooaosis some Wagner at 39 54 Wagner Tr at EFL-til 59 Wagnefs testimony that 0PM and their interagency partners were no longer gaining useful intelligence from the monitoring phase is at odds with the testimony ofBrendan Saulshury an 0PM contractor with IT Security Operations who played a signi cant role in monitoring the attackers during this period Saulsbury stated Q And you and your team were monitoring their penetration And was there any particular danger that precipitated the decision to conduct the Big Bang when it was conducted A Yes So we would sort of observe the attacker every day or you know every couple of days get on the network and perform various commands And so we could sort of see what they were looking for They might take some documentation come back and then access you know somebody else's tile share that might be a little bit closer or have more access into the system We would sort of see them progress as we are doing our investigation And then it got to the point where we observed them load a key logger onto a database administrator s work station or actually several database administrators5 workstations At that point the decision was made that they are too close and 0PM needs to remove whatever they were aware of at the time Q Okay And that precipitated the Big Bang When you say too close A They were too close to getting access to the l IPs system 255 The distinction is signi cant on two levels First if Mr Saulsbury is correct it is possible that 0PM had not yet identified all of the infected systems on their network i e the agency had not yet identified the scope of the hacker s foothold Second if the adversary was getting too close to the PIPS system it is likely the hacker had conducted sufficient reconnaissance of network to access that application but had not yet successfully executed the end-stage of their back and successfully exfiltrated data Regardless of the instigating events the rst phase of the remediation plan the Big Bang was completed on May 27 201-4 2515 0PM took a number of steps in collaboration with to eradicate the malicious actor at least temporarily from network These steps included removing all known compromised systems creating new accounts for 150 known or potentially compromised users and disabling their old accounts and forcing all Windows administrators to use PW card for authentication 155 Saulsbury Tr at 2546 Saulsbury Tr at 48 Wagner Tr at Wagner referring to the end of the monitoring phase as the Big Bang 3 June seen first Incident Report at I 235 60 In addition the Big Bang effort included resetting administrative accounts PIV- enf orcing all admin accounts building new accounts for compromised users resetting all local accounts on all servers taking the compromised systems offline and a stateful reset of all intemet routers 2513 0PM and their interagency partners were effectively attempting to press the reset button and eliminate the adversary s foothold in environment by eliminating their means of mobility user accounts and presence compromised systems 0PM continued remediation efforts and was con dent the adversary had been removed from their environment Jeff 1 hlagner Director of IT Security Operations testi ed DHS remained with their Mandiant tool for another 30 or 45 days We even had regular checkups with where I d go over to the - - and talk to them to see if there was any communication throughout DHS FBI the 1C conununity if anything that was being identi ed related to 0PM and there was no communication whatsoever 259 Documents and testimony show 0PM leveraged both interagency partners and private sector technologies including lviandiant 2m to ensure their systems particularly the PIPS system were clean of any malicious presence Saulsbury testi ed The NSA blue team came into 0PM and they ware performing both vulnerability scans and scans for malware artifacts on the network 1Wagner and Saulsbury admitted however that the attack 0PM discovered in 2015 - which led to the ea ltration of background investigation data in the summer of 2014 was already underway during the 2014 incident response period and continued after the Big Bang 51 On or about May 2014 and while 0PM was closely monitoring the 0PM network the attackers had established a foothold and dropped malware Jed Wagner new 0 It Sent-n Ii 1 n1 35 June 2014 one Incident Report at Hooeosls - 1243 59 Wagner Tr at 40 I Wagner at 54 They also deployed some of their technical staff to deploy the Maudiant tool We didn't have at the time a deployed endpoint search mechanism So they deployed their Mandiant to our environment to do the search for malwarc Actually there's another component They also utilized their forensics learn to do some of the forensic imaging and then malware analysis once they took the drives -- occasionally took the drives back to DHS headquarters DEIS of ce on Globe to do analysis forensics analysis 26' Saulsbury Tr at Wagner Tr at 121123 Saulsbury Tr at i at 61 - 1-- ffilfj it'ljt' 4 Var- 91 whim ale- M 3'3Lut- gram - 1 nurse Hat im Elwii'a' H5 ch fr During the 2014 incident rmponse period while 0PM was monitoring the attackers 0PM observed the ex ltration of data related to the PIPs system The fact that this information was taken makes clear the target rrther this information liker informed the background investigation data ex ltration that was later discovered in 2015 June 2014 Incident Report Appendix lists the data ex ltrated while 0PM monitored their network in 2014 _i mp5 PDF q Ho PIPSHPEILQF IE Protesting DOC Pro Brie ng l'or CID v5 pp Eow amend i no DFM ms Modernluluon lieu mmend-at no yls l rdor - om ms BusinessCase Apps-311L553 also ya 01493 ch 7 veiu-Fhuc 1 Apphcanon Crest-anpdf T m Wit-In 1 I hassle L- V's-o 3 Investigation pd yo I Yum-flu at Last tiles-n gel Ho PM ettsiuhm eicn- tit vat- 159 V a #1 1 5mm 8- frem-cot on sleet I Ho not genie Ho r nitrate tor Match sty- Hovt ames test 4 SEN $93-9 if 1 i PIPE Outside nay ml for Awareness 2314111 Mommies last-I - Programmer Groups doc No pips centrutor lot 2009 I - up P gr roup Met-53 1 I Ho 3 I a group list Ill hip 1 PIPgustihhlriiqf ilm 1h u A I 1 Password Protected Unable toOpon i or IT Awareness-Iii lit 1 He He mes Mt oi-lye ny MP5 EPIC _ne I NE 28 No I Pies-n rue Usage mi rem-03 3 ult PIPSJIusterHCenvenion Plan 11- PM Batch lob requenrv ads No - lo a Erwv By way of background PIPE is a mainframe application on the OPM environment that stores the background investigation information provided by employees and perspective employees on forms SEES and SFSS PIPS interests with several other 3 Wagner Tr at 19 1 1 5 Of ce ofPers Mgmt Federnf Investigative Sentiee Division Information Technologi- Priveejt Impact Assessment 43 Oct zone 62 Federal Investigative Services FIS systems and the connected and component databases contain information and materials that are considered the crown jewels for a foreign intelligence - 2155 service Based on the nature of the information held in the PIPS and related systems it was clearly a target but Jeff Wagner Director of IT Security Operations seemed to downplay the signi cant of PIPS as a target He testi ed Q What is the PIP server or system A PIPS is an application that sits on the mainframe Q Why would that be a target for an adversary that particular application A It s a large data repository Q It s a high-value target A It s currently assessed as a high-value assessment but it s a large data repository Any large data repository is always a target 255 The PIPs system is more than simply a large data repository The data it stores sensitive background investigation information gathered from forms is some of the government s most valuable PILS1 Documents that could in form attackers about the nature of and the architecture of PIPS and related systems should not have been permitted to be es ltratcd from network Appendix as shown above lists documents that were exfiltrated during monitoring effort in 2014 The documents relate to 0PM IT systems including PiPs contractor information and documents with names and the last four digits of those individuals Social Security numberst Additionally the documents listed in Appendix contain information relevant to large repositories information The list of Ex ltrated 0PM Data in Appendix identi es 34 documents Appendix indicates none of the documents contained P11 except in one case where the was password protected and the adversary was unable to open 2 David Perera 3 Joseph Marks his trip Discinsed Hoe Got Crown Jewels Petrrleo June 12 EUIS available at ww politico eo m star-y 20 I Ed hackers-fed era -employees-security-back ground checks l 395 4 3 Wagner Tr at l9 36 According to HIST guidance is any information about an individual maintained by an agency including any information that can be used to distinguish or trace an individual's identity such as name social security number date and place of' birth mother s maiden name or biometric records and any other information that is linked or linkable to an individual such as medical educational nancial and employment information See National Institute for Standards and Technology Special Publication Still-122 Guide to Protecting the Confidentiality ol PersonalIy Identi able Information I Elpdf June 3314 GEM incident Report Appendix at 8 1011454246 1 M 63 it Four of the documents however included the last four digits of individual Social Security numbers makes 2T0 in describing the items ealiltrated in Appendix D June 2014 incident Report clear the target was PIPE The Report stated The attackers primarily focused on utilizing SMB Server Message Block commands to map network le shares of 0PM users who had administrator access or were knowledgeable of system The attackers would create a shopping list of the available documents contained on the network le shares After reviewing the shopping list of available documents the attackers would return to copy compress and ea ltrate the documents of interest from a compromised 0PM system to a C2 server Further there remains the important caveat from US-CERT that additional documents may have been stated that th rt ltrated prior to OPM's monitoring phase which began in March 2014 In should be noted the attackers had access to network since nly 2012 and the documents were exfiltrated during the time period of March 2014 to May 2014 when 0PM stated their advanced monitoring of the infected systems Addition a documents may have been exfiltrated prior to March 20 but there is no way to determine with exact 1Wagner downplayed the signi cance at the information ex ltrated in 2014 arid testi ed information was standard and would not necessarily give an adversary an advantage in a subsequent attack He testi ed2014 the adversary was utilizing a visual basic script to scan all of our unstructured data So the data comes in two forms it s either structured Le a database or unstructured like tile shares or the home drive of your computer things of that nature All the data that is listed here all came out of personal le shares that were stored in the domain storage network And when I went back to the program of ces and had them sit down with us and do an assessment of it and look at the age and the amount of data within these it was not recognized to be critical data or critical information it s pretty standard documentation for the most part 17 June June 2014 incident Report at HUGRDSIB 2014 DPM Incident Report at HGGRUSI El 1235 Notably 0PM produced these documents from Appendix to the Committee in the Fall of 20' 5 with redactions and in camera It was only under subpoena that produced these documents without redactions in February 20115 64 Q When you say standard documentation documentation that would be public accessible A I don t necessarily know if it would totally be publicly accessible I don't know what everyone publishes But like Aich and 3ch packages for the most part are available for review they re traded amongst agencies It s not something you would be you know overly freaked out over 274 When questioned further about the signi cance of the Appendix documents Wagner continued to downplay the significance of these documents in his testimony Q One of the entries includes a document that was es ltrated PIPS contractor list Is that the kind of information that you would want in the hands -- not that you would want in the hands of an attacker but that would give an attacker an advantage A The list of contractors from was just simply a user name list of the system It s not something that s it wouldn t necessarily give them an advantage I mean Q Would knowing the users on a network For a particular system Finding users is not dif cult For the most part if you think about it most companies or agencies utilize a standard type naming scheme So it s fairly easy from a pen tester or an adversary standpoint to glean this information either from initial presence or half the time you can just Google it For instance everybody s Facebook account utilizes a Yahoo or a Google email address it wouldn t be dif cult to nd anyone any individual s credentials in so th form to figure out what your user name to your Facebook is Saulsbury however disagreed with Wagner s assessment of the sensitivity of the Appendix documents that were esliltrated He testified that the documents could be useful to the hackers in a subsequent attack He stated Q So tell me first of all are these public things that 0PM would be concerned about if they were put out into the open A Yes those are not documents that are meant to be public Q And what kind of documents are these if you could generally characterize them l 2 Wagner Tr at 3 Wagner Tr at 42 65 A They are basically sort of system documentation various processes and related to the background investigation systems Q So if an attacker were able to extiltrate this type of data which it appears they did would this give them an advantage for a future attack A Yes And how so A It gives them more familiarity with how the systems are architected Potentially some of these documents may contain accounts account names or machine names or IP addresses that are relevant to these critical systems m Saulsbury's testimony indicates the exfiltratcd documents in Appendth contained information relevant to understanding how the system works These documents included among other things a 2014 list of contractors with access to the PIPS system a CEO level brie ng on the EPIC system and a discussion of the interface between the PTPS and Joint Persolmcl Adjudication System JPAS systems These documents would have improved an adversary s understanding of system its architecture and information on who has access to the background investigation information contained on the PIPS system The Appendix information is significant because it would be useful to an attacker and it provides irther evidence that the hackers were targeting PIPs Nonetheless Mr Wagner s characterization seems to downplay the significance of the Appendix D Given the near certainty that PIPS and the information it hold was a target before and continued during the 2014 incident response period it is noteworthy that network monitoring technology did not have total visibility into PIPS Wagner testi ed guess it would be fair to say that there was minimum visibility ofthe PIPS application Despite this lack of visibility 0PM asserted they were con dent no PII was taken during the course of the 20 4 data breach Wagner testified Q 1Without monitoring tools on the PIP server at that point at least insofar as this is described could data from the PIPS application have been taken prior to March 28th and 0PM had not been aware of that That would not be possible Why is that Saulsbury Tr at 2123 37 Wagner at it 66 A Because it would have to pass through the distributing environment to do so The mainframe sits within the center of the distributed nucleus so in order to get data out it would have to pass through all the other monitoring techniques- And why would that allow you to see it Because we had seen large sums of data leaving And that would be - -- we ve seen large spikes and things of that nature and D118 and us both looked for those large spikes at that time and we did not are see any 0PM has consistently asserted that no PII data was taken in the 2014 breaclt but as US-CERT stated additional documents may have been ex ltrated prior to March 2014 but there is no way to determine with exact At a minimum sensitive data was in fact exfiltrated by the hackers as evidenced by the items listed in Appendix D- The Appendix data cx ltrated provided clues as the data targeted and the tactics techniques and procedures TTPs of the attackers 0PM monitored in 2014 provided hints about the data breach 0PM later discovered in 2015 The attackers discovered in 20M used Tactics Techniques 85 Procedures l Ps such as the type of malware and the attackers ability to move throughout network hinted at the targets of the attack 0PM discovered in 2015 These TTPs also indicate the persistence scope and sophistication of attacks on network Those key pieces of information however were not enough for 0PM to stop the far more serious attack discovered in 2015 A public report by a threat analysis group has said the attackers discovered in 2014 used a speci c and uncommon toolkit or malware designed for late-stage persistence and data ck ltration 23 The malware used by the attackers discovered in 2014 was identi ed as two variants of Hil t malware referred to as HiKit A and HiKit 13 2 Notably an October 2014 FBI Cyber Flash Alert said HiKit malware shouid be given the highest priority for enhanced mitigation and it uses rootkit functionality to sit between the network interface card and the operating system enabling the malware to sniffall traf c toffrom the compromised host M232 Wagner Tr at El June 21314 Incident Report at Houses 1 a coma EM Novetta Operation Avior Threat Actor Group Report at 6 Tr at June ama om Incident Report Appendix at HOGRGBI 34301244 - 1245 1 33 Cyh l' on Fed Bureau oFInvestigation Far Cyber Final arm on 15 2cm ragebeastf in regard hi kitll ash 67 The use of HiKit malware is evidence of a sophisticated attacker that had achieved persistence on the IT environment and was capable ofperforming a variety of functions including data ex ltration within network In the 2014 incident Report US-CERT described lIikit as an extremely stealthy form of malware designed to hide its malicious processes and programs from detection of commodity intrusion detection and anti virus Saulshury described how the HiKit malware was used by the attackers discovered in 2014 He testi ed So the fact that it is still beaconing means that an attacker could use it to still obtain entry into network Itjust means that they could get onto that command and control server and start issuing commands to that infected machine So C2 means command and control As far as it being an IP rather a domain that s not a signi cant issue Basically the way that their malware worked was there is a con guration file that tells the malware where to beacon out to And instead of it having a domain that they created they just put the IF directly in there so instead of doing DNS resolution it just goes directly out so it is just a quirk 134 Wagner described Hikit as a form of a remote access tool or RAG It s a basically a hack door command tool with multiple functionalitics Most malware these days are kind of a Swiss Army knife type effect You don t necessarily have a mctionality like key logger It usually utilizes multiple modules that allow various activities 235 Wagner also said the Hikit malware was mostly used for persistence or maintaining a presence at 0PM though keylogging activity was also observed 2315 Effectively the malware was used so the hackers could still use it to obtain entry into networki m 3 June 2014 0PM Incident Report at metres 13 001234 334' Saulsbury Tr at 1349 Wagner Tr at 31 3 Wagner at 13 Saulshury Tr at 18 68 tampon-had Callback a if 0 mm @0990 Multiple Stages The New Attack Life Cycle Exploitation of system First Callback for malwa re download Malware executable download Data ex ltratlon Mailman spreads laterallyr If-ll FireEye From a presentation by Ashar Aziz Vice-Chairman and FireEye Inc at RSA Conference USA 2GB Feb 23 2Ul3 In other words the Hikit malware is a rootkit or a set of software tools that allow an unauthorized user to gain control of a computer system escalate access and persist in presence on the network without being detected LIB-CERT explained that Hikit allowed the hackers to gain root level or administrator access to network and A llow ed the attackers to create a reverse shell from their C2 conunand and control servers into the infected systems in network from a remote location anywhere in the world The C2 servers are used to proxy the attackers connections from their actual location on the interact in order to keep their real identities and locations hidden Hikit allows the attacker to run commands and perform functions from a remote location as if they had the equivalent of a monitor and keyboard connected to the compromised 0PM system 233 The presence of Hikit on the UPM network was evidence of the adversary s presence and capabilities but it did not reveal the initial point of entry However the use of a rootkit means the attackers had to have high level access to network US-CERT said the attacker was able to acquire high level credentials by exploit a vulnerability and likely obtained access to network using social engineering methods such a phishing attackm Outside threat analysis experts have described Ilikit as a late-stage persistence and data ex ltration tool that 3 June and one incident Report at HOGRDSIS some 339 indicates the final phases ofthe threat actor s operational lifeeycle m The use of Hikit is evidence of a multistage operational lit ecycle that would require the adversary to not only be well resourced but also well organized The attack discovered in 2015 had similar characteristics The Hikit malware allowed the attackers to remain on systems to maintain persistence but in order to move throughout network undetected the attackers used Server Message Block 3MB protocols 292 Hikit and SMBmetocols are TTPs that tend to suggest advanced penetration and a sophisticated actor 29 With respect to the use of the SMB protocols US-CERT said the malicious actors were connecting into the server between the hours of 10pm and l am EST with a compromised Windows domain administrator credential to search for related les on network le servers utilising 5MB conunands 294 Wagner described the attackers use of 3MB protocols during the 2014 attack He testi ed If you do some form of traversal or conununications you run over a normal communications protocol it s not uncommon to change the protocol language or change the protocol ports in which you do traf c And essentially what they did is they tried to hide their activity and the things they were doing in a very highly utilized protocol port So they basically hid their communications in the fuzz of the network traffic 295 Wagner acknowledged that the use of 5MB protocols in addition to other were evidence of the threat actor s sophistication and capabilities Wagner testi ed Malware itself doesn t indicate sophistication The other tactics and techniques that they utilized or other things that they did such as hiding their commands through 3MB shows an advanced penetration It s not a simple attackzg The use of the Hikit malware and SME- protocols by the attackers discovered in 2014 show the attackers had a well-developed foothold in environment and maintained a presence and persistence that indicated an advanced penetration that 0PM was facing in 2014 NIST described the challenge of a persistent late stage penetration threats and identifying modern attacks in their early stages is key to preventing subsequent compromises preventing problems is often less costly and more effective than reacting to them after they occur Thus incident prevention is an important complement to an Novetta Operation Avian Tin-en Actor Group Report at 15 Id 3 Juno com com Incident Report at tiocsosis -oo1231 93 Wagner Tr at 33 19 June 2cm can incident Ropott at nooaosis amass Wagner Tr at Id 96 Wagner Tr at 31 TU incident response capability If security controls are insuf cient high volumes of incidents may occur ability to determine the how and how long of the attackers discovered in 2014 was limited by signi cant gaps in their capability to create collect and review audit logs of their network Consequently the answers to these questions remain unclear Audit logs are collections of events that take place on infomtation teclmology systems attd networks 29 n the course of a forensic investigation a variety of sources produce reviewable log information including antivirus software rewalls and intrusion detection and prevention These sources can help investigators piece together how the attacker gained access where the attacker has been how long theJy have been there and most importantly give clues as to what the attackers are after identi ed numerous gaps in the centralized logging of security events at 0PM during the investigation of the attackers discovered in 2014 stating Currently 0PM utilizes Arcsight as their SIEM security information and event management solution of choice but there are numerous gaps in auditable events being forwarded to Aresight for analysis correlation and retention 3m Gaps in audit logging capability likely limited ability to answer important forensic and threat assessment questions related to the incident discovered in 2014 This limited capability also undermined ability to timely detect the data breaches that were eventually announced in June and Jul ZUI 5 3 2 If IT security teams can track the attackers movements back to the point of entry they can patch the system vulnerabilities that allowed the penetration in the rst place The 0PM team did not at the time of the incident discovered in 2014 have a robust logging capability that would have allowed them to determine the initial point of entry Wagner acknowledged the audit legging gap and how that impacted their ability to identify the initial Paul Cicltonski et al Natl Inst of Standards Sr Tech Spec l ub SUB- lmy 2 Computer Secriirrp Incident Guide Recommendations Nations Institute ofSt rmriorrir and Teeimoiogy 2 Aug s nistgovinistoubsi oeci alPublic atio ns y rindi See generoiiy Karen Kent St Murugiah Souppaya Hat Inst of Standards and Tech Sp Pub SKID-512 Guide to Computer Seem-irv Lo Management 9 sea ails-o Saulsbury Tr at IS testifying that There are many different log sources that we look at during a forensic investigation Eg Wagner Tr at Saulsbury Tr at if June 2014 cam Incident Report at access 1 com 23 3'3 1 1 5 Of ce of Pets Mg th Press Release 0PM to Notify Empioyees ef uent-security Incident June 4 21315 wsircleascsilil Sitl ioorn lo noti iv-emplovees-of cybe rsecttri ty incidenti ILLS Of ce oiPers Press Release 0PM Announcer Steps to Protect Peder-oi Workers and Utiicrs em ther Tin-ants July 9 l l 71 point of entry He stated don t think we ever necessarin found initial point of presence or point of contact Our last log entries at best gave us the evidence of adversary presence was November of 211 33 Wagner also testi ed We did forensics to try to nd the initial point of infection but because we didn t have the full volume of logging that we have today throughout 2013 or 2 or prior to the 2014 breach we just ran into a point where there wasn t logs to give as suf cient evidence or indication of the exact point of presence Saulsbury also acknowledged the limited logging capability He stated Q Okay And after all was said and done and you were looking back when were the earliest actions taken by the hackers relating to the breach And when did they take place And what were they A So we don t know with 100 percent certainty what the initial entry point into the network was and when it was So what we were able to do is look back through some of the logs that can t remember at this point what the actual like our earliest log entry of activity was I want to say that we had stuil' activity at least back in 2013 that was observed but I can t recall at this point what the rst evidence that we have is 5 The gaps in audit logs not only make it dif cult to determine how the attackers perpetrated their hack of 0PM but also to determine with any degree of certainty how long the attackers were in the OPM network and any data ert itrated US-CERT said of the attackers discovered in 2014 it should be noted that the attackers had access to network since July 2012 and the documents below were ea ltrated during the time period of March 2014 and May 2014 when 0PM CIRT started their advanced monitoring of the infected systems Additional documents may have been ex ltrated prior to March 2014 but there is no way to determine with exact certaintym 0PM also could not accurately assess the risks to their IT environment because the agency lacked the necessary logging information and centralization practices to generate a full picture of how the hackers established and then maintained persistence on 0PM s systems Threat and vulnerability infonnation are the l'oLrndational step itt implementing risk based approach 3m Wagner it at 12 13 4 1Wagner Tr at 22' Suulsbury Tr at 14-15 5 Iune 20M orM incident Report at means 1 scorers 3m Comput Sec Div Nat l Inst of Standards and Tech Risk rmuewonl'r Overview last updated Ir tpr I 2014 7'2 The agency's inability to determine what other documents were ex ltrated prior to March 20 2014 revealed two aws in network monitoring practices First from March 2014 forward and 0PM were installing the monitoring equipment including additional logging capabilities to determine what was being esfiltrated going forward This left the agency with limited ability to look backwards Second the gaps in monitoring practices prevented 0PM from determining what exactly was leaving the network and what data had been taken in the nearly two years the attackers had access to network After investigating the attackers discovered in 2014 USE-CERT recommended 0PM implement a robust system audit log data practice and Require program of ces to send critical system audit log data to Arcsight During the system development life cycle security related information and auditing requirements should be identi ed in accordance with 0PM IT Security Policy and NIST recommended guidelines and configured to be sent to Arcsight for analysis correlation and retention The following log sources were identi ed by Network Security as a high priority Linux Secure Logs HRTI Active Directory Logs RACF authentication logs and PIPS access logs Aggregation of audit log data to centralized location such as Arcsight allows for proactive security monitoring and quicker time for triaging and remediating security incidents Low level of effort to implement 3133 Wagner testified that 0PM now as of February 2016 has 100 percent visibility over their systems but it is not clear when 0PM gained this increased visibility He stated o A Did you have total visibility over environment during the 2014 incident I would not say 100 percent We had a great deal of visibility Actually at the time we had full visibility on the perimeter internal visibility is where we had some gaps Why is that As I said it was an issue in which there was a longstanding project to have long entries loaded into the logger Post the 2014 incident that became a major priority and we now have 100 percent visibility It is notable that as Mr Wagner admits they may have had signi cant visibility on the perimeter of the 0PM network but the gaps were more pronounced once the attacker was already inside the perimeter Thus an attacker already inside seemed to have the ability to move 3 33 June 2014 Incident Report at access I tl l 237 Wagner Tr at 33 7 3 undatocted across network In a sol-o trust environment an attackor s ability moire once inside a notwork anviromnont would be limited by a segmented and strong access controls As noted earliar the attacker later discovered in 2015 had ahead established a foothold inside the 0PM network as of early Ma r 2014 74 Chapter 3 0PM Attempts to Mitigate the Security Gaps Identified in 2014 While Iron Man and Captain America Go to Work May 2014 April 2015 After the Big Bang effort on May 2014 there were a number of events that inform the story of the data breaches announced in 2015 These events are also relevant to April 15 20 5 when 0PM rst identi ed an unknown SSL certi cate3 used to communicate with an at the time unknown domain opmsecurityorgf 1' Opmsecurityorg was later found to be registered to Steve Rogers Captain America s alter ego 0PM subsequently identi ed another domain opmlcarningorg which was registered to Tony Stark Iron Man s alter ego These domains were part of an advanced and sophisticated attack infrastructure used to ex ltrate data from 0PM in the summer oi'2014 As 0PM and a multi agency team began to investigate the scope and method of the attack 0PM enlisted the assistance of two contractors Cylanee and CyTech The multi agency team and contractors eventually made ndings that caused 0PM to announce in June and July 2015 that the personnel records for over 4 million individuals and background investigation data for over 20 million individuals had been compromised 3 '2 To fully appreciate the May 2014 through April 2015 period it is useful to establish posture with respect to mitigating the threat of the cyber incident that was identi ed in March 2014 IT Security Feature and Mitigation Efforts After the May 2014 Big Bang On June 22 2014 issued an incident Report to 0PM with fourteen observations and recommendations to address the security gaps identi ed in the aftermath of the 2014 cyber incident The observations and recommendations in this Report highlighted the poor state of IT security at 0PM and the failure to implement basic cyber hygiene practices The Incident Report directed 0PM to redesign their network architecture to incorporate security best practices 3 '1 Brendan Sauisbury an 0PM contractor who participated in OPM's 2014 and 2015 incident response efforts testi ed that US-CERT deemed network very insecure insecurely urchitected and found there was lots of legacy infrastructure m a An SSL is a security sockets layer and is standard security technology used to establish an link between a server and a website 3 June 9 2015 DMAR at 3 Of ce of Pets Padgett Press Release 0PM to Not-fie Eill pi j f ' Incident June 4 US Of ce of Pets Mgmt Press Release 0PM Announces Steps to Protect Federal Wot-item and Diner Front Critter Tin-cots July 9 201 5 othersafrorn-cy lune 2e14 or-M Incident Report at i toonesi Saulsbury Tr at ld-i'i' 75 Saulsbury said this ultimately led to decision to create basically a brand new hardened networ they called the shell 3 '5 According to Saulsbury 0PM intended to eventually move legacy applications to the new shell 2014 Incident Report identified several Speci c technical recommendations to improve network security in the legacy envirotnnent including buying security tools and reorganizing the The Incident Report included the level of effort required from 0PM to implement each rcconunendation from low to high Three recommendations were considered low effort four moderate and two high 313 The US-CERT Incident Report found 0PM did not have the capability to centrally manage and audit rewall access control lists and rules Consequently DHS recommended short and long term actions to combine manual auditing and scanning tools and then buy a network equipment solution to centrally manage configuration settings while also auditing these settings against best practices This recorrunendation was considered high level of effort The Report also found Plvl s network was extremely flat and had little to no segmentation w Thus LIE-CERT recommended a redesign of network architecture with security best practices incorporated including enforcing no direct user access to servers and requiring PW credentials for access in order to limit an attacker s ability to move laterally across the network once initial access is obtained 32' This was a high level of effort recommendation The recommendations that required a low level of effort to implement were related to logging security awareness training and a redesign of Incident Response Plan In rcconunendations related to the OCIO found there is a gap in information technology leadership across 0PM as an agency and that it is not uncommon for existing policies to be circumvented in order to achieve business functions while exposing the entire agency to unnecessary risk 22 In response recommended 0PM undertake a policy review and gap analysis to determine the need for additional policies to manage IT security and business functions and noted a cultural change will need to occur to ensure policies are never circumvented unless absolutely DHS also recommended 3'5 Saulsbury Tr at iti-l'i' an 3 June 2014 Incident Report at Hooaosis-oc 1235 See nan 0PM Cybersecurity Events I'imcline The Cybersccurity Events Timeline states that the UPM Security Dpclations Center SOC began unof cially reporting to the UPM CID in April 2014 and officially began reporting to the 0PM CID in March 2015 after the union approved the reorganization As of March 22 2G1 5 the relevant onions at 0PM formally approved the DCIU reorganisation- June 2014 0PM Incident Report at 3 June 2014 0PM Incident Report at itocncsis-ooizss in 311 M June 2014 0PM Incident Report at Rooms 1 scams 3 Id 76 reorganizing the Among other things the reorganization shifted the Director of Security Operations to report to the 310 325 Documents and testimony show OPM began to implement the DHS recommendations in or around May or early June of 2014 The effort continued through early 2016 Based on testimony front two witnesses involved in reaponding to the 2014 incident it appears 0PM tried to implement ll- SE recommendations but the agency was hindered by the fact that it started with a woefully unsecure network Throughout this phase the attackers involved in the data breaches announced in 2015 had already established a foothold on the OPM network 316 Key 2014 US-CERT Recommendations Highlighted 0PM IT Security Vulnerabilities One of key recommendations was to ensure all OPM users were required to use PW cards for access to the OPM network in a 2015 OMB Report on IT security OPM was identi ed at the end of scal year 2014 as one ofsevera agencies with the weakest authentication a majority of the agency s unprivilcged users logged on only with a user TO and password making an unauthorized access more likelym The OMB Report also stated that at OPM only one percent of user accounts required PW cards for accessm Wagner Director of IT Security Operations stated PW card enforcement did not illy roll out until September 2014 and was being implemented through early 2015 330 He added the FIS Federal Investigative Services contractors who did the background investigations were the last group required to have PIV cards for access Had OPlvl leaders fully implemented the PW card requirement or two-factor authentication security controls when they first learned hackers were targeting background investigation data they could have signi cantly delayed or mitigated the data breach discovered in 2015 The agency rst learned attackers were targeting background investigation data on 3 June 2014 can Incident Report at 3'25 OPM Cybersecurity Events 'I'irneline- 5 Wagner Tr at 75- 3 discussing implementation status of two recommendations Sauisbury Tr at 31-34 discussing implementation status of six recommendations and noting logging capability gaps remain due to technical diffitmlties applying the logging function to mainframes June 9 2015 OMAR at HOGR0724-001154 In August 2004 the federal government initiated several initiatives to enhance cybersecurity across the federal government including Homeland Security Presidential Directive 12 established a mandatory govemment-wide standard for scours and reliable identi cation for access to government 11' systems and facilities that was further de ned as a requirement for personal identity veri cation PW credentials Then OMB directed federal agencies to issue and use Pl v' cards to control access OMB reported that as of the end of scal year 2014 only 41 percent of all agency user accounts at the CFO Act agencies required PW cards to access agency IT systems can rreats and Data Breaches Illustrate Need for Stronger Controls Across Federal Agencies Hearing Before Saba-anon on Research d Tecli and on Oversight ofthe ll Conan on Science Space 0 Tech 114th Cong July 3 2015 testimony Gregory C Wilshuscn Dir oflnfo Sec Issues Oov t Accountability Of ce Of ce of Mgmt d'e Budget Exec Of ce of the President l ll-4 Annual Report to Congress Federal liy onnatioa Security Management Act at 23 Feb 2015 available at docsl nal fv14 sma report 02__2 i' 2015 de' 33 at at an Wagner Tr at 33 '15 3 Wagner Tr at l5 Mei-en 20 2am 2 vet the first data major ea ltration involving 21 5 million individuals background investigation files did not occur until early uly 2014 giving the agency over three months to implement security controls to protect those datam Testimony from the Department of Homeland Security revealed that implementation of twoefactor authentication for remote logons in January 2015 which was already uircd of federal agencies stopped the adversary from taking further signi cant action 4 If 0PM leadership had implemented two factor authentication even earlier for example in April or May of 2014 the agency might have locked out attackers before they had a chance to commit the most signi cant digital violation of national security faced to date In July 2015 0MB launched a cybersprint to require all agencies to expedite implementation of cybersecurity measures including enforcement of PW card access within 30 days According to 0PM 100 percent of their privileged users were required to use PW cards as of April 2015 but only 41 percent of their users were required to use PW cards The agency improved its PIV card compliance by July percent of unprivileged users were required to use PIV cards In August 2015 0PM updated its PW card implementation status in response a request from the Committee The agency reported approximately 99 percent of 0PM users are required to use a PW card or equivalent to access 0PM workstations with two-factor authentication a The agency also told the Committee that 0PM bought 5 000 ActivClient licenses in 2005 j to enable the use of PW card credentials to access 0PM workstations and further clari ed that currently 8 400 such licenses are activated current and operational m The agency s response raised questions as to the status of the 5 000 licenses purchased in 2009 and why PW card enforcement was not a priority earlier particularly given that OMB had identi ed 0PM as an agency with one of the weakest authentication pro The use of basic cyher hygiene practices such as full implementation and enforcement of card access would have limited the damage incurred during the 2015 data letreach incidents Den of Homeland SeetlritleS CERT and cm can Cybersecurity Events Timeline tang as 2015 oeM Production May 13 2015 33 3 Under throes Federal and the 0PM Data Breach Balers tlte S Comm on Homeland See ii A lta l4lh Cong 2015 statement of Andy Oement Assistant Secretary for Cyberseeurity 3 Communications Department of Ilomeland Security adversary activity June 2014 to January 2015 stopped by security control rolled out January 2015 see Dep't ofl-lomeland SecttritleS-CERT and 0PM 0PM Cybersccurity Events Timeline Aug 26 2015 Production May 13 2016 security control rolled out January 2015 was two factor authentication for remote access 335 Of ce of Mgmt Budget L- aee Office of the President CPberSpt'lrir Results July 31 2015 Un file with the Committee letter from Jason Levine Dir Congressional Legislative lit Intergovernmental Aflairs US Of ce of Pcrs to the Hon Jason Cliaffeta Chairman ll Comm on Oversight 3 Gov't Reform Aug 23 2015 3 of ce of a Budget Exec oraee of the President Franteatninet Report to Cortgi ers Federal light-motion Security Management Act 23 Felt 2015 available at docsl nal fvl-fl fisma report 02 2015 odf T3 0PM Efforts to Buy Security Tools to Secure the Lettjacyr Network and Rebuild Very Insecure Insecurely Architects-d Network In response to US-CERT observations and recommendations in the 2014 Incident Report 0PM launched a multi-phasc IT Infrastructure improvement project to I buy security tools to secure their legacy network and 2 create an entirely new network environment Former 0PM C10 Donna Seymour testi ed to the Committee this project began alter the March 2014 cyber incident In May 2014 Seymour contacted lmperatis an IT schrity contractor to discuss the project In an email to former colleagues at Imperatis Seymour wrote Djo you recall all the work we did at MARAD Maritime Administration to straighten out a very messy network with poor security Well I'm looking for an expert consultant who can guide me and my team through the enact same thing m Seymour and two Imperatis employees worked together at Ultimately these discussions led to a sole source contract award to Imperatis for the multi-phased IT Improvement project in June 2014 343 The project included four phases Tactical securing the legacy IT environment 2 Shell creating a new data center and IT architecture Migration migrating all legacy IT to the new architecture 4 Cleanup decommissioning legacy hardware and systems Phase I or the Tactical phase supported effort to buy security tools to secure the agency s legacy IT environment immediately following the 2014 incident The Tactical phase of the project began in June 2014 and was completed in September 2015 343 efforts to buy security tools involved interactions with a number of contractors including Cylance and CyTech which would later provide cybersocurity and forensic solutions to 0PM Doro Brooch Hearing Before lire H Comm 0n ver-sight and Gov r l4th Cong June 16 2015 gtestimony ofDomia Seymour Chief Info Of cer US or lice olPers 4 Email from Donna Seymour Chief Info Of cer US Office of Pers Mgmt to Patrick Mulvaney and- Imperatis May It Elli-4 9 46 am Attach 12 at 101463 Imperatis Production Sept- 1 2m 5 Id Imperatis Proposal Volume Staf ng and Management Attach So at 262-264 ass-arc Apps A Key Personnel Resumes Imperatis Production Sept 1 2015 Letter Contract June is Attach at scones Imperatis Production Sept 1 2915 The oral DIG raised concerns about the sole source nature of this contract but did acknowledge given the urgency need to secure the UPM legacy network making a sole source award for purposes of buying security tools Tactical phase was reasonable 15 Of ce of Pent Report No 41 Hash U3 G ice of Personnel Improvement Project 5 June 2D 5 hereinafter DIG Flash Audit Alert glunc 2015 43 Letter From mperatis to H Comm on Oversight 3t Gov t Reform Majority StafF Fcb IE Bill 6 on le with the CommittEe 79 Documents and testimony show Cylance began conversations with 0PM about their products through a reseller and Cy l ech was introduced to 0PM through Imperatis The Committee obtained documents that show 0PM was buying and deploying at least ten security tools to the legacy IT environment Websense is one such tool in 2014 Websense had limited functionalit and simply ltered users web traf c to prevent access to certain sites like gambling sites 4 The agency had to upgrade Websense because according to Saulsbury the old version wasn t performing and did not include the advanced capabilities such as web ltering email and data security rnctionalitym Saulsbury also testi ed that in 2014 the Websense server was not the primary target in Saulsbury believed the Personnel Investigations Processing System PlPs was the target 43 The Websense upgrade was identi ed as a Priority 1 task and 0PM quickly made a purchase in June 2014 but the phased deployment of this tool was not completed until September 2015 349 As of February 2015 there were continuing challenges with the Websense pilot and as of April 2015 the project status for Websense was only at about so percent completem Saulsbury testi ed one ofthe deployment challenges was balancing usability and security but after the 2014 incident there was less resistance from users and security became the higher priority 351 In April 2015 according to 0PM the rst indicators of compromise were detected including the unknown SSL certificate that was beaconing to the domain opmsecurity org during the roll out ofthe upgraded version of Websensti z The agency purchased another tool to improve network access control 353 The agency purchased on July 23 2014 and deployed it from September 2014 September 2015 354 Documents show the_ deployment was delayed at least in part by required noti cations to relevant unions In August 2015 an lrnperatis Weekly Report stated that project sponsor for is in noti cation stage with the Union and the proposed mitigation strategy to prepare updated project timeline plan a memo to pilot to non- Union Agency users 35 in the aftermath of the 2014 incident 0PM attempted to implement reconuncndations including buying new security tools and building a new IT environment but 4 See Infra Chapters 4 The Role of Cylanec and Chapter 5 The Cy'l'ech Story 5 Saulsbury Tr at lit-13 3 Saulahury Tr at 49 Saulsbury Tr at 1113 3-43 9 0PM Tactical Toolset Purchase Kick cit and Completion Timcl'rarnes Oct 21 2015 1mperatis Production Jet 21 Saulshury Tr at 50 35 Imperatis Weekly Report Apr 13 acts Attach a at coarse Imperatis Prudu iun Sept 1 Imperatis Weekly Report Apr 2t Z l Apr 24 Attach ti at lrnperatis Production Sept I 2015 35 Saulsbury Tr at 53 35 Saulsbury Tr at 58-59 3 Imperatis Program Review July-Aug 21314 Attach a at 1039 Imperatis Production Sept 1 2015 DPM Tactical Toolset Purchase Kick-oil and Completion Time 'amcs Oct 11 2015 lmperatis Production Imperatis 1 Weekly Report Aug 3 T 21315 Attach 3 at lmperatis Production Sept 1 2015 80 because of the state of IT security at 0PM was so poor there was much to do The agency however missed opportunities to prioritise the purchase and deployment of certain cutting edge tools that as Cyiance CEO Stuart McClure testi ed would have prevented this attack 3m Meanwhile as 0PM worked to deploy badly needed security tools Captain America and Iron Man were ex ltrating sensitive data from OPM's unsecure IT environment in the summer of 21 4 0PM Missed Key Developments The Committee obtained evidence that shows 0PM was working to respond to the attackers discovered in the spring through the summer of 2014 while the attacker groups who ultimately stole background investigation and personnel records data were moving through the agency s network 0PM did not discover the attackers responsible for the background investigation data breach until April 2015 when it was too late These attackers had already established a foothold in network as of early May 2014 and began to exfiltrate this data in early July 2014 Meanwhile 0PM continued its mitigation efforts in response to the attackers discovered in 2014 Documents and testimony show a timeline of key events that provide context for data breach discoveries made beginning in April 2015 I July 2012 Attackers had access to OPle network I November 2013 The rst known adversarial aetivitly begins in network that led to the breach identi ed by LIB-CERT in March 2014 53 I December 2013 Adversarial activity to harvest credentials from 0PM contractors begins by the attackers later identi ed in April 2015 I March 20 2014 USHCERT noti ed 0PM of malicious activity and 0PM initiates investigation and monitoring of adversary I March 2014 to May 2014 0PM under guidance investigated 2014 incident and monitored attackers I April 25 2014 The domain is registered to Steve Rogers aka Captain America 3551 This domain was later used to ex ltrate data from network I May 7 2014 The attacker poses as a background investigations contractor employee KeyPoint used an 0PM credential remotely accessed network and installed malware to create a backdoor The agency s forensic lo gs show infected machines were accessed through a connection which was how background McClure Tr at 18 June a 2015 smart at 154 ha Hearing rm 0PM Dram Breach Fru'l Statemerll ofDonna Seymour Chief Info Of cer US Of ce 01' Pets Mgmt Tn Ex 4 81 investigation contractors accessed network At the time 0PM gave contractors a username and password and investigators would log in with this 0PM credential 360 I May 27 2014 OPM initiates Big Bang to eliminate attackers and complete remediation This decision was made after 0PM observed the attackers load a key logger onto several database administrators workstations and they got too close to getting access to the Meanwhile the attacker that established a foothold on May T 2014 remained in the OPM network I June 5 2014 Malwarc is This malware installation appears to have been facilitated through the backdoor established on May 2014 3 I one 2014 0PM contractor USIS self-detects a oyher attack on its IT system and noti ed USIS investigates and blocks and contains the attacker by early July and invites us cear to osrs facilities to investigate by late July 2am 365 I June 20 2014 Attackers conduct a remote desktop protocol RDP session indicating the attackers had escalated their access and began moving deeper into the network contacting important and sensitive servers supporting background investigation processes This RDP session was not discovered until 2015 I one 23 201d First known adversary access to mainframe according to US- CERTFW I to August 2014 Attackers successfully ea ltratc 0PM background investigation data 0PM contractor Brendan Saulsbury testi ed that forensic lo 5 showed they are sort oftouehing or accessing the data during the summer of 2i 53 Wagner Tr at Iii 128 Saulsbury Tr at OPM Cybersecurity Events Timeline Brie ng by LIE-CERT to H Comm on Oversight dt Gov t Reform Staff Feb I9 2i KeyPoint CEO testi ed that there was an individual who had an 0PM account that happened to be a KeyPoint empioyee and the credentials of that individual were compromised to gain access to Hearing on 0PM Doro Brooch Pro-r statement of Eric Hess KeyPoint The OPM Director of IT Security Operations Wagner said multiple credentials were compromised during the E l incident but a KeyPoint credential was likely used for the initial attack vector Wagner added the adversary utilizing a hosting server in California created their own FIS investigator laptop virtually They built a virtual machine on the hosting server that mimicked and looked like a IS investigator s laptop and they utilized a compromise KeyPoint user credential to enter the network through the FIS contractor VPN portal Wagner Tr at 35 35' Saulsbury at 25-25 at 25-25 3'53 Letter from KeyPoinl Govemment Solutions to the Hon Elijah E Cummings Ranking Member H Comm on Oversight 8e Gov t Reform July 2 Brie ng by to H Comm- on Oversight dc Oov't Reform Staff Feb l'ii 2D Id 3 Hearing on 0 9M Dora Breech For statement of Robert Giannetta Chief In fo Officer U S Investigations Serv s Despite a contractual obligation to notify contractors immediately of a new or unanticipated threat or hazard OPM did not notify their contractors KeyPoint and U513 of the March 21314 incident rid 36 Hearing on OFM Doro Breach Part II statement of Robert Giannetta Chief info Of cer LLS Investigations Serv's LLC Coulter Tr Ea IE 36 OPM Cybersecurity Events Timeline 32 I uly 239a 2014 The domain Opinulcarningorg is registered to Tonyr Stark Iron Man a I August 2014 Following public reports of a data security breach at another contractor 0PM requested access to KeyPoint facilities and KevPoint agreed 3m I August 16 2014 The malware installed on June 5 2014 appears to cease operational - - - 3n capabilities I October 2014 Attackers move through the OPM environment to the Department of Interior data center where 0PM personnel records are stored I December 2014 Attackers ex ltrate 4 2 million personnel recordsm I March 3 2015 - is registered by attackers Attackers would use this domain for C2 and data ex ltration in the nal stage of the intrusion M I March 9 2015 Last beaconing activity to the unknown domain opmsecurityerg registered to Captain America attackers switched their attack infrastructure to wdc- newseposteom as their primary C2 domain for the remainder of the intrusion I April to June 2015 Primal I incident response and investigation period The timeline outlined above sets the stage for the incident response and forensic investigation that took place in the spring of 2015 In April 2015 0PM Realized They Were Under Attack Again On April 15 2015 0PM sent an email to reporting the presence of four malicious binaries and what would later turn out to be the rst indicators that systems had been compromised in the largest data breach in the historyr of the federal government 3 Saulsburv Tr at 70 the 0PM Director of IT Security Operations admitted 0PM did not have a fully logged environment in the summer of 2014 but the r were working award that end during the summer and through the fail of 2014 Wagner Tr at 173 Saulsbur r TL Ex 4 Hearing m'l' 0PM Dem Breech For statement of Eric Hess Chief Exec Of cer KeyPoint Gov t Solutions Letter from KevPoint Government Solutions to the Hon Elijah E Cummings Ranking Member Comm on Oversight Gov't Reform Julyr 2 2015 citing Report Aug 30 2015 '3th 'vaersr curitgr r Events Timeline 3 oomm a woe newsrosrcom 'l'linEnTCaonmpao last visited as aole 5 Saulsburv n at 59 sea area oomam wnc-newsrosreom available at hrcatcrowd orgfdomai php idomain-wdc-news po st com 3 us Dep'l ofHomeland Preliminary Digital Media Analysis-465355 May 4 2015 0PM Production Oct 23 Brie ng by US Office ofP'ers Mgmt to H Comm on versiglit Er Gov t Reform Staff Papa 13 2016 83 Documents and testimony show the initial discovery of the indicators of compromise IDCs involved a number of parties including US-CERT the FBI 0PM contractors the OPM IG and several private companies In April 2t 1 5 0PM discovered and began investigating the rst indicator that its systems had been compromised m Director of IT Security Operations Jeff Wagner testi ed that the rst indicator of compromise was an unknown SSL certi catef and was discovered during the rollout of a new version of the security application A Secure Socket Layer BEL certi cate is used to establish a secure channel between an individual s browser and a website In this case an 0PM computer had been communicating with an unknown website or domain opmsecurityorg The Committee obtained documents that show the unknown domain opmsecurityorg was initially brought to the attention of 0PM by a contractor Assurance Data during the roll out of a new functionality for Websense technology m Assurance Data identi ed opmsecurityorg in an email with the subject 0PM Daily Health on April 14 2015 33' 0PM was adding groups of users to Websense as they were transitioning towards filtering all outbound traf c through Websense m During the course of this rollout Assurance Data observed a certi cate error for the domain called opmsecurityergf m The next day April 15 0PM responded to Data Assurance In an email an 0PM employee described the domain opmsecurityerg as sketchy at best m The agency looked up the domain details and observed that it was what appeared to be a spoof domain or a domain that was purposely named to emulate legitimate looking websites belonging to or af liated with 0PM- There were clues that was a spoof domain it was a randomized email addressim and it was registered to Steve Rogers aka Captain America 0PM provided to the Committee a document entitled Timeline that provided more information about their ndings on April 15 and 16 related to the unknown SST certi cate 3 June a 2 15 DMAR at 154 see also Saulsbury n at 51-53 Wagner Tr at 3D Saulsbury Tr at 53 330 M 1 Email from Chief Sec 51 Strategy Di cer Assurance Data Inc to et al US Of ce ot'Pers Mgmt Apr 14 2 36 at 133 Production Apr 29 it Saulsbury Tr at SS 333- Email from Office of Pers Mgmt to Chief See a Strategy Of cer Assurance Data Inc and et al US Of ce of Pers Mgmt Apr 15 211115 9 51 am at 6 1886 0PM Production Apr 29 31 Saulsbury Tr at 59 ThreatConnect Research Team 0PM Breech Airoivsis June 5 2615 available at 84 According to this document the unknown SSL certi cate as identi ed and attached to domain oapmsecurityerg and six machines were identi ed as communicating with this domain 3 The AAR Timeline also reported that the domain opmsecurirtyerg was registered to a fake email address under the name Steve Rogers 3mg Further the AAR Timelinc noted that an alert related to this unknown SSL certi cate was initially discovered on February 24 2015 and the original beaconing traffic to this domain began in December 2014 339 The MR Timeline also indicated 0PM had identi ed three work stations and three servers on the OPM network that with the suspicious domain opmsecurity org 39 The investigation revealed that these machines had also contacted another potentially malicious domain - which was registered to Tony Stark a k a Iron Man and wdc news post corn Two of the three suspicious LP addresses each registered to a Marvel comic book character was a really big red flag for security team After running forensic scans 0PM was able to determine the suspicious 1P address registered to Tony Stark was in fact communicating with malwarc that was trying to under the radar as if it was a McAfee antivirus executable 2 This was noteworthy because 0PM did not use Beginning in 2005 US-CERT had issued alerts that APT attacks often used malwarc speci cally designed to elude anti virus software and rewalls and mentioned the use of McAfee and Symantec names in connection with these attacked After identifying the false IP addresses and the malware 0PM alerted At 6 53 p m on April 15 2015 Computer Incident Readiness Team led a report 1140428069 identifying four malicious binaries les that 0PM considered to potentially be malwarc or other malicious code Three of the four malicious binaries reported to US CERT on April 15 2 115 were identi ed as having the potential for a breach or a compromise passed a malware infection 3% Wagner Director of IT Security Operations also contacted the FBI's CYWATCH to report that the 11 addresses and domains associated with the incident as potential C2 servers the infrastructure necessary for an adversary to conduct an attackm The first evidence of the attackers' presence comes on May 2014 when the attackers dropped malwarc Ping onto an 0PM server that was one hop away from a machine with 1 MR Timeliae Unknown SSL Certificate April 15 21115 at Hoeaozos I o 1922 ow- 4 Production Apr 29 2015 Id 139 Saulsbury Tr at 59 3 Saulsbury Tr at so 192 an M 39' Teciim cai Cyber Security Afar 39 1 Targeted Iiujaa Email Attacks July 2 11215 39 Saulsbury Tr at Coulter at 14-15 7 Email from Fed Bucrau ot'investigation Cyber Div to Jeff Wagner Dir Info Tech Security Operations LLS Office of Pete Mng Ape lti 2015 2 19l am at 1910 Production Apr 29 2 1ti saa m se AAR Timeline Unknown SSL Certi cate April 15 2'1115 at HOGRG20316- 1922 Production Apr 29 2016 35 direct access to the background investigations and nger print databasem Ultimately these attackers were able to access Local Area Network LAM the foundational component of internet infrastructure and drop 1 ng malware 31 The malware which is a sophisticated piece of malware allowed the attackers to maintain a presence on system and network as of May 2015 and it also provided the attackers with other functionality This malware has an estimated 19 000 lines of code and comes with 13 default modular plugins m It provides an attacker with a range of functionality including the ability to log keystrokes modify and copy files capture screenshots or video of user activity and perform administrative tasks such as terminating processes logging off users and rebooting victim machines 4m has the ability to give attackers complete control over the infected The malware which was the primary piece of malware used in the 2015 data breach was engineered to covertly beacon back to the host s network resources and establishing a 331 connection to malicious domains opmsecuritymorg and wdc news and setting the state of a TCP connection 4m In effect an SSL connection establishes a secure or link between a server and a website which in this case was established between the malware and the malicious domains opmsecurity org and also found these attackers uscd opmsecurityorg primarily associated with the address as part of their attack infrastructure the intemet components necessary For the attackers to corrununieate with their 13 1ng malware throughout the life-cycle of the intrusion m Further found based on domain rewall logs that the compromised machines on network connected with known malicious 1P on January 12 and January 20 201 sacs Other variations of were found to have been active within the 0PM environment throughout the 201432015 intrusion The attacker placed additional modified versions of Plugxwdubbcd by investigators as the rst and second variations on victim machines on October I0 2014 and January 31 2015 respectively 4 These versions of 131ng were installed months after the key objectives of the intrusion were already achieved This shows the attacker was continuously modifying and customizing in order to better customize the malware to JPl vt s network environment maintain access and conceal malicious activities June 9 2015 DMAR at IIOGRGTE4-001154 Cybersecurity Events Timeline Roman Vasilcnko tit Kyle Creyts An Analysis LASTLINE Lass Dec 17 2013 Ryan Angelo Ceneza Pulling the Plug on Plugx Oct 4 2012 fofusr'threat-enc yclop eeliat'wcb-attacl-o'r Zipul Iing-the-plug-on-plugx 4433 Id June 9 2015 charts at rrocaoraa oer 154 June a 2015 DMAR at 4115 4 June a 2015 DMAR at 154 86 On a related matter the security research firm published a February 2015 analysis of the Anthem breach announced on February 4 2015 that mentioned the upm learningorg domain Anthem is a health insurance company that held data on as many as St million Americans current and former members of Anthem health plans and some nonmembers m ThreatConnect attributed the Anthem hack to a threat actor group variously described as Deep Panda 4m in February 2015 over one month before emu April 2015 discovery ThrealConnect found that this group may have also registered the domain opm learningorg as part of an intrusion campaign and noted had been compromised by a likely state sponsored Chinese actor in mid-March of ThreatConnect warned that because the domain was registered alter the breach occurred on July 29 2014 could be an ongoing direct target of Chinese state-sponsored cyber espionage activity In March 2015 it appears that the attackers changed their attack infrastructure The attackers switched their command and control servers installing a new updated version of malware on infected systems 2 Consequently on March 2015 the attackers registered the domain wdc news-posteom resolving to the IP address 413 The domain would switch IPis to on May 1 l 2015 after the intrusion was already discoverede The switch from to wdc news post com was accompanied by a new version of I lng nralware dubbed the third version by which would be programed to call-back to the newly-created wde-news-postcom domain 5 The March 2015 change in the attack infrastructure could rave been prompted by a number of factors First it is not uncommon for attackers to use different infrastructure during different stages of the intrusion life-cycle It is possible largceseale data had been completed by spring 2015 and the attackers were moving to a new infrastructure wholly unconnected front that used to effect the initial entry into network In the event this intrusion and theft of data was discovered the infrastructure used would be compromised Second changing the infrastructure would allow the attackers to maintain access to the network should their previous infrastructure be discovered it is possible open-source threat researchers were dangerously close to independently discovering infrastructure used in the 0PM intrusion 4m Threalcunnee Research Hie Anthem Hock AH Ronnie Lead to China Feb 2G l-roads-lead-to-chinaf Michael Hiltzilt Anthem is Warning Consumers About its Huge Doro Branch More s o' Translation LA TIMI-15 Mar 6 1915 m- is-warni rrg corrsumers Z l 503 'l'lireatconnoet Research Team The Anthem Hock AH Roads Land to Clrirro Feb 21315 are My or M 3 June 9 2 315 paras at 15 3 DOMAIN WDC-NEWS-PUST COM raeaarmowuoeo last visited June as sets lu eatcrowd orgt'tlornain php i'do main wdc-news- postn om June DMAR at rsr Id 8 The version of used in the intrusion had a suite oi capabilities that were likley customized for the OPM environment In describing the malware delineated the capabilities at the particular version of used in the 2014 2015 intrusion T his version of also is capable of remote access control enumeration let directory creation process creation enumerating the host's network resources establishing a SSL connection to malicious domains opmsecurityworg and and setting the state of a connection The ability to establish an connection to malicious domains would become a critical component in the ltaclter's ability to execute command and control maintain access and ea ltrate data out of OPlvtis network Hackers used the Ping to create fake SSL certi cates that would allow host machines to connect to the malicious domains opmsecurity org opm- learningerg and The use of these SSL certi cates eventually led to the discovery of the intrusion In Apn l 2015 0PM security personnel began installing Websense which gave 0PM an enhanced ability to lter SSL certificatesm During the Websense roll out the newly installed system was able to flag fake SSL certi cates to opmsecurityorg and other malicious domains It is not entirely known how or even when the attackers gained access to an 0PM network credential held by contractor Keyi oint but the attackers were able to use that credential to gain initial access into network using a virtual private network login to access an 0PM SQL server The attackers also setup remote desktop protocol EDP sessions from the SQL server to move laterally infected additional systems and gained additional footholds until nally connecting to their primary target the background investigation and ngerprint databases The KeyPoint credential was utilized for the initial vector of infection m but a number of compuomised credentials were used over the course of the data breach ' The credential that was used at the initial vector of infection the point at which the adversary drop ed malware to obtain persistent presence was being used by a KeyPoint employee s account But that KcyPoint employee did not have administrator credentials which are necessary to conduct higher order functions on IT environment JeffWagncr testi ed So the adversary utilised tactics in order to gain domain administrator credentials Exactly how they obtained the credentials we don't have forensic evidence for but they needed to gain another set of credentials to do operations It's not the only set of credentials they utilised to perform operations So there are multiple stages where various 5 June 9 2015 omen at noc acres not 154 4 June 9 sets DMAR at HOGRtl'i'Ztl not 154 Saulsbury Tr at 53-59 4 9 Saulsbury Tr at 53-59 Wagner Tr at 85 m Wagner Tr at 36 3 Wagner Tr at Ed 88 credentials were used and though as enforcing PW killed the capability of them utilizing the KeyPoint credential they still had persistence from the malwarc So they were able to get into the environment through another method to maintain persistence and then utilize domainfm After gaining access to the SQL server the attacker opened a RDP and dropped malwarc to maintain a presence on the SQL server The SQL server itself is significant for its use as the back end storage for various 0PM applications including a Jumpboa server used by the administrators that had access to background investigation data Saulsbury testi ed this jumpbox had access into the environments into the network segments that contained the background investigation systems 424 The attackers used an RDP to enter the jumpbox and use it as a pivot point to access all of the systems that were firewalled off from the normal network 425 The move from the SQL server to the jumpbos was a lateral movement by the hackers and it demonstrates their ability to maintain a presence on systems and also to gain the necessary administrator credentials necessary to move from system to system from computer to computer Using the jompbox as a pivot point the attackers were able to access the PIPS mainframe which stored the background investigation data and all the FTS boxes which are related to the ngerprint transmission system and nally the human resources department s systems with personnel records stored on systems hosted by the Department of the Interior- 425 These lateral movements as evidenced by RDP sessions and the timestamps on the 1ng variants continued from May into one of 2614 42 With access to mainframe as early as June 23 2014 and less than one month alter the May 2014 Big Bang the attacker would have had access to mainframe applications such as the background investigation data stored on the system By early July 2014 the attackers began to ex ltrate the background investigation data Evidence of data exfiltration would appear to 0PM and US- CERT in the form of RAE archives stashes of stolen datam The attackers continued to estiltrate the background investigation data through August of 2Itl14 4m but the fingerprint transaction system data was not taken until March 26 2015 43 43 Wagner Tr at 36 43' Tr- at 75 41 35 Saulsbury Tr at Coulter Tr Ex 13 433 0PM Cybersccurity Events Timeline Coulter Tr at Mr Coultcr would go on to describe the attackers use ot'R R files to ex ltrate data Saying so as is common in a lot cases or actually a lot of breaches iftlreir end goal is to collect data then they're going to search for it and bring it back to a central point for aggregation A lot of times data like this email if you were to compress it it would be you know potentially one l tlr ofthe size So EAR which is a compression Format is used to shrink data You can also then apply a password cases where there is data ea ltration or a con rmed breach it s very common to find these compressed stashes of whatever bad guys were after See also June 9 2015 DMAR at 156 0PM Cybersccurity Events Timeline June a 2015 DMAR at 153 39 The time period from early July 2014 when the attackers begin to exfiltrate the background investigation data to April 24 2015 when 0PM successfully eliminates the adversary from their systems represents the data breach In this nal phase where the attacker achieves their primary objective whether it is accessing and esfiltrating data or some other malicious activity it is important to note this end-stage would have been preceded by an initial penetration through 0PM's defenses an intelligence gathering phase to learn about network systems and security measures Then after all of this activity the attacker would nally drop the maiware and set up the domains necessary to collect and extract data The details of the initial phases of the attack and how the 2015 attackers penetrated defenses and gained sufficient knowledge of systems so as to quickly begin ex ltrating data likely will never be known What is known is how 0PM discovered the data breaches announced in 1 une and July of 2015 and how 0PM their interagency partners government contractors and private sector incident responders took 0PM from the initial indicators of' compromise discovered on April 15 2015 to remediation of the incident in June 2015 Between the rst si of the attackers foothold on May 2 2014 3 to the rst exfiltration of data in early July 2014 4 0PM would complete the Big Bang 435 to expel from their network the attackers discovered in 2014 From perspective by the end of May 2014 the 2014 incident was over little did 0PM know that the 2015 data breach operation was underway The following chapter provides additional details on 2015 discovery and incident response effoits that ultimately led to the discovery of background investigation and personnel records that were exfiltrated from the perspective of an 0PM contractor called Cylance which was brought in to assist 0PM in April 2015 0PM Cybersecurity Events Timeline 3 0PM Cybersecurity Events Timeline m_ 0PM Events Timeline 43 Email from Press Secretary Df cc of Pers Mgmt to Jeff Wagner Dir of IT Sec perations 1 1 5 Of ce of Pers Mgmt June 18 2015 8 01 pm at HUGH 020310-000206-6 0PM Production Feb 16 2016 432 90 Chapter 4 The Role of Cylance Inc Cylance Inc s information security tools detected critical malicious code and other threats to network in April 20l5 While Cylance tools were available to 0PM as early as une 2014 0PM did not deploy its preventative technology until after the agency was severely compromised and the nation s most sensitive information was lost OPlvt's IT security operations rccounnended deploying Cylance s preventative teclutology CylaneeProtect Protect to insulate enterprise from additional attacks after it became aware in March 2014 of a data breach whereby sophisticated adversaries targeted background investigation datafm The Committee obtained documents and testimony that show internal bureaucracy and agency polities trumped security decisions and that swifter action by 0PM to harden the defenses of its enterprise architecture by deploying Protect would have prevented or mitigated the damage that systems incurred to June 2014 0PM began evaluating numerous products including two Cylance products for possible use in its legacy environment The agency s consideration of these tools occurred at a time when the agency was aware its existing environment had been compromised and vulnerabilities had been exploited by a sophisticated adversary On March 20 2014 US-CERT noti ed 0PM that data had been eit ltrated from system 433 Agency of cials later testi ed this data breach resulted in the loss of security documents and manuals about high valued systems and applications on its enterprise architecture but downplayed the significance of these documents m June 2014 0PM Incident Report highlighted the sophistication of the attackers which used an extremely stealthy form of malware a Hikil rootkjl designed to hide its malicious processes and programs from the detection of commodity intrusion detection and anti virus products 440 A rootkit is malicious piece of software that uses administrator or root access to modify system settings to hide malware and malicious code at lower layers of an operating system rendering itself and adversary activity almost undetectable by common anti malware software H From March 20 2014 to May 2014 0PM and US-CERT observed the attackers to learn more about their tactics techniques procedures and objectives including the ex ltration of date - In the sets US-CERT June 2014 0PM incident Report stated Wagner Tr at 92 McClure Tr at 14 3'3 June rota Incident Report at Heeaosts-oo 1 233 Hearing on 0PM Dara Breach Part II exchange between Chairman Jason Chaffetz and UPM Dir Katherine Arehuleta and 0PM Chietilnfb Uli' Donna Seymour June 2914 0PM Incident Reporl at see supra Chapter 2 The First Alarm Bell Attackers Discovered in 2 14 Target Background Information Data and Ea ttrate System-related data What is Remit-ft AVG available at UStEturlNamc What is rootkit 3 June 2am ortvt Incident Report at Houses 1 scenes 91 T he attackers primarily focused on utilizing Server Message Block commands to map network le shares of 0PM users who had administrator access or were knowledgeable of Personnel Investigations Processing System system The attackers would create a shopping list of the available documents contained on the network le shares After reviewing the shopping list of available documents the attackers would return to copy compress and ex ltrate the documents of image-gt from a compromised 0PM system to a Command and Control SBWEF The discovery of a successful intrusion and data breach in the spring of 2014 put 0PM on notice Sophisticated attackers defeated their information security measures and practices and remained unnoticed as far back as July The attackers had a clear objective the background investigation material contained in PIPE In other words 0PM had every incentive to take swift decisive action to immediately fortify its legacy systems against a persistent threat that already had secured an advanced understanding of environment including its highest valued targets The agency purchased select tools from various vendors in June 2014 445 but declined at this juncture to purchase a key preventative tool recommended by the OPM Director of IT Security Operations called CylanceProtectm' and only bought its more limited tool CylanceV 4 The agency s security personnel remained interested in Protect and Cylanee arranged an extended demonstration in early 2015 443 When 0PM identi ed an indicator of compromise on April 15 2015 the agency turned to Cylance for assistancem As soon as 0PM began using the Cylance tools in April 2015 it immediately began nding the roost critical samples of malicious code on its networkm Cylance tools identi ed a signi cant amount of malware on network within 48 hours 451 and Cylance personnel quickly recognized the agency s cyber situation was direm 4ylanee personnel even con ded to each other internally over e-rnail They are fucked btw 53 By April 2015 it was too late to undo the damage Following the May 2014 Big Bang 0PM decided not to purchase and deploy Protect as a result of internal bureaucratic 3 June 2014 0PM tneinent Report at HDGRUS l U l134-35 June 2014 0PM Incident Report at Hoosost-ootzss 5 0PM Tactical TooIset Purchase Kick-off and Completion Timeframcs Oct 21 21315 Imperatis Supplemental Document Production Oct 21 2015 on le with the Committee Wagner Tr at 91-92 see ciao McClure Tr at 35-36 4 McClure Tr at Ill-2'0 tl-IH Coulter Tr Ex 2 lj-mail from Matthew Morrison Assurance Data Inc to Jeff Wagner Dir Info Tech Security Operations 11 5 Of ce ofPers Mgmt Apr 15 2015 10 43 pm at 0PM Production Apr 29 Z lti 0 Coulter Tr Ex Saulsbury Tr at 72 Email from to Brendan Saulshury Senior Cyber Sec Engineer ERA Apr 5 19 pm at 75 Production Dec 22 ISouller Tr Ex 3 Saulshury Tr at 7'2 53 McClure Tr Ex 9 Coulter Tr lint 5 45 1 m 92 hurdles and political challenges on the The Big Bang remediation proved unsuccessful the malicious actor linked to the theft of personnel records background investigation data and ngerprint eit ltration had already gained a foothold in system by May 33 14 455 The malicious actor downloaded 1 1ng malware on May 7 2014 on a key Microsoft SQL serverls ' at 0PM and had moved laterally across the network to access the mainframe which holds background investigation data on or about June 23 Mild The attackers ultimately es ltrated background investigation data from early July through August 2014 33d then exliltrated personnel records in December 2014 and ngerprint data in March 2015 Overview of the Gylanee Cyber Tools In June 2014 Cylance and 0PM personnel began conversations about the potential use of Cylance s products in the agency s legacy existing information technology environment LJI At this time Cylance offered two products to the marketplace CylanccV V is a detection product used on end-point devices desktop computers laptops etc First available to the marketplace in October 2013 software scans endpoints to determine whether or not something is malicious on a Deployment of v is limited to one endpoint at a time The product is focused on detection rather than prevention ofa cyber threat Cylance CEO Stuart McClure testi ed that will find where an infection might already be or exist and that will help IT operations to go into the computer clean whatever they want to that system But is not preventive It just is after the fact it will catch somethingm ' Protect on the other hand is designed to prevent malicious activity It is distributed throughout an enterprise where it utilizes mathematics and algorithms to determine good from bad That is it seeks to identify and address items that do not belong within an enterprise that could be a threat The agency s tlu'eat detection and initial response efforts in the wake of the March discovery revolve in part around the two modes available tlu'ough Protect Alert and Auto Quarantine In Alert mode Protect places the onus on the administrator running the tool to determine whether or not Protect has identi ed a malicious computer process that should be quarantined or if it should be white listed and remain operating on the environment When 4 McClure Tr Ex 4 McClure Tr at 44-45 5 0PM Cybersecurily Events l'irneline 55 June 2014 0PM Incident Report at 154 0PM Cybersecurity Events Timeline Coulter'fr at Eat it Email from Christopher Coulter to Jonathon Tends 0PM Cyberseeurity Events Timeline '53 Cybersecurity Events Titneline Brie ng by us-csnr to H Comm on Oversight a Gov t Reform stair Feb I9 June 9 2315 DMAR at 153 4 McClure Tr at 14 The Cylanec sales team was introduced to IT security personnel at 0PM through Assurance Data L'ylance s sales staff Nicholas Warner was introduced to IT security personnel through Mathew Morrison at Assurance Data McClure Tr at 124 3 Assurance Data maintained a re-seller arrangement with Cylancc McClure TL Ea 1 McClure Tr at E 4 McClure Tr at E 93 Protect is operating in Auto Quarantine mode it automatically removes and quarantines threats thereby requiring no intermediary action McClure testi ed Proteet sits on a computer in real time and watches everything that happens on a computer And every single element of the computer determines whether it s good or bad whether it's safe or unsafe malicious or not And ifit s malicious it stops it it blocks it It doesn t even allow it to start So true true prevention 4m According to McClure V R equires a user to actually hit a button that says point to this drive or point to this computer or this share whatever new hit scan It requires a physical body to do something like that Whereas CylanceProtect the agent can be completely hands-free If you just set it into auto quarantine mode just forget it If you have an alert mode of course then you have to review the alerts hopefully and then try and quarantine whatever things you nd that are bad in theref 63 On April 15 2015 0PM reported to LIE-CERT the rst indicator of compromised This led to June and July 2015 announcements regarding the loss of 4 2 million personnel records 21 5 million background investigation and 5 6 million ngerprints At this time 0PM owned V but had not yet purchased Protectfm 0PM Director of 11 Security Operations Jeff Wagner described how malware was discovered in 2015 Wagner testi ed that an indicator was found then it was followed back to an infected server and then the search began for the malware on the infected server 4M1 Wagner testified Tlhe initial malware discovery on an infected machine is normally not done by say a tool It s done once you nd an indicator and that indicator points back Then you use a tool such as Mandiant or Carbon Black or Cylance or various tools to do an overall search because once you nd one piece and you at additional indications you can then look for other indications as well r Wagner testified that the unknown SSL certificate was discovered by Websense and that Cylance would have found the speci c malware on the machine And then one engineers would have reverse engineered the malware to find it written within the malware 53 53 McClure Tr at 8-9 McClure at 46-41 June a 2015 nman at Hosanna-am 154 455 McClure Tr at 21 45 Wagner r at 54 1 Wagner in at 54-55 Wagner Tr at El 94 an older Websense to enhance the capability to include protection of remote users while attached to foreign networks Documents show the upgrade started on September 9 2014 and was completed by September 2015 Web-sense and during this deployment process identi ed an initial indicator of compromise On June U 2014 the agency purchased an upgraded version of Websensem to replace I14 47 By April 2015 DPM's IT Security Operations began to deploy the upgraded version of Saulsbury testi ed We originally detected a problem during the course of the Websense rollout as we were sending groups of users adding more and more groups of users to the pilot group to have all of their outbound traf c being ltered through Websense One of the things that we were doing was SSL Because that is such an intrusive method of inspection we were monitoring for errors with $31 certi cates that were potentially breaking access to applications updates and things like that 1 3 Saulsbury continued to describe the ndings while rolling out Websense saying W e also looked at the LP sic domain resolved to and put it into NetWitness We were able to see that going back we had these three machines that were going through Websense but we also had tlu'ee servers that had been contacting this IP address It looked very strange because there wasn t any business connection between these users' work stations and these three different servers So that is when the red ag started to go up as this could potentially be malicious activity At 6 53 pm on April 15 2015 Computer Incident Readiness 't eam OPM-CIRT led a repo with LIE-CERT and it was assigned incident number 415 is Nou'Forcepor'ur FDRCEPDINT available at On January 14 201d Raytlteon announced that it was rcbrauding the product FonsepointTM as part of a new venture between Raytheon and 1v ista Equity Partners List of Tactical Security Products Imperatis Production Clot 2 HHS Sauisbury Tr at 53 Id 4 Souls-bury Tr at 59 E-mail from to CERT torso Apr 15 2015 5 54 pm at rosin Production Dec 22 2-1015 95 From Sent Wednesday April 15 201R ti 'u-t PM To t'lFtl Subject follow Up on lririrlent rel- r'iimriei W1 le4 Fl lfi l hm wieiveil you report and has assigned Incident number for lulure reference incident Sulimlt Dare 6 53 13 PM lhanlt you US-Cllt'l Operations Center As 0PM began to grapple with the developing cyher incident the agency also discussed the possibility of using Cylance tools to stop the malware from functioningm The documents show there was already a high degree of familiarity with the ijlance products and their capability but that 0PM did not have full access to the tools 1 Message From Matthew Morrisrm Sent art utS 10 4313 PM To Wanner Jeffrey -xcnanet AWIMSTFIMWE caour JPWagner Subject Eviante I also have Cvlancc on ready to deploy protect to the venom- rs desktop and servers it WILL step malwaie from ru niatt 959% n- 7- ri r -- i wiry As of the evening of April 15 2015 0PM owned but did not have the latest version of nor did 0PM have access to Protect the preventative tool 3 The next morning April 16 Cylanee offered assistance to 0PM as the agency was attempting to point at endpoints and soon thereafter provided technical support to 0PM via conference call to help 0PM overcome incompatibility issuesm Chris Coulter Cylance s Managing Director of Incident Response and Forensics testi ed that was trying to use against a forensic image and the methods to do so aren't E mail from Matthew Morrison Assurance Data Inc to Jeff Wagner Dir Info Tech Sec Operations U-S ofPers Mgn1t Apr 15 21315 10 48 pm at 0PM Production Apr 29 Z lfi I Coultcr Tr Ex 2 In this email Matthew Morrison with Assurance Data wrote to Grant Moerschel Cylance Sales Engineer seeking the latest Cylance versions copying Nicholas Warner Cylanee sales director OFM and DPM contractors including Jeffrey Wagner Director of IT Security UpcralicnsD Coullcr Tn Ex 2 McClure Tr at 65 96 clearly documented because it's more of a trade craft to know how to do Coulth offered to be onsite at 0PM the following morning if the incompatibility issue with was not resolved ' Jonathan Tonda then an 0PM contractor in IT Security Operations replied We were able to resolve the issue and obtain results from Cylanec Thanks for your Hi ohl 3 Confidential 3 - From Tonda Jonathan D na i ltm Ir Saint Thursday April 1 6 2015 4 15 PM a To Chris cutter-I 3 9 Cc Sau lsbury Brendan 5 on subject RE cylance versions as Hi Chris we were able to resolve the issue and obtain resu1ts from Cylance Thanks for your help v- Jon At 3 56 p 1n Saulsbury sent Wagner a list of Four malicious caecutables identi ed by iv that were residing on 0PM servers and each malicious executable was assigned a score under the Cylanee rating systemfm3 McClure described this rating system in his testimony to the Committee He stated So we rank and score les and executional elements in a Spectrum from positive 1 to negative 1 Anything from a positive 1 to a zero is considered safe mathematically Anything from zero to negative 8 is considered abnormal And then From negative 3 to negative 1 is considered unsafei Three of the four malicious executables found by on April 16 2015 were rated r and the fourth was rated 93 on the Cylance scale 115 Coulter testi ed that the les showed That there s a potential for a breach or a compromise past a malware infection 4a One of the four files included a Windows Credentials Editor WCE Coulter described the signi cance of the nding So malware while as nasty as it can be is fairly common at least in a broad sense Somebody actually has to use that malware for it to be malicious most of the time When you see something like a con rmed Whidows Credentials Editor of other types of credential dumping tools that s usually a sign of an overt act so something that somebody with ill intent actually was trying to achieve versus just a presence of a Coulter Tr at l Coulter Tr Ex 2 is M- Coulter Tr Ex 3 43 McClure Tr 131 3188 435 Coulter Tr Ex 3 I joulter Tr at 14-15 97 malicious le which may or may not have been used A WCE 64 doesn t just appear for -- just to have it there It usually is used 431 US-CERT would later con nn WCE as a hack tool 433 On April 15 0PM found another suspicious le a McAfee dynamic link library DLL called macutildll that Saulsbury recalled in testimony as being integral to the attacks So we took Cylanee and put it on the known infected machine with the MeAfee rnaeutiLdH malware -- so the machine with the moutildll malware and then we ran Cylance on it to scan the machine for malicious artifacts And what it came up with is it successfully identi ed that moutildll le as malwarefmg The McAfec le was highly suspicious because 0PM did not use McAfee in its systems Saulsbury stated It was basically trying to y under the radar as if it was a MeAfee antiyirus executable The problem is that 0PM doesn t use McAfee so that stood out right there to us that at that point I was 100 percent certain that this is malware that is beaconing The next day con rmed the malicious nature of this le On Friday April 17 2015 at 1 1 39 am Saul sbury processed a new malware submission to US-CERT for its review that included the les he shared with the night before At 5 19 reported to 0PM is initial analysis of the executable lesm US-CERT reported that the mcUtill dll was a loader an operating system component that copies programs to memory When executed by a seemingly innocuous executable mesyncexe mcutill dll decompresscs and loads a third le into memory This le is the primary le - or payload for a remote access tool RAT called Plugx Each ofthese les was contained within a McAfcc VC folder which also contained an output file for the keylogger used the malicious domain wdc- newsportcorn for conunand and control 493 In other words the four les contained in the folder which resided within a directory called worked in concert to harm 0PM and did so in a way that was hard to detect Each of the four flies had a speci c function leii r Tr at 16 U S Dep'l of Homeland SeetiritnyS-CERT Malwarc Analysis Report-46635 April 17 21315 at HOGRUD92 50PM Production Dec 22 2'015 9'9 Sauisbury at as Saulshury Tr at email from to Brendan Saulsbury Contractor 0PM Security Operations Apr 2015 pm at 7'5 DPM Production Dec 22 2015 4 Email nm to Brendan Saulsbury Conlraclor orM rr Security Operations Apr ices 5 19 pm at rs orM Production Dec 22 2015 2 Id at June 9 2015 earns al 157 98 I is an dll file and Plug malware considered malicious After analysis of the Master File Table USE-CERT found that the file was time- stamped Documents show the creation date was March El 2U15 at 6 13 l am I Mesmeexe is a binary itself and is innocuous however it is used to load the 1ng malware through lv lcUtildll- Analysis of the MFT shows the le was time-stamped Documents show the creation date was March 9 2015 at 15 13 01 am I is a binary that has been identi ed as a 131ng loader lt attempts to connect to the malicious domain which resolves to LP- US-CBRT found the attacker time-stamped the le Documents show the creation date was March 9 2015 at 6 13 01 am - was found to be the output file created to store the key strokes recorded by mesynceal in addition to key-logging this version of is capable of remote access control enumeration filcfdircetory creation process creation enumerating the host s network resources and establishing a SSL connection to malicious domains reported 1 1ng was located in two 0PM directories a McAfee folder and a dimmrt 361 f l495 um Sent Frtoav 015 5 as PM TD 5- B'L 'dil Le Sol-lion i1 mart-ii 5 llif' a 15115 3 tilt-ill INLIH Update flu via-thin the inltlers loll - or- eat serve-i vow sirnllar tn lim- ini ilware mist-i iatml the folders contained two loaders named liilrillil all was small lnRElE-I'i are written ll'1 assembly- tanguage and are certr ln design and as the loaders withm the other Mle the leader themselves are loaded with the volrtl Matteo tool tl-us Iool 3 not malwarc they in lurn load and decode the file found in the li lci ifeesvti folders Thr- decoded files will If turn launch the RfiT conlamorl within the In this Last the utilized for command and control Milt the PtUGlt' RATS is as follows June 9 2015 areas at seasonal act 154 a os-essr Digital Media Aliases Report provides detailed analysis and insight into the speci c tactics techniques and procedures observed on the media submitted for June 9 lots DMAR at 155 99 On April 20 5 Coultcr arrived at OPIWs headquarters in Washington D C to provide on-the ground assistancem That day 0PM decided to deploy Protect but only in Alert mode not in auto-quarantine model Since 0PM had been familiar with the product since June 2014 but still did not execute a purchase Cylance staff was skeptical about whether this time the agency was truly moving to purchase and deploy Protect Cylance sales engineer Grant Moerschel emailed Coulter Is this a Proof Concept in their mind or the stalt of a real deployment m Coulter replied Not entirely sure what the back stories are all Iknow is they want this on all systems by the end of today Director of Sales Nick Wamer replied It s go timel 5m To Nicholas 'Wamer inbjeer RE 0PM Protect Access From Ntcholt'ts Warner Sent 2015 Til AM To Slim-rt McClure Subject Fwd 0PM Protect Access It's go time NW Begin forwarded messagi- an1 Date Apn - To Chris E'oulter - 1 1 Without Wilmer Grant Moerschel Subject Re 0PM Pratect Access Cll-t Keep Supp-an _nd I the loop We will do what we can to lie-1p glnut On Apr 3015 at 1'13 AM Chris Cooker wrote Not entirely sure what the back stories are all I know is they want this on all systems by the end of today Sent from my iPhone On Apt 1015 at 10 11 AM 'wl'ole Unis 7 - - Director of IT Security Operations Jeff Wagner testified that we initially started using Cylance for malware analysis Within a day or two we obtained the Protect It was part Coultcr TL Ex 2 see m xa 1PM 1 it isitor Log Washington DC April 1 2015 to July It EDIE at 0PM Production Feb 16 Coulter TL Ex 17 95 McClure TL Ea a 100 of our license lbelieve SDI As oprril 2015 0PM had not purchased a Protect license and did not purchase such as license until June 30 2015 32 Nonetheless Cylance provided 0PM full access to Protect in mid April 2015 on a demonstration basis and without purchasing a license because as Cylance testi ed it was evident 0PM was under attack and they deemed it the appropn'ate course of action McClure testi ed A Yes So typically like we say an evaluation of this sort would be a small evaluation However when it s under these kind of incident response emergency situations we allow them to install on as many boxes as they want Because we just want to help them provide them the support get them to be able to identify the problems and then prevent them clean it as quickly as humanely possible get the bad actors out of the company organization- So we allowed them to install on all of them as many systems as they had a little unusual for an evaluation but not completely unusual especially under these circumstances Q Those circumstances being A That they were under severe attack and had been for quite some time Q And you just described incident response efforts going on Are you aware of the sense of urgency in how 0PM was responding to what they found and flagged for your attention the day before A Once we were engaged on April 6th l'r'th it was very much a re drill every 24 hours And they were taking it very very seriously from all of our observations and reacting as quickly as possible and getting as much help as they could and engaging with us and getting the technology out there and trying to quarantine as quickly as possible It s actually one of the poster-child examples of how to do it properly in an investigation just as soon as you humaner possibly know that you ve been breached to try and roll out this new tech 1 think they did an admirable jobm With respect to why 0PM utilized Cylance tools in April 2015 Wagner testi ed We were uncomfortable with just trusting that we knew all the indicators of compromise And so we obtained the Cylancc endpoint client and 5m Wagner Tr at 95 McClure Ex see also Cylancc Purchase from Assurance Data Inc June 3G at CYLANCE DDGDIE Cylancc Production Dec EDI McClure Tr at 53-59 101 d pl y d it and then a Cylance engineer helped make sure we got it con gure i Directly in get proper information out of it s Wagner also testi ed that Cylanee was able to find things other tools could not because of the unique way that Cylance functions and operates It doesn't utilize a standard signature of heuristics or indicators like normal signatures in the past have been done it utilizes a unique proprietary methodf j On April 13 2015 one day alter deploying Protect 0PM rapidly escalated its use throughout the enterprise McClure wrote checked in on the deployment and we are at 2226 devices at last count Tons of ndings Chris is working through them already quarantining It is juicy l McClure testi ed Wle were nding a ton of malicious attacks on -- on the boxes that we were getting deployed to -Jr On April 13 however 0PM was not yet utilising Protect s full capability- The agency was using the product in alert mode and not r auto quarantine mode ug Agency personnel therefore had to determine what should be stopped from operating in environment after reviewing alerts When McClure stated in the April Sm email that Cltris is working through this statement describes the steps that must be taken to evaluate each item 0PM was alerted to before agency personnel could then consciously address them extracted from the environment white listed etc McClure testi ed that only about ten percent of Cylanee s customers use the alert-only mode and in alert only mode the product will alert only when an attack is present or happening in the systemi m Wagner testi ed that 0PM was running Protect in passive mode because we didn t want the tool to automatically end up deleting forensic evidence that we needed m That is not how Protect works McClure testified W1hen we quarantine a le we don t actually delete it yet The rationale is if' we quarantine something by mistake that s a false positive In that rare instance the customer would want to unquarantine it to put it back in production So we keep it in a secure untamperable space on disk that allows us to perform that unquarantining Unfortunately that does take up Space as part of the quarantine area 1 Protect identified 39 Trojans on various parts of network that were rated a negative one on the Cylance rating scale the worst possible rating and Cylance sta ' recommended quarantinng these items 5 2 The finding of 39 'l'l oj ans was significant because as Coulter testi ed the Trojan s functionality allows the attacker to bypass to some degree 5 Wagner Tr at 32-33 5'35 wagl'l l' Tr at at Ex 3 if McClure Tr at 25 McClure Tr Ex 3 McClure at lot 1 Wagner Tr at 94 5 McClure Tr al 71 Coultcr TL Ex 4 102 security controls and allow a bad actor in some cases unrestricted access to a netsvorkFS'3 Coulter stated Any one Trojan could have that capability m in fact when reviewing the work ticket that identi ed these 39 Trojans Coulter testified To say it bluntly Protect lit up like a Christmas According to lCoulter Cylance s team concluded these were downloader les which are typically associated with malware and multiple Troj ans 516 When asked these results caused concern Coulter stated Having gone through security clearance process many times I know what 0PM does And dealing with APT almost on a daily basis you put two and two together You can just assume the risk that you know what could unfold or what could he there 5 It quickly became clear to Cylance that the l l security situation at 0PM was dire 5 '3 By April 19 2015 malicious items continued to he found in enterprise Front Chris millet Sent Sunday April 19 1015 Alt-I To Stuart MctC ltu'c Cc Subject 0PM They are fucked hrw 1Itt- alleiog their threusic guys through some analysis and I pointed them to an rar archive of scene bad stuff Stu can we use Brians rig to crack thenl' Not seeing the common that would give as the password easily Chris Conlter I Consulting Director In an April 19 email Coulter reported to McClure that he had identi ed an rar archive of some had stuf McClure told the Committee a file is a compressed archive of other les that he recalled seeing evidence of an attack that had already been there been successful and it was nasty and that t here were signs of exr ltration of data In order to address the rar archive nding Coulter asked for assistance with another tool to help break the McClure testi ed W hen forensic folks like us get on-site and take a look at these things we can t easily open them and see what they ve been able to steal and push out of the environment without using something like a GPU Graphics Processing Unit password cracking rig which is what s 3 Coulter Tr at so 5 Conlter Tr at so 5 lCoultcr Tr at 26-21 5 Coulter Tr at 2 f McClure Tr_ Ex 9 Coultcr TL Ex 5 McClure at 103 referenced here So he s saying you know I m not seeing the common BAT or VHS les that would give us the passwords easily So typically BAT is short for batch les and they are Windows batch les And VBS is short for visual basic scripting or script both of which help automate certain commands that are run on a computer system And oftentimes because hackers are lazy they ll put into the batch or the VBS scripts the actual hard-patted password of the RAE so that they can help automate both and of it in their tasks m On April 19 the signs of a signi cant compromise at 0PM were clear Coulter testi ed They re in a severe situation It s an incident now It s much more than just a malware incident So when I was talking earlier about you know credential dumping tools and overt actions this is again another overt action If you don t usually -- if you can t explain why you have a large RAR archive in a location that most administrators would recognize there s -- it s likely a stash of somethingcommon in a lot of APT cases or actually a lot of breaches if their end goal is to collect data then they re going to search for it and bring it hack to a central point for aggregation A lot of times data like this email it you were to compress it it would be you know potentially one-IDDth of the size So RAR which is a compression format is used to shrink data You can also then apply a password cases Where there is data ex ltration or a breach it s very common to nd tgcse compressed stashes of whatever bad guys were 5 after Like McClure Coulter also testi ed that as of April 19 2015 a signi cant chance existed that data from 0PM had been ca ltrated 523 analysis validated their concerns According to Analysis of the image revealed that several variants of Plugx once resided on the victim machine with the last variant from downloaded folder RAF SFX2 still residing Several password protected EAR files were found on the victim machine which have been identi ed by the customer as ex lh'ated data 524 McClure Tr at 2123 53' Couller Tr at 25-25 513 Toulter at 2154 Coulter at 2T 51 June a 2m 5 DMAR at tse 104 The RAE les that had been identi ed were notable because these les were ultimately linked to the data exfiltration of the background investigation and ngerprint data and personnel records For example RAR SFXE appears to contain FTS data held on the attackers primary foothold WDC new-postcom Another RAF SFXZ when downloaded created the MoAfeeS le folder in a directory located on a key Microsoft SQL server and its duplicate server This location gave attackers access to a key jump box that facilitated access to other segments of environmentmsegments that house sensitive information m found the attacker was active on that server stating the first appearance by the actor that was observed on the victim images was on 5 2014 at I I l2 25 PM from a SQL Server 528 analysis of this string of malicious activity would later point out the liability to the country It is interesting to note the machine had an remote desktop protocol session with United States Governmen system_ on In other words was pointing out a remote deskt0p session that occurred in October 2014 on the system that led to a tunnel Interior Business Center at the Department of Interior D01 and to the federal employee personnel records that were stolen US-CERT and 0PM would later affirm that the attacker pivoted to the data center at DUI in October 2014 with the personnel records subsequently being exfiltrated in December 2014 530 In an exchange with Rep Robin Kelly IL DOl s CID Sylvia Burns would later testify before the Committee ab0ut how the attacker traversed onto network and stole the personnel records Ms KELLY Thank you Mr Chairman Ms Burns the two data breaches 0PM recently reported have been particularly concerning to us because of the national security risk involved According to testimony you 35 June 9 2015 BM el 53 U S Dep't of Homeland Digital Media Analysis Report-465355 June 9 Elli-15 at 000090 Production Dec 11 2015 Saul abury Tr at 7345 35 June 9 2015 omen at 154 us oep' of Homeland Sectlrityl'US-CERT Digital Media Analysis Report-465355 June a sols at cleanse S-CERT Production Dec 1 1 2015 an 0PM Cybersecurity Events Timeline 105 gave at a recent hearing on the OPM data breaches the OPM personnel records that were compromised in one of those bleaches were hosted in the data center maintained by the Department of Interior Did the cyber attackers who gained access to those records also gain access to the Interior Department data center Ms BURNS So the adversary had access to our data center It was exposed There was no evidence based on the investigation that was led by DHS and the FBI there was no evidence that the adversary had compromised any other data aside from the OPM data Ms KELLY Okay so the same cyber intmder who breached personal data which the Department of Interior hosted on its servers also breached the defense s ot the Interior Department data center Ms BURNS So this the intrusion that you re refen'ing to was a sophisticated breach And my understanding based on assessment was that the adversary exploited compromised credentials on side to move laterally and gain access to the Department of Interior s data center through a trusted connection between the two organizations- Ms- KELLY So the cyber intruder did they gain access it to 1301 s data center through 0PM or was it the other way around Ms BURNS The adversary gained access to 1301 s infrastructure through 0PM as far as I understand based on investigation Ms KELLY- In addition to hosting OPlvl s personnel records the Department hosts data from other agencies in its data center is that correct And if so which agencies Ms BURNS Yes Actually the Department is a the data center in question the biggest customer of the data center is actually Interior So it s the Interior Business Center what we call IBC- They re a shared service provider and they are the majority user of the data center And we also host some applications for the Of ce of the Secretary in the data center 53 I The same day RAE les were being discovered April 19 2015 Protect also identi ed command shells 532 Command shells are signi cant because they provide a means for the attacker to remotely control a victim machine On April 19 2015 McClure wrote to Coulter 53' Cybersecurr ge The Department ofthe Interior Hearing Bar y-s the Subcomm on burnt-marina their and Sitbeomm on Interior ofthe H Comm on Oversight d c Gov't Reform 114th Cong 21-22 July IS EDI 2 McClure Tr at 31 Email from Stuart McClure Chief Exec Uf cer Cylance to Chris Couller Managing Din Cylance Apr 15 EDIE 9d at CYLANCELUUEI 12 Cylancc Production Jan l id 106 They quarantined one of the medexe les but I found two more Might want to recommend they quarantine those torrid McClure explained the signi cance of finding med exe files A Sure So XCMD -- so CMD stands for command and they usually stand for conunand shells And what that allows you to do is actually have remote access of their computer on your own computer So when you start XCMD on the victim box it will then create a shell to you on your remote computer wherever you are in the world and you can then type commands as if you are sitting right there on the computer Q And why did you recommend quarantining another two mentioned in the message A Because that s that s as nasty as you can get I mean they can do anything that they want with that accessic'34 Cylancc and 0PM made additional ndings about the breach on April 19 20 5 535 Then on April 20 2015 a Cylancc expert contacted Coulter about 0PM data collected and a backdoor Thus began a chain of events eventually leading to the discovery background investigation data had been stolen Speci cally the Cylance expert wrote to Coulter Give me a call when you have some time l rn going through the data now Wanted to ask some questions about the system WCE was sitting on and a few others You may want to have them get an image of is a backdoor that looks like the command and control server was active around $2014 corresponding to when they came out and said they had a problem Callback was to resolved to if they have any kind of network or DNS logs going back that far m This communication in particular would start the process of revealing how the background investigation materials were compromised More evidence would unfold and become clear in the coming days 533 McClure Tr at 29 Email from reituart McClure Chief Exec Of cer Cylance to Chris Coulier Managing Dir of Incident Cylancc Apr- 19 5 9 9 at Cylance Production Jan 27 Ellie 5 McClure Tr at 29 30 535 The same day that Cylancc identi ed RAR les and was working to decode the passwords Protect found a fraudulent attempt at making this look like a Bit iI signed binary Set the signed by Bit And website 1 v'irns Total calls it quite evil McClure 'l'ranscribed interview Ex 1'v irtls'l'otal a subsidiary of oogic is a free online service that analyzes les and URLs enabling the identi cation of viruses wonns Trojans and other kinds of malicious content detected by antivirus engines and website scanners About Virus Total available at 36 Couiter Lia s 10 The agency continued to expand its use of Protect through April 21 2015 The tool was on 6 25 hosts and it was expected to roll out to 10 000 hosts soon thereafterd On April 21 Cylance also identi ed two Trojans sitting on key servers Fran Chris Coulter Sent ruesdn a rt 3 2015 12 51 AM To c Subject 10 5 for 0PM gon Gross flagged these please make sure they are tagged correctly as Mdlware Trojan tailback to Team - At thatpoint 0PM also began utilizing more outside help CyTeeh's Enter rise was installed on the servers where Coulter had identi ed new pieces of Trojan malwarc 53 then imaged malware and artifacts residing on these servers that were subsequently supplied to These ndings were covered in US-C May 4 2015 Prelimir i ry Digital Media Analysis Report and June 9 2015 Digital Media Analysis Report Cylance also discovered remnants of malware used by adversaries in the 2014 intrusion against 0PM CylanceProtect found dormant variants of Hikit which was the primary malware used by the attackers discovered in 2014 on systems during the discovery phase ofthe 2015 investigation Jeff Wagner Director of IT Security Operations stated Cylanee In doing a full analysis of the entire nd an older version oinkit It also found library fragment les of rearward S 41 Wagner testi ed regarding the Hikit malware found by Cylance and its relevance to the 2015 intrusion A So the llikit variant discovered in 2015 was not an active piece of malware it was a dormant piece of malwatc That because Cylance was utilized to analyse the entire environment we discovered the maiware was dormant within one of the servers It was believed to have been an abandoned piece of malware that was previously installed at some other time Q Was it related to the incident in 2015 5'37 McClure Ex 1 l Coulter Tr Ex Briefing by 1 1 3 Dfiice ofl ers Mgmt to 11 Comm on Oversight St Gov t Reform 13 2016 u 1 1 5 Dep t of Homeland Security lS CERT Preliminary Digital Media Analysis Report - 1NC405355-A May 4 2015 at Production Dec 1 1 2015 Brie ng by 1 1 3 Of ce oFPers Mgmt to H Comm on Oversight r'i r Gov t Reform 18 2016 54' Wagner Tr at 120 108 A We don t have direct evidence it was necessarily related to the 2015 incident It was discovered in the 2015 incident ear-e Q Sorry So did you have any indirect evidence that the Hikit Found referenced in the 2015 DMAR was at all involved in the 2014 breach A No We don t don't remember the exact quote bent on date of the malware which shows the initial point of infection but it was not during the HHS timeframe of adversary activity So we really didn t have a recognized idea as to when it showed up It was one of those pieces of malware as well as additional fragments of former malware that Cylance identi ed and we proceeded to eliminate along with everything else 542 One of the two Trojans found on April 21 contained what called a uniitlue 5'13 file named winrsvesdll with a compile time of 5 34 4s EST on March 18 This file was a malicious Windows Dynamic Link Libraries le designed to run as a service When running the DLL allows a hacker to pass and execute executables and DLLs to a victim system at willim This rst unique Trojan file winrsvesxill contained a plugin framework that allowed it to import and load DLL les described the file as follows The DLL which is identi ed as a Hikit Remote Access Tool is unpacked and loaded into memory while never being written to disk During execution this ELL will attempt to read a con guration file in the same folder in which it was executed This con guration is expected to have the same name as the originally executed le but with a eonf extension In this case the expected con guration le is winrsvesconf If this le is not found the malwarc will create a configuration file which contains its default The Cylance found on April 19 would reveal that the con guration le contains the command and control location The con guration le contains the configuration string 5 13 Wagner Tr at 134 135 us Dcp't of Homeland Malware Analysis aspen-aeessr-s corrected April 24 2e15 at EUGENE-001065 UPM Production Dec 22 2015 5 LLS Dep t of Homeland SecurilnyS CERT Preliminary Digital Media Analysis Report - May 4 2315 at Production Dec 11 2015 1 1 3 Dep t of Homeland Security JS Elt'l Malware Analysis Report do 352 E corrected April 24 21315 at 0PM Production Dec 22 2tl15 5'15 11 3 Dep t of Homeland SecuritnyS-CERT Malware Analysis Report-4603ST-A April 24 21115 at QUE-CERT Production Dec 1 I 21915 U3 Dep t of Homeland Secttrityi'US-CERT Mal ware Analysis April 24 2 15 al 1601 90-91 Production Dec 11 2015 a June 2015 DMAR at 154 This particular HiKit uses the some strong in the output con guration le as found in DMAR 3551711 109 The second Trojan was located on a server and was called According to LIE-CERT this was a Dropper Generic TIC Hikit found to have resided on the victim machine since September 15 2012 at This binary also pointed tn the malicious domain 550 The cybersecurity event that was developing at 0PM was serious it was not until April 22 2015 however that the agency noti ed the Of ce of the Inspector General that it was dealing with a breach 55 In fact the notification occurred entirely by accident 552 And while the Protect deployment was successfully identifying critical malicious items the product was still being introduced into OPlvl s system conservatively Protect was in Alert mode meaning threats were not automatically In addition Protect was not yet on all 0PM hosts in April 23 2015 Coultcr emailed an 0PM of cial Just letting you know we do not have Protect on the following key hosts servers 554 On April 24 2015 0PM upgraded Protect to autouquarantine mode At 4 1 1 pm on April 24 Coulter emailed several colleagues to announce the upgrade He wrote Guys 0PM hit critical mass today and is burning the house - literallyl They just hit glohal-quarautine for every threat I think it was around 1180 threats in the queue This was done per senior orders They are also pulling the power on every device starting Saturday at 9am Sunday at 5pm I need cveryone's help to make sure what they quarantined will not he mission critical les I have been up for 24 hours so 1 really do need help 555 on Apr 21315 at 4 11 PM C lu'is Coulter wrote 3 Guys - 0PM hit critical mass today and is banning the house - literally 9 They just hit global quarantine for evety threat 1 think it was around 1 tlu'eats in the queue This was done per senior orders They are also pulling the power on every device starting Saturday at 9am - Sunday at 5pm l I need eveiyoue's help to make sure what they quarantined will not be mission critical les I have been up for 2-1 hours so I really do need help l'n- f f June 9 sins DMAR at n3 3 55' DIG Memo Serious Concerns See hi 'n Chapter Ophelia C10 and its Federal Watchdog 3 McClure at 33 55 Coulter Tr Ex 3 555 McClure Tr Ex 12 Prior to April 24 0PM manually considered whether each item that Protect agged should he removed from the system McClure testified My recollection was was processing all the alerts themselves along with the help of us at Cylanee our alert management team as well as Chris Coulter myself and others to help them triage and process the alerts to make sure that they are malicious and not safe and just trying to empower 0PM themselves to make the judgment call on whether to quarantine those files and move them out of alert-ordy 556 Thus while Protect was operating in alert mode the burden was on 0PM staff to determine what les should he quarantined or he allowed to remain operational in environment ure testi ed Q Can you de ne when you said that 0PM was processing things on their owo can you de ne processing A Yes They were in our management console looking at each alert trying to understand if they should actually quarantine it delete it or just allow it to continue to be on the system and study it for whatever purpose Q So 0PM was making the decision on what to delete out of the items identified prior to April 24th 2015 A Correct All customers manage their own quarantined Saulsbury who was on site at 0PM on April 24 2015 provided similar testimony So after we observed that Cylance was able to detect the APT malware in this case it was in the 2015 incident it as a malware family called Plng And once we were able to determine that was able to detect Plng at some point there was a decision made to deploy the Protect agent to all of machines So that was done with the assistance of the vendor of lCylance And so the guy that I am emailing on that is Chris Coultcr So Chris was really good about helping us getting Protect deployed throughout the environment and then also analyzing all the findings that it is coming back with So Cylancc is detecting not just the APT malware but every type of malicious like adware toolbar that somebody downloads and things like that as well as the false positives here and there ll McClure Tr at 34-35 5 McClure Tr at 35-35 111 So Chris was really good about helping us triage through that list and separate what we want to quarantine versus what is false positive and whitelisted So at a certain point we were con dent enough that we had identi ed all of the malware and had whitelisted the business critical applications that needed to be whitelisted And so IelT instructed us to quarantine all of the identi ed ndings What that quarantine means is so when Cylanee detects something we just had it in alert mode So it would see it and say hey this is bad but it is just alerting us on it it is not actually doing anything about it So what we essentially did on April 24th was press a button in the Cylance console and says everything that you ve seen that is bad take that and quarantine it so it is not operable on the machine Wagner also con rmed that 0PM quarantined all the identi ed malware on or about April 24 2015 With respect to why the quarantine did not happen before Apri124 2015 Wagner stated So once you identify inaiware functionality or adversary activity you try to get a sense of the adversary s intention activities and exposure You look to see how deep they are in the environment So once you discover something on the l5l we didn t want tojust start shutting things off We didn t understand the depth in which the adversary had been in the environment- With the deployment of the Cylanee tool a full accountability of all binaries we had discovered identi ed and all the malware was placed into the uarantinc queue by I think it was the 19th of April And by the 24 we had a full understanding that it had discovered everything that was to he discovered and we no longer necessarily needed the adversary to have an active presence within the environment So we ordered Cylance to destroy the malware sg The auto-quarantine did not apply to all of systems however For certain systems 0PM made a value udgment as to whether they should be included in the auto-quarantine or remain subject to the human command quarantine in auto-alert mode Coulter provided guidance to his colleagues at Cylance on April 24 2015 regarding what les to quarantine He wrote I would say anything on desktops are ok to quarantine Servers should be the only thing questioned at this point If they can live without it keep it blocked They are sening up some help desk protocols to identify issues that come out of this Mission critical items that Iknow of Saulsbuly Tr at T233 Wagner Tr at 121-122 112 USA JOBS related apps - they said if we bring that down senators will come for us LAN Desk at SCCM SQUOracie components and connectors to mainframes Past that they can live without for a few weeks This is a desperate move tomorrow is even more desperate by unplugging ever device and moving over to new networks They will blame any issues on the power outage 3 550 McClure testi ed that in auto-quarantine mode mission critical items may stay in alert mode so as not to undermine the system in the event of a false positive McClure also testi ed that 0PM should have considered shutting down mission-critical items given the severity of what Cylance was nding He testi ed Yes they should he 1 Documents and testimony show 0PM used Protect as its quarantine tool and that Protect was not put into auto-quarantine mode until April 2 4 2015 Documents and testimony also show some 0PM systems were not placed into auto quarantine mode at all Contrary to this evidence leadership testi ed before the Committee in June 2015 that the quarantine was hilly in place by an earlier date and stated that the malware was latent and merely being observed 33 The term latent means the malware is not active on the environment it is frozen or othenvisc not leaning on active computer processes The quarantine status was not activated until April 24 2015 when 0PM gave Cylance the authority to place Protect into auto-quarantine mode 5 Unless Protect is in auto-quarantine mode malicious items are not latent wan action is required to stop malicious items um functioning in the environment j According to Wagner in the days that followed the deployment of Proteet s auto quarantine function 0PM had discovered everything that was to be but significant discoveries continued The new discoveries were noteworthy because they provided evidence related to the loss of background investigation materials On April 26 2015 Coulter and Jonathan 'l onda an 31PM contractor at the time in 0PM IT Security Operations engaged in an entail exchange about a segment of the OPM networks This was the same segment that a Cylancc expert asked Coulter to image on April 20 writing Give me a call when you have some time l rn going through the data now Wanted to ask some 5 ocular TL Ex 55' McClure at 6T 3 McClure at as 3 Hearing on 0PM Dora Brooch For at 69 see In o Chapter 5 The CyTech Story for more on quarantine statements by 0PM of cials before the Committee 5 McClure TL Ea 12 Coulter Tr at Til 7'5 E s McClure Tr at 34-36 Coultcr Tr at 34 36 55 Wagner Tr at 121 122 5 Coulter Tr Ex 13 H3 questions about the system WCE was sitting on and a few others You may want to have them get an image of is a backdoor that looks like the command and control server was active around 612014 corresponding to when they came out and said they had a problem Callback was to resolved to if they have any kind of network or Domain Name System logs going back that farfd In this April 26 email exchange between Coulter and Tends Coulter was investigating a Remote Desktop Protocol session that dated back to June 20 2014 and accessed a particular segment environment Conlter asked Tonda what was hosted on the segment Couiter was investigating Tonda responded the segment Cylance identi ed was where lot of im ortant and sensitive servers supporting our background investigation processes are located 0 This was an important development because this server provided access to the PIPS mainframe where background investigation data was stored 71 CERTIOPM would later confirm the rst known adversarial access to mainframe as occurring June 23 2014 in 5 Coulter Tr Ex 5 Coulter Tr Ex 13 M 5 Conlter explained in the email that the segment he had identi ed was a key jump box at 0PM identi ed as -- means a server that manages access between two different network sections of the larger information technology environment Saulsbuiy Tr at 7436 In 0PM this partieularjumpbox enabled access to various parts of the 0PM environment Saulsbury Tr at T4 Tii and Cylance's Couiter was letting 0PM know on April 26 that the jumpbox had a Remote Desktop Protocol session to a signi cant server that gave access to the portion of network where background investigations are stored Couiter Tn Ex 13 Brie ng by LIE-CERT to H Comm on lEnter-sight d Gov t Reform Staffilieb 19 0PM Cybel'sccurity Everth Ti meline 114 To China Ceu1ter CC Date 4fztij2t115 3 45 23' PM I Subject Re oi rect L-inlt Potentiall There is an application called EPIC but that 15 accessible from more I than the server Question if an exe or 111 currently has a process running will quarantine I completely shut it down 9 the mcaFee which was injeeted into also can we comp ete it send malware and any of its remnants from a system via ey'tance on a 25 2015 at 5 15 PM Chris Coulter wrote I Then on that is helpful for us there's an sston from to on at 04 22 21 as user EMS is instance that we sax _ used on at system we also noticed an odd controlset k bein enerated com in just coincidence Would web browsers he used for accessing juicy items i From Tends Jonathan D mailto_ Sent Sunday April 26- 2015 6 0 To Chris Coulter subject He otrect Link This 15_our our Buyers FA data center It covltains various werkstatwns servers punters etc This site is also where re located A ot of important and sensitive servers supporting our I I background investigation processes are located here why Jon i 2 on agar as 2015 at 6 05 vs Chris Coulter 42 wrote Jen m what mm hosts he on i Thanks equity With respect to this jump box US-CERT found another related directory infected with Plng reported Malicious hinaries no longer reside on the victim machine which has been identi ed as a jump server however analysis displays the system was once infected by malware Remnants of malicious les were found in the director r with les and located on image Also metadata displays malicious domain found on imagem As was the case with the directoryr that contained malware this directory ontained four files one output kcyloggcr le an innocuous file that 1 ng used and two binaries that were 131ng malware files By the end of April the situation at 0PM began to stabilize and Cylance personnel prepared to leave the agency s headquarters On April 29 20 5 Cylance reported to Wagner and others at 0PM that i will be working remote today r as I think everything is resolved that would have required me to be ensue 5 June a 2015 ones at 155 f June a 2015 DMAR at 5 Coulter Tr Ex 14 115 As part of a close out email Coulter updated on the work that Protect was doing Coulter wrote We have been working diligently to permanently assign new threats into either blacklist or safe - list que There are roughly 225 les that I would like to go over before we take any action 1 will send the spreadsheet of these tonightm Cylance also provided instructions to other entities who were remaining on site at 0PM Coulter wrote If 0PM can commit to having all output script results back before Thursday next week this plan will work 1 will have 2 of my best guys scheduled to come down Thursday and Friday next week to help in analyzing the results of the bat script deployments We will be done on Friday around Close of Business and would like to have a formal meeting with the CyFir St the other team members to close out While the situation appeared to be contained 0PM continued to face new and evolving threats For example on May 1 2015 Coulter wrote Wagner and Tonda we just saw the very first instance of a prevented Upatrea'Dyre Trojan infection due to settin auto quarantine Completely unknown to industry and stopped before it could do any harm 5 3 The Decision to Purchase Cylanec-Proteet CylanceProtect was the first tool that 0PM used after the agency learned its network was compromised and the tool immediately found malware and set about cleaning OPlv l s This raises a question as to why 0PM did not purchase and deploy the tool sooner in June 2014 when it may have been able to prevent or mitigate the attack especially given the fact that 0PM knew its most sensitive data was being targeted by sophisticated hackers Documents and testimony show internal agency politics and procurement challenges made it dif cult to quickly purchase and deploy security tools On June 12 2014 less than done months after becoming aware ofa signi cant cyberattack 0PM executed a Cylance product evaluation agreement allowing to test the functionality ofboth and Protect for a limited period of timem McClure testi ed that Cylance s demonstrations typically last 30-60 days and in rare exceptions extend to 9G days in With respect to why 0PM was considering their products McClure stated It had been conununicated to me through Cylance staff that had a speci c use case or otcntial problem that they wanted to test new technology that might be able to help them 5 I However 0PM delayed a decision about acquiring either product for months even after key officials knew Coulter rm Ex 14 Coulter TL Ex 22 McClure Tr 2 53 McClure Tr at IS McClure at 13 116 the agency was under attack and despite allocating resources to procure tools to secure legacy IT envirorn'nent 2 After the March 2014 data breach OCIO launched a multi phased project that included buying security tools to secure the legacy IT environment and create a new IT environmenth In June 2014 0PM made a sole source award to a contractor called Imperatis for this project and C10 Seymour was designed as the OPM of cial to manage the contract 534 The estimated cost of the initial project hases was $93 million and $18 million was allocated immediately with the one 2014 award 35 The rst phase of this contract referred to as the tactical phase was focused on purchasing security tools for the legacy IT environment to strengthen legacy systems but Cylancc does not appear to have been considered as part of this contract despite the immediate need for tools like Cylanee Separately and three months after initially viewing Cylance s products 0PM decided to purchase one Cylance product for use in its legacy system on September 2014 The agency opted to purchase V which is the product limited in scope when compared to Protect and that did not provide preventative capabilities 5815 This decision was made despite the fact that information security personnel within 0PM wanted to aequ ire Protect because they recognized its potential to detect threatsm Brendon Saulsbury a contractor in OPl'vl s IT Security Operations testi ed I believe Cylance Protect is very useful The fact that they do heuristics based analysis as opposed to signature-based was bene cial in that they are abie to detect our APT malware which was undetectable at the time by traditional signature-based antivirus Saulsbuzy testi ed he shared that impression ot Cylanee s products in 2014 long before 0PM was in crisis mode and that he communicated that belief to his 5 By the end ofJune 2 14 agency of cials received nal incident report which made clear that sophisticated attackers were working to acquire information related to the PIPS system See June Z ld 0PM Incident Report 0PM was also keenly aware of other de ciencies in its system by this time that it needed to address such as the 0PM Inspector General warning the agency in its scal year 1 13 audit that problems in its information systems constituted a material weakness See Of ce of Inspector Gen US Of ce of Pers MgmL Report No Federal tyre-mantle Security Management Act Fine Audit 1 2t 3 at ii Nov 21 2GB available at I 3-4a-ci- fl- 3-021 pdf 5 GPM Doro Breach Hearing Before the H Comm on Dversight Gov 't Re n-m 1 14th Cong June 24 2015 testimony ofDonna Seymour IChief Info Of cer U S Of ce ofPers Mgmt see In -n Chapter 3 for more on the IT Infrastructure Improvement project and contract 5 lrnperatis Letter Contract June 16 Attach at tl tltl lmperatis Production Sept 1 2015 Id at designating Seymour as the contracting of cer representative 53 UFM Date Breech Hearing before the H Comm on Oversight Gov't Reform l4lh Cong June 16 2015 testimony of Donna Seymour Chief Information Of ce Of ice of Personnel Imperatis Letter Contract glunc 16 20M Attach I at Imperalis Production Sept I 2315 3'3 McClure TL Ex 3 Wagner Tr at 91-92 533 Saulsbury at ST-63 5 Saulsbury Tr at sass ll'l' Documents and testimony show internal politics contributed to inability to swiftly purchase the too that its IT security personnel wanted to acquire specifically political challenges on the desktop at the With respect to the meaning of that term and why it would have prevented 0PM from acquiring Protect in 2014 McClure testi ed Typically in larger environments there are other people that own the desktop So security people don t own the desktop Security people make recommendations to the desktop teams You got to do this You got to do that You got to install this You got to install that And the desktop preparations people usually come from the IT side the information technology side of the house versus the security side that usually tries to come outside of the IT to be sort of the watch guard of IT and make sure that what they re doing is secure So there s always a rewall unfortunately between them virtually between the IT guys that try and own the desktop and run the desktop and the security guys who just want the thing to be secure Because priorities are around availability predominately not always con dentiality or integrity and security is all about confidentiality integrity and things like that so that becomes unfortunately a challenge between those organizations And unless they report separately all the way up to the top its always going to favor the folks that own the desktop The decision-making the way that they go about trying to nd solutions and what they deploy they control the desktop they own the desktop so ultimately they have the last word on what gets installed l McClure testified A necdotally what I have been told was that they had had challenges getting this installed on the endpoint on the desktop during that initial tiineframe in 2014 So because of that they purehase d -- they could only purchase which is just this detection product And I had been told that they were not happy with having to only buy V that they really wanted to buy PROTECT 592 McClure testified these political challenges 593 prevented 0PM from acquiring Protect and that had the product been acquired It would have prevented this attack 515M 59 McClure Tr Ex 4 591 McClure Tr at 44-45 593 5 McClure Tr at 1-5-17 594 McClure at 16-13 118 Director of IT Security Operations Jeff Wagner testi ed that political reasons were not why 0PM failed to purchase Protect Wagner stated the primary reason that OPM did not acquire Protect was because Cylance didn t currently have a FedRAMP-ccrti ed cloud 595 The Federal Risk Authorization Management Program or is a federal government program that provides a standardized approach to security assessment authorization and continuous monitoring for cloud products and servicesm A December 2011 guidance memo issued by the OMB de nes the requirements for executive departmean and agencies using in the acquisition of cloud services 5W Wagner testi ed that OPM had the capability of deploying the Protect tool We just didn t because of the issue we felt it wasn t necessarily critical at the moment It would have been a risk deploying it to a non fed ramp environment qu While Wagner acknowledged that Protect doesn t necessarily upload sensitive data or PM data or anything of that nature he testi ed that a lack of FedRamp authorization was the primary reason for not securing the tool Wagner testi ed In a perfect world we would have deployed it earlier but because we were trying not to break rules and trying to live within structures correct we didn't deploy 3 59 Wagner s assertion that the reason 0PM did not buy Cylance tools was because they were not complaint is not supported by the facts The fact is that OPM ultimately deployed and purchased CylanceProtect without being FedRamp compliant Protect was not FedRamp compliant when it was rst deployed throughout enterprise on April 201550 and it was not FedFtamp compliant when it was ultimately purchased in une 3t 2G1 5 6 In other words 0PM swiftly broke the rules once its house was already burning down but not when it was in a position to save it Further at the same time 0PM apparently declined to purchase Protect because it was not compliant OPM which was a cloud-based product and not Wagner Tr at ll-92 Wagner also said that funding contributed to the decision However the funding ultimately obligated to CylanceProtect was a mere 'action of what OPM began immediately spending to build out a new infrastructure In late October 2015 0PM reported to the Committee that it had spent an estimated million in FYEDM and FYZGI for the new IT in 'astructure project About 30 percent of the funds originated from revolving fund and the remaining 20' percent from a variety of discretionary and mandatoryr funds areas Email from US Of ce of Pers Mgmt to H Comm on Oversight 35 Gov t Reform Staff Oct 23 2015 - on le with Committee 5961 s learn more about FedRAlva visit Memorandum from Of ce of and Budget Exec Of ce of the President to llilhief Info Of ccls Seem-try Authorisation Syn- sz in Cloud Computing Environments Dec 3 rampmemoprlf Wagner Tr at ill Wagner at 144 lm McClure Tr at 23 am Telephone Interview with Stuart McClure Chief Exec Of cer Cylance Feb 18 See also Cylance Purchase Order from assurance Data Inc June 3t 2615 at CYLAINICE _il ll l$ Cylance Production Dec 2015 119 compliant at the time 0PM purchased on September 2014 and the invoice covers Cylance ln nity API which is the application programming interlace for V lCylance has both a local- gpd cloud model McClure stated the model was cloud-based and local based compliance is an important part of federal agencies efforts to ensure security and realize ef ciencies with cloud-based products In the case of 0PM however its compliance efforts were inconsistent when acquiring tools The agency did not comply with requirements when it purchased Cylance s non FedRAMP compliant Then a mere six months after 0PM declined to purchase Protect 0PM asked Cylance for another demonstration of Protect in the spring of 20 while the product was still not compliant On March 20 ZUIS 0PM executed a evaluation agreement that McClure testi ed is our internal process for managing somebody that s evaluating our software so that it doesn't stay in evaluation mode forever So since had disengaged on the Protect side the prior year at a certain point they had come back and said they wanted to retest so we re-cngagcd with them through that process In other words interest in Protect did not diminish with time despite the lack of FedRamp compliance Then after 0PM had been breached 0PM deployed Protect which again was not at the time compliant UPlv l ultimately deployed Protect in April 2015 once the agency was in crisis mode despite its lack of compliance Director of IT Security Operations eff Wagner testi ed that 0PM took this action because Protect was able to nd malware that nothing else could and he acknowledged that he would have purchased Protect earlier had he been able He stated Q So since they didn t have a FedRAMP certitied cloud that would meet all the Federal requirements we felt it would be less than Optimal to go with the PROTECT right away A Cylance was in the process of getting a cloud and we thought we d utilize the as much as we could until they got to that point I think they re still working to get certi ed however we moved to utilize the PROTECT because it was able to find malware that nothing else could Q Is it fair to say that if it was up to you you would have gotten PROTECT at the earliest convenience I A Absolutelyms The agency purchased Protect on June 30 ZUIS when it was still had not been deemed compliant- t at mum 2016 salaries application is Peasants in vacate McClure at 15 at 6 McClure at 19 20 Wagner Tr at 91-92 120 with 0MP acting as Cylance s sponsor It sponsorship path in one 2014 is not known why 0PM did not pursue a similar In sum Wagner s assertion that 0PM did not deploy Cylance's preventative tool Protect sooner because it was not compliant is lacking given actions at the time in buying other non-FedRAMP compliant products Despite Cylanee s signi cant support to 0PM in April through May 2015 following discovery of the attack 0PM was slow to execute payment for services rendered lgr or execute a purchase agreement for Protect In addition 0PM and their contractor responsible for building the new IT infrastructure was reticent to consider Cylance tools despite their proven record during the 2015 incident response period OPM's contractor imperatis which was responsible for building out the new IT infrastructure asked Cylance on May 12 2'315 to conduct a demonstration in order to be considered as a security too for the new infrastructure m mm- sermons-ww Senl Mammal 1 cc 7 Sunset Fwd stance Info- and meeting request I'or IShell A Really NW Begin torwarded message From Pam's Mummy s at 1 39 23 PM PDT Tu Matthew Moro-Inn Subject RE glance- Infn and meeting request lur 0PM Shell We can p iilbl'lf take a look although It may be a couple weeks out we haul all 0 our engineers engaged with other vendor Installs at the moment and are on a tight schedule It you could reach back out in 2 weeks we can assess where Mr bandwidth is at to support a demo In the meantime I have sent the information out to my team McClure TL Ex 1 see also Cylance Purchase Order From Assurance Data Inc June 3U 21315 at CYLANCF 3 Cyiance Production Dec 5 5m FedRamp Cylance Inc Last accessed 090216 Id Em McClure at 35 McClure testi ed that Ifl recall I think it look about 4 or 5 months to get fully paid Coulter TL Ex 23 12 The documents show Cylance employees were Surprised by the way was handling the procurement process On June 22 2015 Cylanee CEO McClure emailed a business pattner I am having flashbacks to 0PM one year ago when they couldn t pull the trigger on Protect because of political challenges on the desktop so instead only bought which is detection only So of course it didn t prevent the hack they just suffered through it only noti ed them after the fact Then we installed Protect a year later in April of this year and it detected cleaned and is preventing new attacks every day there Jeff Wagner is kicking himself that he didn t deploy us when there wasn t an imminent threat 1 0PM was also slow to ensure they could maintain access to Protect and eventually purchase this tool On June 30 2015 Cylanee warned C10 Donna Seymour that the agency would lose access to Protect that evening because the demonstration status was ending and no purchase had been made Front Sqnmur Donna ll miller Sent Tuesday June 30 1015 3 23 PM To Smart McClure Subject RE Impotlant Braiding you ylanuProtect Evaluation ti 0PM 511ml Thank yarn for contacting me I am getting some on Ins smuttiea new and someone will he in touch with you soonest Take care Donna From 5mm McClure Seal Tuesday June 30 2015 4 15 PM To Donna K Subject hip-outwit Extending your CylanceProteel Evaluation 0PM Denna In the imuest creations seetme and mderstandiug the grainy of the situation you are dealing with can We please gel on the phone today to discuss extending your ylaneePretet-I deployment which began on 4112015- The 'aluatien is scheduled to end tonight at many PST after days of deployment to over 10 350 devices when we've til-timed and blocked almost 2 000 pieces of malware inelutling the critical samples related to your breads elite were completely missed with your prim reelection Ieetumlegies Please let use latow ifwhen we can jump on a rail today'tonight Thanks Satan Met lure 5 Email from Stuart McClure Chief Exec Df eer Cylance to June 22 2015 149 am at CYLANCE Cylanee Production Jan 2016 122 McClure wrote to Seymour The evaluation is scheduled to and tonight at midnight PST after 74 days of deployment to over 10 250 devices where we ve detected and blocked almost 2 000 pieces of malware including the critical samples related to your breach which were completely missed with your prior protection Seymour responded Thank you for contacting me I am getting some intel on this situation now and someone will be in touch with you soonest 513 In July 2015 0PM nally purchased a perpetual license for Protect and access to one year of support and update services that must be renewed on an annual basis where the initial support services will expire in September 2016 The agency while now current in payments to the vendor took four-to ve months to compensate Cylance for its product and work provided The signi cance of the cutting edge preventative technology offered by Cylance in responding to the OPM data breach cannot he overstated Wagner testi ed as to why 0PM did not nd the 2015 attacker who accessed system as early as May 7 2014 prior to the Big Bang Wagner cited the fact that 0PM did not have a tool like the one lCylance provided He stated Q Is it possible that FBI DHS and the other folks that were advising you in 2014 that they were unable to detect a latent malwarc or other parts of that foothold in other directories or portions of the network A Once again the detection of malware prior to a tool like Cylance is based on what you know So it s very plausible that there would be instances in which detection would go unnoticed because you have to know what you re looking for to nd it Perhaps most importantly given documents that demonstrate the tool s effectiveness Cylance would have likely been able to find variants of the malware already on system in early 1 one 2014 and prevented irther compromise Given that the attackers did not appear to move laterally into the background investigation system until une 23 2014 if 0PM had used CylaneeProtect in early June 2014 there is a distinct possibility the ex ltration of data such as the background investigation data could have potentially prevented andfor the data losses ineun'ed in the fall and early 20i 5 could have been mitigated The Committee obtained documents that show federal agencies are facing a dilemma On June 18 2015 the Washington Post published a story in which government of cials described the challenges that agencies deal with when purchasing cyber technologies m The story stated But one challenge was a bureaucracy that made it dif cult to buy security tools quickly 3 McClure Tr Ex 20 so 1 McClure Tr at 35-36 Coulter Tr at 139 Ellen Nakashima G ieialr Chinese to S cm i l Clear-noes Dora-for One Year Wat-Ell June 13 2016 available at 123 officials said can t get through government procurement that fast said a U S official who was not authorized to speak for the record 5 The Committee obtained an internal 0PM email that shows Director of IT Security Operations Jeff Wagner was the anonymous official quoted in the story The email from Wagner to the Washington Post reporter regarding acquisition of tools following the breach identi ed in March 2014 stated The following month in March 2014 the Department of Homeland Security noti ed 0PM of the first heel of the security clearance database in May that year the agency did a remediation Big Bang Wagner said to try to make improvements to the system But one challenge was a bureaucracy that made it dif cult to buy security tools quickly he said can t get through government procurement that fast Wagner said He noted an Of ce of Inspector General audit suggested we were breaking rules by failing to have key systems certi ed Well I couldn t go any faster without breaking procurement rules ISIS The documents and testimony show IT security personnel identi ed tools they believed would make the agency s enterprise more secure and failed to purchase and deploy the most effective and cutting edge preventative technology As the record demonstrates the Cylance tools later proved invaluable after T4 days of deployment to over 10 000 devices these tools detected almost 2000 pieces of malware on system and later blocked new threats Unfortunately the most effective preventative tool Protect was not deployed until long after the attackers stolc background investigation and ngerprint data and personnel records from system The next Chapter describes the assistance another contractor provided to 0PM during the 2015 incident response period an Id Email from Press Secretary f ce of Pers to Jeff Wagner Dir Info Tech Security Operations LLS O ice ofPers Mgmt June 13 2015 3 01 at HOGR 02031000020067 0PM E'roduction Feb 115 2015 124 Chapter 5 The CyTech Story On June 10 2015 the Wolf Street Journal reported four people familiar with the investigation said the breach was actually discovered during a mid-April sales demonstration at 0PM by a Virginia company called CyTech Services Inc which has a network forensics platform called The agency on the other hand issued a press release that said the breach was discovered as a result of an aggressive effort to update its cybersecurity posture adding numerous tools and capabilities to its networks in April 2015 0PM detected a eyber-intrusion affecting its information technology systems and The Committee has investigated the seemingly conflicting statements and as is often the case the truth is somewhere in between and the story more complicated than it appears The documents and testimony do not definitiver resolve this dispute They do however support the following ndings 1 CyTeclr a service disabled veteran owned small business contractor participated in several meetings with 0PM in early 2015 to discuss the capabilities of their CyTech Forensics and Incident response tool and to provide a demonstration of their tool on April 20 I 5 at 0PM headquarters 2 During CyTech s April 21 2015 demonstration CyTech identi ed or discovered malware on the live 0PM IT environment related to the incident There is no evidence showing CyTech was aware at the time of the April 21 demonstration that on April 15 0PM had reported to an unknown Secure Sockets Layer SSL certi cate beaconing to a unknown site opmsecurityorg which was an initial indicator of compromise related to the background investigation data breachfi'21 The record con rms the agency reported this nding to newer on April 15 201 Further there is he evidence CyTech was aware that 0PM in consultation with Cylance deployed Cylance vr on April lb and then deployed on April both of which identi ed additional key malware samples related to the breach 23 3 Begirming on April 22 2t 1 5 CyTech offered and began providing signi cant incident response and forensic support to 0PM related to the 2015 incident The documents and testimony show 0PM and Cylanee recognized ability to quickly obtain forensic images CyTech provided an expert to manage the tool and continued to provide onsite support through May 1 2015 was not paid for those services 5'9 Damian Palette dz Siobhan Hughes US Spy Agencies Join Probe ofPersormei-Records Tfie WALL STREET JOURNAL June it 2915 1433935969 1 1 5 Office of Personnel Management Press Release 0PM to Motif onybei-secorior Incident June 4 EDI 5 AAR Timeline Unknown SSL Certificate April 15 2015 at Production Apr 29 2916 '53 at Email from to cart torso her 15 2m 5 6 54 pm at or-hi Production Dec 22 2915 '33 See Sigh-n Chapter 4 The Role onylance 125 4 There is no evidence showing CyTech leaked information about their involvement in responding to the OPM breach to the media In fact after the Wet Street Joni-not contacted CyTech on June 9 2015 the day before the paper reported CyTech discovered the breach CyTech immediately contacted 0PM CyTech coordinated with 0PM Director of IT Security Operations Jeff Wagner on CyTech s response to the reporter and CyTeclt s clari cation that they did not advise 0PM personnel concerning the incident a year ago Wagner responded to CyTech s proposed response to the Wall Street via entail He wrote correct away 624 5 Testimony from former 0PM Chief Information Of cer Donna Seymour to the Committee on one 24 2015 regardin the CyTech matter is inconsistent with documents and testimony from other Seymour testified that OPle purchased CyTech licenses In fact 0PM did not malte any purchases frotn CyTech Seymour also testi ed that CyTech s appliance was installed in a quarantine environment for the demonstration In fact the tool which runs against programs running in live memory was running on a live environment when it identi ed ntalware on April 22 2015- Seymour testi ed that CyTech was given some information regarding indicators of compromise prior to installing the appliance on the live IT envirorunent for tlte detnonstration In fact CyTech was not given information on indicators of compromise until after they discovered rnalware on April 22 2015 CyTe-oh Is a Small Business Contractor with Significant Cyber Tool Capabilities CyTech is a service disabled veteran owned small business The company was started in 2003 by CEO Ben Cotton Prior to starting CyTech Cotton served for ntorc than twenty years in Army Special Forces and Specialized in computer forensics Cotton told the Committee that after he retired he started CyTech to provide computer forensics e diseovcry collection sensitive site exploitation support to the US Government the intel community and SDCOM Special Operations Command as well as commercial entities 526 Over the course ofhis career Cotton has been quali ed as an ex ert witness on computer forensic matters in a number of matters at the federal and local level 32 CyTech s clients include military and intelligence entities as well as a major commercial CyTeeh offers cyber related services that include a tool referred to as CyTech Forensics and Incident response The IR tool was released for public sale in 2014f Cotton described in his testimony to the Committee He stated fundamental to is a concept we call speed to resolution which is the ability to identify rnalware or breaclt '34 Cotton Tr Ex 9 Hearing on Bren-sir Part II statement of Donna Seymour Chieflnfo Officer Of ce of Pets Cotton Tr at ti Cotton Tr at 15-2 Cotton Tr at Cotton Tr at S 126 conditions inside of a network to investigate those anomalies to isolate them and to remediate He also stated The value add to is the speed that we can perform these discovery investigative and remediation inctions specifically in the incident response and the network forensics realms We have the ability to simultaneously conduct searches and do assessments on every single end point inside of an enviromnent EnCase a competing tool due to its teclmology limitations can only search a limited subset of that and the number of end points that it can search is dependent upon basically the network infrastructure and the ability for it to pull that data front the end points back to the investigative console our search results can come back to us in as little as 45 seconds where with the other competitive tools which EnCase is one of them that typically takes days or weeks to get that information lCotton also stated that is designed to run in a live environment and it is not a dead drive forensics wolf He testi ed about the challenges of modern eyber threats He stated we need to eliminate the time constraints that are imposed by using dead drive forensics tools to investigatg ncident response And so we ve done that with We operate strictly on live systems In 2014 CyTeeh began promoting the tool through outreach to various partners and an exhibition at the 2'314 RSA Security LLC confereneei'i' 4 This outreach ultimately led to the demonstration of the tool at 0PM on April 21 2015 CyTeeh Was Invited to Conduct a Demo at 0PM In response to the 0PM cyber incident rst identi ed in March 2014 and after subsequently identifying serious vulnerabilities in the OPM network 0PM initiated the IT Infrastructure Improvement project 535 In June 2 l4 0PM awarded a sole source contract to Imperatis to serve as prime contractor for the proj ect mm As part of this contract the prime contractor was directed to identify evaluate and recommend security tools to secure legacy IT environment and design and build a secure new IT envii'omnent Cy'l ech was among the tools that lmperatis and 0PM considered as pan of this effort 635 Cotton Tr at 3 Cotton Tr at 9 533 Cotton Tr at it 633 I 4 Cotton at ti Cyt'l'tt RSA CONFERENCE snonsorstexhihitor-listfl l gtcv r last visited r'tpril 1t 2016 list of products available at 2014 RSA Conference 635 0PM Doro Breach Hearing Before the H Comm 0n Giver-sight and Gov '1 Reform 1 14th Cong June 16 EDIE statement of Donna Seymour Chief info Officer US Of ce ol'Pers Letter Contract June 20'14 Attach I at oceans lmperatis Production Sept 1 2015 A sole source contract is a contract that was awarded without being subject to the competitive bidding pro-sees See Chapter 3 The lT Infrastructure improvement Project Key Weaknesses in UPM's Contracting Approach Security ToolNendor Demonstrations Attach at 101441 42 lmperatis Production Sept 1 2015 127r Documents and testimony show 0PM had interest in the tool beginning in February 2015 and meetings were scheduled to learn more about the lmperatis coordinated two meetings for 0PM at CyTech headquarters to discuss the tool on March 22 2015 and April 2 2015 At the March 2 meeting according to Cotton Wagner s reaction to the tool was very positive and OPM requested another meeting to include additional OPM staff m At the April 2 meeting according to Cotton Wagner s reaction was again extremely positive and 0PM told CyTech they wanted CyTech to bring the appliance to 0PM for a demonstration to let them kick the tires on inside their environment 6 Wagner testified that was a potential replacement of our current EnCase capability because they were indicating that their client tool was able to take the forensic ima remotely and then transmit the image le back instead ofa piece ofthe image tits at a time After these two meetings the onsite demonstration was scheduled for April 21 2015 at OPM headquarters In preparation for the demonstration at OPM headquarters CyTech ordered and con gured a appliance Then on April 20 2015 Imperatis employee in formed Wagner that the tool was ready for the UPM team to give it a run through and that Cotton was available to be on site with demo licenses for about fifty agents On the morning of April 21 2015 Cotton solved at OPM headquarters for the demonstration 645 635 Email from Jeff Wagner Dir Info Tech Sec Operations US Of ce of Pers Mgmt to Matthew Morrison Assurance Data Inc Feb 23 2015 1 51 at HOGRDEDSIEHUUDEQE Production Feb 15 2016 639 Security Toolf'v endor Demonstrations Attach 1 at 101441-42 ltnperatis Production Sept 1 2015 Cotton Tr Ex 1 Email from Imperatis to Jonathon Tonda Contractor US Office ofI ers 3t 2015 1 51 at 0PM Production Feb 15 2015 Imperatis Weekly Report Mar 3D 21115 to Apr 3 2315 Attach ti at noose-4 Imperatis Production Sept 1 EU 15 Cotton Tr at 12-13 Email from Ilnperntis to H Comm on Oversight dc Gov t Reform Majority Slan Sept 1 2f 5 stating after the March 27 2i 1 5 meeting Wagner requested an additional follow up meeting for several members of his staff to be briefed on on le with the Committee Cotton Tr at 13 Apr 2 EDIE Meeting Acceptance by Brendan Saulsbury Senior Cyber Security Engineer Sim Mar 31 2315 at Production Feb Id 21316 Email from lmperatis to H Comm on Oversight d Cov t Reform Majority l 2015 stating OPM interested in the tool and a subsequent meeting was arranged for an onsite demonstration on file with the Committee 2 Wagner Tr at toss 3 Cotton Tr Etc 2 Appliance and Con guration Invoice for Apr 3 21315 Email from Imperatis to Jeff 1 Wagner Oi r info 'l'eelt See Operations and Jonathan Tonda Contractor 1 1 3 Of ce of Pers Mgmt Apr 21 2315 4 22 at Production Oct 23 2015 5 ortvt visitor Log Washington o c Apr 21 2015 at Hoorte2os 1 5 1300522 om Production Feb is it On September 23 2 15 0PM produced a highly redacted version of the above cited visitor log in response to a July 24 2015 request The initial version was so heavily redacted that no names were provided including 128 Wagner testified that he forgot the demonstration had been scheduled but he decided to go forward with the demonstration because we had something interesting going on it would be interesting to see what the too could The decision to conduct a demonstration in the midst of an incident response effort is interesting given the severity of the incident During a demonstration of the tool CyTech usually provides a license with a limited number of agents to be deployed For purposes of the 0PM demonstration that began on April 21 Cotton testi ed l we had a very limited license on the number of agents 6 Cotton stated Cy'I ech arranged for twenty agents to be pushed out by OPM for the demonstration Cotton stated that 0PM did not give him any speci c instructions or con gurations prior to the April 21 2015 demonstration nor was he given indicators of compromise to look for when the appliance was installed The a ency later claimed that indicators of compromise were given to CyTeeh prior to installation The documents and testimony show however that CyTech was recruited to provide assistance to 0PM and given indicators of compromise only after it had successfully identi ed malwarc in the live environment With respect to where the appliance was installed on April 2 2015 Cotton testified we left it up to OFM as to what computers or what environment we would be put in other wordsdecide where to deploy the agents Cotton stated he spent a significant amount of time waiting for permissions and access to IT facilities on April 21 By the time the appliance was installed it was late in the da and Cotton s escort had to catch a bus so the demonstration had to continue the next day 5 2 Before he left Cotton activated the tool s cyber threat assessment function which takes a snapshot of all the computers where is installed and then compares the snapshot against known good known bad and unknown processes 653 There is no evidence that shows C ech received specific information about where on the OPM network was deployed Documents and testimony do show however that on April 21 2015 the tool was deployed to a live production environment where it identi ed malwarc when results of the demonstration were examined the following day Wagner Cotton s After multiple requests and almost seven months after the initial request the Committee nally obtained a readable version ofthe 0PM visitor log in February 2016 '54 Wagner Tr at 99 Cotton at 16 43 Id 9 Cotton Tr at 14 is E's Notabiy 0PM appears to assert that an April 23 2015 email exchange supports the statement that 0PM provided the indicators of compromise to IyTech to nd the malware gig to the April CyFlIt demonstration See Email from Jonathon 'l'onda Contractor US Office of Pers to Jeff Wagner Dir Info Tech See Operations 1 53 Of ce of Fers Mgmt June 15 5 2 35 p rn with Attach Email from Brendan Saulsbury Senior Cyber Security Engineer BRA to Irnperatis Apr 23 2015 12 4 at re Production Feb 16 2016 Cotton Tr at 16 Id '55 Cotton Tr at 16 T 129 testi ed the tool was deployed in a live production environment and that the tool did identify malwarcfm In fact Production Change Request Form for the April 21 2015 demonstration was signed by Wagner that day It states that the Change Request was Urgent that the Need Justification for deploying was because Security needs to stand up and deploy to investigate incident and that the Implementation Plan was to Rack con gure and deploy products and test in production environment M555 Q to malatidm drone-stay Cva o Irivestlpiitewit - jitnalnri at P F$t1lnti it11t Q 3 Menu AK hi lth- dorm 1 2 Users J J iicleo 1th ttiplf ale-dqut- dlit11@ F I't'dlicitm environment I 35 The Change Request Form lists five areas where the 1R tool was to be deployed on April 21 201 S all live were live production servers The next day on April 22 2015 Cotton returned to 0PM to continue the demonstration 55 Upon arrival Cotton accessed the threat assessment screen and found the tool had identi tied known malware as well as a subset of unknown processes masquerading as MeAfee eaecutables according to the categorization system Cotton testi ed he put the malware found on a thumb-drive and gave it to- - who worked for Imperatis and was escorting Cotton at 0PM 553 Cotton stated that he believed- provided the information to 0PM IT Security Operations Wagner testi ed was able to nd malware within the environment and was deployed in a live enviromnent w USS-CERT con rmed Cotton s assessment that found malware on a key server in fact four of the five servers that was loaded onto April 21 2015 were implicated in the personnel and background investigation data breach w While CyTeeh s CEO was not told f Wagner Tr at 1132-103 The one Director ofIT Security added that did not nd specifically anything that we hadn t already found Id at 115 6 5 0PM Production Change Request Form for Apr 21 Eli Demonstration at Production Get 23 21315 5 ortd vieirer Leg Washington 11c Apr 22 sols al Production Feb 16 2015 Cotton Tr at l9 rd in February acid the Committee inquired with ihtper-etie employer about the status ofthis thumb drive but the thumb drive was not located Notably lmperatis stated Mr Cotton did not provide a thumb drive to with incident response data but_ was told by another CyTech employee such a thumb drive was given to the FBI Imperatis Memo to Majority Staff Feb 3 2016 on le with staff 559 Wagner Tr at l E-lfiil The Director Wagner added that it did not nd speci cally anything that we hadn't already found Id at 162- CIPM Production Change Request Fitz-rm For Apr 21 2G15 Cylilli' Demonstration at to 91 0PM Production Cat 28 2615 130 going into the demonstration that all of the malware Cylancc identi ed on April 21 2015 had been previouslyr identified with the Cylance tools it is indisputable that did identify inalware on four of the five servers it was deployed to during the April 21 2015 product demonstration The documents show 1- was installed on server on April 21 2015 6 On this server which is believed to be a workstation Cylance found the_ malwarc on April 21 2015 and discussed it via email at 12 51 emf 52 was a_ Hikit that pointed to the malicious domain identi ed malwarc on this server April 21 2015 1 53 This information was provided to US-CERT and it appeared in May 4 2015 Preliminary Digital Media Analysis Report I was installed on server on April 21 2015 151'5 011 this server CylaneeProteet also found the Tre i an on April 21 21115 and discussed it via email at 12 51 are 5 5 was a Hikit RAT Remote Administration Tool and the DLL Dynamic Link Libraries would attempt to read a configuration file in the same folder it was erteouted tam Cmih identified malwarc on this server This information was provided to US-CERT and it subsequently ap pcared in May 4 2015 Preliminary Digital Media Analysis Repor 15 I was installed on key Microsoft database server It was on this server that Cvlanee vr initially identified the malicious executables on April 16 2015 that would affirm as a malicious package on April 17 2015 669 eyreeh identified malwarc on this server - eyFiR was installed as server_ on April 21 2015 Cylaneei rotect would identify a SFXE folder on this server that was created in a us DeP'l eri tome-land seeiiriiyius ess'r Preliminary Digital Media May 4 2015 at Production Dec 22 2015 Brie ng by U S Df ce of Pers to H Comm on Oversight dc Gov't Reform Iii 201a Coultcr Tr Ex Mi 1 us Dep't of lolneiand Preliminary Digital Media Analysis me4ii5355-A May 4 2e 15 at HOGR0224-001032 Production Dec 22 2015 Briefing hy 11 5 Office ofPers 1v1gmt to 11 Comm on apersight Gev t Referm Stafl tiipr 13 20161 M Coulter Tr Ex 2 See also lBiz-utter Ex 3 its pearl of Homeland sari-riiyius esirr Malware Analysis aspen-4enss'i-A April 24 213115 at min as Production Dec 1 1 2015 3 us Dep't ofl lomeland Preliminary Digital Media May 4 2n15 at Production Dec 22 2015 Briefing by US Office oft- ers Mgmt to 1-1 IComm on Oversight d Gov t Reform Staff Apr 13 20113 Email from to Brendan Saulsbury Senior Cyber Sec Engineer SKA Apr 2015 5 19 pm at HUGR0224-000372- 75 0PM Production Dec 22 2015 1 1 3 Dep t of omelan Securitvr'US-CEKI' PreliminaryF Digital Media Analysis-4155355 May 4 2015 Production Oct 23 2016 Briefing by 1 1 5 Of ce of Pcl's Mgmt to H Comm on Oversight r'i Gov t Reform Staff Apr 18 2016 131 folder in a directory a folder that was part ol'a malicious 1 1ng package This RAR SFXZ would also be found on its atorementioned duplicate server CyTech identi ed malware on this server - was installed on sever on April 21 2015 The documents obtained by the Committee do not make reference to this server According to Cotton around lunchtime on April 22 2015 there was a brief meeting between 1Wagner and- Cotton s escort Wagner asked they found it nodded Cotton testi ed that Wagner requested an emergency purchase order for inside ofthe legacy environmentl for a license with 15 000 agents and several appliances as well as 1 000 hours for personnel support m Cotton testified that on April 22 2015 he offered incident response and forensic assistance to OPM and OPM accepted n Cotton subsequently met Iorie y with and the to describe ndings and said it was his understanding that had turned over the malware that we had imaged that morning to them Late on April 22 2015 Cyiance began working with CyTech and requested that CyTeeh pull system les to support forensic analysistTlEr Cotton testi tied that he contacted CyTeeb's senior incident response expert Juan Bonilla who was not part oftbe original demonstration and directed him to y in as early as he could to assist with the incident responseffn The documents and testimony show OPM quickly escalated the use of within the agency s environment after successfully identi ed malware For example on April 22 2015 at 3 53 was loaded on server 5n This server provided access to the PIPS mainframe On April 23 2015 was loaded on its duplicate server_ mgr was put on servers and on April 2015 and the images extracted from these two servers were supplied to appeared in May 4 2015 Preliminary Digital Media Analysis Reportfgu These Cotton at 20 to an Cotton Tr at 39-41 5 Cotton Tr at at CyTech Dclnonstraticni esuhs Participants at Production Sept 25 2015 showing CyTech demonstrations results participants included 1 13 OPM OPM contractors Imperatis and Cyteeh Email from Chris Coultcr Managing Din Cylanee to Ben Cotton Chief Exec Of cer Cy'I'ech Apr 22 2015 7 101 p llt at HOGR0203 lit 000008 Production Feb 16 2016 Cotton at 25 Cotton noted that Cy'I'ech s expert Bonilla as a senior member of the CyTeeli team is typically billed at between $450 and $350 an hour Id 6 Dep't ofHomeland Securityi US CERT Preliminary Digital Media Analysis-4 5355 May 4 2015 Production Oct 23 2015 Briefing by US Of ce of Pet's Mgn'it to H Comm on Oversight 65 Gov't Reform StaffijApr 13 20M Id at 132 servers are also critical because it provided access to the PIPS mainframe reports show was placed on an additional key server and its duplicate on Apri123 at 2 27 p m m This sever is a critical jump box that provided access to the portion of environment segments where the PIPS mainframe resides 632 While Cytance was installed on these servers at 6 21 p rn on April 2015 was assisting with forensic work Documents show 0PM after reviewing the results of the CyTech demonstration deployed to key servers that gave access to critical parts of environment including one of the most important and sensitive servers that gave access to the PIPS mainframe where sensitive background investigation data was stored This suggests 0PM believed CyTech could assist the agency in the incident response situation By April 24 2015 and in response to Wagner s verbal request for services Cy'l'ech submitted a quote to 0PM through lmperatis m CyTech quoted $818 000 for a perpetual license with 15 000 agents g l The documents show there was a serious effort to nalize verbal request for services and that the in the April 22 meeting understood intent Sometime the week oprril 27 Imperatis reported coordinating equipment installation and con guration with security vendors including i woi'kirtg to nalize BUM bill oi'materials for In an interview with the Conunittcc Wagner testi ed that he did not say 0PM would buy but acknowledged that he likely asked for a quote mm CyTech relied on the request for services that exceeded the scope of a typical demonstration and expanded the services it provided to 0PM during the 2015 incident response period Consequently on April 22 2015 CyTech provided a license to 0PM for 1 000 endpoints that expired on June 30 2015 Cotton testi ed that CyTech provided incident response and forensic assistance to 0PM out of a sense of duty and with the expectation that there would be a contractual arrangement put into place 633 Cotton stated there was a promise of a contract but execution was delayed With respect to why CyTeeh provided these services without a contract in place Cotton testified us Dep't ofHonteland caauriiyms-cssr Preliminary Digital Media Analysis 465355 May 4 2015 oars Production Del 23 2016 Brie ng by LLS Of ce oi'Pers Mgn rt to H Comm on Oversight 8 Gov t Reform Staff Apr 18 201 1 3 Saulsbury Tr in 75 76 633 Cotton Tr Ex 3 4 CyTech Price Quote $313 000 for Emergency Purchase Order Apr 24 2015 and CyTeeh Egansrnittal email to lmperatis for Ccheh lQuote Apr 24 2015 Id 535 Imperatis Weekly Report Apr 201 S-May l 20l5 Attach 6 at 000253 Imperatis Production Sept 1 2015 5 35 Wagner at 104 Cotton Tr at 25 see also Email from Ben Cotton Chief Exec Of cer CyTech to H Comm on Oversight Gov t Reform Majority Staff Apr 16 2016 continuing the nature of the licensing arrangement as of April 22 2015 on file with the Committee Cotton at 41 539 Cotton Tr 11 10 133 Typically there is a contract in place it s also atypical that we are doing a demonstration and we ind live malware on the end points of a government agency that quite frankly controls my security clearance I knew immediately once it was determined that this was malware what the implications could be for the country So you know maybe For a bad businessman maybe I'm too much of a patriot at this point but didn t want to leave them in the torch and didn t want to let this breach go without a capability that would help minimize this to Just days before 0PM denied Cy'l'ech s role in the response to the media 0PM personnel and imperatis shared internally the clear expectation that 0PM would he compensating CyTech l'or and incident response and forensic support based on the conversations CyTech had with 0PM in mid-April 2o 5 On one 5 2 015 Imperatis inquired about the status of the CyTech quote An lmperatis employee asked an 0PM of cial do you want for the existing network I assume yes to compliment your Encase tool sting Message Sent 6153' 11-5 8 45 0 PM 10' Wagner HUM-NET rilit IItP Subject Mir 353 Q5 of Icitfion ir Had a conversation with the Cytech team today who were following up on a tow item-3 I told th I lcnow you are in the thick of it right now Wanted to get some clarification and direction with regarcl J rensics and r hell we had some time belore we were ire-caring forensics You may have a higher immediate need for l jot would trump our trrneline Can l rou answer some of these below E Thanks Patrick The Of the leaner appliance Do you want them to pick up the auction ts it currently supporting an active investigation on 1you want to possibly leave it ln place assuming a ming procurement with CyFir I was under the impresslon the licenses or it have aspired Do you want CyFir tor the network I assume yes to C mplirggf our name tool It so how quickly do you need it and do you iorcsec that toning procured off our contract yours and scoped to support both sides I can't recall won the current BUM wlwre the E- ippiiancos we for somehow we got to that number but lrlun'l recall thejostit'ication HA config or physical Inca for them I need to no cure Ilse-re is ennugh im- Shetland Existing Q- Mulveney 36 Cotton Tr at ate-41 59' Email from Patrick Mulvaney Imperatis to Jeff Wagner Dir Info Tech Security J ifvperations 1 1 5 Of ce of Pers Mgmt June 5 2015 8 45 at Production Get 23 2015 i 134 The CyTeoh Demo Turned into Incident Response and Forensic Support In mid-April through May 2015 signi cant incident response and forensic support activity was underway at 0PM Documents and testimony show Cy l eeh was part of that effort Other contractors that Were onsite confirmed CyTech s role Cylance was one such contractor A Cylance of cial testi ed CyTech was providing assistance onsite with a tool that can make it easier to obtain evidence and that having that tool actual r was useful It sped up the initial triage process of trying to obtain critical forensic Another contractor who staffed the OPM IT Security Operations group said OPM made a decision to have the product assist with gathering forensic images of some of the servers that requested the image 693 Yet another 0PM contractor reported that 1R forensics tool was installed in legacy environment through operational testing and has proven to be extremely bene cial in the reduction of man hours required with an active security issue 94 The Committee obtained documents and testimony that show CyTech provided speci c incident response and forensic support activities to 0PM On April 23 2015 after the demonstration Cotton returned to 0PM to provide assistance Cotton also brought a CyTech expert Juan Bonilla whose services are billed at $350 to $450 an hour to assist 0PM with the tootw BOnill remained onsite at 0PM through May 1 2015 55 Documents show that it was an incident response and forensic support environment at that time The FBI and CERT were also onsite on April 23 2015 and returned for several days thereafter- 693 In testimony to the Committee and in public statements 0PM officials downplayed CyTech s role in the incident response and forensic support operation in April-May 2015 For example Wagner testi ed Bonilla wasn t really part of the investigation m In an email from April 28 2015 however Wagner noti ed 0PM IT administrators that Bonilla would be 3 Coultcr Tr at 53 59 Saulsbuty Tr at 34 94 Imperatis Weekly Report Apr 2C1 Ems-Apr 24 2m 5 Attach 6 at lmperatis Production Sept 1 2015 DFM Washington no Apr 23 2015 at trooaoscata-oocsso Production Feb ta 2cm 5% Id Cotton Tr at 25 E's Cotton Tr at 26 Email from Iuan Bur-ills Senior Sec Consultant Cy'l'ech to Jonathan Tonda Contractor and Jeff Wagner Dir Info Tech Sec Operations U S Of ce of Pers May 1 it 5 12 43 at IIOGRUEOSI 0PM Production Feb 16 2016 showing Bonilla coordinating collection of images with 0PM prior to May 1 departure Email from Juan Bonilla Senior Sec Consultant Cy fech to Jonathan Tonda Contractor U S Office of Poor Mgmt May 1 EDIE 5 09 at ltd itititi Production Feb 16 l ld indicating Bonilla left credentials for use UPM 1visitor Log Washington D C Apr 23 21315 at 0PM Production Feb 16 2616 Wagner Tr at 1131 135 assisting with an investigation over the next two weeks and asked what needed to be done to obtain system access for himm Wagner also testi ed Bonilla and Couiter worked together during the incident response Wagner stated we threw everybody into a giant room and Juan Bonilla was the CyTeeh engineer much like Coultel' was the Cylanee engineer Clearly Cylance had a signi cant role in incident response and the comparison between CyTech and Cylance personnel onsite suggests at the very least CyTech played a suppoiting role in incident reaponse that 0PM has not publicly acknowledged In terms of other speci c CyTech activities Cotton testi ed CyTeeh was initially asked to image all the random access memory from approximately lly computers image the hard drives for those computers and pull event logs for CyTech also worked with Cylance to fulfill their requests for les For example on April 24 2015 Cylance asked CyTech to pull a bat lem Cotton testified that bat les are commonly used as part ofa breach to autontate the infestation or the installation ofmalwarefnm 5 object - Ftla Fl 37411851 Date a '13 9015m154 PM To Cc WI tar Jotter fonds Jonathan Ben I I Would you be able pull this file want to verify something Bonilla worked with 0PM to deploy and coordinated with 0PM staff to address connectivity issuesm Documents show that as of April 28 2015 Wagner prioritized deployment to at least thirty eight serversmr Documents show CyTeeh collected thousands of images in its forensic support role Indeed the documents show the appliance was literally running out of memory space to retain all of these images On April 29 2015 Eoniila requested information from about a Email from left Wagner Dir info Tech Sec IDperations LLS Jilice ot'Pers tolames Anderson U S lElt tice ochrs Mgmt Apr 23 2015 5 43 at 0PM Production Mar 1e EDI-5 ml Wagner Tr at Cotton Tr at 2123 Email from Chris Coulter Managing Din Gylance to Ben Cotton Chief Exec Of cer CyTeclt Apr 24 2tl15 5 54 pm at Production Feb 16 2516 Cotton Tr at 29 Emails between Juan Bonilla Senior See Consoltant CyTeclt and Brendan Saulshury Senior Cyber Security Engineer SEA Apr 2015 at Production Feb 16 21 116 705 Message from_ Contractor U S Of ce of Pete Mgmt to Jonathan Tonda Contractor U S Of ce of Pets Mgmt Apr ES 2015 9 134 pan at Production l- eh 1e 13a list of images that needed to be retained because the appliance only had fourteen terabytes of storage space and was quickly nearing capacity Cotton testi ed that 0PM asked CyTech to collect all this information and we were running out of storage for that THIS on Apr 29 ems at and PM Juan canine wrote All storage - is rap- le reaching 12h 11 5TB out or MTB have asked the customer to complle a ilst ol images that can be deleted from but I have not received a reply yet With the FBI lully involved 5 agents onsitel in this case and based or the conversations the have shared i think we head to plan on getting extra Storage for as the cu'stomer most likely doest not have and extra ISTB oating around for storage has been oushrnu agents and as at tins writing we have 55 agents checking in with CYHR server from 23 we had a Hooch today Th'FSJ'ost means more work and that is always welcome but i used to be able to at least deliver what the customer needs Full Forensic Images selected hmeitne Files and most importantly memory dumps thoughts Juan tortilla Sr Security Consultant 9720 Capital Court Suite 200 I Harnesses VA 201 v v v v It is worth noting during what would turned out to he most damaging data breach in the history of the federal government 0PM was making decisions about what forensic evidence to retain without it appears consulting the DIG or counsel in a meaningful way In late April 2015 CyTech and Cylance continued to assist 0PM On April 29 2015 Cylance and CyTech updated 0PM on the status of Cylance s analysis Coulter testi ed that there were three teams working on incident response with 0PM Cylance CnyR and law enforcement With respect to role Coulter stated as Cylance through CylanceProtect was identifying new instances of rnalware that were related we would then rgguest to install an agent on that machine to then collect the data for further analysis An April 29 2015 email from Coulter stated that would install agents on the scoped hosts and collect data for the other team and suggested a formal meeting with the other team members to close outfm Email from Juan Bonilla Senior Sec Consultant CyTech to Brendan Saulshury Senior Cyber Security Engineer SEA Apr 29 2015 5 26 pm at Production Feb 16 201d Cotton Tr at 31 Cotton Lin 5 showing internal discussion about storage options and how such costs may he covered under a contract Test Message from Jeffrey Wagner Dir Info Tech Sec Operations US Of ce ochrs Mgmt to Jonathan Tonda Contractor LLB Of ce of Pets Apr 30 201 5 at Production Feb 16 2016 showing internal 0PM discussion on options for to dump images Coulter Tr at Email from lChris Coniter Managing Din Cylance to Jonathan Tonda Contractor US Office of Pete Mgml and Jeff Wagner Dir Info Tech Sec Operations LLS f ce ofPers Mgmt Apr 29 2015 4 40 pm at Production Feb 16 20145 137 In sum CyTech was onsite at 0PM from April 21 to May I 2015 During that time CyTech identi ed malware and provided incident response and Forensic support to 0PM that exceeded the scope of the product demonstration that began on April 21 Wagner testified that once Bouilla left the site we never utilized Cy'l ech s product again 7 Documents suggest otherwise A cr Bonilla left 0PM on May 1 2015 CyTech continued to provide assistance on an as needed basis On May 3 2t115 Bonilla emailed Wagner to follow up on the work he did the week before and offered to ptevide additional assistance with the tool The documents show DPM continued to use the tool from May 2015 through early June For example on May 7 2015 Cylance requested be deployed to a particular one hoot T13 on May as 2015 an 0PM contractor stated that had collected images from a key production server On lttne l 2015 an 0PM contractor wrote all other securityr agents are currently running CylanIcJe Forescout m Documents show the forensic capabilities of the tool were a continuing topic ot discussion For example hnperatis the OPM contractor who introduced CyTech to 0PM described a May 15 2015 Forensics capabilities meeting with Documents show there were continuing interactions with CyTech and use of the tool through June 2015 7 Wagner minimized the scope of the deployment in his testimony to the Committee He stated we only deployed their client to a select number of ntacltines 13 Documents show however CyI- llt s deployment was fairly extensive The Committee obtained documents that show the 1R tool was tested on more than sixty different servers including key servers connected to the personnel records and background investigation data that was Wagner Tr at IDS Email from Jean Bonilla Senior Sec Consultant CyTech to Jeff Wagner Dir Info Tech Sec U S Of ce of Pcrs Mgml May 8 5 49 at Production Feb 16 21315 3 Email from Chris Coulter Managing Din Cylance to Jonathan Tonda Contractor U S Of ce of Pers Mgmt May 2015 3 56 pm at Production Feb to 20113 Email from Jonathan Tonda Contractor US Of ce of Pcrs Mgmt to Brendan Saulsbury Senior Cyber Security Engineer May 23 2015 1 43 pm at Production Feb 16 2315 Email from Contractor LLS Office of Pers Mgmt to LLS Office of Pet's Mgmt Employees June 1 2015 3 28 a1 l tltl iit ifi 0PM Production Feb 16 201d lmperatis Weekly Report May 13 22 EDIE Attache at Inipctatis Production Sept 1 2015 Email from LLB Of ce ot'Pcrs Mgmt to Jonathan Tonda Contractor U S Off of Pet's Mgmt gJune 2 2015 12 01 pm at Production Feb 15 '3 Wagner Tr at 151 List of locations on which was tested at 32 Production Sept 25 2m Initially this document was provided with reductions that did not allow a cross reference with key 138 Documents show the tool was deployed on the OPM system through one 2015 and that it was not fully uninstalled until August 2015 On June 25 2015 an 0PM IT of cial contacted Bonilla for instructions on how to uninstall the Cyfir software installed a month ago from a list of more than forty servers including several servers involved in the background investigation data breachm This request for instructions to uninstall occurred the day after former C10 Donna Seymour and Director Katherine Archuleta testified before the Committee about Cy l'ech s involvement in the discovery of the data breach Seymour and Archuleta testi ed that CyTech was not involved in the discovery of the data breach and they did not disclose the involvement of Cylanee who like CyTech also did not have a contract in place when leadership was testifying before the Committeem Be gin Dru-'arded message Subject Uninstall Cy r Date lune 25 2015 at I l2 24 PM EDT To Juan 1 ant to uninstall the Cy r software 1 installed a month ago for the l'ollowinlbJ servers is there a special process to remove them' don't see the Cylir software listed in the add and remove pro gram Feature Please let me knt tw Thanks Server list lCant ping servers involved in the breach with where the tool was deployed in response to the Committee s February 3 31216 subpoena 0PM provided an unredacted version ofthis list on April 15 20th Email from Contractor US Of ce of Pers Mgmt to Juan Bonilla Senior Sec Consultant Tech June 25 Cotton Tr Ea ti Wagner Tr at 32 33 Hearing rm 0PM Doro Breach For statement ofDonna Seymour Shief Information Of cer Office of Personnel Management statement ofKatherinc Arehuleta Din Office of Pers MgmL 139 - cumming Idp think ll1 nih l Wurk all in In f nnt ping I'dp Sy xlums Mlluinintl'ulm f'1lT1uL- nl' Purxurmul Hauugcmunl R'L'lu'url u l Surwr Hpcrul mm Hiram-I Hashing-In 311-115 SHE-L Inlm'nnliunul In shew 0PM did um nish unlil Auguat 21 5 The Cummilluc ublaincd inlurnal agunuy emails 111mm the uninstall cl fnrt hcg n am June 26 Hill 5 140 and was partially complete by June 29 2'3'15 722 As of August 13 2015 0PM determined that as many as twenty four devices were still communicating with the server m The documents show CyTech provided significant incident response and forensic support from April 23 through May 1 2 115 CyTech continued to provide services as needed after CyTech personnel were no longer on site at 0PM Further 0PM deployed the tool beginning in Apri12015 and did not fully uninstall it until August 2015 24 The documents also show the tool was still installed and communicating with the server as late as August2015 CyTech relied on request for assistance on April 22 2015 and provided incident response and forensic support services Then CyTech became the unwilling focus of media attention The Wall Street Journal Reports on CyTeeh's Role in the OPM Incident on June 10 2015 Pieces of the CyTech story became public when the Wall Street Journal published a story under the headline Spy Agencies Join Probe of Personnel-Records Theft on June 10 212115 725 The story stated Last week the Of ce of Personnel Management disclosed that hackers had breached its networks warning that the personnel records of roughly four million people many of them current or former government have been stolen At the time 0PM said the breach was discovered as the agency has undertaken an aggressive effort to update its cyberSecurity posture adding numerous tools and capabilities to its networks But four people familiar with the investigation said the breach was actually discovered during a mid-April sales demonstration at 0PM by a Virginia company called CyTeeh Services which has a networks forensics platform called CyTech trying to show 0PM how its cybersecurity product worked ran a diagnostics study on network and discovered malware was embedded on the network Investigators believe the hackers had been in the network for a year or more An 0PM spokesman didn t respond to a request for commentm 733 Email from Administrator 11 5 Of ce of Pers Mgmt to Jonathan 'l'onda Contractor U S Df cc ochrs Mgmt Aug 19 2015 11 34 am at Production Oct 23 733 Email From Administrator US Office of Pers Mgmt to Brendan Saulsbury Senior Cyber Security Engineer SKA and Jonathan Tonday Contractor U S Of ce of Pars Aug 18 2D15 11 32 am at 01 10125 Production Oct 23 Cotton Tr at til Damian Palette 8 Siobhan Hughes 13 5311 Agate-fax Join Probe anersomrel Reeorrh- Tiler WALL STREET JOURNAL June 10 2015 available at records-theft-1433936969 I41 The Committee obtained communications between 0PM and CyTech related to the media inquiry The documents show that before the article was published CyTech coordinated with 0PM There is no evidence to suggest CyTech was the source of the story Cotton testi ed We did not intend to find ourselves in the middle of these hearings And I am just very concerned about the representations that may or may not have been made around this Hill that have actually been relayed to me that 0PM is maligning my company s reputation and our capabilities m On June 9 2015 Cotton received a call from a reporter regarding CyTeeh's role in the discovering the 0PM data breach 23 The reporter told Cotton he had four sources saying that CyTeeh discovered the OPM breach and that CyTech had been advising 0PM about this matter for the last yestm The reporter requested a commenth Cotton said the reporter could email him about the story but that he would not comment 3 Cotton wanted something in writing to confirm the identity of the person on the call 132 Late on June 9 2015 Cotton reviewed the email from the reporter and immediately forwarded it to Wagner for guidance Cotton asked whether he wanted CyTech to make corrections 34 Wagner said Correct away ust give me a heads up as to the response so we can discuss 735 Cotton proposed a response to the reporter is CyTech policy to not discuss clients or operational matters with the press Cy'l eeh can categorically deny that personnel from CyTech advised 0PM personnel concerning this matter a year ago na Wagner responded early the next day and suggested what amounted to a no comment response Wagner wrote if you need anything feel free to fire back 1Keep the faith Damian Palette dc Siobhan Hughes US Spy Agencies Join Probe ofPersnirtieLRa-eords The WALL IDURNAL June 10 3015 1 433 93 I596 9 Cotton Tr at IGT Cotton Tr132 Cotton Tr at 64-65 4 Cotton Tn Ex 9 Email from Ben Cotton Chief Exec or car CyTech to Jet f Wagner Dir Info Tech See Sperations US Office of Pets Mgmt June 9 am Id 736 Email From Jeff Wagner Dir Info Tech Sec Operations U S Office of Pets lvigmt to Ben Cotton Chief Exec Ititt ticcr Ccheh June It 2015 am at 2 4 Cy'I'ech Production Aug 19 2015 142 On June 10 2015 the story was published It stated Fjour people Familiar with the investigation said the breach was actually discovered during a mid-April sales demonstration at 0PM by a Virginia company called CyTeeh Services which has a network forensics platform called Wagner testi ed that this portion of the story was not accurate in any The story Further stated CyTech trying to show 0PM how its cybersecurity product worked ran a diagnostics study on network and discovered malwarc was embedded on the networkfm Coulter the Cyiance engineer onsite at the time of the CyTech demonstrationf testified with respect to that portion of the story that s actually accurate They did They ran a diagnostic study They may have discovered malware that was embedded on the network but it was likely already known at that point N2 On unc 12 2015 Wagner emailed CyTech about the story Wagner wrote i cannot express how bad this is going down for you We should talk about this Cal my Cotton quickly responded just tried to call THE LEAKS ARE NOT emphasis in the original 4 ln teaponse Wagner suggested a call with public affairs of ce to work something that will benefit both organizations 45 Cotton agreed to discuss the situation From Hem Cotton Still Friday June 12 2015 910 AM TO Wagner J 'l'rey Subject Re talking to prions and muting rlalrns about Jc i Junl ll iutl ttI Edi ii Lli LEAK- 5 lit-1t i en Hen L'oltmr I Cytoch Sort-ten Damian Paletta 3r Siobhan Hughes US Spy Agencies Join Probe ofPei'soiitief Reeoro s The WALL STREET JOURNAL June ID 2315 available at reeords-tlte - 4 339 3 959 Wagner Tr at 156 Damian Palclta r9 Siobhan Hughes US Spy Agencies Join Probe ofPersormei-Reeoi'tir Theft WALL June 10 2013 14339315969 DPM 1trinitor Logs Washington DC aprii 21 22 201d at noonozcsia ocosst 524 Production Feb 16 2m 6 Coultcr Tr at til Ex 9 3 Cotton Tn Eii to Email from non Cotton CitioiEnno orator CyTech to no Wagner on r into Tech Sec gperations LLS Of ce of I ers June 121 d its M Cotton Tr at on no lo 143 In describing phone conversations with CyTeeh to the Committee Wagner testi ed he had two calls with Cotton on or about June during which the CEO acted shocked assured me it was not him or his company who had leaked the storym Cotton testi ed he was surprised by reaction on the rst call and learned 0PM was concerned about the story because the account in the Wall Street Journal was inconsistent as to how 0PM leadership had already testi ed to Congress 743 Wagner testi ed that during the second call with public affairs staff Cotton again said CyTech was not the source of the story but he believed Cotton was tellin the Wall Street Journal that CyTech did in fact have some role in the discovery of the breach 9 Cotton on the other hand testi ed that 0PM wanted CyTech to sign on to a joint statement that in essence it was that Wall Street Journal was totally without basis Without fact and was a lie TSU Cotton also testi ed he requested a written draft ofUPM s suggested statement but 0PM declined and ultimately CyTech did not agree to their approach because it was not what actually occurred 5 Cotton testi ed that he explained the whole situation to public affairs staff including the April 2 2015 product demonstration and Cy l ech s role in incident response and forensic sol-morn 52 Cotton testi ed that press spokesman seemed surprised and said he would he in touch but CyTech did not hear from 0PM againm After multiple press inquiries following the story CyTech issued a press release on June 15 2015 The press release stated It is CyTech s policy not to discuss our clients or their sensitive operations However due to extensive media reporting we wanted to clarify involvement and the assistance we provided in relation to breach response in April 2015 CyTeeh was initially invited to 0PM to demonstrate Enterprise on April 21 2015 Using our endpoint vulnerability assessment methodology quickly identi ed a set of unknown processes running on a limited set of endpoints This information was immediately provided to the 0PM security staff and was ultimately revealed to he malware CyTeeh is unaware if the OPM security staff had previously identi ed these processes CyTeeh Services remained on site to assist with the breach response provided immediate assistance and performed incident response supporting 0PM until May I 2015 Wagner Tr at 153 Cotton r at so Wagner Tr at 15d Cotton Tr at 58 TSI rs Cotton at ease Cotton Tr Ex 14 CyTech Press Release CvToel Services Con rms Assistance to 0PM Breach Response June 15 CyTech did produce a draft press release dated I one 1t 2D 5 to the Committee that the Cy'l'ech CED quickly identi ed as a draft document when questioned about it This draft press release did not precisely describe CyTeeh s involvement The lC yTeeh CED explained that he revised this draft to the version released one 15 since this was a public statement against a very large and very powerful government organization i needed to 144 The Wet Street Journnt covered CyTech s public statement in a follow up article on June 15 3315 55 In the story an 0PM of cial stated the assertion that Cytech was somehow responsible for the discovery of the intrusion into network during a demonstration is Cotton testi ed that when he heard statement he was concerned because the dispute was starting to impact our corporate reputation and our capabilities and he speculated that 0PM was parsing words by using the term discovery of the breaching Cotton testi ed that the challenge we had here was clearly you don t want to get into a ght with in the news with one of our clients But at the same time to say we had no part in the discovery was clearly false f 3 Cotton testi ed that discovery ofthe breach is not precisely de ned and that in his mind CyTech had discovered malware on the system rill Cotton stated it was possible that had somebody noticed a packet going out to an unknown Web site that they could then say well we discovered that because we saw this The documents show the statement issued by CyTech on June 15 2015 is consistent with the facts The documents show CyTech did play a role in identifying malware in the live 0PM IT environment and providing incident response and forensic suppoit to 0PM beginning in mid April 2015 The documents show CyTech did not publicly claim to have discovered the intrusion but rather that it played a role in identifying malware The agency s strong reaction to the June 10 2015 stoty in the Watt SrreetJournoi was based on a concern that it contradicted statements senior of cials made to Congress about the data breach it is troubling that Cy l ech appears to have in good faith worked to coordinate with 0PM on responses to the press while 0PM worked to kill this cytech 0PM press of cials also detnandcd that the WSJ print a retraction of the CyTech story on June 10 the day the story be very precise about what my company did and what we didn't do to avoid any entanglements with de nitions over breach discovery Cotton Tr at 34-35 5 Damian Palette Cybersecrtrity Finn Says It Found Spvware or Government Nam-tori in Apt-ii WALL ST June 15 1015 available at 1434359994 Id Cotton Tr at it its Cotton Tr at 7'1 its Cotton at as Email from Sam Schumach Press See US Off of Pers Mgmt to Jeff Wagner Dir Life Tech Sec Upcrations 1 1 5 Office or Pers Mgmt and Donna Seymour Chief Info Of cer LLB- Df ce ot'Pcrs Mgmt- June 13 2015 1 25 pm at 0PM Production Feb 16 2016 appears to have become frustrated with the CyTech story 111 a June 23 2915 email the 0PM Dir of Communications was coordinating a response to the WSJ on a cybersecurity issue and said to Mr Wagner do you have time to get on the phone with the reporter for 10 minutes I want to make sure he s not trying to resurrect the CyTech Dracula here in a subtle way Email from Jackie Kosecauk Dir otComm LLB Of ce of Pers Night to Jeff Wagner Dir Info Teelt Sec Operations US Of ce of Pcrs Mgmt- June 23 E l 5 113 0 pm at Production Feb 16 2016 145 was published without apparently verifying all the facts surrounding the story and Cchelfs role in incident response and forensic support 0PM Description of CyTeeh s Role- Was Misleading Testimony and public statements by 0PM officials regarding Cy l eeh s role in the data breach incident response and forensic support activities from April to May 5 were confusing and misleading 0PM was also slow to respond to document production requests regarding this issue Further compounding the confusion 1When 0PM produced documents in early 2016 and as the investigation proceeded the CyTech narrative became clear However when the CyTeeh story was first reported in June 2015 the details were less than clear and further confused by senior 0PM officials testimony In June 2015 the CyTech story was the subject of various press reports including the June It 2015 story in the Wall Street Journal On June 16 2015 former 0PM Director Katherine Archuleta testi ed before the Committee that detected the intrusion and denied that contractors did so 754 Alchuleta omitted the fact that Cylancc and CyTech played critical roles in identifying the actual malware and providing forensic support On June 23 2015 the House Permanent Select Committee on Intelligence HPSCI referred evidence to the Committee obtained from CyTeeh m 11 light of the press developments and the information from IIPSCI Rep Turner questioned Se nour and Arehuleta about CyTeeh when they appeared before the Committee on June 24 2015 5'5 Rep Mike Turner questions and Seymour or June 23 MES hearing Email Jackie Dir ot'Comm U S Of ce of Pers ight to Damian Paletta Reporter Wall St 1 June It Elli 5 7 15 pm at Production Feb 16 2016 The WSJ declined to print a retraction solely on the basis of the agency s assertion that it is inaccurate Email from Robert Durlian News Editor Wall St 1 to Jackie Koseesult Dir of Comm U S ltiltl ice of Pets Mgmt June ID 21 5 9 26 pm at HUGRDEGS ld t't 63 Production Feb 16 2016 0PM Doro Breach Hearing Before the H Comm on Jr'ersight rt iov t Reform 14Lh Cong June 16 2015 statement oFKatherine Arehulcla Din US Office of'l'ers Mgmt The House Permanent Select Committee on Intelligence also referred information related to the Cy'l'eeh matter to the Committee Letter from the llon Devin Hones Chairman and the Hon Adam Schil'f Ranking Member H Perm Select Comm on Intelligence to the Hon Jason Chall'etz Chairman and the lion Elijah Cummings Ranking Member H out Cl'versighl d Gov t Reform June 23 20 5'5 Hearing on 0PM Hum Branch F'an H 146 Rep Turner asked Archuleta and Seymour was CyTech involved in the discovery of this data breach Both witnesses responded no CyTeeh was not involved Documents and testimony do show 0PM identi ed and reported to on April 15 2015 that an unknown Secure Sockets Layer SSL certi cate was beaconing to a site opmseeurityerg not associated with 0PM of cials left out the fact that Cylance and CyTeeh also identi ed malware related to the data breach In the case of agents were deployed on April 2 l 2015 to several production servers where images were collected and transmitted to Subsequent analysis showed the presence of malicious les related to the data breachm Rep Turner also asked Archuleta and Seymour whether Cyteeh was ever brought in to run a scan on equipment m Seymour testi ed that CyTeeh was engaged with and added that 0PM was looking at using CyTech s tool on the OPM networkm She stated her understanding was that 0PM gave them some information to demonstrate whether their tool would find information on network and that in doing so they did indeed nd these indicators on OPM's network m She testi ed Seymour W e had purchased licenses for CyTech s tool We wanted to see if that tool set would also discover what we had already discovered So yes they put their tools on our network and yes they found that information as well Turner So you were tricking them You like already knew this but you brought than in and said Shazam you caught it too That seems highly unlikely don t you think Seymour We do a lot of research before we decide on what tools we are going to buy for our network Turner At that point you hadn t removed the system from your system I mean you knew it was there you brought them in and their system discovered it too which means it would have been continuously running and that personnel information would have been still at risk- Correct Seymour No Sir We had latent malware on our system that we were watching that we had quarantined hi it was Time-line Unknoivn Certi cate apriI 15 2015 at Production Apr as sole US Dcp t of Homeland SecuritthS-CERT Preliminary Digital Media May 4 2015 at 0PM Production Dec 22 2015 Brie ng by US Of ce of Pers Mgmt to H Comm on Uversight d Gov't Reform Eitaflr Apr 13 3316 m Hearing on 0PM Dora 11' reach Part II M 14 Turner You had quarantined it So it was no longer operating Seymour That is correctm Seymour- s testimony raised several questions First documents show 0PM had not purchased licenses or anything else from CyTech despite a verbal request for an emergency purchase order TM Second testimony obtained by the Committee shows CyTech was not given the indicators ofcompromise prior to running on network on April 21 21115 Documents obtained from 0PM suggest indicators of compromise were shared with an 0PM contractor lmperatis - on April 23 2015 days after the April 21 CyTech demonstration A11 Imperatis employee escorted Cotton when he was nnsite at 0PM but there is no evidence showing he provided Cotton or CyTech with indicators of compromise prior to the April 21 demonstration Third Seymour s claim that the tool identi ed latent malware on systems that had been quarantined is not accurate Wagner testi ed the 1R tool was deployed in a live production environment Documents show 0PM prioritized deployment of the CyFir tool to servers in the OPM production In fact the IR tool is designed to run in a live environment and runs against programs running in live memory Seymour s ciaim that the malware in the UPM system had been quarantined is not accurate Cotton testi ed there was no quarantine in place when I found the malware live on the system on the morning of the 22nd 779 The agency did not move the primary tool used to identify malware enterprise wide CylaneeProtect from alert to auto quarantine mode until April 24 The tool did in fact identify malware and contrary to Seymour s testimony the tool did so in alive Data on CyTech s Appliance Collected During the 2015 Incident Response Period was Deleted After two hearings in June 2015 the Committee requested additional information and documents from 0PM related to the data breach incident announced in 2015 including specific Hearing on 0PM Darn Breach For Statement of Donna Seymour Chief info Of cer U S Office aners Mgmt Wagner Tr at 1113 Cotton Tr at 14 15 Email from Brendan Sanlshury Senior Cyber Security Engineer SEA to lmperalis April 23 2015 12 47 pm at HDGRU2G316-11110254 0PM Production Feb 145 21116 escorted Cotton for the April 21 demonstration Wagner Tr at 103 Message Conlraetor 1 1 5 Of ce ofPers Mgmt to Jonathan Tends Contractor U ice ofPers Mgmt Apr 23 2015 at Production Feb 212116 7 Cotton Tr at If lCotton Tr at W Saulshury Tr at See disc McClure Tr Ex 12 7 Wagner Tr at 1112 148 information about CyTeeh and the use of the tool at 0PM The Committee requested information about CyTech s role in this incident in a July 24 ZGIS letter to 0PM then Chairman Chaffetr issued a preservation order to 0PM on August 21 2015 and on September 9 2015 the Committee requested specific additional information about CyTech s tool after learning data on the too was deleted before it was returned to CyTech m Deepite a ctear obligation to preserve documents and evidence relevant to the Committee s investigation 0PM deleted data an CyTech s appliance before returning the appliance to CyTech on August 20 2015 The appliance was used to collect forensic images that would assist the investigation of the data breach Those images are relevant to determining the scope of the intrusion and data extiltration On June 23 2015 HPSCI advised the Committee that 0PM was still in possession of the appliance Documents show that on June 25 2015 0PM requested instructions 'om CyTech to uninstall the agents CyTeeh subsequently requested that the appliance be returned but it was not returned until August 20 201 S one day after Committee investigators visited CyTech's of ces 5 In mid-August 2015 0PM deleted data on the appliance and arranged to return it On August 13 2015 lmperatis the contractor that introduced CyTech to 0PM wrote Wagner and advised that CyTech wanted the appliance and offered to help coordinate its retum m An 0PM contractor who worked for We ner on IT Security Operations wrote we need to scrub HDs hard drives prior to pick up After some internal discussion about the best way to remove sensitive 0PM data from the appliance Saulsbury and Tonda two 0PM IT security operations contract employees handling security operations requested permission to secure delete all sensitive 0PM data from the demo server including memory images disk images and any individual les or Letter from the Hon Jason Chaffetz Chairman and the Hon Elijah E Cummings Ranking Member H Comm on Oversight d Gov t Reform to the Hon Beth Cohort Acting Din US Of ce ofPers Mgmt July 24 2315 Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight d Gov't Reform and the Hon Michael Turner to the Hon Beth Cobert Acting Din U S Office of Pcrs Mgmt Sept 9 EDI Iettcr from the Hon Devin Nunes Chairman and the Hon Adam Sci-tiff Ranking Member H Perm Se ch lComm on Intelligence to the Hon- Jason Chaffetr Chairman and the Hon Elijah E Cummings Ranking Member H Comm on versighl is Govit Reform June 23 Z l 5 ii Cotton Tr Ex 5 Email from Contractor us Of ce ol'Pers to Juan Bonilla Senior Sec Consultant Cy'liech June 25 5 Cotton Tr at '32 Email from Patrick Mulvaney Imperatis to Jeff 1 i v agncr Dir Info Tech See Operations LLS Of ce of Pers Mgmt Aug 13 2 15 11 26 at 0PM Production Oct 23 2015 Email from Jonathan Tonda Connector US Of ce of Pers MgmL to Patrick Mulvaney Imperatis and Jeff Wagner Dir Info Tech Sec Operations U S Of ce ofPers Mgmt Aug 13 2015 41 am at Production Clot 13 2015 149 metadatg pstracted from OPM devices TBS On August 2015 Wagner approved this request The process of deleting the data was tedious On August 18 2015 Saulsbury who had been directed to delete the data on the appliance reported to his colleague Tonda that the secure delete is only about 30% ccmpiete Saulshury and Tonda were aware that the Committee was investigating the breach at this time- In an email Saulsbury asked Tonda do you need help with anything for the HOUR stuff 1 Tonda responded N ot yet I m reviewing it with Jeff now Maybe later So at the same time the data on the appliance was being deleted they were aware that there were outstanding Committee requests for information Nonetheless OPM made the decision to delete the data on the appliance On August 19 2015 the same day that Committee investigators met with CyTech staff at their of ces a counsel from the OPM OIG told staff in the Of ce of General Counsel that CyTech was complaining that OPM still has not returned the servertapplieation thingee that CyTeeh built and left with OPM after the demonstration 93 He further stated heard something that will create unpleasant work for both our of ces unless it s headed off looks like a badnpublicity lawsuit coming down the pike unless assuming of course that OCIO has it OPM returns it Just saying Wagner forwarded this exchange to an lmperatis employee and said want this CyFir appliance gone today 5 There is no evidence showing any 0PM of cial recommended that the data on the appliance should be preserved in light of the ongoing congressional investigation After the appliance was returned on August 20 2015 examined the appliance to determine what data was on the appliance for the purpose of responding to the Committee s requests for information CyTech determined that 1 1 035 les and directories were deleted by OPM personnel or contractors on August 13 and 19 Enlim Cotton testi ed that Email from Brendan Saulsbury Sonior Cyber Security Engineer SRA to Jonathan Tonda Contractor 1 1 3 Of ce ofl ers Mgmt and Jeff Wagner Dir Info Tech Sec Operations 11 3 Of ce of Pers Aug 2615 at OPM Production Oct 23 2015 Email from Jeff 1 Wagner Dir info Tech Sec Operations U S Of ce of Pets Mgmt to Jonathan Tontla Contractor 1 1 3 Office of Pers Aug Bill S 2 130 pm at Production Oct 23 2 15 Messages between Brendan Souls-bury and Jonathan Tonda OPM IT Security Operations contractors Aug IE at 52 Production Oct 31 Id 793 Email from Jeff Wagner Dir IT Sec Operations US Office of Pers Mgmt to Jonathan Tonda Contractor US Of ce of Pets Aug 17 2015 2 013 pm at nonsense-comer 0PM Production Oct 28 2015 Email from 01G Counsel LLS Of ce of Pers to Associate Gen Counsel 11 5 Of ce of Pers Mgml Aug EDIE pan at Production Oct 23 Email from OIG Counsel US Office of Pers Mgmt to Associate Gen Counsel US Of ce ofPers Aug 19 it'll 5 1 27 pm at Production Oct 23 2GB Email from Jeff Dir 1T See Operations Of ce of Pers Mgmt to Patrick Muivaney Imperatis and Jonathan Tonda Contraolor LLB Of ce ofPers Mgmt Aug 19 21315 foil pm at OPM Production Oct 23 2315 Cotton Ex 12 Forensics Report OPM Server Analysis Report Sept 10 2315 The Forensics Report included a ti t page Appendix A that iisted in detail the LEGS le names and any data or artifacts related to those les that was recoverable Cotton Tr at Tilt-75 150 when CyTech examined the device they were interested in recovering certain database information in order to answer the Comm ittee s questions and to provide clarity as to the scope of their activities while onsitc at 0PM in April-May Cotton stated the tool was not in a functioning state when it was returned to using 5 Cotton also testi ed that the information on the CQIFIR server would have been covered by the Committee s August 21 2o 5 preservation order Message From Patrick Mulvanoy Sent a o ols 12 56 14 PM To Wagner Jeffrey P GE nommsr am WE eaous _recaiemrm- Subject Eyfir Fyi is out ol the building and on its way to On October 23 2015 OPM responded to the Committee s September 9 2015 request for information about the IR appliancegm The agency disclosed they sanitized the appliance prior to returning it to CyTech 8m The agency stated it did so in accordance with best practices and applicable information security policiessch without regard for the ongoing congressional investigation The agency knew as ot'luly 24 2015 that there was an ongoing congressional investigation and that CyTeeh s role in the data breach incident was a subject of the Further the Committee issued a preservation order related to the investigation on August 21 2015 Em The agency deleted the data on the appliance between August ande 2015 Cotton at T3 Cotton Tr at '34 Cotton Tr at 106 Letter from the Hon Jason Chaffet'z Chairman Comm on Oversight 5r Oov't Reform and the Hon Michael Turner to the Hon Beth ICober t Acting DEL US Of ce of Furs Mgmt Sept 9 Eli Letter from the Hon Beth Cohort r tcting Dir Office ofPerS to the Hon Jason Chaffetz Chairman H Comm on Oversight tit Gov't Reform and the Hon Michael Turner Oct 23 2 315 Em Letter from the Hon Beth Cohen Acting Dir 121 3 Of ce of Pete Mgmt to the Hon Jason ChaITcta Chairman Comm on Oversight Gov t Reform and the Hon Michael Turner Oct 23 2015 M Letter from the Hon Jason Chaffetz Chairman and the Hon Elijah E Cummings Ranking Member H lComm on Oversight ri'iov t Reform to the Hon Beth obert Acting Din L13 Of ce oFPers Mgmt July 24 5 a Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight i5 Gov t Reform to the Hon Betlt Cohort Acting Din 11 3 Office ofPeIs Mgmt Aug 21 2015 15 0PM Violated the AntieDefieiency Act Documents and testimony show CyTecl'r provided a service to 0PM and 0PM did not pay for this service The Anti-de ciency Act ADA prohibits a federal agency from accepting voluntary services without obtaining an agreement in writing that the contractor will never seek payment The ADA generally does not permit a federal agency or department to accept services from a contractor free of charge The relevant section of the ADA states An of cer or employee of the United States Government or of the District of Columbia government may not accept voluntary services for either government or employ personal services exceeding that authorised by law except for emergencies involving the safety of human life or the protection of The ADA was enacted to prevent the use of voluntary services to avoid congressional scrutiny The ADA that passed in 1884 and substantially amended in 1950 and 1982 represented a desire to set strict limits on executive branch payroll and procurement officials 5mm Executive branch employees often worked overtime in excess of the agency's congressionally approved budgets and the agency would subsequently request back pay for the Employeessn Congress found it politically and morally problematic to deny payment to individuals who had rendered valuable services to the federal government a fact the agencies well knew To eliminate this tactic for increasing departmental budgets Congress prohibited voluntary services altogether While voluntary services are prohibited by the ADA courts have distinguished voluntary services from gratuitous services Gratuitous services are offered under an arrangement in which the government receives uncompensated services in accordance with an advance written agreement or contract in which the provider of the services agrees to serve with out compensation Bag A contractor or individual can thus provide gratuitous services free of charge without violating the ADA so long as the contractor signs a written agreement in advance stating that the 31 1342 tors See Gov t Accountability UF ce B-3 93 l Recess Appointment For June 8 200 set M SHE im Id 152 services are being offered without expectation of payment and waiving any future pay claims against the govertunentf' The ADA allows the federal government to benefit from personal services exceeding what is authorized by law in the event of emergencies involving the safety of human life or the protection of The exception has historically been understood to require two factors in order to be invoked a reasonable and articulable connection between the Jnetion to be performed and the safety of human life or the protection of property and some reasonable likelihood that the safety of human life or the protection or property would be compromised in some degree by delay in the performance of the function in question 2 Previous invocations of the emergency exception have required a close nexus between the service being provided and the life or property protected For example the arbiter of ADA violations the Government Accountability Of ce found an exception when a municipal health officer disinfected a federal government compound to prevent the further spread of diphtheria that had already resulted in four deaths in that speci c compound When the service provided is merely convenient or helpful in avoiding a future emergency it does not qualify under the exception GAD ruled in 1930 that a man who offered to tow a Navy seaplane to a nearby island after a forced landing did not qualify under the emergency exemption ELM GAO found the rendering of service to avoid a potential future emergency was not enough to invoke the exceptionf The ADA applied to the 0PM and CyTec-h Situation in April 21 2015 CyTech provided a demonstration of its tool at OPI vi s facility in Washington CyTech CEO Ben Cotton conducted the demonstration using CyTech equipment most notably a computer forensics tool known as For the demonstration CyTech brought a server to 0PM which would be connected to network and provide forensics services on up to twenty machinesgl Gov t Accountability Off 3624214 Decision Department of Transom Acceptance of V flii'l'f i'fl Services Jan 27 2014 3 31 use ii 134 2012 3 43 cc Att y Gen 293 3-32 1931 3 12 ecnt Dec 155 Gov t Accountability onicc ccnt Inc 243 Gov't or cc lass 3 ecm ecu 243 Gov t Accountability otncc 193m 0PM visitcr ch Washington no Apr 21 2015 at Hoeaesesia-ceoszs 0PM Feb 15 as Email from_ Imperatis to Jeff Wagner Dir Info Tech Sec Operations and Jonathan Tonda Contractor U S Office of Pete Apr 4 22 pm at Production Oct 23 2 15 Cotton Tr at 43 153 At that time 0PM had not purchased any licenses from CyToch CyTech only provided a limited licensing arrangement hr the purposes of the demonstration for which typically there is no expectation of payment to enable the installation of the tool on twenty 0PM machines for thirty days thereby allowing the machines to be scanned for malwarc and unknown software processes On April 22 2015 Cotton reported the results of the demonstration to 0PM staff and to of Imperatis another contractor retained by The CyTech system had identi ed three unknown processesm The results of the scan were copied to a thumb drive and taken to security eitperts Em Around noon that day Cotton had a conversation with Jeff Wagner Director of 1T Security Operations about the findings Wagner asked for a purchase order for the tool that would cover 15 000 agents sis appliances and 1 000 data Cotton agreed to immediately expand the number of licenses to 1 000 before a purchase order was formalizedim in this conversation with Wagner Cotton also oorrunitted a CyTeclt expert to provide incident response and forensic support for the investigation 824 purchase order for Cy'l eclt services was to be made via a preexisting contract vehicle with Imperatis l15 Consequently Cytech provided a quote to Imperatis on April 24 for 15 000 IR licenses sis appliances six training vouchers and 1 040 onsite engineering support hours that would cost a total 1n the meantime CyTech relying on the government s verbal request for services beyond a typical demonstration situation began espanding its services to 0PM and provided a license to 0PM on April 22 2015 for 1 000 endpoints that expired on one 30 2015 Eu The documents show specific incident response and forensic support activities that CyTech provided to 0PM for which 0PM should have compensated CyTech The documents show 0PM confirmed that the CyTeeh es ert Juan Bonilla would be assisting with an investigation over the next two weeks in terms of speci c lCyTeclt activities Cotton 3 Wagner at 102 103 33 Wagner Tr at 102 103 32 Cotton at 19 333 Cotton Tr Ex 3 4 CyTech Price Quote $318 000 for Emergency Purchase Order Apr 24 2015 and CyTech Transmittal email to Imperatis for CyTech Quote Apr 24 2015 Email from Ben Cotton Chief Exec Officer CyTeclt to H Comm on 8t Gov't Reform Majority Staff Apr 16 2015 con rming the nature ofthe licensing arrangement as of April 22 2015 on le with the Committee 1 Cotton Tr at 25 Cotton noted that Cy'l'ech's expert Bonilla as a senior member of the CyTeclt team is typically billed at between $350 and $450 an hour fr 3 Cotton Tr at 23 Cotton TL Ex 3 4 CyTech Price Quote $313 000 for Emergency Purchase Dialer Apr 24 2015 and CyTeel-i Transmittal email to Imperatis for CyTech Quote Apr 24 2015 Email Han Cotton ChiefFixec Officer Cy l'ech to 11 lComm on Uverisght 8e Gov't Reform Majority Staff Apr 15 2016 con rming the nature ofthe licensing arrangement as of April 22 2015 on le with the Committee Email Jeff Wagner Dir Info Tech- Sec Operations ffiee of Pei-s to 12' Administration 1 1 5 foice of E ers Mgmt Apr 23 2015 at Production Feh Id 2015 154 testified that was initially asked to image all the random access memory of about fi computers and then image the hard drives for these computers and pull event logs for 0PM S 9 CyTeeb also worked with Cylance an 0PM contractor to ful ll their requests for les 330 Documents show CyTeeh s role in providing forensic support was signifieant CyTech collected thousands of images in its forensic support roles Documents show the agency continued to use the 1R tool in May 2015 through early June For example on May 2 2015 Cylance requested deploying to a particular 0PM host machine In another email on June 1 2015 an 0PM contractor con rmed that all other security agents are currently running Forescout 333 Documents show the agency and its contractor Imperatis expected 0PM would be compensating CyTech for incident response and forensic support based on the conversations CyTech had with 0PM in April 2015 For example during the week of April 22 2015 an Imperatis weekly report stated coordinating equipment installation and con guration with security vendors including working to nalize BUM bill of materials for Cylillt 334 Then as late as June 5 2015 Imperatis inquired about the status of the CyTech quote An lmperatis employee emailed an 0PM official do you want for the existing network I assume yes to compliment sic your Encase tool 33 The documents show Cy I ech provided a demonstration and Following that demonstration 0PM requested a purchase order for CyTech services to support incident response activities including forensic support Based on the agency s apparent intent to nalize a purchase order CyTech expanded the licensing arrangement beyond what would normally be provided in a demonstration and provided onsitc incident response services from April 23 through May 1 2015 0PM also retained the for months after the demonstration and used at least some of the licenses for 3 The record demonstrates CyTech was never compensated for these services and CyTech did not sign an agreement stipulating that its services would be provided for free 319' Cotton Tr at 22-23 53 Email from Chris Coultcr Managing Din Cylence to Ben Cotton Chief Exec Officer CyTech Apr 24 2015 5 54 pm at HDCR020316-000010 0PM Production Feb 16 2016 Email from Juan Eonilla Senior Sec Consultant Cy l ech to Brendan Saulsbury Senior Cyber Security Engineer SEA Apr 29 2015 5 26 pm at 0PM Production Feb 16 2010 Email from Chris Couller Managing Din Cylance to Jonathan Tonda Contractor US Of ce ochrs Mgmt giviay 2 2015 3 56 pm at Production Pet 16 20115 53 Email from Contractor US Of ce of Pers- Mgmt- to US Of ce of Pers Mgmt Employees June 1 2015 4 42 pm at Production Feb 145 2016 4 Imperatis Weekly Report Apr 27 2015-May 1 2015 Attach a at 000758 Imperatis Production Sept 1 2015 a Email from Patrick Mulvaney Imperatis to Jeff wagnar Dir lnf'o Tech Sec Operations 1 1 3 Of ce efPers June 5 2015 3 51 pm at Production Oct 23 2015 335 See Email From Contractor US Of ce ofPers Mgmt to 1 1 3 001cc oi'Pers Mgmt Employees June 1 2015 4 42 pm at HUGR0203 Production Feb 16 2016 contractor listing as a security tool running on an 0PM server see also Trial of Locations on which Cy'l'ech s 111 was Tested at 000320-321 Production Sept 25 2015 155 The ADA prohibits a transaction of this nature All the services that were unrelated to the product demonstration including the provision of 1 303 additional licenses after the demonstration was over should have been paid for The agency also kept CyTech s hardware for months after the demonstration CyTeeh did not sign any written agreement that might have converted its voluntary services to gratuitous services because it expected to eventually receive payment This scenario raises the same concerns that the authors of the ADA had in mind when the bill was originally passed The agency accepted a valuable service from a company that expected to he paid but never was The agency s actions placed the federal government in the uncomfortable position of either approving retroactive payment for voluntary services or forcing CyTech a small disabled veteran owned business to hear the sole burden for thousands of dollars in expenses incun'ed in good faith to help 0PM respond to a signi cant cyber incident 156 Chapter 6 Connections Between the 2014 and 2015 Intrusions There has been signi cant public commentary on the source of the data breaches at The Administration has chosen not to make any otlieial assertions about attribution 333 Some Administration of cials have hinted at the source behind the cyberattacks Director of National Intelligence James Clapper has referred to China as the leading suspect stating you have to kind of salute the Chinese for what they did 339 The documents and testimony gathered over the course of the investigation as well as analysis of private sector threat research show the data breaches discovered in 2014 and 2015 are likely connected potentially coordinated campaigns by two threat actor groups This conclusion is based on evidence that indicates the threat actors tactics techniques and procedures TTPs and attack infrastructure share a common source or benefactor The documents show a broader campaign against federal workers associated with the hacking collective Axiom Threat Actor Group Axiom and the threat actor Deep Panda This conclusion is based on a multifactor analysis of the threat actors and the tools they used to perpetrate the data breaches in 2014 and 2015 I First the data breach discovered in March 2014 was likely conducted by Axiom based on the presence of Hikil malware and other TTPs associated with this group - Second the data breach discovered in April 2015 was likely perpetrated by the group Deep Panda aka Shell_Crew a k a Deputy Dog as part of a broader campaign that targeted federal workers This conclusion is based on commonalities in the 2015 adversary's attack infrastructure and common to other hacks attributed to Deep Panda including attacks on VAE Inc and United Airlines However the cyber intrusion and data theft announced by Anthem in 21 15 is a separate 3 Brian Krebs Catching Up an the OPM Breach Raises on Sst uanv June 15 11 25 AM available at sac atso Ellen Nakashima US Decides Against Pabtt'ctv Btaating Chinafar ata Breach POST July 21 so 5 available at ational-sccuritvtu in-esoionaset l St it l 1 le5-33 53-12154 5949f4 a Ellen Nakashirna US Decides Against Chinafor Data Breach July 21 EDIE available at aswf'air-Hame-in-esniona aetl St 'ttll t037't939t i -2 see l citing a Senior Administration Of cial 339 David Welna tn Data Breach Reluctance to Point the Finger at China PUB RADIO July 2 2015 arallels l St 'i't td 1945 hina Director Clapper s and towards China as the perpetrator of the DPM data breaches gained credibility when the Chinese government arrested a hand Ful ot'hackers it says were connected with the breach Ellen Nakashima Chinese Government Has Arrested Hackers t't Says Breached 0PM Database WASI Dec 2 EDIE available at of-breaching oonr-databaseti 15tl H 2t 295b9 1 1 chill-Itch storv html 15 attack by a separate threat actor group unrelated to the hack against 0PM discovered in 2015 a Third both Axiom and Deep Panda are believed to be state sponsored threat-actors supported by the same foreign government I Fourth based on these facts the Committee nds that the 2014 and 2t l4t20 5 cyber intrusions into networks were likely connected possibly coordinated campaigns One Group Several Names There is an inherent challenge in associating a data breach to a particular hacking group as threat researchers and governments do not have a common naming convention for cyher threat 341 actors Threat intelligence researchers generally name threat actor groups based on intrusions called campaigns that share characteristics Over time analyses of campaigns performed by different rms may result in the same threat actor group being given multiple different names Only later are these different names linked or identified as the Same group The groups that will be discussed in this report Axiom Deep Panda Shell_Crcw Deputy Dog APT6 ete were created by threat researchers For instance researchers have relied on the naming convention of Deep Panda 3 2 while other groups term the same threat actor groups as PinkPanther Deputy Dog Shell_Crcw Group 72 Black v ine etc 1L3 Finally because naming conventions of threat actors often revolve around intrusion campaigns rather than membership and affiliation the analysis is unable to account for major changes to the threat actor group s membership funding TTPs malware or infrastructure over time This may result in one group being misidentified as another or two actor groups being identi ed as one 0 Noveila Operation Axiom Threat Actor Group Report at 8-9 See e g Brian Krebs Catching op on the DFM swat-s KREBS oN Serum-v June is 11 25 available at Novetta Operation SMht' Axiom Threat Actor Group Report at 3 9 ThreatCoImect Research 'l'earn 0PM Breach Analysis June 5 EDIE available at 3 Dmitri Alperovitch Deep in Thought Chinese Targeting of National Security Think Tanks BLDG gluly T 2014 43 Deopf ontfo or Shell Crew Who is Behind the Cyhet'Attoehs on US Nit-Morales RESEARCH MOE June 22 2015 RSA Incident Response Emerging Threat Pro ts Sheff Crew 5 Jan Note A set of common characteristics in these groups eyber campaigns and intrusions led to the belief that they are all actuaily the same group with several different names 58 The Axiom Group has been found reaponsible for a series of highly sophisticated cyber campaigns against public and private sector targets throughout the world in the last six years The de nitive technical and behavioral report on Axiom s history and methods of attack was conducted by the threat research group at Novctta in 2014 345 which found in part that the Axiom threat group is a well-resourced disciplined and sophisticated subgroup of a larger cyber espionage group 345 The data breach at 0PM in 2014 like other attacks perpetrated by Axiom or one of its subgroups involved the use of Hikit malware as the primary means of maintaining presence in Oi M s environmentw According to Novetta Hikit malware is a tool only seen used by Hikit malware is a sophisticated remote access tool RAT that offers attackers the ability to create covert backdoors into target computer networks and eventually take full control of target computer netviiorks W Hikit is purposefully built to evade detection and circumvent sin protections offered by rewalls and network monitoring tools Similar to most sophisticated cyber intrusion campaigns Hikit can be modi ed for tailored-use in a target s network and optimized to operate within and take advantage of the vulnerabilities of the sottware hardware or operating system in the victim s environment 351 Additionally con guration les extracted to Hikit binaries indicate that command and control domains C2 callbacks are tailored towards the geographic and network environment in which the target network is located According to Novctta 32 domains will consistently be named and hosted in such a way that traffic appears legitimate likely in an to fool network security operators of target 3118 0 PM incident Report From June 2014 positively identi ed the malware responsible int the 2014 intrusion at two variants of Hikit Hikit a and Hikit 13 353 Hikit a and Hikit differ primarily in the methods they use to communicate with their C2 servers Hikit it uses a unique 4-byte XOR key for each packet while Hikit compresses its network traf c Novetla Operation drioiti titrertt Actor Group Report at 3-9 5 Novctta and the Cyber Security Coalition that conducted Operation published an executive summary of the operation on October 15 2314 The nal report was released in November Elli- 1 and is the product of an industry led effort to identify and disrupt a threat actor group a Novetta Uperotr'ori SMi'u'i' Axiom Threat Actor Group Report at 4 H Comm on Oversight ti Gov t Reform Transcribed interview of Jeffrey P Wagner Feb Iii Ellie at 31 32 3 Novetta Dperotiou SW Ari om Thretit Actor Group Report at 19 9 Novetta Operation SMN Axiom IiireotAetor Group Report at 23 Novetta Dperution SMN Axiom Actor Group Report at 24-25 35' Novetta Uperotiott SMN Axiom Threat Actor Group Report at 4 The Novelta report makes man Ir references to liKit customization by the Axiom group and consider it a tier 1 custonr piece ofmalware id at 4 Gperution SMN Axiom WireutAc-tor Group Report at 5 June 2on4 0PM Incident Report at I59 with quieklz then it is with a hash concatenated with itself in a loop six times 354 The actors responsible for the 2014 intrusion used a wide variety of command and control servers C2 throughout the entirety of the intrusion Forensic investigators were able to identify C2 Servers active and in use during 2014 by detailed deep inspection of network traf c in and out of environment Analysis of the Hikit malware used in the attack provided a granular comprehensive picture or the command and control infrastructure that was created to support the campaign The domains and IP addresses were hard ended as call hack functions within the Hikit malware used in the campaign 3 Bandwidth Monitoring Hikintt ServEr hooliservicetlchatnook I I m a Sever -t-lssm Backup I Ilikit A Storage Manager Eton _l I Network i Performance I Hikil It Monitor Ecsl com - Hints -Server - Server Hilutt'i Ber Hikit A we 1 Server Hit-cit a Server Domains and We used in the 20H intrusion and their associated Hiktt motworo oorrnterportsw I Iikit malware is extremely unique to a speci c threat actor group Hilcit is known as a Tier implant which means that it is a custotn piece of malware that can be strongly attributed to one particular threat actor group Axiom uses a variety of tools in varying stages of the intrusion cycle which fall generally into four families These families ofmalware range in uniqueness from extremely common Poison Ivy Ghost ZXshell to more focused tools used by 354 M 53 5 June 2am 0PM Incident Report at Hooaos 1 84031144 1245 33f Jone saw one Incident Repo at 1 S-GD1244 - 1245 55' Novctta Dporotion SMN Axiom Threat Actor Group Report at 160 Axiom and other threat groups directed by the same organization Dcrusbi chcl to tools only seen used by Axiom The use of Hikit in the 2014 intrusion strongly indicates that a group associated with Axiom is responsible for the IBM intrusion Analysis by open-source threat researchers is consistent with this finding attributing the attack to a state sponsored actor 359 the Novetta report highlights that the Axiom Group s targets Asian and Western governments responsible for government records journalists and media organizations et elm Hikit was rst detected in 201 1 and has evolved and developed into multiple versions since thong Hikit splits into two generational variants Hikit generation one which dates back to 2011 and Hikit generation 2 which spans between 2011 and 2013 31 Both generations of Hikit allow a great deal of functionality for threat actors Once Hikit is dropped on a system the attacker will have a variety of capabilities1 including 1 File management upload and download 2 Remote shell 3 Network tunneling proxying 4 Ad hoc network generation connecting multiple Hikit infected machines to create a secondary network on top of the victim s network topology In addition to there being two generations of Hikit there are also variants All the malware found in 2014 were two variants of Hikit malware termed Hikit A and Hikit 3 354 According to the 2014 DHS incident Report the Hikit malware A llow ed the attackers to create a reverse shell from their CZ command and control servers into the infected systems in network from a remote location anywhere in the world Wagner reaffirmed the Hikit rnalwarc Was mostly used for persistence or maintaining a presence at 0PM though keylogging activity was also Effectively the inalware was used so that the hackers could still use it to obtain entry into OPIWs networka Hikit in particular has shown to take particular advantage of poor Novetta Uperotr'oo SMM Axiom Threat Actor Group Report at 19 '1'hreatConnect Research Team Breach Analysis June 5 21315 th reatconnectcomr'o pm-hreach-anal ysisr 3 50 Novena Operation SMN ltriom Threotztetor Group Report at 10 Novetta Hikit Analysis at 1 Nov 21314 available at contenb uploads 14H 1 linKitpdf as 363 Novctta Spar-noon SMN Axiom Thine Actor Group Report at Saulsbury Tr at 1 Wagner Tr at Saulsbury Tr at 18 161 internal rewalls and network According to one of the earliest analyses of Hikit malware conducted by FireEye Inc an attacker was able to tunnel via Remote Desktop and proliferate across the network using previously compromised credentialss 3 This allowed attackers to create hop points among internal and external network segments by installing copies of the rootkit in strategic locations to establish new footholds within the target network The Hikit malware was well-suited for use on network DHS found 0PM did not and may still not have tiered network architecture with segmentation between users databases applications and webservers network is extremely flat at this time and has little to no segmentation m DHS ultimately recommended the server environment should be segmented via rewalls into logically separate internally and externally accessible DMS web server application server and database environmentfm The flat network architecture that Ol M s legacy environment employed made the agency an ideal target for exploitation by the Hikit malware Security researchers have suggested a variety of possible threat actors are responsible for the 2015 data breach at While much of the evidence that would support attribution of the actor to a particular threat actor or actors remains classified public source documents indicate a grou referred to as Deep Panda is likely to have been involved based on the attack infrastmeture- 3 Unlike the 2014 data breach where l Iikit malware could be uniquely linked to the Axiom Group the use of malware in the 2015 data breach alone is not suf cient to positively identify Deep Panda as the culprit The employed by the 2015 attackers is commonly used by eybcr threat actors and has only become more prevalent since the initial 3 Saulsbury Tr at 13 353 Christopher ICilyer 5 Ryan The Niki Advanced and Persistent Artur-it Techniques For 2 22 21312 available at at vanced persiste nl altack teehniques-part-Z 9 Id June 2cm 0PM Incident Report at Hooaosi s-ooizse hi all Jeremy Wagstaff Deep Pearle Intensi es in Trencher of IS Chino Cyberwor REUTERS June 21 2015 available at Security researchers have many names for the hacking group that is one of the suspects for the cyberattack on the LLB government s Df ce of Personnel Management PinkPanther KungFu Kittens lGroup T2 and most famously Deep Panda But to Jared Myers and colleagues at cyberseeurity company RSA it is called Shell Crew see oiso David Perera Agency Dido 'i Fetis' Doro Hooked by Chinese June 4 2315 available at ji oip ion-federalemployees-hackedl 13655 The massive data breach there affected the records of 4 1 million current and former federal employees and may be linked to a Chinese State backed hacker group known as Deep Panda which recently made similarly large scale attacks on the health insurers Anthem and RSA incident Response Eill i gii'lg Threat Pro le SireiLC'rew 5 2Dl4 available at a 62 intrusion in An analysis of the infrastructure used to hack network in 2015 however points toward the liker responsible actor The adversary s attack infrastructure which includes the websites used to hack networks and cx ltratc data was similar to attack infrastructure used in seemingly unrelated intrusions The malicious domains registered for the 3th back had three distinct characteristics Marvel comic book superhero names GMX throw away e-mail accounts and domain names tailored to appear as legitimate portions of network and training resources 375 An advanced persistent threats APT attack infrastructure is visible to cybersecurit experts in the form of domain names and their corresponding IP address hosted on C2 servers 15 How when and by whom domain names and IP addresses are created registered and used in conducting a are therefore important factors in attributing a hack to a particular actor The adverstarv that perpetrated the data breach against 0PM in 2015 used an attack infrastructure similar to cyberattacks tied to Deep Panda Cybersecurity research rms and have exposed a number of characteristics of Deep Panda s attack infrastructures These characteristics were identi ed during the analysis of several intrusions including attacks on VAE Inc 1m and United Airlinesm These attacks bear a striking similarity to the 20 5 data breach at The attacks share several common elements I Registrant Names Domains were registered under names associated with Marvel s Avengers or actors related to the Iron Man franchise and Marvel universe Chris Brook PingX Gowto Mohenrefor ngered Attacks More Prominent Than Ever Feb It 2015 available at evenIf 1 10936 8 75 ThreatConnect Research Team 0PM Brooch June 5 2015 available at Wagner testi ed that one oflhc reasons he considered the 2015 attackers to be sophisticated was because the E l attackers used speci cally U S based IFI hosting addresses to prevent geolocation rules from being effective Wagner Tr at 32 Threat Connect Research Team The Anthem Hock nit Roads Lend to China Feb 27 20'15 available at save ails-o Matt Dahl i on Ironmon DEEP PANDA Uses Soirtrio Mniwm c to linger Organizations in Muttipie Sectors BLDG Nov 24 2014 available at tea l 3 Drew Harwell 3t Ellen Nakashima Chino Stt 'pECrlEd in Major Hocking ofHaat'ti insurer PDST Feb 5 2m 5 available at 5i02i 5i25fbb36c-ad56-1 le4 9c91 e9d2f9fdo644 inl Elizabeth Weisc Massive Breach nt Henith Care Company Anthem USA Toonv Feb 5 2015 available at mistorvitec hill 1 tDZi itfhcalth c 99392 St 3W Ellen Nakashirna Security Finn Finds Link Between Chino anti Anthem Hock WASH PUST Feb 2015 gtonpost condnewsithc-switc hiwpt 5t 02 'itsocuritv- rm -an anthem hacki Threat Connect Research Team raa- Aaraam Hack an Lead to China Feb 21 2015 g ailable at Id 163 - Registrant Emails The domains were registered using emails that were a combination of pseudorandom ten digit alphanumeric usernames and e-mail accounts 332 I Faun Domain Names Registered domains were tailored to look like legitimate domains hosting resources that belonged to the target organization or portions ofthe target s network 1m With respect to registrant names Deep Panda s use ofa comic book themed naming convention was previously documented by during their analysis of a 2014 eampaign against among other targets the healtheare and govemment sectors 834 The agency using a variety of network monitoring tools identi ed three domains as the primary attack infrastructure opmseeuritverg wdc-news-postcom and opm leamingerg - L L Warning th term I stark 0PM Breach upmaawMJorg Steve Rogers W hp Lh gmeImIn 0PM Breech Mki-vael'thm Ten r Stark VAE Inc Targeting Campaign shampotnt-vaelt eorn Natasha Roman-off Inc Targa ng Campaign Duhal Tymon VAE Inc Targe ng Campaign John Nels-m VAE Inc Targeting Garnpalgn Hark Waring Unldenli'l'ted united-al inaamnet James Rhodes Unidunll' ad Ntr'eafcomteer chart shows similar registrant trainer e mor ir and evidence ofrr forget triers corrapler enriipnignm Deep Panda registered their attack infrastructure using the names of Marvel s Avengers characters and other names associated with the lm franchise I Tony Stark aka Iron Man I Steve Rogers ales Captain America I Nalasha Romanoff a k a Blaek Widow 1- James Rhodes aka War Machine I John Nelson the visual effects supervisor for the Marvel lm from Mam PMBrenefr Annivsr s Update last visited June 15 EDI at Threat neaeareh Team The Anthem Haea as Reese Leas to China res anti available at Matt Dahl I um DEEP PANDA Uses L'i'rrinn o Mathews to Target Organizations in Madriple Sectors BLDG Nov 24 2014 available at sakula mal ware target or anination s mn llinle-seetorsf' ea l 1465319953 ThreatConnec-t Research Team 0PM Breach Anrrfeviv June 5 2915 available at 355 John Nelson Biography available at fn Ell 11m l 164 I Dubai Tycoon the name of an uncredited role in the Marvel lm from Men portrayed by noted rapper and Wu-Tang Clan member Ghostface Killah 33 With respect to registrant email addresses and domain names the original registrant s email was always a random alphanumeric with a @gmxeom email address and the domains had 0PM themed names On April 25 2014 actors registered the malicious domain opmsecurityorg under the name Steve Rogers using the e-mail address Shortly after the Bi Bang concluded and just eighteen days after the New York Times broke news of the breach on July 9 2014 339 another C2 node was established by the same actors On July 29 2014 the attackers registered the OPM-themed domain The domain was registered by Tony Stark using the e-mail address in addition Deep Panda s attack infrastructure typically involves domain names tailored to look like legitimate domains that belong to the target organization For instance the security rm ThreatConnect has tied the use of Wellpoint look-alike domains to a series of targeted attacks launched in May 2014 that appeared designed to trick 1lilicllpoin t employees into downloading malicious software tied to the Deep Panda hacking gang 392 Domains such as we lpoint com or myhr wel pointeom were used in the course of a campaign against Anthem Em Security expert Brian Krebs stated appeared that whoever registered the domain was attempting to make it look like Wellpoint the former name of Anthem before the company changed its corporate name in late 2014 394 These victim-centric domains could easily fool network monitors as they at rst glance appear legitimate but under further analysis are proven to be malicious 83 Iron Man Trivia last visited June 3t Ebi Ghostfaee Killah a long-time fan ofthe Iron Man comics he uses the aliases Ironman and Tony Starks titled his 996 album Ironman and sample clips of Iron Man had a cameo as a Dubai tycoon However his scene was cut from the nal lm Jon Favreau apologized to Ghostface and used his We Celebrate video in the a 0PM Breech Aimiysis Update last visited June 15 2015 available at 335' Michael E Schmidt David E Sanger 3 Nicole Perlroth Chinese Hooks Pursue Kev Dam on US Wm'kem NY TIMES July 9 2D14 0PM Branch Aimiysis Update available at analysis-update 39' 'l'breat Connect Research Team The Anthem Hock AH Roads Lead to China TI-IREATCOHNECT Feb Bill available at 392 Brian Krebs Premiere Cross Hrencii Exposes Financiei' Msdieni Records KREBs 0N SECURITY Mar ii 5 42 PM available at medical recordsittmore-E 0380 Threat Connect Research Team The Anthem Hack Roads Lead to China Feb 27 2015 available at Brian Krebs Anthem Branch May Have Started a spin 2am asses on Secunn'v Feb 15 sets 10 34 are available at 21's Mi 165 Deep Panda also appeared to name the domains to emulate portions of the target s network or to mimic organisationally-related resources hosted outside the target s network In the case Deep Panda made the domains look like company-related Sharepoint or Wiki resources by naming them sharepoint vaeit com and in the 2015 0PM breach the malicious domains used for command and control and opmsecurity org resemble the websites 0PM uses for its annual information technology security awareness training opmsecnrity goleaming org and security This training is required for all ill time and part-time federal employees and contractors who have access to networked g The fans domain naming used in these hacks is a Deep Panda calling card but it also reveals information about Deep Panda s TTPs These victim-centric domains could slip past network monitors as they at rst glance appear legitimate The domains are designed to fool employees into thinking they are legitimate After clicking on a link sent through a spear phishing e mail attackers can download malware into the company's network by exploiting vulnerabilities in the victim s web browser This technique called a watering hole attack 399 is a strategy that uses hacked websites or fake lcgitimate looking domains to download malware into a victim s computer 9m Watering hole attacks are a technique heavily favored by though not exclusive to the Deep Panda threat actor groupFD Another common element of Deep Panda s campaigns is it often relies on some of the same attack infrastructure for multiple intrusions including the breach into network The following domains were active on systems during the course ofineident responsezm Entry IP Domain Entry 1 Wiki-vaeit com Sharepoint vae eom ssl-vaeit com Wiki-vaeit com Entry 2 Wei lpointcom Threat Connect Research Team rsa Anthem Haeir an Lead to China Tnssareoansrr Feb 27 2015 agiEailable at id' 39 0PM Breach Analysis Update last visited June 15 2016 available at rn-breach-analysis-updatct 3 Saulsbury Tr at 34 So named because it resembles a strategy employed by predators who will lie in wait to ambush prey at a site they are known or expected to frequent like a watering hole win Gragido Lions ar the Watering Hirier The retro A air ass Jul 2o sols gm Adam Greenherg Watering Hoie Attacks ore Becoming Increasingr y Popular Says Study SC MAGAZINE Sept 2 13 available at saysustudvr altieletfi 35lth quoting Nick Levay chief security officer with Eit9 Watering holes have been on the rise in the past few years and a lot of hackers that were using spear phishing attacks to target people have started using watering holes said Levay explaining that while watering holes typically target a speci c group or community he has seen narrower variants that for example will only target a certain range of iP addresses See eg ThreatConnect Research Team 0PM Breech Anoiysis June 5 available at 0PM Domain Name Log Unredacted at 0PM Production Dec 22 21315 166 Exteitrixwel Ipointeom Myhr wel lpoint eorn Hrsolultionswel pointeom Entry 3 drongobasteom efueliaeom gandabandeom kopirabuseom maerosascom mustufackaeom gainaScom ns8 gaina5 net nsa gaina5 net Entry 4 as org en Entry 5 smtp outlookssl eom Entries 1 and 2 in the above chart are malicious domains also used by Deep Panda against VAE and WellpoinUAntbem systemsgm Seven of these domains Wild-vaeitcom Sharepoint vae eom ssl-vaeit com We lpointeom Exteitriawel lpointeom lpoint com Hrsoluitions wel lpoint com were active on systems during the 2015 data breach and share common identi ers with the primary infrastructure used tn perpetrate the breach against 0PM discovered in 2015 including Avengers-themed names and GM email addresses Threat researchers tied attacks at VAE and Anthem to a grou known by a number of names including Deep Panda Axiom Group 72 and the 5 Testimony shows 0PM security personnel also cormeeted the 2015 attack to Deep Panda Saulshury testi ed Q So my question is as a result of the April ZDIS eyber intrusion was 0PM SOC able to draw any conclusions as to whom or what organization might have been responsible for the malicious activity And again to the extent you can answer without revealing any classi ed information- Right so to clarify I do not have a ciearanee 1 do not have access to any classi ed information The only unclassi ed information that we have was that some of those Marvel character related domain names or domain registrants they showed up in a -- I believe it was a Mandiant report incident response report regarding a publicized data breach for a healtheare provider but I can't recall speci cally which it was at this time But the Mandiants dubbed the attacker Deep Panda emphasis added so at Threat Connect Research Team The Anthem Hock AH Roads Lend to China l eb E li available at 9'55 Brian Krebs Anthem Breech May Have Sim-ted in April 2014 K1st 0N SECURITY Feb 15 24315 Ali-ii available at http chrebsonseeuritycom 4f 16' based on that domain registrant correlation that is the only indication or at least on the unclassi ed side that we have that that may be the same Saulsbury s testimony was corroborated by Coulter who testi ed about the Plug malware and other evidence Cylanee found on systems Couiter stated A So I ll use the word aetor the ones that were identi ed in prior exhibits You had Shell Crew or sometimes known as Deep Panda as well as Deputy Dog and it has many many other names So those were tlte two that at least as it relates to the industry research being done that the malware that we band was closest related to it By no means are we saying it was them it's just it was a relationship or similarity Q Okay Are those two generally associated with a particular country A In the industry yes Q Can I ask which country I A - 91 The 2015 0PM attackers use of malicious domains similar to or even the same as those used in attacks against VAE and Wellpoint Anthem show Deep Panda likely perpetrated the data breach against 0PM that was discovered in 2015 The similarities in the pseudorandom 10 digit GMX address OPIv l-themed domains and Avengers-themed registrants are evidence that the was created and utilized by the same group Documents and testimony connect Deep Panda and Axiom and therefore the 2014 and 2015 data breaches at 0PM were likely connected and possibly coordinated 2014 8 2015 Lil-relyr Connected Possibly Coordinated While 0PM has maintained the cyberattacks conducted against their systems in 2014 and 2015 were separate occurrences documents and testimony show a broader campaign against the information of federal workers by state-sponsored hacking organisations Deep Panda and Axiom were responsible Under a theory advanced by threat researcher FireEye many seemingly unrelated cybern attacks may in fact be part of a broader offensive fueled by a shared development and logistics Saulsbury at 33 Couiter Tr- at 93 168 infrastructure a finding that suggests some targets are facing a more organized menace than they The overlapping use oimalware and exploits or as FireEye called it a shared malware- builder tool 9 9 by Axiom and Deep Panda show the data breaches were likely connected possibly coordinated If FireEye s theory is true either Axiom and Deep Panda s efforts to collect data from systems in 2014 set 2015 were connected via a common supplier of cyber resources or that Axiom and Deep Panda s efforts were actively coordinated by that supplier While FireEye terms this common supplier a digital quartennaster other threat researchers have identi ed a similar shared resources model A researcher at PricewaterhouseCoopers LLP stated In our experience very few attackers have the patience to maintain completely distinct infrastructure with multiple registrars name servers and hosting providers at the same time in our view the hypothesis with the highest probability is that groups of attackers share resources leading to overlaps this appears to be an ever more common feature with malware families builders and even sometimes hosting infragtlgucture being shared between disparate actors with a common goaL Documents show Axiom used Hikit malware to attack network in 2014 and were targeting the background investigation data stored on the PIPS system that was eventually stolen by Deep Panda using malware Documents show Axiom and Deep Panda had more in common than their target Both have been tied to the use of Plug and Hikit nalwar'e 'm Among the challenges in making this assertion are the naming conventions used by the tlueat researcher community in analyzing data breaches and persistent threat actors For example threat researchers at Ciseo stated that hikit according to our data is unique to Group T2 and to two other threat actor groups Group i2 is an alias associated with a state-sponsored espionage group known by a number ofnames including Deep Pandafg12 But llikit is not the only malware that Axiom and i reEye Supply Chain Analysis From Qnorrennoster r o SnnsiropFireEye at 3 available at us fww w ii reeve comicontentitlamff'i ohalie nice rrent M Chris Doman Sc Torn Lancaster SeorrBoI Fronrework Who A ecrert' and Who 15' Using It PWC Oct 2014 available at security whos-using it- html g FireEye Chain Analysis Fm Quartermaster SunshopFireEye at 3 available at dt sfrot-m alwarc-suoolvuchaimotif Brian Krebs Anthem Breach Moi Hove Starred in Apr'ii sore KJLEBS UN SECURITY Feb 15 2015 19 34 AM available at It is noteworthy that Brian Krebs links Deep Panda and Axiom see also Andrea Allievi et a1 Cisco Deconstructing and De ending Against 5 2 l4 available at lesipublications and presentation sfpapersiCisco security Group'ii wopdt' 169 Deep Panda use Malware Name Ghost Rat oudour deer Poison I km Brent HydraQ 913021231 Maid Roanlr Mdmhot ZaShell Sensode Do excl Derusbi Midi-tilts Thoper Sega Korplug Kaba Destre RA Sakula Sake Sakurel Mivast RAT Huria In addition to an overlapping repertoire of malware Axiom and Deep Panda have both been linked to the use ofthe Elder-wood Framework l l Symantec Security Response identi ed attackers employing re use components of an in 'astructare which they named the Framework after a source code variable used by the The Eldenvood Framework is effectively a library of exploits that hackers can use to conduct malicious operations lam Novetta cited Axiem s use ofsimilar tools and other attack infrastructure including Elderwood platform attacks in 201 1 2012 and 2014 9 According to Symantec Black Vine a k a Deep Panda also used the Elderwood The overlapping TTPs malware and attack infrastructure that Axiom and Deep Panda use suggests these groups share a digital quartet-master a central supplier ol malicious tools tactics and techniques to a variety of state-sponsored espionage groups This explains why the same group of hackers has launched attacks under several ditTerent names Axiom Deep Panda Shell Crew Deputy Deg etc With respect to the OPM breach the attack infrastructure and common malware indicates Axiom and Deep Panda are probably connected The overlapping timeframe of the attacks on 0PM also suggest that a connection between the perpetrators See Novella Operation SMM Axiom ThreetActor Group Report at 4 see also ThreatConnecl Research Team OFM Breech Arielvst's June 5 2015 See also Brian Krebs Anthem Breach May Have Started ta April 23H Knees ON SECURITY Feb 15 13 34 See alto Liam Tang Anthem Health Insurance Heehetis are Well Funded Bast ut t CED July 29 2015 http llw ww cso com aularticlel53 eas Slanthem-heal th i nsu ranee-hac kers well mded-busv -outfi tt' 9 Gavin D Gorman 3 Geoff McDonald Symanlec The Eldentrood Project last visited June 15 21316 915 lcl uis ld Novetla Upet'ett'ott SMN 'lhreot Actor Group Report at 12 Liam Tong Anthem Health lttsarmtee Hackers are Well Funded Busy Out t CED July 29 2015 available at l'l0 Documents show that while 0PM was monitoring the 2014 attacker s movements in May 2014 the 2015 attackers were able to drop malware onto servers connected to the background databases the 2014 attackers were targeting Within days of their initial entry into networks the 2015 attackers were able to gain access to the personnel records and background investigation databases establish a late stage attack infrastructure and begin data extiltration The speed at which the 2015 attackers were able to escalate access from initial entry to end-stage presence and ex ltration suggests a level of familiarity with environment This creates the appearance that the 2015 attackers relied on information obtained by the 2014 hackers who had access to network for years and were unable to compromise the most sophisticated systems such as those holding background investigation data According to Saulsbury the documents the 2014 attacker esl iltrated from 0PM provided an attacker or any associated group with directly or indirectly - an advantagem As Mr Saulsbury explained the documents provide more familiarity with how the systems are architected Potentially some of these documents may contain accounts account names or machine names or IP addresses which are relevant to these critical systems 921 The documents the 2014 attackers stole may be characterized as documents that provide overviews of key systems such as PIPE and Fingerprint Transactional System and provide information as to who has access to those systems 922 The documents effectively provide a roadmap to how the background and personnel data is ingested into systems how 0PM integrates those systems with the government contractors working on them and who has access to those aysterns It is the kind of information that would accelerate an attacker s familiarity with most highly sensitive information and could explain the speed with which the 2015 attacker was able to establish access orient themselves escalate network authorities and penetrate the most highly sensitive data repositories on network Documents obtained by the Committee show additional evidence of a connection between the 2014 attacker and the 2015 attack For example the 2015 attacker persisted in their intrusion even after the public announcement of the 2014 data breach on July 9 2014 and continued ex ltrating background investigation data This shows the 2015 attackers had sufficient awareness of security protocols and were not worried despite the heightened state of security that was put in place This suggests a degree of collusion or shared tasking between the two attackers enough so that the 2015 attacker would be comfortable that earlier efforts wouid pave the way and the subsequent mitigation steps taken by OPM would not disrupt the 2015 attackers1 ongoing operation Regardless of the names of the threat actor groups that were conducting malicious activity on systems it should have been clear to 0PM in the wake ofthe 20 4 data breach June a 2015 name June 2014 cam madam Report at perms rm Saulsbury Tr at 22-23 913 June 20 I4 0PM Incident Report at 3 001245 171 that they were facing a sophisticated well-resourced adversary with connections to a spectrum of state sponsored threat actors Private sector threat researchers were connecting the dots between the targeted campaign against federal employees as evidenced by the data breaches at Anthem Premera USIS KeyPoint and should have heightened awareness of federal agencies like 0PM holding large sensitive data repositories 17 2 Chapter 7 OCIO and its Federal Watchdog Pursuant to the Inspector General IO Act of 1923 Inspectors General provide a means for keeping the head of the establishment and the Congress fuily and currently informed about problems and de ciencies relating to the administration of such programs and operations and the necessity for and progress of corrective action m When President Carter signed the IG Act of 1928 he charged the 165 to always remember that their ultimate responsibility is not to any individual but to the public armrest The relationship between Office of the Inspector General BIG and its 0010 became strained while Katherine Arehuleta served as Director and Donna Seymour as 310 In fact the relationship deteriorated to the point that Patrick McFarland took the drastic step of issuing a memorandum to Acting Director Beth Cobert to share serious concerns regarding the OCIO on July 22 2am 925 The memorandum was issued just 12 days after Cohort was appointed Acting Director of the agency During her nomination hearing before a Senate Committee 926 Cobcrt was emphatic that she takes the relationship with the 1G seriously especially as it relates to enhancing cyberseeurity 92 Cobert met with the 10 on her first day at and she instituted regular meetings with the OIG thereafter Despite serious concerns raised by the IG and Congress about Seymour s fitness to serve as C10 in the summer of 2015 93 Cohort maintained support for Seymour and allowed her to remain on the job until her retirement on February 22 2016 31 The Committee obtained testimony in October 2015 that shows problems between the OCIO and the OIG persisted through the fall of 2015 An 01G employee testi ed that the relationship was strained and the onus was on DIG staff to chase down information from the 0010 933 3 Inspector General Act orlars 2 5 use app 21121212 as amended - 9 Council of the Inapeetors Gen on Integrity and Ef ciency It Act History available 01G Memo Sci-ions Concerns Nomination oj'rne Honomhie sen F Colbert to be Director O ice anei- sonnei Management Hearing Before gr 3 Comm on Homeirmri Sec Gov Affairs l4ll l Cong 2916 r 39 Incorporating Scent-i Media into Federo Background investigations Hearing Before the Subcomm on Gov 't Ugemfions and Subcomm on Not See of the H Conan Di'ei sight r i Gov 't Reform 1 14th Cong at 1 12 35 93 Letter from the Hon Jason Chaffetz Chairman ii Comm on l five-might Sr Ciev't Reform to the Hon Beth Cohort Interim Dir US Of ce of Pcrs Ivigrnl Aug ti see oiso Letter from 18 Members of Congress to Earack Oberon President United States June 25 2315 raising concerns about 0PM Director Katherine Archuieta and 0PM Chief Information Officer Donna Seymour 99 Aaron Boyd 0PM Chi-f Sevinoin Resigns Days agen- Gvenrr'gnr Hearing FEDERAL Feb 22 2016 available al resignsfii TtitE-44 f Billy Mitchell U ice ofPer'nmincf Management Donna Seymour Retires Feb 22 Mid available at Ian Smith 0PM CH3 Donne Seymour Resigns FEDSMITH Feb 22 2Dl available at rcsignst 1 Special Agent Tr at 4a 5566 123 Overall however the relationship with the DIS steadily improved under Acting Director Cohert s leadership and as of this report s publication both o ices report it to be without conflict The - G's Memorandum of concern On 1113' 22 5 the OPM IO wttote Acting Director Cobcrt to call attention to four situations where he felt the OCIO hindered his of ce s efforts and ve instances where he contended the OCIO provided incorrect andtor misleading irttf'orttr'tation 9'l'l MEMORANDUM FOR BETH F rtctinj I Director l' E FAREspecter General Serious Concerns Regarding the Of ce ofthc Of cer Wm The memorandum stated In certain situations the actions have hindered the ability to ful ll our responsibilities under the Inspector General Act of IQTS as amended 1G Act Further we have found that the OCIO has provided my of ce with inaccurate or misleading information some of which was subsequently repeated by former OPM Director Katherine Archulcta at Congressional 5 McFarland pointed out that the breakdown in the relationship stood in stark contrast to the relationship the OIG had with the OCIO in the McFarland served as the agency s watchdog for twenty six years Docutnents show the relationship between the OIG and OCIO did in 1 act deteriorate after being strong for years 0PM Data Breaches Part Ht Hearing Before H Comm on Oversight d r Gav 't Reform 114lh Cong Feb 24 2016 prepared statement of Norbert Viol Of ce of Inspector Gen US Of ce of Pets- hearing cancelled see atso Incorporating Social Media taro Federal Boekgronnd Investigations Hearing Before Hubert-nan on Gov Operations and Snbcontar on Nat Sec oftrte H Comm on Oversight Er Gov't' Reform 1 Cong at 12 35 ants 9 LLS Of ce or Pcrs Of ce of Inspector Gem Memorandum from Inspector Gen Patrick McFarland to Acting Dir Beth Cobert Serious Concerns Regarding the Office ofthe Chief Information Of cer July 22 Lhereina er DIG Serious Concerns Regarding OCIO Jul r 22 2015 35 at at t 936 Carton Cerdell 0PM IEFetnerarl Resigns Leaving in February FED TIMES Feb 3 2016 federalti mes eomtstorytgovernrne an agem ot t topm insp ector general resig ns leaving-februarvt' 91'553221 1T4 For example in the April 2008 Semi-Annual Report to Congress McFarland reported that then-Director Linda M Springer had initiated a series of actions to make sure that all 0PM employees clearly understood what PII meant the importance of protecting P11 and their responsibilities in protecting it 933 The was to play an integral role in the efforts The report stated Director Springer requested that the DIG conduct an audit of one of largest program of ces to ensure that they had developed and implemented effective controls over PII has also become a routine topic of discussion at the Agency s Information Technology Security Working Group meetings The group was set up by the Chief Information Of cer to ensure that information technology IT security and privacy policies procedures and directives are communicated to all 0PM program offices On the technical side 0PM has made significant progress in implementing 0MB requirements to safeguard 1311 939 Pl AND Former Inspector Illis'neroir Patrick MeForionrt testi es about data brooches In 2015 however McFarland had to resort to a public noti cation to Acting Director Cohert to call attention to the fact that his of ce was being undermined McFarland wrote In the past the GIG has had a positive relationship with the UCIO Although the MG may have identi ed problems within the areas of responsibility we all recognized that we were on the same team and the OCIO would leverage our ndings in an effoit to bring much needed attention and resources to information technology IT program 933 Of ce of Inspector Gen US Of ce ofPers Sentioiinnai Report to October i 200 to March 33 Mill Mar a M 13 5 Unfortunately this is no longer the case and indeed recent events make the DIS question whether the OCIO is acting in good faith MCI McFarland s memorandum was released to Congress and the public Chairman Chaffeta shared the G s concerns a a letter to Cohort Chairman Chaffetz stated that he lost con dence in Seymour in the wake of the agency s announcement of the breaches that his concerns were ampli ed by the 16 s memorandum and keeping Seymour in place only added insult to inj ury to those whose personal and sensitive information was stolen in the breaches On June 26 I communicated to President Obama that I have lost con dence in Ms Seymour s ability to execute her role as C10 Despite repeated warnings from the OPM Inspector General lvls Seymour failed to prevent breaches of personallynidenti ahle infonnation harming over 22 million federal employees and other individuals and weakening our national security Its a result I asked the President to address this serious issue by removing Ms Seymour from her position 1 am deeply troubled Ms Seymour remains at her post over a mouth a cr this request was made My concerns about Ms eymour's ability to serve are amplified by a communication the Committee received from the Inspector General In a letter dated August 3 EDIE Ut t vt's IG notified me that on July 22 Etll a was sent to you and the letter advised me that there have been situations where actions by the OCID have interfered with and thus hindered the Ulii s work Further the has repeatedly provided the DIG with inaccurate or misleading information ' Excerpr om August 6 201 5 fetter 'mn Chairman Cba etz Acting Director Cobras-r Cohort did not remove Seymour ln fact Cobert gave Seymour a vote of con dence Fert'Newerrdi reported An 0PM spokesman said by email that Cobert is pleased with Seymour and the entire CID team s efforts to improve OPM's cybersecurity The spokesman said Cobert responded to the G s letter saying In her f'u'st four weeks at 0PM she has observed that the team including the Office of the Chief Information Of cer worlcing sidesby-side with experts from across the federal government has been working incredibly hard to enhance the secun'ty of our information technology systems and support those who have been affected by the recent cybersecurity incidents The recent results of the Cybersecurity Sprint demonstrate the progress that has been made although everyone recognizes there is more to do 1943 DIG Serious Concerns Regarding DCICI July 22 5 at l 9 Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight 3r Gov t Reform to the Hon Beth Cohort Interim Dit 1 1 5 Of ce of Pers Aug 5 3 Jason Miller re ma a sit-lass Here on 0PM cro Aug 6 2m 5 available at The Cybersccurity Sprint was meant to increase the security of agencies systems For additional infonnaticn see Exec Of ce of the 176 Cobert said she was committed to ensuring a cooperative relationship between her teams and the Cobert added that she discussed the importance of the issue with her leadership team and said they are fully supportive of rebuilding a productive relationship and Fully understand how that will help us collectively deliver on mission The extremely serious nature of the concerns however raise questions about the decision to stand by Seymour Four Instances Where the OCIO Failed to Cooperate Fully McFarland s letter to Cobert on July 22 2015 identified four situations where the OCIO failed to cooperate with his of ce to the detriment of the agency In April 2015 the agency identified an unknown Secure Sockets Layer certificate beaconing to a site opmsecurityerg that was not associated with The agency reported this nding to on April 15 2015 on Frida April 17 2015 at 11 39 am one submitted several more questionable files to 3 and by 5 19 pm that evening US- CERT confirmed the malicious nature of the executable les that 0PM reported The IG was not noti ed by OClG or anyone else at OPM until one week later on April 22 2015 95 Under OPlvl s Incident and Response and Reporting Guide the 016 is an integral part of incident response 95 1 For exam lo the Guide states that the DIG must be noti ed immediately if criminal activity is suspectedgs The Guide instructs lcey 0PM personnel to be trained in how to make notifications in a manner that serves the best interests of forensic investigations It states that the OPM Computer Incident Readiness Team OPM-CIRT must be trained in such areas as whom to contact when an incident occurs how to preserve forensic evidence and how President Press Release FACT-SHEET Enhancing and Strengthening the Federal Government '3 Q'bersecurity Tune 12 2015 sheetsr'enhancing 4 Memorandum from the Hon Beth Cobert Acting Din 1 1 3 Of ce of Pets Mgmt to Patrick McFarland Inspector Gem LLB Of ce ofPers Mgmt Your Memo offiin 22 ENE Aug 3 1015 hereinafter Cobert Egsponsc to DIG Serious Concerns Regarding Id AAK 'l'imeline- Unknown SSL Certificate April 15 21315 at Production April 29 mild 94 Id Email from to CERT April IS sols 6 54 pm at DPM Production Dec 22 EDIE Email from to Brendan Saulsbury Senior Cyber Security Engineer SEA Apr 17 3315 5 19 pm at 75 Production Dec 22 Id 95 Serious Concerns Regarding ocro July as 2015 at a 95 1 ILS Of ce of Pers Mgmt incident Response and Reporting Guide at 3 July EDGE 9 1 Id The Special Agent testi ed in October Ell 5 that this Guide was still the most current despite being dated July 2609 See Special Agent al 8 to eradicate the various types of incidents The training must also include when incidents are reported to US-CERT the OPM IG and appropriate law enforcement agencies The Guide states that e omputer incidents are generally a lot easier to handle when reported and requires the Network Management Group Chief to help notify in a timely manner all responsible parties including the Assistant Inspector General for Investigations in the Documents and testimony show the OCIO failed to notify the in a timely manner in April 2015 In fact the 16 found out about the breach by coincidence The OIG Special Agent in Charge SAC ran into OCIO Director of IT Security Operations Jeff Wagner in the hallway Wagner tagged the SAC to meet later in the day at which time the SAC was informed of the rst breach The SAC noticed Wagner on the sixth oor of OPM around lunch time which was unusual because Wagner worked on a different floor The SAC testi ed As I recall it it was tmly a chance encounter I was exiting from the elevator on the sixth oor I was walking down the hallway Jeff Wagner and a coworker -- I don't recall who the coworker was or to this day don t remember -- was walking into the Federal Investigative Service Of ce which is in the hallway of the sixth oor and as I was approaching Jeff waved nodded as I know who Jeff is And Jeff said Hey when you get a chance come down to my of ce And we -- or I continued on into my officer156 The SAC testified that the entire conversation lasted no longer than thirty seconds and that would describe this as a conversation in passing Literally he was walking into an of ce 1 was walking towards my office 95 The SAC testi ed to not knowing what Wagner wanted to discuss at the meeting Wagner requestedw n fact the SAC thought Wagner may have wanted to discuss Federal Employee Health Bene ts FEHB program carriers The SAC stated So I immediately went back to my office and as 1 recall I thought this was in reference to another potential breach- We had the Anthem breach earlier I believe February 2015 March of 2015 you had the Premera Those were large FEHBP catriers We were still trying to sort out what the impact to not only FEHBP subscribers but the as a whole and its nancial integrity I immediately thought this was another breach of a FEHBP carrier when I left Jeff 959 1 on met no of Pets Mgmt Incident Response and Reporting Guide at 12 95-1 9 DIG Serious Concerns Regarding DCID uly 22 2015 at 3 Special Agent Tr12 13 173 When the SAC visited Wagner later that afternoon the SAC learned 0PM had suffered an intrusion Wagner handed the SAC a security incident timeline that included a series of dates and bulletsgw The earliest date was April 2015 and there was an attached deseiiption that stated Zero day malicious activity found 961 The SAC testi ed what immediatelyjumped out to me was internal noti cations were made The FBI was called Also the United States Department of Homeland Security US-CERT team the Computer Emergency Response Team had been called and noti ednw The SAC recalled being shocked that law enforcement was in the building and that the 01G was unawarem With respect to why it was important for the DIG to receive timely notice the SAC stated A There are several reasons why First the Act It s the agency s responsibility to notify the 1G of potential incidents or situations that impact the agency so the IG can timely timely matter of notifying Congress You have the FISMA Act which is the Federal Information Management Security Act which requires noti cation of the appropriate 10 of what I recall of a potential or what I recall and believe it states of a potential situation -- we would be the appropriate 16 in that situation and by their own incident and reporting guide of 2009 The other thing is just basically common courtesy I would expect lef f s of ce -- capecially if you have people walking into the building with guns I m also responsible if there is an active shooter in the building of deploying assets and it can obviously be a very terrible situation if we don t realize what other people are in the building that are armed at that particular time Q So you re saying if other law enforcement officers were in the building -- A Sure Q you would be the one responsible For coordinating with those individuals A Correct Id at 13-14 at 5 15-15 l79 The SAC testi ed that Wagner said 0PM had no intention of notifying the public and that the DIG disagreed with that planfm5 The SAC testi ed that Wagner said there was no need to notify the public and that Wagner believed there was no evidence the agency had lost information to the attackers and that the situation was being carefully monitored By April 22 2015 however 0PM already found evidence of a serious breach 0PM eventually announced that it lost the personnel records of 4 2 million federal employees on June 4 2015 96 The failure of the OCIO to notify the in a timer manner undermines the important role Congress has established for the H35 Like all federal watchdogs McFarland s ultimate responsibility during this time was not to any individual but to the public interest 513 Being prevented from taking part in the investigation into the cyber inuusion from day one hampered the 10 s ability to effectively carry out its work on behalf ofthe public and also undermined the public s trust that the agency was acting in good faith As conveyed by McFarland Failure to include GIG investigators and auditors from the beginning of the incident impeded our to coordinate with other law enforcement organizations and conduct audit oversight activity With respect to the loss of background investigation materials the Special Agent testi ed that the 010 was noti ed unintentionally The SAC testi ed So it was another right place at the right time type of situation On or about May 18 2015 I had received information that there was another breach at an FEHBP carrier this time being CareFirst CareFirst is an extremely large FEHBP carrier and this caused us great concern- I called Jeff Wagner on or about May 13th May 19th that evening asking if he had heard anything about the CareFirst situation The SAC stated that Wagner had not heard anything about CareFirst and they agreed to continue checking-in with each other Two days later on May 20 2015 the SAC saw news about a breach at CareFirst and tried to contact Wagner several times that day m The Special Agent recounted watching the news and deciding to call Wagner The SAC statedrecall it was approximately 6 to 6 30 that night before I was leaving for the day I called Jeff Jeff picks up the phone I was -- almost jumped through the phone as I recall 9 5 Main-13 95-5 a Of ce of Pars Mgmt Press Release 0PM to Notify Employees Incident June 4 2015 available at g1 St topm-to-notifv-e 9'53 Council of the Inapectors Gen on Integrity and Ef ciency It Act History available at last visited June 4 201d ml DIG Serious Concerns Regarding DCID July 22 at 3 Special Agent Tr at It an 1 ta a 19-20 180 saying Jeff have you heard anything about CareFirst And Jeff s initial response was Where are you And I said l m still up in the oliice And Jeff said Ineed to come see you So I met him at the door it was only a few minutes Jeff was obviously in the building It was a few minutes He came up I escorted him into the conference room Jeff sat down And the best way to describe it was it was totally different than the April meeting that had occurred I knew something was up just by his body language and sat down And Jeff initially said They got it Hooked at him and he then repeated They got all of it And I asked the question CarcFirst And he was like no I said something to the effect of How big is this And as I recall Jeff said Homeland Security or US-CERT is down here FBI is down here We had a couple of questions but Jeffjust didn t have a lot of information It was truly different than the April meeting whereas you know we were asking questions eff seemed to be able to respond this one was certain not that way And did he Specifically at this time indicate that background investigation records may have been compromised He speculated that yes they had But we were I was also asking about other systems that are controlled by the Office of Personnel Management but yes Jeff did speculate that background investigations the The SAC testi ed that the scene on May 20 2f 5 was dismal and that it looked like somebody was defeated I mean this was a man who was defeated The shoulders were slouehed and it had obviously been a my recollection from what I recall I would classify as a long day The SAC accompanied Wagner to meet personnel from the FBI and LIE-CERT The Special Agent testified that Wagner said law enforcement personnel were on site and that Wagner willingly introduced the SAC to the law enforcement officials on sitem after the April 22 2015 discussion with Wagner until the May 20 2015 conversation in the OlG s conference room about the loss of background investigation material the two had no 15974 Later that day when the SAC reported the news to 01G colleagues nobody was aware of the cyber investigation that was underway just a few oors The SAC stated that substantial The SAC stated We WIS Id at 20 21 M at 45 emphasis added Id at 21 In at 12 Iniat 45 181 It was just more work was going on in reference to that Our conversations primarily focused on again the FEHBP carriers and nding out more information about the Anthem breach finding more information about Pregmera breach working with the FBI and what information they needed W The 10 s notification to Acting Director Cobert did not follow an isolated incident but rather a series of incidents where it was not noti ed immediately or by the DCIO In addition to failing to notify the BIG about the breaches in April 2015 and May 2015 the SAC also testi ed that the DCIO failed to provide timely noti cation concerning a breach that identi ed on March 20 2 114 at 0PM The SAC stated Q Okay Would you characterize the 10's noti cation of this March 2014 incident as being timely A No Q Would you characterize it as being in keeping with 0PM policy and rules governing noti cation to the A No Q Today we have discussed three separate cybersecurity incidents occurring at 0PM since March 20M From your perspective having been involved with all three events how would you characterize noti cation to the Of ce of Inspector General for these three incidents A I would characterize it as nonexistent There was my opinion there was no formal noti cation to any of these incidents It was -- the rst one the March 2014 we were notified by another agency the April 2015 I was just getting off the elevator and happened to be there and then the May 2015 I proactively reached out to the agency in reference to another issue and that's how we were In summary when McFarland wrote Cobert to raise concerns about the failures to notify his of ce in a timely manner about major cybersecurity events as the Act FISMA and own guidance direct the IG could have cited even more examples The 0010 s repeated failure to involve the DIG eroded the relationship between the two of ces and prevented the 01C from conducting its important work on behalf of the American public it at at 4344 9 at at 25-27 182 Under Incident Response and Reporting Guide the BIG is responsible for providing law enforcement authority and investigative support to any incident handling The Guide makes clear that the GIG must be notified immediately if criminal activity is suspected and that As detennirred by the 010 other law enforcement support may be called in to assist in the investigation of an incident 93l While the guide clearly states the DIG should he an integral part of any law enforcement activity and determine the need for law enforcement support the DIG was not even consulted about the need to bring in law enforcement support for this particular incident response In fact the GIG was prevented from even attending key meetings with other federal law enforcement agencies- McFarland raised these concerns to Cobert- He wrote During the investigation of the second breach involving background investigation les the DIG requested to attend meetings between DCIO staff the Federal Bureau of Investigations FBI and the DHS U S Computer Emergency Readiness Team Former Director Arehuleta stated that the GIG could not attend these meetings because our presence would interfere with the FBI and aces This action is a violation of the hrspector General Act of IQTE as amended 10 Act The DIG contacted the FBI and US-CERT directly and did indeed meet with them without adversely affecting the progress of the investigation These meetings provided the DIG with critical information necessary for our own investigatory and audit work What the former Director considered interference was simply the BIG ful lling our responsibilities 3 1 The SAC told the Committee that on May 20 2015 after Wagner relayed that they got all of it 934 the SAC asked Wagner Can i go down and meet llaw enforcement The SAC testified immediately asked because I did not meet the investigators from the previous breach I wanted to go down introduce myself and meet the investigators 9815 Wagner responded Absolutely no problem and escorted the SAC to a room where a large number of investigators were sitting and that most had been sitting there and had their laptops US Of ce of Pcrs Mgmt Incident Response and Reporting Guide at 3 id 933 DIG Serious lLioneerns Regarding DCID July 22 EtllS at 3- 933 rd at 34 34 Special Agent Tr at 2U 935 at at as 93 at 133 up and running sl The SAC testi ed that Wagner introduced him to the law enforcement of cialsm The SAC offered assistance and Ieft gg The following day on May 21 2015 0PM Director Katherine Archuleta requested a meetingO occur McFarland and his Deputy Norbert Bert 1Vint attended the meeting with with 16 McFarland in the situation room a small room where classi ed brie ngs can Arehuleta and they debriefed DIG staff immediately afterwards will The SAC testi ed that Vint recalled the Director asked McFarland to stop interfering with the investigation The SAC stated My personal recollection as I recall I was stunned at this because the investigator that they were talking about was me I was there that night receiving the noti cation from Jeff I reiterated to both Pat McFarland and Bert Vint that the May 20th date I was trying to get ahold of Jeff There were several times that day I reached out to Jeff I emailed Jeff I called Jeff It was not in reference to this I had no idea this was going on Again I was under the impression that Wagner was working the CareFirst breach and wanted more desperately wanted more information about thisgg have never had a situation where the agency has -- I perceived -- as I recall I perceived it as the fonner Director Archuleta was telling Pat McFarland that he had a heavy-handed agent who was going down there demanding information And as I recall there could be nothing further from the truth That s why it stands out in my mind This is such an outlier of anything or any feedback that has ever come from our of ce And I recognise there are situations where agencies and IGs may not agree but to the point where there was a complaint that aesened we were interfering no I was just stunned by that w' KeyPuint Audit Documents and testimony show the OCIO also interfered with the audits McFarland wrote In October 2014 due to concerns raised after a security breach at United States Investigative Services U315 was identi ed in June 2014In at 46-41 in at at 23 at 993 at 3124 99 9 rd al 25 184 Of ce of Personnel Management 0PM Of ce of the Inspector General DIG informed the OPM Chief Information of our intent to audit KeyPoint Government Solutions KeyPoint At an October 16 2014 meeting the C10 requested that we delay this audit stating that the US Department of Homeland Security DHS had just completed a comprehensive assessment of KeyPoint which was also in response to the USIS breach Therefore she was concerned that our audit would interfere with KeyPoint's remediation activity The BIG tries to coondinatc our oversight work with the OPM program of ces to the maximum extent possible and so we agreed to delay our audit We later discovered however that 0PM became aware in early September 2014 that KeyPoinl had been breached Despite knowing this the CIO did not inform 01G staff of the breach in the October 16th meeting when she requested that we delay our audit work m Our audit which was a comprehensive evaluation of the information technology IT security posture of Key Point was delayed for over three months The DHS review was focused on incident response objectives and did not have as wide of a scope as the CID alluded in fact our audit identi ed a variety of areas that were not part of review where KeyFoint could improve its IT security controls The interference with our audit agenda resulted in additional time passing with these vulnerabilities still present in KeyPoint s environment The delay also prevented us communicating important information that may have been relevant to the recent Congressional hearings regarding the OPM data breaches g This situation is signi cant and a concern because the 01G has a track record of conducting valuable work related to security posture There is no basis legal or otherwise for 0PM of cials to delay or otherwise interfere with the 16 s work Noti cation Concerning New IT Infrastructure The IG alleged the OCIO prevented the from being involved in the development of its new IT in 'asttucnire front the start After a March 2014 cyber incident OPWOCIO launched a project to overhaul IT infrastructure This project involved a multi phase approach including Tactical improving the existing security environment Shell creating a new data center and IT architecture Migration migrating all 0PM systems to the new GIG Serious Concerns Regarding DCIO July 22 21115 at 3 rs oic l- lash Audit Alert June tr 2015 at 5 185 architecture and Cleanup decommissioning existing hardware and The agency awarded a sole source contract for this multi-phased project and the contract was initially managed by C10 Seymourfm The IG stated that the OCIO again failed to work in good faith with the BIG on this initiative McFarland wrote The UCIO failed to inform the 01G of a major new initiative to overhaul the agency s IT environment We did not learn the full scope of the project until March 2015 nearly a year after the agency began planning and implementing the project This exclusion from a major agency initiativg stands in stark contrast to OPM's history of coerperation with our of ce The IG found about the 1T infrastructure Improvement project on March 2 2015 when the Deputy met with the OCIO Chief of Staff regarding a special mding request mm Speci cally the IG learned for the rst time at this meeting that he was expected to pay the agency approximately $1 16 million in FY2015 funds to support the project m1 The OCIG Chief of Staff told the Deputy IG that this would be a one-time assessment but then later was told the assessments would be annual mm The IT Infrastructure Improvement project implicated a significant amount of money In late October 2015 0PM advised the Committee that it had spent approximately $60 million in FY2014 and 2015 on the projectum About eighty percent of the funds originated from revolving fulritlgl and the remaining twenty percent from a variety of discretionary and mandatory lnds areas 5 According to McFarland despite the high stakes of the project for IT security delivery and costs the excluded the 01G McFarland wrote The role of the BIG is to promote economy etticiency and eilectiveness in the administration of the agency s programs as well as to keep the Director Congress and the public informed of major problems and de ciencies Because the GIG was not involved agency of cials were denied the bene t of an independent and objective evaiuation of the 3 Id Imperatis Letter Contract June 16 2014 Attach 1 at 000002 Imperatis Production Sept 1 2015 Ed Attach 1 at 000011 A sole source contract is a contract that was awarded without being subject to the competitive bidding process an DIG Serious Concerns Regarding UCIO July 22 2015 at 4 mm Of ce of Pete Mgmt Background Information 0PM Infrastructure lOverhaul and Migration Project June 2015 on file with the lCommittee 1m Id 1003 me Email from 1 1 5 00 ofPers Mgmt to H Comm on Oversight 3 lt'jov t Reform Staff 0ct 23 2015 on le with the Committee m M requested million in FY2010 to implement and sustain these improvements The FY2016 omnibus requires 0PM to use $21 million ofits $222 million appropriated dollars for IT security improvements 186 project s progress from the beginning The audit work that we have performed since learning of this project has identi ed serious deficiencies and flaws that would have been much easier to address had we been able to issue recommendations earlier in the project s lifecycledm The decision to exclude the It hurt the agency because it lacked information that could have informed the decisionamaking and planning stages for the IT in 'astructure overhaul The project was exposed to waste fraud and abuse partly because of the posture with respect to involving the DIG Five Incorrect andier Misleading Statements McFarland s Jul r 22 20 5 Memorandum cited ve incorrect andior misleading statements to Congress In the public version of the memorandum the descriptions of those ve incon'ect andr or misleading statements were full r redacted At a hearing before a Senate Committee on Appropriations Subcommittee on Financial Servcies and General Government former Director Katherine Archuleta stated that 0PM completed a Major IT Business Case formerly known as the OMB Exhibit 300 for the in 'astmeture improvement projectdm r McFarland also wrote that indicated in response to the flash audit that they have been in eontinua consultation and discussion with OMB the Of ce of Management and Budget regarding this project was According to McFarland however 0PM has not cempleted a Major IT Business Case and has not provided us with any evidence that it has consulted with OMB regarding the full scope of the project and that OMB approved OPM's approach In its June 22 response to the ash audit alert 0PM acknowledged that it has not completed this document and actually disagrees with our recommendation to prepare one After the hearing the DIG again requested documentation supporting OPME statements and again the agency r has failed to produce an r evidence whatsoever that it has It 0MB apprised of the full scope and scale of this project I 9 Serious Concerns Regarding OCH JulyI 22 21315 at 4 m 01' on Technoiogv Spending andr Join Seenrirv Hearing Before Snbeomni on Financial Set-vicar Lt Gen Gov 't after 3 Comm on 14th Cong at 1 40 June 23 5 hereinafter Hearing on DEM In Technologth Spending and Data Security DIG Serious Concerns Regarding UCID July 22 2015 at 5 M 13' Former Director Archuleta testi ed at a June 23 2015 Senate subcommittee hearing that my CID has told me that we have indeed an inventory of systems and data mm According to McFarland however Both our flash audit alert and Fiscal Year FY 2014 FISMA audit noted that 0PM does not maintain a comprehensive inventory of its information technology IT assets We con rmed with the Chief information Of cer C10 on June 23 2015 and again with her staff on June 2911 that 0PM is still in the process of developing a comprehensive information system inventory and this process is not yet complete m Archuleta and Seymour testi ed before the Senate Appropriations Committee and the House Committee on Oversight and Government Reform that the sole-source contract with Imperatis only covered the first two phases of the IT Infrastructure Improvement project and that contracts for the migration and cleanup phases of the project had not yet been awarded m According to McFarland however The document that justi ed the sole-source contract clearly stated that it was intended to he used tor the full scope of the project and that full and open competition would he pursued if and when it became appropriate to do so Further the statement of work contained in the contract itself speci cally states that t he Contractor shall complete the work within this statement of work in four dilferent phases Tactical Shell Migration and Clean Up When OIG personnel met with the OCIO on May 26 20l5 to discuss concerns regarding the use of a sole-source contract for all phases of the project the C10 argued strongly in favor of this approach She informed us that she wanted the same contractor to oversee all four phases of the project for continuity purposes Hearing on 0PM information Technology Spending and Dam at 1 40 BIG Serious Concerns Regarding DCID July 22 20 5 at 5 Hearing on 0PM norm-mans Technology operrding and Data Security at 2 14 former 0PM Director Archuleta would like to remind the Inspector General that contracts for the Migration and Cleanup have not yet been awarded Hearing on 0PM Dom Breach Pair at 2 19 134 former Director Arcliuleta would like to remind the that the contracts for Migration and Cleanup have not yet been awarded And we will consult with him as we do that id at 2 53 0 Seymour that's why we only contracted for the first two pieces and we said as we work through this project to understand it we ll be able to better estimate and understand what needs to move into that Shell GIG Serious Concerns Regarding July 22 2315 at ti 183 During a hearing before the Committee on Oversight and Government Reform in response tn a question about the eleven systems operating without a valid Security Assessment and Authorization Authorization2014 Seymour stated this was no longer a concern because she had granted an interim Authorization to these systems 10 According to McFarland however OMB does not allow interim or extended Authorizations Therefore the ClO s exteusion from the 16 s perspective was not valid and the eleven systems identi ed in the 2014 audit have still not been subject to the Authorization process m At a June 25 2D I 5 Senate hearing former Director Arehuleta stated that 0PM had received a special exemption from OMB related to system Authorization because of the ongoing infrastructure Of ce of Management and Budget CID Tony Scott was unable to confirm this during the hearingm' After the hearing however the found 0MB submitted a request to 0PM for evidence supporting this claim According to McFarland 0PM of cials responded by telling OMB that AJ'chuleta did not make such a statement McFarland found This is incorrect as the statement can be found at timestamp 1 4 ofthe The agency disagreed with McFarland with respect to the truthfulness ot these statements to Congress The 16 s allegations however are very serious and they are supported by documents and other evidence Providing false testimony to Congress is a crime and these statements should be evaluated by the Department of ustiee to determine whether a prosecution may be justi ed Current State of Relationship McFarland wrote to Cobcrt it is imperative that these concerns he addressed if 0PM is to overcome the unprecedented challenges facing it today mm Indeed 0PM has taken actions to improve communication with the DIG Following the July 2015 memorandum Cobert m 0PM Dara Breech Hearing- Befoie H Conn on Chiersigin' ctr Gov 'i Reform 1 14th Cong at 2 213'0 June 16 2015 available at form UPM CID Donna K Seymour Sir I have extended the Authorizations that we had on these systems because we put a number of security controls in place in the See ciao Hearing on 0PM Technoiogv Spending and Date at 1 36 former Director Archuleta can tell you that all but one ofthose systems has been Authorized Hearing rm 0PM Doro Breech Part statement of former Director Archuleta 0f the systems raised in the 2014 audit 1 of those systems were expired One of those a contractor system is preSenlly expired All other systems raised in the audit have either been extended or provided a limited Authorization DIG Serious Concerns Regarding UCID July 22 EDIE at 6 l l id at t H313 Luis mm In' at l 139 instituted regular meetings between the OCIO and DIG to cover key issues such as planning and new projects 21 I In addition to the mi weekly meetings we have recently established between you and ICE Director Meetings and the weekly meetings we have recently established between your senior staff and mine Senior sen Meetings we believe we would also both bene t-from separate regularly scheduled meetings between your IT team and Meetings We propose at the outset that we would meet once a month and can adjust the Frequency as needed We would propose leadership involvement in those meetings whenever possible as well tut Kilt team will come prepared to bricfyou on recent events and progress on ongoing activities and you will have the opportunity to raise any questions or concerns on a regular basis 'l'ypieal agenda items would include but not be limited to a Short term and long-term planning l b Proposed new projects c Updates on ongoing projects gaps in dcliverables and plans to address any such galls ti identi cation and mitigation of any technical issues that might develop e audits and compliance wrw - u J wr vwvj DIG Monro Serious Concerns Jilly 2015 In testimony prepared for a February 2016 Committee hearing that was canceled following the resignation of OPM C10 Donna Seymour two days prior Acting Inspector General Norbert E Vint stated The productivity of those meetings has improved over time and through these meetings we have been able to work through celtain issues The OCIO has also begun to consult with us more often such as when they instituted the recent Authority to Operate Sprint 323 Vint stated the relationship improved under Cobert and that there were no further problems with respect to accessing information 323 Vint was prepared to testify that Consequently we have no reason to believe that they have intentionally provided us with inaccurate information or withheld material fleets m I ll 33 ops Dora Breeches Part or Hearing Before a Comm on Oversight Gov 'r Reform 114th Cong Feb 24 2616 prepared statement of Norbert E Vint Of ce of Inspector Gem US Of ce of Pers Mgmt hearing cancelled 113 190 Cotter testifies about the agency is relationship with the Inspector General before the Committee on Matt-F i3 sore It is also noteworthy that Cohort added cyber talent to the agency 125 McFarland attributed improvement in the OCIO-OIG relationship to one of these staff additionsl z'5 On November 4 2i Cobert announced the addition ofCli on Triplett to the OPM cvber team m Reporting directly to Cobert is tasked with advancing the state of enterprise architecture and eyberseeurity including information technology investments capabilities and services'mE Working alongside CID currently Acting CID Lisa Sehlosserum Triplett supports the ongoing response to the 2015 incidents completing the development of plan to mitigate future incidents and recommends further improvements to best secure 0PM's architecture m Triplett has thirty years of broad executive management experience including work on Top Secret and other advanced technologies in the protection and defense of the US Nuclear Command and Control Systems m Vint s draft testimony stated that Triplett helped to mend internal relationships lv int s testimony stated We believe that the new Senior Cyber and Infonnation Technology r Adviser Clifton N 'l'riplett has helped facilitate this improved 115 US Of ce of Pers Mgmt Press Release 0PM Director Announces Key New Cyhei'Arl'trixar Nov 4 w oprn go vinewsrrelea sesfl 1 5i 1 Hop m-director-announces- key-new-c veer ad visor-2f 0 an Date Breaches For HI Hearing Before H Forum on Oversight d c Gov r Reform 114th Cong Feb 24 2016 prepared statement of Norbert E Vint Of ce of Inspector Gen U S Of ce ofPers Mgmt at 5 hearing cancelled m LLS Of ce of Pete Press Release 0PM DirectorAnnounces Kev New Cyber Adviser Nov 4 2015 1 5 cv ber-advisor Ei Id 1039 U S Office of Pers Mgmt Lisa Sein'osser Acting Cirieffry ormorion D icer May 11 201 5 mm 1 3 3 Of ce at Pets Press Release 1PM Director nnonncar Key New Nov 4 5r 1 at 191 relationship as well as create additional avenues of communication between the OIG and the agency s IT staff It appears that Triplett s role is to provide high level advice to assist the Acting Director in developing a strategy to address the multitude of IT challenges facing 0PM I and other senior OIG officials meet with Triplett on almost a weekly basis From what we understand he agrees with the 016 that the agency needs to have a comprehensive plan moving forward that would include a short-term plan to address the needs of critical IT systems as well as a long- terrn plan for the implementation of agency wide Infrastructure Improvement Cohen testi ed that the relationship had improved from her perspective In response to a question from Rep Mark Meadows R-NC at a hearing on May 13 2016 Cobert testi ed We have been working across the agency to strengthen our effectiveness ofour dialogue with the CIO and I believe we ve made real progress in a number of different areas We ve set up a cadence of regular conununieations at my level with the Inspector General currently Acting Inspector General- On a bi-weeklyr basis we meet and get an overview of the issues We have speci c working teams that meet on a periodic basis as well - both around the C10 around procurement we ve set up that same kind of mechanism on the stand-up of the NBIB given the oversight issues there and wanting to make sure we get those right So I think we ve made considerable progress in terms of the dialogue the clarity of the communications We welcome their input on what we could be doin as better As we welcome input from our colleagues here and elsewhere 33 Cobert characterized the relationship as much While the OIG reported being pleased that communications have improved the of ce was still concerned about overall IT strategy lms 1 v int conunitted that the OIG would continue to monitor the activities and work with them to ensure that actions discussed at meetings are in fact implemented and implemented in accordance with proposed timelines m36 333 OFM Dara Breaches Parr Hf Hearing Before H Serum on Oversight dE Gov 'r Reform 114th Cong Feb 24 2016 prepared statement of Norbert E Vint Of ce of Inspector Gem U S Office of Pers Mgmtat 5 hearing cancelled m3 Incorporating Social Media into Federal Background Investigations l'Iearing Before Subcomm on Gov t Operations and Snbcontrn on Hat l Sec of the H Comm on Oversight div Government Reform 1 14m Cong at 12 35 May Iii Hilfi investi gationsf own Hater Breeches For Hearing Before n Comm on hersighr a crow Reform 1 14th Cong Feb 24 201 3 prepared statement E Vint Office Gen US Office of Pers 5 hearing cancelled nus m 0PM Darn firs-riches For Hearing Before H Comm on Oversight Iii Ger 't Reform 1 14th Cong Feb 24 Z lti prepared statement of Norbert E Viol Of ce Gem US Of ce of Pers Mgmt at 5 hearing cancelled 192 Summary of DIG and OCIO relationship Federal watchdogs play a critical role in the federal government one that is statutorily- driven by the Inspector General Act of 1923 Despite the key role Gs play the relationship between 0PM OIG and its OCIO became strained while Katherine Archuleta served as Director and Donna Seymour as CIO Despite serious concerns raised by the OIG in July 2015 and despite concerns raised by Congress about Seycnour m Acting Director Cobert maintained support for Seymour allowing her to hold a leadership role until her retirement on February 22 2016 133 Overall however the relationship with the steadily improved under Acting Director Cobert s leadership and today is reported by both entities to be without con ict m The future effectiveness of the agency s information technology and security efforts will depend on a strong relationship between these two entities moving forward m Letter from the Hon Jason Chairman H Comm on Oversight do Gov't Reform to the Hon Beth Cobert Interim Din Of ce of Pets Aug 15 2'015 Letter from 13 Members of Congress to Barack Obama President United States June 26 5 raising concerns abetlt OPM Director Katherine Archuleta and OPM Chief Information Of cer Donna Seymour Aaron Boyd 0PM Cit Seymour Resigsrs Days Before Oversight Hearing FEDERAL TIMES Feb 22 2 lt'i available at ht 2 1 322th cio-se mour- Qt Billyr Mitchell U ice ofPersoanel Management CID Donna Seymour Retires FEDSCDUP Feb 22 2 16 available at Ian Smith 0PM Donna Sawtonr Resigns PEDSMITH Feb 22 2015 available at resignsi OHM Data Breaches Fart ill Hearing Before H Comm on Oversight dl Gov 't Reform Cong Feb 24 2G I In prepared statement of Herbert E Vint Of ce of Inspector Gen US Of ce of Pcrs- Mgmtat 5 hearing cancelled Incorporating Social Media into Federal Background lat-artigatiaas' Hearing Before Snbaomm on Gov 't Operations and Sabeomm on Not See oftlre H Comm on Oversight ti Gov 't Reform 114th Cong May l3 2t 193 Chapter 8 The IT Infrastructure Improvement Project Key Weaknesses in Contracting Approach On March 20 2014 informed 0PM that a third party had cxfiltrated data from network 14 In response to this discovery and after identifying serious vulnerabilities in the 0PM network the agency initiated the IT Infrastructure Improvement project Seymour testi ed before the Committee that this project began as a consequence of the March 2 14 incidentlw This project was intended to quickly secure legacy IT environment with the urgent procurement of security tools Tactical phase I and to fully overhaul IT infrastructure with a new IT environment that included security controls building the Shell phase 2 After building the new IT environment the Shell the plan was to migrate entire IT infrastructure into the new IT environment Migration phase 3 and then decommission legacy IT hardware and systems Clean Up phase 4 In June 2014 0PM made a sole source award to Imperatis to execute this project 4 As of May 2016 multiple security tools have been purchased some with only limited due diligence to secure legacy lT environment and a new lT environment has been built the Shell After the agency paid a contractor over $45 million for the Tactical and Shell phases the June 2014 contract was terminated in May 2016 and as the IG predicted 0PM had two IT environments legacy and the new Shell to maintain 343 Meanwhile 0PM continues to address concerns first raised by the 1G in June 2015 about contracting approach Speci cally the IG expressed consent that this investment was made with limited consideration of alternatives and without a full understanding of the scope of existing IT assets and potential costs to execute the entire projectlm The taxpayers return on this investment is now further in question after the creation of the National Background Investigations Bureau NBIB which will absorb existing Federal Investigative Services and new that the Department of Defense will assume the responsibility for the design development security and operation of the background investigations IT systems for the m These developments present a binding challenge for this project because 0PM initially planned to rely on funds from revolving fund 14 June 2014 0PM Incident Report at Breach Hearing Before rat a Comm on versighr Gov Reform 114th Cong June 16 3315 testimony of Donna Seymour ChiefInformatien Of cer Of ce oFPersonnel Imperatis Letter Contract June 16 2014 Attach at Imperatis Production Sept 1 2015 43 DIG Flash Audit Alert June 2015 at 5 stating in this scenario the agency would be Forced to inde nitely support multiple data centers further stretching already inadequate resources possibly making both environments less scourE and increasing costs to Email from mperatis to 11 Comm on Uvelsiglit Er Gov t Reform Majority Staff unc T 201-5 con rming total paid to lmperatis from June 16 21314 to May a 2016 is $45 1 million on file with the Committee mo Flash Audit Alert June 17 sols 145 White House Press Release The Way Forward for Fedora Background Investigations Jan 22 2 16 w whitehouse gow blogt 2t 1 so way f orward Federul baek ground -i nvestigations 194 which is largely derived from background investigation fees 0PM collected from other an agencres The documents and testimony show IT infrastructure project would have benefited from more robust communications with the 1G particularly in responding to cybersccurity incidents Former 0PM CEO Donna Seymour testified she was not aware of a requirement to notify the IQ of every project that we take Oilfim Given the significant funding for the IT Infrastructure project which initially had an overall estimated cost of $93 million the agency-wide nature of this project and the fact that this project was launched as a consequence of the 2014 data breach 0PM should have involved the DIG so that the expertise of his of ce could help the agency deter problems before they arose Because agency did not communicate with the on the front end 0PM found itself spending signi cant time and effort responding to concerns after the fact In this case the 1G found out about the project a year after it was launched 343 Shortly thereafter the 16 issued a Flash Audit Alert that contained serious concerns m The 1G and 0PM continue to have discussions about these concerns The documents and testimony show there should be pre established contract vehicles for cyber incident response and related services Instead of issuing a sole source contract to facilitate the procurement of security tools to secure a compromised IT network in the midst of an emergency situation and without the bene t of competition there should have been a goverrunent wide contract vehicle already established to ful ll this need Just as emergency preparedness of cials learned the value of establishing contract vehicles to support emergency response to natural disasters prior to such disasters after Hurricane Katrina so too should similar resources be established for responding to cybersecurity emergencies m The state of IT legacy envirortment leading up to the 2014 and 2015 breaches illustrates the pressing need for federal agencies to modernize legacy IT in order to mitigate the cybersecurity threat inherent in unsupported end of life 1T systems and applications The GAO recently observed that in cases where vendors no longer support hardware or software this can create security vulnerabilities and additional costs 3'5 In testimony before the Committee then- UPM C10 Seymour admitted the vulnerability ot'OPM's legacy She stated mm Data Breach Fart Hf Hearing Before the H Comm on Oversight riGov Reform Feb 34 2016 prepared statement of'Norbert E Viol Office of inspector Gen U S Of ce of Pers Mgmt hearing cancelled 4 GFM Data Breach Part Hearing Be tre the H Comm on verst'ght 601 Reform 114th Cong June 24 2015 testimony of Donna Seymour Chieflnfo Officer LES Of ce of Fers m LLS Office of Personnel Management Of ce of Inspector Gen Eackgrotmd Ityformatt'on 0PM ttL astraetare Overhaai antiI Migration Project June 17 Bill 5 on le with the Committee ore Flash atlas Alert June 17 ants '50 In Detobcr an 5 OMB released a Cyberseeurity Strategy and implementation Plan that reported an effort to establish a contract vehicle in order to develop a capability to deploy incident response services that could be used by agencies on an expedited basis Memorandum from Shaun Donovan Din and Tony Scott Fed Chief Info Of cer Of ce of Budget Exec Of ce of the President to Agency Heads Strategy and implementation Pianfor the Fedora Civitian Government Clot 3D 2015 available at titted pdi' m Gov t Accountability D ice GAO-164 63 act'srnt Agencies Need to Address Aging Legacy Systems 2 May 2D 16 195 0PM has procured the tools both for of its databases and we are in the process of applying these tools within our enviromnent But there are some of our legacy systems that mayr not be capable of accepting those types of in the environment that they exist in today 1 52 Further in making the ease for updating aspects ofOPM s legacy IT environment in the context of this contract Imperatis said certain servers could no longer be patched and hardware had to be replaced in order to mitigate the risk of catastrophic failure since the cun'ent hardware was woefully out of 53 The need to modernize is clear however the modernization of such systems should not he done through a sole source contract in an emergency situation and without a full assessment of alternatives and understanding of the scope and cost of such an effort The IG Issues a Flash Audit Alert and Interim Reports on the IT Infrastructure Project On one 2015 the IG issued a Flash Audit Alert to then-Director Katherine on the sole source 1T contract to secure and update legacy IT infrastructure 154 The 16 raised serious concerns about this project and identi ed substantial issues requiring immediate action and urged the C10 to immediately begin taking steps to address these concerns mss McFarland wrote 0 ur primary concern is that the OCIO has not followed the US Office of Management and Budget OMB requirements and project management best practices - the DCIO has initiated this project without a complete understanding of the scope of existing technical infrastructure or the scale and costs of the effort required to migrate it to the new environmentlm McFarland also expressed concerns with the nontraditional Government procurement vehicle that was used to secure a sole-source contract with a vendor to manage the infrastructure 105 overhaul These two themes lack of project management and the sole source contracting approach have been present throughout the 16 s oversight of this project with varying levels of cooperation from 0PM Over time and more recently 0PM of cials have become mote responsive to the 16 s concerns particularly as new 0PM leadership was put in place m 0PM than Breach Hearing Before the it Comm on h'ersight it Gov 't Reform 1 14th Cong June 16 2015 of Donna Seymour Chief Information 'Jf cer Of ce of Personnel '53 Email from_ Tmperatis lo Donna Seymour Chief Info LLB Of ce of Pers Mgmt July 31 2014 3 13 Attach its at lt i3 Imperatis Production Sept 1 2015 Email from Dir Stragctic Growth Imperatis to_ US Of ce ofPers Mar El EDIE 3 12 pm Attach 9a at 10 lmperatis Production Sept 1 2015 etc Flash Audit Alert June 17 2015 55 Id at 1 Ian ItlS M- 196 With respect to the project management concerns the 16 observed at the time that 0PM had not identi ed the full scope and cost ofthis project and had not prepared a Maojor IT Business case document which is an OMB requirement for major IT investments 1 53 As a result ofthe inadequate project management the IG found a high risk that this Project will fail to meet the oh'ectives ofproviding a secure operating environment for 0PM systetns and applications $59 The 16 recommended that 0PM complete the Major IT Business case document as part of the FY 20 budget process m The 10 predicted the failure to plan and understand the full scope of the project also would introduce schedule and cost risks For example 0PM did not have a complete IT inventory of existing applications and systems for migration and redesign 352 In addition the cost estimate at the time for the Tactical and Shell phases was approximately $93 million and did not include the cost of migrating legacy applications to the new environment ' 53 The source of funding was also unclear The 1G stated when we asked about the funding for the Migration phase we were told in essence that 0PM would find the money somehow and that program oilices wpliiglgl be required to fund the migration of applications that they own from their existing budgets With respect to the sole source contract award issue the IG questioned the use of a sole source contract for all four phases of the network infrastructure improvement project 1 5 The acknowledged that the sole source approach may have been appropriate for the rst Tactical phase of the project given the immediate need to secure the legacy IT environment The IG did not agree however that it was appropriate to use this sole source contract for all four phases of the project Chairman Chaffetz raised those concerns in a June 24 2015 hearing lie stated when it is a sole source contract it does beg a lot of The 1G recommended against using a sole-source contract for all four phases cfthis project because without submitting this project to an open competition 0PM has no benchmark to evaluate whether the costs charged by the solewsource vendor are reasonable and appropriate mm On June 22 20 S former Director Katherine Archuleta responded to the G s Flash Audit Alert and generally disagreed with G s concerns mm She argued that a business case was 053 our Flash Audit Alert June 17 2015 at a 11159 Id at s 35Inf-3 I test M at as 'MMmi m Hearing on OFM Dam Breach Parr Statement of Chairman Chai f ctz mo Flash Audit rum rune W 21315 a s 'm Memorandum from Katherine orchuleta Din LLS Of ce of Pets Mgmt to Patrick McFarland Inspector Gen S- lliil'fl icc ofT- ers Response to insir A edit A iert US U ice ofPersonnei alfonogemenf 's 197 not necessary and would take too long With respect to the concern that 0PM lacked a full understanding of the size scope and cost 0PM said and the OCIO have always been very clear that the unclenaking includes factors and costs that will be understood more clearly as the Project proceeds essentially we will gure it out as we 0PM also disputed the 16 s characterization of the contract as a sole-source award covering all four phases of the IT Infrastructure Improvement project and took the opportunity to state the contract for the Migration and Cleanup phases of the infrastructure improvement project have not yet been awarded n The Concerns Continued through the Fall of 2015 On September 3 2015 the DIG released an Interim Status Report on the Flash Audit Alert mu The 016 s Interim Status Report acknowledged developments related to this effort that in the 16 s view emphasized the need for a disciplined project management approach mn Such developments included former Director Archuteta s resignation Senate appropriators rejection of million funding request for accelerated migration of IT systems in July 2015 and the fact that 0PM had identified serious security vulnerabilities in several systems including which is the electronic questionnaire systems for background investigations m 4 In the Interim Status Report the IG reiterated the recommendations in the original Flash Audit Alert and pointed out that 0PM has not yet determined the full scope and overall costs of the Project and without completing a Major IT Business Case proposal for the Project the concluded there is a high risk of project failure jms Further the 10 said the sole source award for all four phases and the original justi cation for making such an award violate d federal acquisition regulations because any involvement that is not required to eotreet the urgent and compelling circumstances would not be Ijusti ed under the urgent and compelling exception authorizing certain sole source contracts m6 IG Reports Progress in Responding to Concerns but Challenges Remain as of May 2016 Almost one year after the 0PM lG issued a Flash Audit Alert on IT Infrastructure Improvement project Acting IG Norbert Vint issued the Second Interim Report on this project in Improvement Project Report No June 22 Arehuleta Response to IE Flash Audit Alert Archuleta Response to one Flash Audit Alert Office of the Inspector Gen US Of ce of Personnel Mgmt Report No art croo 1 5 355 Interim Status Report on Responses to the Finish Audit Alter US J ice ofI ersonnel' iltniiogemem 's In Jrovenmnt Project SopL 3 2o153 hereinafter GIG Interim Status Report Sept1-2 ' iira at 2 5 1015M at I emphasis in original citing 4E sees 41 U S C 198 May 2016 The Acting reported some progress with submission of a major IT Business Case during the FY 201' budget process but the Acting also said there were lingering overall concerns about the project related to the insuf cient capital planning process and unsubstantiated lifeeycle cost estimates m The Acting IG made two recommendations 0PM should conduct an Analysis of Alternatives to determine whether the Shell which is now known as Infrastructure as a Service or IaaS is the best approach to modernizing the IT environment given changes in the internal and external environments and 2 0PM should continue to leverage the application profile scoring framework developed boy 0PM in order to develop reliable cost estimates for modemiaation and migration activities i In May 2016 the Acting IG reported that 0PM had submitted a Business Case for this project as part of the 201 budget process in response to the Ga prior recommendation However after reviewing the document the Acting 16 said the document was insuf cient because 0PM did not perform capital plamring activities such as a perfonning an to the ShellfIaas and had not developed a solid cost estimate for modernization and migration mm The Acting It said 0PM still had not determined the full scope of the project but there had been some improvement in developing an inventory of legacy systems and estimating costs to modemize these systems 103 in addition the Acting 16 identi ed a new complication to funding the IT Infrastructure improvement project Speci cally the decision to create the NBIB and designate the Department of Defense as responsible for the IT systems to support the background investigation process altered the potential funding options 0PM had planned to rely on its revolving fund which is primarily funded through revenues from the background investigation process to support the IT Improvement project m3 With the creation of the N313 the background investigation processing function will no lon er be part of the Consequently this funding source is no longer available 53 The Acting concluded that while it was not too late for 0PM to complete the capitol planning activities which should have been done prior to project initiation the IG remains concerned that there is a very high risk that the project will fail to meet its stated objectives of delivering a more secure environment at a lower cost m On April 22 2016 Acting CID Lisa Schlosscr offered response to the Second Interim Report and said OCIO appreciates the detailed analysis and feedback provided in the report and generally tumours with the recommendations m The m Office ofInspector Gem u s Office of Pers Mgmt Report No 4a-cr-oo-1s-osr Status Report on the US Mice ofPersomref s Project Major IT Business Core May 18 2016 hereinafter DIG Second Interim Status Report on Infrastructure Improvement Project May IS 2016i 1013 IdI033 Id at '35 US Of ce nl'Personnel Mgml Acting Chief Info Of cer Lisa Schlosscr Response Apr 22 ti to Office of Inspector Gen US Of ce of Pers Mgml Report No Second Interim Status Report on the US 199 Response then proceeded to provide details on ongoing efforts and planned next steps to address the IG reconunendations For example the Acting CID said 0PM has engaged in on-going efforts to inventory IT systems and identify plans to mitigate migrate or modcmiac these systems 1 8 Further 0PM agreed that this project would bene t from a more rigorous lifecycle cost estimating process and pointed to a plan to use an application pro le framework developed by Senior Cybersecurity and IT r tdvisor to inform lifecycle cost estimates for IT modernisation 337 In sum 0PM has come a long way from the state of affairs in June 2015 when the 16 released the Flash Audit Alert on the IT Infrastructure Improvement project Today 0PM is currently working cooperatively with the to mitigate concerns raised by the IS The agency appears to be making progress on completing basic capitol planning activities that should have been completed prior to the launch of this project and these efforts should be acknowledged However the if continues to have concerns about this project and unfortunately some of the risks identi ed early on by the IG seem to have played out during the course of the Iinpcratis contract The Story of OPIWs IT Infrastructure Improvement Project and the Sole Source Contract Over the past two years 0PM has made progress toward securing legacy environment and building a new IT environment but there were significant concerns raised by IG about the IT Infrastructure contract that were validated and expanded upon based on review of the documents obtained by the Committee which included more than LTUU pages of documents from lmperatis The agency did procure updated security tools to secure the legacy IT environment although not all such interactions were handled through this contract including Cylance and the new IT environment Shellilaas that lmperatis built appears to be an improvement over the legacy 1T cnvirorunent However there were schedule and cost challenges as the IG wanted and questions remain as to how 0PM will realize the bene ts of new Shell aaS and at the same time maintain the legacy IT environment in a cost effective way Further 0PM has no clear assessment of whether the costs paid to data under this contract over $45 million were reasonable given the lack of competition for the contract Finally the long-term plan for securing and modernizing IT environment remains unclear especially given ongoing efforts to complete an analysis of alternatives and establish reasonable cost estimates for modernisation The following is a timeline of events related to the IT Infrastructure Improvement project contract and more details that validate some of the concerns initially identi ed by the IG U ica ofPersomte 's in 'osn'ncmra Improvement Project Major ITanfness Case at hereinafter Schlosser Response to Second Interim Status Report mg Schlosser Response to Second Interim Status Report Apr 22 201 3 at l toss M at 3 200 Timeline IT Infrastructure Improvement Proiect - May It 2014 Then-0PM C10 Donna Seymour contacts former colleagues who she knew from her time at the US Maritime Administration around 2006 at Imperatis about thelgaseeurity situation at OPM and a potential IT project to address the situation - May 22 2014 In response to the malicious activity identi ed in March 2014 OPM executes the Big Bang remediation plan Director of IT Security Operations Jeff Wagner and team members provided an unclassified brie ng to Imperatis employees '39 In June 16 2014 Letter contract statement of objectives for Imperatis contract describes activities under the contract in all four phases of the IT Infrastructure Improvement project mm The base year of the contract plus options included a period from June 2014 through December 2016 Initially $13 million was allocated under the letter contract - Iune 22 2014 issues the OPM Incident Report and makes fourteen recommendations to improve IT security including a general recommendation to redesign their network architecture to incorporate security best II October 14 2014 Solicitation for IT Infrastructure Improvement contract issued as part of the process to de nitize the June 2014 Letter contract 192 - November 12 2014 Imperatis submits a proposal in response to October 14 2014 solicitation II January 30 2015 contract for IT Infrastructure Improvement project is de nitized l I February 2015 OPM FY 2016 Congressional Budget Justi cation requests $21 million to implement and sustain agency network upgrades initiated in FY 2014 and security m Email from Donna Seymour Chief Info Of cer U S Of ce of Pers Night to Patrick Mulvartey and lrnperatis May It 2014 9 46 Attach 12 at 4301463 Imperatis Production Sept 1 2015 Letter front Maj General ret Mastin Robeson President 3 Chief Exec Of cer Imperatis Corp to the Hon Jason Chaffetz Chairman H Comm on Oversight Er Gov t Reform Sept I 2615 at S m Imperatis Letter Contract June 16 2314 Attach I at DUUDUE Imperatis Production Sept 1 2015 0PM used a DHS contract vehicle but the former CIO Donna Seymour 1rrvas designated the contracting of cer representative COR and thus was responsible for contract performance management In' at I designating Ms Seymour as June zero OPM Incident Report at Hoeaesrs-omzso 93 Letter from Mari General reL Mastin President a Chief Exec Of cer Imperatis Corp to the Hon Jason Chaffetz Chairman H Comm on Oversight 3t Oov t Reform Sept 1 2t 5 at 9 193 Irnperatis Proposal Volume I Statement of Work and Technical Attach 5 at DDUITS Imperatis Production S gpt 1 2015 lrnperatis De nitized Contract Jan 30 212115 Attach 2 at Production Sept I 2015 201 software maintenance to ensure a stronger more reliable and better protected 0PM network architecturem gs I March 2 2015 hnperatis coordinates initial meeting with CyTech and 0PM to evaluate CyTech s tool for possible use in the new IT Infrastructure the Shell ma I March 2015 GIG becomes aware of the IT infrastructure Improvement Project when the UCIO meet with DIG to discuss the special assessment the OCIO would be collecting from all 0PM program of ces to partially fund the project I April 2 2015 Cy feeh meets with Imperatis and 0PM at CyTech of ce in Manassas 398 I April 15 2015 0PM noti es regarding potential indicators of compromise my I Aptil 21-22 2015 product demonstration at 0PM facilitated by Imperatis ' I June 15 2015 The rst six month option to continue Shell phase 2 work is exercised This option expired December 15 2015 1 '01 I June 16 2015 The Committee holds rst hearing on the UPM data I June 2015 McFarland issues Flash Audit Alert to then-Director nachulcta to aleit her to serious concerns the has regarding the OCIO infrastructure improvement project The IG nds OCID launched project without a complete understanding of the scope of existing technical infrastructure or the scale and costs ofthe effort required to migrate it to the new environment The 16 also expresses concern that a sole source contract award had been made 3 '03 ms 1 1 5 Of ce oI'Pers Mginl Congressional Budget Justi cation Performance Earliest 1 20 ti at 2 Feb 2015 available at Imperatis Weekly Report Mar 30 2015 r tpr 3 2015 Attache at 000204 Imperatis Production Sept 1 2015 mm U S Of ce of Personnel Management Office Gen Background Itgfotartatiott ' 0PM iry rasn-nemre Gueritani and Migration Project June 201 on le with the Committee ma Imperatis Response to H Comm on Uverisght ti Gov t Reform Majority Staff Regarding Clari cation on Sept 1 2015 Production Sept 10 2015 on file with the Committee mg MR Timeline Unknown SSL Certi cate April 15 2015 at 0PM Production Apr 29 2016 1m Imperatis Response to H Comm on Overisght 3 Gov't Reform Majorin Staff Regarding Clari cation on Sept 1 2015 Production Sept 10 2015 on le with the Committee Memorandum from the Hon Belh Cohort Act Dir U S Of ce ofl ersonnel Mgmt to Patrick McFarland Inspector Gen 1 1-5 Of ce of Pers Mgmt Response to interim States Report on Responses to the Flash Ano'it Aiert US Mice oft ersonnei Management 3 improvement Pian Report No reason 15 055 Sept 9 2015 at 3 03 0PM Data Breach Hearing Before the it Conan On Usersignt and fiat Reform 114th Cong Jane 16 2015 11 on Flash Audit Alert June 17 2015 202 - June 22 2015 Then Director Archuleta responds to G s Flash Audit Alert regarding the IT Infrastructure Improvement Project 0PM generally disagrees with the recommendations in the Flash Audit Alert saying there was no time to do a business case and activities associated with the Shell are extensions of existing IT investments ' 'm I June 24 2015 The Committee holds a second hearing on the 0PM data breach Thene C10 Donna Seymour testi es we only contracted for the rst two pieces of the four- phase IT Infrastructure Improvement prog'eet She also says the estimated cost of the initial project phases was $93 million 1m I July 22 2015 0PM IG McFarland issues a memorandum to Acting Director Cobert on serious concerns regarding the CIO including statement to Congress that she was not aware of a requirement to notify the 16 of every project we take on in response to a question about the IT Infrastructure Improvement project and incorreeti mislcading information provided by 0PM on the sole source contract ' '0 - Augpst IS 2015 Committee sends letter to Imperatis requesting information about the IT Infrastructure Improvement project I 4- September I 2015 Imperatis provides documents to the Committee in response to August 18 requestum - September 3 2015 BIG issues Interim Status Report on the Flash Audit Alert on IT Infrastructure Improvement project I mg a September 9 2015 Acting Director Cobert responds to the G s September 3 Interim Status Report on IT Infrastructure Improvement project I September 2015 Imperatis completes buying cybersecunty tools to secure the legacy IT environment Tactical Phase 1 1 m Archuleta Response to DIG I lash Audit Alert Us Hearing on Dotti Breech Part II testimony ofDonna Seymour Chief Info Officer US Of ce of Pers- mt DIG Serious Concems Regarding DCID July 22 2015 m Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight St Gov t Reform to Major General Incl Muslin Robeson President St Chief Exec Officer Imperatis sing 18 2015 Letter from Maj Genera retJ Mastin Robeson President ChielExec Of cer Imperatis to the Hon Jason Cha etz Chairman H Comm on chrsight Sr Gov t Reform Sept 1 2015 DIG Interim Status Report Sept 3 2015 Memorandum from the Hon Beth Cohen Act Dir US Of ce ofPersonnel Mgmt to Patrick McFarland Inspector Gen U S Of ce of Pets Mgn rt Response to Interim Status Report on 0PM 's Responses to the Wash Audit Atari U S O i cs ry'Pst-sotme Management '5 firtprovemcnt it th Report No 055 Sept 9 2015 Imperatis Response to H Comm on Ctverisght Se Gov t Reform Majority Stall Questions on Status of the Project Feb 12 2016 on le with the Committee 203 - September 28 2015 Imperatis completes initial operational capability of the Shell Phase 2 Imperatis had planned to complete Full Operational Capability early summer 2016 Performance tuning and staff training on new technologies for the Shell were planneldl It continue through the end of the contract period of performance December 2016 I October 15 2015 lmperatis provides briefing to Committee staff on their interactions with CyTeeh and status of the IT Infrastructure Improvement project - December It 2015 Chairman Chaffetz calls for Seymour to resign for the sixth time citing in addition to previous concerns IT Infrastructure Improvement project 3 I Januagy 22 2016 The White House announces the creation ofthe NBIB which will absorb existing Federal Investigative Services FIST1 and stated the Defense Department will assume the responsibility for the design development security and operation of the background investigations IT systems for the I '4 Februagg 24 201d 0PM Acting IG Norbert 1v int prepared testimony for a Committee hearing entitled 0 PM Data Breach Part canceled and highlighted continuing concerns about the IT Infrastructure Improvement Project and the sole source 1 I I5 contract In April 22 2016 0PM Acting CID Lisa Sehlosscr issues a memorandum to the BIG responding to a draft of the Second Interim Status Report on the IT Infrastructure Improvement project and outlining next steps to implement the Ga recommendations I a - Ma 6 2016 lmperatis reports payments from 0PM totaling million for the period June 16 2014 through May 6 2016 1 I May 9 2015- 0PM terminates lmperatis contract for nonperformance lmperatis is precluded from public comment due to Non Disclosure Agreement with IE I imperatis Response to 11 Comm on Uverisght Gov't Relbnn Majority Staff Questions on Status of the Project Feb 12 21216 on le with the Committee Letter from the Hon Jason Chaffetz Chairman 11 Comm on lDversight 3c Gov t Reform to Beth Cobel't Acting Din U S Of ce of Pers Mgmt Dee l0 2-315 White House Press Release His Way Forworrffor Feds-rot Background Investigations Jan 22 2 316 available at segovlblog on 1L2 0PM Doro Breeches Part Hearing Before H Comm on Oversight it Gov 'rf Reform 114th Cong Feb 24 2 115 prepared statement of Norbert E Vial Office ofInspeetor Gem U S lDf ce of Pet's Mgmt Ecaneclled 5 Schlosser Response to Second Interim Status Report Apr- 22 2tllt'i n Email from Impertis to H Comm on Oversight 3s Gov t Reform Majority 2016 on file with the Committee 3 Jack Moore Connector on Ul rtf's year Upgrades Snddenfv Quits Citing Financier Distress May 13 2016 available at 123103 Based on information provided to the Committee 204 I May 13 2016 The Acting IO issues the Second Interim Status Report on the IT Infrastructure Improvement project noting continuing conccm regarding the lack of critical capital project planning practices required by OMB for this project but also noting some positive actions by OPM '9 I June 2016 Original end date for the rst option period for the Imperatis contract I December 2016 Original end date for the second option period for the Imperatis contract 0PM Initiates Contact with Imporatis and Awards Sole- Source Contract On May 10 2014 then OPM 310 Donna Seymour initiated contact with two lmperatis employees with whom she had previously worked on a prior IT project at the US Maritime Administration 1 '10 She explained that she was looking for assistance to help straighten out a very messy network with poor security '21 Initially Seymour offered to hire one of these individuals as an OPM employee but he declined citing a commitment to his supervisor at Imperatis and offered instead to provide assistance as an expert consultant I '22 Seymour said she investigate potential options for such assistance adding wantr need you on the team 0PM and Irnperatis continued discussions about the scope of the project and potential costs through late lvlay Then on May 22 2014 Imperatis received an unclassified brie ng from Jeff Wagner Director of IT Security Operations and members of the team regarding the network security incident OPM learned about in March 2014 1 '25 In a letter to the Committee imperatis told the Committee that this briefing conveyed an urgent and compelling need for immediate action on both the operational network and for the development of a new separate and distinct information systems architecture % the contractor may be experiencing nancial dif culty due to an accounting issue for a separate and unrelated contract with another agency '1 Oltj Second interim Status Report on Infrastrcuture Improvement Project May 18 2016 Email from Donna Seymour Chief Info Of cer 1 1 3 Of ce of Pers Mgmt to Patrick Mulvaney Senior IT Manager and_ Dir of Strategic Growth Imperatis May 10 2014 9 46 am Attach 12 at 001463 Ilnperatis Production Sept 1 2015 lIEl 23 Email from Patrick Mulvaney Senior 11' Manager Imperatis to Donna Seymour Chief htfc Of cer 1 1 3 Of ce ofPers Mgmt May 12 2014 10 01 am Attach 12 at 001479 In'tpcratis Production Sept 1 2015 11' Email from Donna Seymour Chief Info Of cer US Of ce of Pers Mgmt to Patrick Mulvaney Senior IT Manager Imperalis May 12 2014 10 10 Attach 12 at 001429 Imperatis Production Sept 1 2015 For example on May 2014 Imperatis provided labor rates information to Ms Seymour See Email from Dir of Strategic Growth Ilnperatis to Donna Seymour Chief Info Officer US Of ce of Pers May 17 20l4 1 1 14 am Attach 12 at 001432 Imperatis Production Sept 1 2015 13' Letter from Maj lOcneral ret Mastin Robeson President d Chief Exec Of cer Imperatis to the Hon Jason Chaffcta Chairman H Comm on Oversight tit Reform SeptId Imperatis also noted that a decision was made to use a OHS contracting vehicle given their cybersecurity role for the federal government Id 205 On June lo 20 4 just over one month after initially contacting Imperatis a letter contract award was made to Imperatis n the days leading up to this award Wagner followed up on a phone call with lmperatis He emailed am looking forward to having you guys come in My team and I have been working this issue with no funding and limited assistance for four years it will be awesome to have better opinions and solutions 1 In Wagner testi ed to the Committee that lmperatis was contracted to build out a new environment and in building out the new environment they were given the initiative to find new technologies and innovation 1 lmperatis and 0PM Buy Security Tools to Secure the Legacy IT Environment Documents obtained by the Committee from Imperatis show a list of ten tools that 0PM purchased through the Imperatis contract to secure Dears legacy network I Purchases were made beginning in June 2914 up through October 2m 4 There were challenges in deploying tools including delays and technical challenges 32 The documents show the time elapsed between the purchase of these tools and completing deployment ranged from almost three to fifteen 1110111115le The reasons for the extended period of time between purchase and full deployment varied and are not entirely clear from the record Wagner testi ed that when 0PM rolled out certain tools such as PW cards these deployments caused certain applications and certain functionalities to break and it was something that we had to work through ' '34 Further in the ease of completing the roll out of a tool called oreScout the documents show some delay can be attributed to a requirement for notifications to applicable onions ForeScout which is a tool to manage network access control for devices was purchased in July 3 Letter Contract June 15 20 I4 Attach 1 at GUUDUZ lmperatis Production Sept 1 Email from_ Contracting D icer Dep t of Homeland See to lmperatis June 16 2014 3 41 pm at Imperatis production Sept 1 Etll 133 Email from Jeff Wagner Dir Info Tech Sec Operations 5 Of ce of Pers Mgmt to Patrick Mulvaney Senior ET Manager lmperatis June 13 EDI-4 1 59 Attach 12 at @1539 Impelatis Production Sept l sets - 29 Tr at 30 0PM Tactical Toolset Purchase Kick-off and Completion Timeframes Oct 21 2015 imperatis Supplemental H cument Production Oct 21 2915 on le with the Committee Id In Inperatis told the Conunittce their role in buying security tools during the Tactical phase of the contract was limited to acting as a procurement agent to purchase 0PM-selected security tools and associated vendor professional services Letter from Maj General retJ Maslin Roberson President St Chief Exec Ufftcer Imperatis to the Hon Jason Chaffetz Chairman l-i Comm on Sc Gov't Reform Sept I 2 15 at 4 The record indicates that Imperatis while acting as an agent also provided justi cation for tools and typically did pertorm some due diligence on these purchases Email from lmperatis to Donna Seymour Chief info Of cer U S Office of Pers Mgmt July 29 2314 3 1ilI Attach 9a at 1160 1161 Imperatis Production Sept I 2015 explaining the bene ts of Palo Alto Networks Next Generation Firewalls 33 0PM Tactical Toolset Purchase Kick-off and Completion Ti tneframes Oct 21 so 5 Imperatis Supplemental Document Production October 21 on le with the Committee 34 Wagner Tr at T2 206 2014 but it was not tlly deployed until September 2015 ' '35 Imperatis stated in a Weekly Report for August 2015 that approval has not yet been received for Agency-wide memo and project sponsor is in notification stage with the t1 nion I '36 The mitigation strategy for this situation was to prepare updated project timeline plan meme to pilot ForeSeout to Non- Union Agency users The documents show there were also situations where hnperatis was not able to perform due diligence because of the expedited nature of a purchase For example in July 2014 Imperatis described a riski cbailenge area desire to purchase tactical gear without Imperatis being able to perform true due diligence on tool and fit into current as is network ' '33 Part of the proposed mitigation strategy for this challenge was to collect more information from Wagner and request his assistance in setting priorities This limitation on due diligence and lack of priorities was identi ed as a Risktr Challenge beginning in July 2014 through November 2014 until 1m eralis stated implementations are proceeding and most roadblocks have been cleared lmperatis Role in Responding to 0PM Data Breach Incidents Imperatis stated to the Committee that they did not perform incident response activities related to the June and July 2015 data breach announcements 1 Imperatis said 0PM and other 0PM contractors were responsible for operations security and maintenance of the legacy IT environment The record does show other contractors with a more significant role in incident response and security of the legacy 1T environment I '42 Imperatis did facilitate meetings with vendors who played a role in incident response and also did provide 24 man-hours of assistance for security incident response and clean up according to a Report for the Week of April 2011' '43 While lmperatis did not perform signi cant incident response activities they did have some visibility into the incident response and the security challenges related to the data breach incidents announced in 2015 Imperatis was aware of the March 2014 security incident as demonstrated by documents provided to the Committee For example documents show lmperatis was invited to assist 0PM 35 0PM Tactical Toolset Purchase Kick-off and Colnplelion Timeframes Och 21 2015 l Imperatis Supplemental Document Production October 21 2015 on tile with the Committee - Ilnperatis Weekly Report hug 3 201 S Aug T 2015 Attach 6 at 000942 Imperatis Production Sept 1 2015 113 My Imperatis Weekly Report July it 2014 Iu1y 14 2am Attach a at some Imperatis Production Sept 1 2015 1139 M mperatis Weekly Report Nov- 10 14 2014 Attach 6 at 0004't3 lmperatis Production Sept 1 2015 Id Attach 6 at 000492 Imperatis Production Sept 1 2015 Letter from Maj General rot Mastin Robeson President Chief Exec Of cer Imperatis to the Hon Jason Chaffetz Chairman H Comm on Oversight dc Gov t Reform Sept 1 2015 at 12 '13 Saulsbury an employee of sax explained his role at ores saying he had worked at one since 2012 as an sea contractor and worked in network security He said BRA provides supplemental staf ng under a contract to provide a variety of management services Saulsbury Tr at 3-10 lmperatis Weekly Report Apr 201 S-May 1 2015 Attach 6 at 000 53 lmperatis Production Sept 1 2015 20 after the primary incident response period for the March 2014 incident I The lmperatis proposal also stated Unfortunate y 0PM experienced a recent security incident that occurred because the network was neither set up to easin recognize potential intrusions nor quicka react with the necessary incident response to stop attacks from becoming major data breaches 1 1 lmperatis said by the time of the June and July 2015 0PM breach announcements the procurement of security tools for legacy network under the Tactical phase of this project was nearly 100 complete hnperatis said they did not generally provide incident response services during this period I However lmperatis did report that at request during this period lmperatis arrange d the procurement of Palo Alto rewalls and associated professional services to support the bolstering of network defense around the teP applications and completed this procurement by July 1 2015 1143 Sole Source Schedule and Cost 16 Concerns Related to IT Infrastructure Improvement Contract Validated Documents and testimony obtained by the ICorru'riittce show 0PM Of cials Made Statements to Congress that were Inconsistent with the Record When the IG raised concerns about 0PM making a sole source award for all four phases of the IT Infrastructure Improvement project 0PM officials insisted that a contract award had not been made for the latter two phases of the project Migration and Clean-Up Donna Seymour testi ed before the Committee that we only contracted for the first two pieces of this multi-phased project I '49 Former Director Katherine Arehuleta made similar statements before the Committee and elsewhere I 13 44 Letter from Maj General ret Mastin Robeson President at Chief Exec ISifficer lmperatis to the Hon Iason Chairman 11 Comm on Oversight r3 Gov't Reform Sept 1 21 115 at T-E Imperatis Proposal voiumc - Staf ng and Mangement Attach 5a at 000233 lmperatis Production Sept 1 2015 16 Letter from Maj General reL Mastin Robeson President Chief Exec Of cer lmperatis to the Hon Jason Chairman Comm on Oversight Gov t Reform Sept 1 2015The Electronic Questionnaire For Investigations Processing System is used to collect infom tation related to Federal background investigations On June 29 21115 0PM shut down the EQIP system which was offline until august 21 115 Assistant 1G Michael Esser said of the shut down of cial statement on this issue claims that the agency is acting proactively by shutting down the system However the current security review ordered for this system is a direct reaction to the recent security breaches In fact the e- QIP system contains vulnerabilities that 0PM knew about but had failed to correct for years Is the 0PM Doro Breech the Tip ofrhe Iceberg Hearing Before the Hearing Before S'trbconrm on Research ti Tech and Subcomm on Gvecrigtr of 're H Corner on Science Space dc Tech 114th Cong July E 2015 statement of Michael Esser Assistant Inspector Gen US Of ce of Pers Mgmt An 0PM constructed diagram of how the attacker navigated system identi ed as one of the affected servers See 0PM data breach diagram dated Sept 1 2015 at unretlacted version production Dec 22 2015 An UPlvl contractor noted in a transcribed interview that he believed related to accessing Lt Saulsbury Tr At re Hearing 0PM Darn Breech Par-r1 testimony of Donna Seymour Chief Information Officer Of ce of Personnel Management in Hearing 0PM Dom Breech Parr stating would like to remind him the 15 that the contracts for Migration and Cleanup have not yet been awarded Hearing on 0PM Infomrarr'oa Tecfirtot ogu Spending and Data Security 208 Later 0PM admitted the contractor did have a role in the latter two phases of the IT Infrastructure Improvement project On September 3 2 015 Acting Director Cohort supplemented the former Director s response to the regarding the sole source contract and Imperatis role in tlte later phases Migration and Clean up ofthe project '15 Acting Director Cobett explained that although the contract contemplates that lmperatis will have work to do in all four phases not all aspects of the work required by OPM in phases three and four is included in the contract with Imperatis M152 The documents show that while not all work for the project is covered 0PM did in fact make a sole source contract award to Itnperatis for work in phases of IT Infrastructure Improvement project Thus from the beginning this sole-source award was to cover aspects of work from all four phases of this project Indeed the IG pointed out in the June 1 Flash Audit Alert that the original documentation justifying the sole source award covered all four phases of the work Tactical Shell Migration and Clean '53 The 1G also pointed out that in a May 26 2015 meeting the former CID ar ued in favor of an approach where the same contractor oversaw all four phases of the project H 4 The Committee obtained the contract le which calls into the question the truthfulness of certain statements by OPM of cials to Congress- The contract documents outlined in detail the contractor s role in each at the four phases of this project The Statement of Objectives SOD for the June 2014 letter contract states the work is focused in four primary hases and then listed tasks that the Contractor was expected to perform under each phase 5 For the Migration phase the $00 stated Contractor shall work with 0PM to plan for oversee and assist in the migration of existing 0PM network and business applications and services into the new IT infrastructure 1 '55 For the Clean Up phase the SOD stated Contractor shall work with DPM to cleanse all data and applications from unused hardware and shall prepare it to be The Statement of Work tor the contract stated t hc Contractor shall complete work within this SOW in four different phases Tactical Shell Migration and Clean Up l '53 The SOW also is similar to the SOD in that the 30W outlines speci c contractor tasks in the later two phases of the project I 159 stating i would like to remind the Inspector General that contracts for the Migration and Cleanup have not yet been awarded Memorandum from the Hon Beth Cohort Acting Din U S f ce of l ers Mgmt to Patrick McFarland Inspector Gen U S lL'tht'tce ot Pers to Response to Flash Audit Alert -- US O ics ofPst-sonns Improvement Project Report No Sept 3 2015 hereinafter Cohort Response Sept 3 EDIE to DIG Interim Status Report 52 Cobert Response Sept 3 2015 to Interim Status Report at l 53 GIG Flash Audit Alert June 2015 at 5 6 55 Letter Contract Statement of Objectives June is 2014 Attach 1 at cannot Imperatis Production Sept 1 2015 ms at Imperatis Detinitized Contract Statement of lWork Jan l5 Attach at lmperatis Production Sept l N15 1' Id of st 209 The Committee obtained documents that show the contractor had every expectation that they would be providing services through all four phases of the project In their November 2014 proposal the contractor said o ur response to the sow directly responds to each of the four phases of the program and describes the ways in which our team has begun fulfilling these requirements to date and added that their proposal provided a detailed response and solution to each of the Four phases of the Infrastructure Improvement program Ml 50 In addition the contractor outlined in their proposal a live step process with an illustrative diagram for the Migration phase Finally as the contractor began to perform under the contract the documents show the contractor was performing tasks related to the later phases of the project In February 2015 the contractor first identified stand of Migration PMG office as a high risk area and proposed a strategy to mitigate potential risks to include working closely with ACIOs to ensure IT program managers 8 application teams are engaged with project plans and a migration schedule is in In early April 2015 the contractor s Weekly Report included a Migration Process diagram and discussion oI Migratiorr Phase 2 options with pros and cons I In May 2015 the contractor provided updates on the Migration PMO office sayin Initial engagement happened There were 2 questions from the application groups l These activities clearly show the contractor understood the work covered under this contract included tasks related to the Migration phase I The Concerns about Schedule Risks Were Validated In the June 2015 Flash Audit Alert the IG raised a concern that 0PM had signi cantly underestimated the time to complete the Migration Phase 3 of this project and did not consider the complexity and process to complete this phase ' ii According to the Ga Alert 0PM estimated the Migration of all of legacy would take eighteen to Monty four months Imperatis immediately recognised the schedule challenges and identi ed schedule risk as a concern in the proposal they submitted Imperatis s proposal stated the duration of the current period of performance is insufficient to accomplish a complete migration into soars Imperatis Proposal Volume II Staf ng and Mangement 5a at 000233 Imperatis Production Sept 1 2015 at at coarse 53 Imperatis Weekly chott Feb to 2a to 5 Attach a at coats-ts autocratic Production scpt t 2015 t3 Ilt tperatis Report Act a rots Apt to 2015 Attach a at noon s-su nopctotis Production Sept 1 2015 '54 mperatis Weekly Report May 4 2015 May it 2015 Attach 6 at 000224 Imperatis Production Sept 1 2015 55 Imperatis stated in a letter to the Committee that while they were engaged in some role for all four phases of the project their most signi cant work related to the Shell or Phase 2 Letter From Maj General rot Mastin Robeson President St ChicfExec Of cer Impetatis to the Hon Jason Chaffetz Chairman H Comm on Oversight tit Gov't Reform Sept 1 2015 at 3 ti DIG Flash Audit Aron June 2015 at 3 lmperatis Proposal ll t ohurre I Statement of Work and Technical Attach 5 at 000219 Imperatis Production Sept 1 2015 210 imperatis also cited in particular challenges with applications requiring modernization including the Federal Investigative Services and Retirement Services' '53 These applications alone are complex and will take signi cant time and effort to migrate to modernized solutions Two years after the June 2014 award the tactical phase has been completed a new IT environment appears to have been delivered but perhaps not fully testedftrained on and 0PM is still working to inventory and fully scope the alternatives of mitigating or migrating legacy IT to the new Saulsbury testi ed to the Committee that he did not work on the Shell but reported that lmperatis has some of the infrastructure up and running and added lmperatis is starting to train SRA station how to operate some ofthe tools within the shell environment m39 The 16 s Concerns about Cost Risks Were Validated In the June 2015 Flash Audit Alert the 16 also said there was signi cant cost uncertainty with this project due to the unknown scope of the work required including a full inventory of IT assets 1 According to Weekly Progress report documents obtained by the Committee the contractor identi ed funding for the Shell phase as an area of high risk beginning in Febmary 2015 through at least August 2015 i Ft-ent March 2015 April 2015 the contractor updated this high risk area by saying still awaiting Mod for additional funding l In In early May 2015 the contractor repolted Mod received Now discussing additional material funding needed let- the rest anv and FY 2016 through nee 3 Then in July through August 2015 the contractor update was need additional Funding quickly to ensure no delay ill procurement The documents show funding for the Shell was a signi cant ongoing concern The uncertainty with respect to total cost of this project has persisted although 0PM now appears to be taking constructive action aimed at improving long term cost estimates In the June 2015 Flash Audit Alert the IG rcportcd that 0PM had estimated the Tactical Phase 1 and Shell Phase 2 portions of the project could cost approximately $93 million which included million to be collected from major 0PM programs as a special assessment with little information as to the scope of the projectm ile Saulsbury at 11 m ore Flash Audit Alert June IT 2015 at s Imperatis Weekly Report Feb 23 2015- Feb Attach at lmperatis Production 1 2015 lmperatis Weekly Report Aug 10 2015- Aug 14 2015 Attach til at 000953 meeratis Production Sept 1 2015 ii Imperatis Weekly Report Mar 23 2o15- Mar 2015 Attach- it at Imperatis Production Sept 1 2015 Imperatis Weekly Report Apr 20 2015 Apr 24 2015 Attach 6 at 000746 imperatis Production Sept I 2015 Imperatis Weekly Report that 27 2015 to May 1 2015 Attach a at coarse Irrtperatis Production Sept I 2015 74 lmperdlio Weekly Report July 13 2015- July 2015 Attach at Production Sept 1 20 llnperatis Repolt Aug 10 2015 Aug 14 2015 Attach a at 000953 Imperatis Production Sept 1 2015 oto Flash Audit Alert June tr 20l s at 21 As of late October 2015 0PM reported to the Committee that overall it had spent about million in FY2014 and 2 115 for this project 1 175 The contractor has reported being paid a total of $45 1 million for the period oflune is 2014 through May a 2016 In May 2016 the IG reported that FY EDI Business Case for this project outlined costs already incurred with some reasonable short-term estimates to nish developing the laaS portion a However the IG expressed concerns about the cost estimates for the long term efforts to modernize and migrate to a new IT environment and called these estimates unsubstantiated because of the incomplete inventory and technical analysis At the same time the IG did acknowledge as positive 0PM efforts to develop cost estimates for modernizing and for migrating all 0PM information systems by leveraging a new application pro ling scoring framework l 9 In I anuary 2016 the Administration announced the creation of the NBIB and the designation of the Department of Defense as responsible for the IT security of background investigation data This announcement has further complicated efforts to identify a de nitive plan to hand IT modernisation at 0PM given that background investigation program is being moved to the NBIB and DOD will be responsible for IT security and mding for these functions likely will not be available for modernizing other 0PM IT assets The Status and Future Plans for New 1T Environment Shellflaasi are Unclear In the lune 2 15 Flash Audit Alert the 010 predicted 0PM could nd itself in a situation where it could be incurring costs to maintain two IT environments legacy and the Shell In June 2015 the 1G said without a disciplined planning process or a guaranteed funding source in place to complete this likely complex and expensive process the agency would be forced to inde nitely support multiple data centers nther stretching already inadequate resources possibly making both environments less secure and increasing costs to '81 The DIG added such a scenario would be inconsistent with the goal of creating a more secure IT environment at a lower cost l '32 This appears to now be the ease with the creation of the Shell and continued uncertainty about plans and costs for mitigation modernization andfor migration ofUPM's legacy IT environment The goal of achieving a more secure environment at lower costs appears to be at risk hi May 2016 the OIG reported that 0PM had allocated a limited amounth funding to Email from us Office of Pets to n Comm on Oversight a Gov'tAffairleet 23 20151ton le with the lCommittee 1m Imperatis Response to H Comm on Uverisght El Gov t Reform Majority Staff one T EDI ti on file with the Committee 13010 Second Interim Status Report on Infrastructure Improvement Project at T m Of ce of Inspector Gem US Of ce oFPers Mgml Report No Second Interim Status Report on the US G ice ofPersomtef r lal'gmt 3 fig 'asn'uemre hnprovemem Project Mayor 1 Business Case at May 13 2616 Brerreher For Hearing at are H on Oversight at Gov Reform 114111 cong Feb 24 2016 prepared statement of Norbert E v int Office of Inspector Gen U S Of ce of Peas Mgmt cancelled ore Flash Audit Alert June 1'1 2015 at 5 ll 3 212 modernisation and migration efforts' '83 According to the IG Business Case for the lT Infrastructure Improvement project allocated only twenty to twenty- ve percent of this project s cost for modemizationfmigration with the remainder allocated to securing and maintaining the legacy and IaaStShell environment The 016 questioned this approach because it does not acknowledge maintenance cost for the dual environments will not likely remain fitted I '34 The 016 speculated that as the costs to maintain the legacy environment increase this could result in limited funding for modernization and migration Meanwhile 0PM is now currently spending approximately $25 million annually to maintain the '35 According to the DIG 0PM is considering a plan to save money by hysically moving legacy systems From old data center envirorunents to the new environment I 5 5 Such a plan would include keeping the legacy systems in a separate logical envirorunent from ShellilaaS It is reasonable to consider such a plan for the purposes of saving money but as the IG pointed out serious consideration should be given to the security risks of maintaining security controls in two logical environments In sum lT Infrastructure Improvement project which was motivated by the laudable goals of securing the legacy IT environment and creating a more secure lower cost modernized IT environment fell victim to a awed contracting and planning approach Two years after this effort began and after much time and effort to acknowledge and mitigate DIG concerns 0PM is only now making progress toward a disciplined planning and assessment of the altematives and establishing a teasonable cost estimating process 33 DIG Second Interim Status Report on Infrastructure Improvement Project at Hit213 Summary of Investigation The agency s posture with respect to the Committee s investigation has been consistently uncooperative until the later stages of the investigation especially as it compares to the level of cooperation from other agencies and contractors who had relevant documents and information Committee hearings on the data breaches in June 16 2015 the Committee held its rst hearing on the OPM data breach which was entitled Data Breach 33 The hearing occurred twelve days alter 0PM publicly announced the breach of personnel records for approximately four million current and former federal employees 1 39 The hearing included testimony from witnesses from 0PM the 0PM DIG the OMB 131-18 and 1301 This hearing provided the Committee an opportunity to learn what occurred based on the information available at that time but responses from some witnesses increased concerns about the data breach Following the hearing Members were invited to a classi ed brie ng on the data breaches Twenty days after 0PM announced the breach affecting personnel records the lCommittee convened a hearing on June 24 2015 entitled Data Breach Part The Committee heard testimony from 0PM the UPM OIG U S Investigations Services LLC a former 0PM background investigation contractor and KcyPoint Goverrunent Solutions a current 0PM background investigation contractor During the June 24 hearing the Committee received an update on the investigation and learned background investigation data also had been compromised but 0PM declined to provide speci c information on the number of individuals impacted citing an ongoing investigation The Committee also teamed more about the OPM data breach discovered in March 2014 Speci cally the Committee heard testirnony that manuals about the servers and environment had been taken from network during the incident 119' Then-C10 Donna Seymour admitted the manuals about the servers and the environment would provide enough information that the adversary could learn about the platform the infrastructure of system 1192 On the same day as the second hearing then 0PM Director Archuleta sent a letter to Chairman Chaffetz clarifying the number of former and current federal employees whose personnel records were compromised by saying roughly 4 2 million individuals were impacted and stating an unspecified number of former and current federal employees background investigation data had been compromised 1 It was not until July 9 2015 that 0PM publicly announced the backgmund investigation data of21 5 million current tbrmer and prospective 0PM Doro Breach Hearing Before the H Conan rm Firerzsr'ghr rt Gov 'r Reform 1 14th Cong June 16 2015 us otaee ofPers Mgmt Press Release 0PM to Notify Empioveer maiden June a 2015 govl ne warreleasesr Ztll 1' Hearing on 0PM Doro Breach For H ll'JI Id 93 Letter from Katherine Archuleta Din 1 1 3 Of ce of Personnel to the Hon Jason Chaffetz Chairman H IComm on Oversight dc Gov l Reform June 24 2015 214 federal employees contractors and related non applicants had been compromised 1 19 Then on July 15 2015 just over a month after the breach was rst announced the Committee's Subcommittee on Information Technology and Subcommittee on the Interior held a joint hearing entitled at the LLB Department of Interior 95 Since DOI held OPM personnel records that were stolen in a shared service data center facility this hearing allowed the Committee to better understand the impact of the breach on how its systems interacted with those of OPM and more detail about how the breach occurred The agency s CIO and Inspector General testified in order to learn more about the incidents described at these hearings the Committee continued its investigation and made multiple requests for information and documents from relevant stakeholders Committee request for information regarding identity theft services On July 21 2015 Chairman Chaffeta and Ranking Member Cummings sent the first letter to OPM requesting information about I the contract for the identity theft protection services for 4 2 million current and former federal employees whose personnel record data had been compromised and 2 OPlvl s plans to provide identity theft services to the 21 5 million individuals whose background investigation data had been compromisch On August 21 2015 OPM provided an initial response related to the identity theft contract for the 4 2 million personnel records victims to the Committee 1 OPM declined to provide detailed information regarding plans for an identity theft services contract for the 21 5 million until a contract had been awarded On September 1 2015 0PM and the Department of Defense DOD announced a new identity theft protection and credit monitoring contract award to provide identity theft services to US Of ce of Personnel Press Release 0PM Announced Steps to Protect Fedora Workers and others -oni Titre-on - Jul Ir 9 2015 available at ste 9 Cvoet'secnritv Tite Department oftita interior Hearing Re ne the on infoJ'eeii and Sithcottnn on Interior oftiie H on Oversight die Gov 't Reform 1 14th Cong July 15 2015 Letter from the Hon Jason Shaffer-2 Chairman and the Hon Elijah Cummings Ranking Member H Comm on Oversight 3t Gov t Reform to the Hon Beth Cobcrt Acting Dir 11 5 Of ce of Pets Mgmt July 21 2-315 97 The Committee reviewed the documents OPM provided and confirmed the contract award to WinvalciCSm was not a solosouree award as was originally suggested l-lowever as the later reported there were some contracting irregularities but it was unclear whether these irregularities would have changed the awardee On December 2 2315 the IG completed a Special Review in response to the Committee's request during the true 24 5 hearing on the $20 miliion contract to provide credit monitoring and identity protection services to the initial 4 2 million victims ofthc Ol l'vl data breach The G s Special Review determined that in order to meet the lune Ii 2015 requirements due date the contracting of cer failed to comply with FAR requirements and 0PM policies and procedures in awarding lhe Winvale contract and their the Hi identi ed ve areas of noncompliance Office ofthe Inspector Gen Of ce of Fe rs Mgmt 4K-lt3-U l D24 apertoi Review ofUFrl-t 5 Award ofo Credit Monitoring and identity the Services Contract to Winmte Group and its Subcontractor CSitientitv Dec 2 2014 215 the 21 5 million individuals impacted by the background investigation data breach 1 '93 After further inquiries to 0PM regarding the contract information 0PM deferred to DOD for the detailslgug this contract The Committee obtained relevant records from DOD on October 20 2015 The DOD award was made under a government-wide contract velticle established by the General Services Administration This contract vehicle provides agencies with access to contractors capable of providing identity monitoring data breach response and protection services This contract vehicle is available to agencies for up to ve years and has an estimated value of $500 million In contrast to the rst contract arrangement for the 4 2 million individuals tlte September 1 2015 contract award established a government-wide vehicle for these services so that agencies are not trying to establish a contracting vehicle to provide identity theft services in the middle of incident response DOD handled the notification process directly for thezg l 5 million victims and the initial notification process was completed in December 2015 Productions related to the OPM data breaches and CyTech On July 24 2015 Chairman Chaffcte and Ranking Member sent a second letter to 0PM requesting information and documents in response to questions about speci c details of the data breaches announced in June and July 2015 mm The letter covered a range of issues including information about Plvl s relationship with and the work conducted by CyTech Services information on 0PM security tools and user credentials for 0PM information systems and additional information related to the data breach The request related to CyTech was prompted by a referral from the House Permanent Select Committee on Intelligence HPSCI and press reports On June 15 2015 the Watt Street published a story on the OPM data breaches alleging that CyTeeh had discovered the breach during the demonstration of their security tool um Then on June 23 2015 just before the Committee's second hearing on the OPM data breaches where the Committee heard testimony about CyTech the received a memorandum from Rep Devin Nunes Chairman of 93 1 1 3 le ce of Fers Mgmt Press Release 0PM DOD Announce Identity Tire Protection and Credit Monitoring Conn-act Sept 1 2015 available at 1m Letter from the Hon Jason Chairman li Comm on Oversight St Gov't Reform to the Hon Ray Mabus See Of ce of the Sec of the Navy Sept 22 2015 Letter from R L Thomas Din Navy Staff Dep t of the Navy Dep't of Defense to the Hon Insert Chaffetz Chairman H Comm on Oversight 32 IGov't Reform Oct 20 2015 mm In the Consolidated Appropriations Act for Fiscal Year 2016 language was including requiring CIPM to provide individuals impacted by the CPM data breach with 10 years of identity protection services versus three years under the Sept 1 2015 award and five million in liability insurance Jason Miller Pay mire tr nnsit bene ts parity gins- 7 feds optinrisnrfor 2026 FEDERAL NEWE Raoro Dec 17 2016 Letter from the Hon Jason Chafl eta Chairman and the Hon Elijah E Cummings Ranking Member H Comm on Oversight 3r Gov t Reform to the Hon Beth Cohort Acting Director US if lice of hers July 24 2015 13m Damian Palette Cybm seeur it'y Firm Says It Found Environ on Government Nehr- or k in Apr'if WALL ST 1 June IS 2015 available at 1434369994 216 HPSCI and Rep Adam Sehit t Ranking Member regarding the information from CyTeeh 12m l'il Ii Home World LLB F n-lities Fer-nerdy business Tech Mallets Opinion Arts life RenlEslota LI NHI SECURE Cyberseeurity Firm Says It Found Spyware on Government 1 Network in April t vtt't ll HUI tea t Linn mitt'a titlt'fil mm tiff-5 int-iv tannins-I tlinil war that -7 As a resolt of these events the Committee sought documents and information to better understand the facts and any role CyTeeh played at 0PM during tho 2015 incident response period Pursuant to this effort the Committee requested information from 0PM about CyTeol t as part of a broader July 24 2015 letter to 0PM On August 14 2015 Chairman Chaffeta also sent an information request to Ben Cotton Chief Executive Officer of CyTech The letter requested all documents and conununications bemeen 0PM and CyTeeh details about the product demonstration that CyTech conducted at 0PM in April 2015 and any additional activities conducted by related to incident response um Cy'l eeh responded to this request on August 19 2015 by providing documents to Committee staff during a visit to CyTeeh headquarters in Manassas Virginia The Committee also conducted a transcribed interview with Cotton on September 30 While CyTeclt responded to the Committee s request for information 0PM dragged its feet initial response to the Committee s July 24 2015 letter did not include information in response to questions about CyTeeh On September 25 2015 0PM made a second production in response to the July 24 2015 request producing a nine page narrative in response to questions posed about CyTech and only one relevant document more than 175 pages olivisitor logs from Washington D C headquarters for the month oprri 2015 that were almost entirely redacted L203 Letter from the Hon Devin Nunes Chairman and the Hon Adam Sehiff Ranking Member H Permanent Select Committee on Intelligence to the Hon Jason Chaffete Chairman and the Hon Elijah E Cummings Ranking Member H Comm on Dversiglitt Gov t Reform June 23 2015 um Letter from the Hon Jason Chaffetz Chairman Comm on Oversight Er Gov t Reform to Ben Cotton President Chief Exec Of cer CyTeeh Aug 14 2015 Ranking Member Cummings did not sign this request 'an Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight 5 Gov t Reform to Ben Cotton President 3 Chief Eaee Of cer CyTeeh Aug 14 2015 '36 Cotton Transcribed Interview 12' August 23 2015 GEM document production 1m Letter from Jason Levine Dir of Cong Legislative d intergovernmental Af fairs LLS Office of Pers Mgmt to the lion Jason Cliaffetz Chairman 11 Comm on Dversight th Gov t Reform Sept- 25 20l5 Production Sept 25 2015 Office of Personnel Management Visitor Log April 1- July 10 2015 at Production Sept 25 2015 21 Heavily redacted visitor fogs provided by 0PM on September 25 201' 5 UPM made a third production to the Committee on October Z l 5 that included a less redacted version ot'the visitor lo and a corresponding analysis of entries for staff from CyTech hnperatis DHS and the FBI 11 9 I l ll l' October 23 2015 0PM made a substantial production of redacted documents made documents available in eastern and responded to a September 9 Eli 5 letter regarding a deleted drive on Cv'l'ceh s appliancemu On August 19 2015 Cy'l ech told Committee staff it had requested the appliance be returned multiple times but it was not returned until August 20 2t 1 S'ZH one day after Committee investigators visited CyTech offices The appliance 1was returned to CyTech sanitized that is with all information deletedmz The agency did not provide a copy ofthe drive s contents to the Committee despite the fact that there was an ongoing congressional investigation and preservation order in place The status of the deleted contents of the drive and whether 0PM preserved a copy was Of ce at Personnel Management v'isitor Log April rally at one Document Production lfillet 3 2015 Additional responsive documents were also made available to the Committee in the UPM liaison office at this time '31 Letter from the Hon Jason Chatfetz Chairman H Comm on Oversight d Gov t Reform and the lion Michael 'l'urner to the Hon Beth Hebert Acting Dir U S Uf ee ot Pers Mgmt Sept 9 2015 Cotton Tr at T2 Email from Brendan Sanlsburv Senior Cyber Security Engineer saw to Jonathan Tonda sea as oincc of l ers Mgmt and IeilWagner Dir Info Tech Sec Operations US Office ofl ers Mgmt Aug 1 5 l pan at Production Get 23 2315 218 discussed at length at a January 7 2016 Conunittee hearing 2'3 It was not until April2t116 that 0PM made a sample of the images collected by available for an in eastern review 0PM had obtained this information for the in eomem review from US-CERT vs CHAFFETI nu Choir-man Chemists questions on 0PM witness about Federations Despite Committee requests For information anti an Au gust 21 2015 preservation order 0PM did not preserve all relevant evidence The preservation order covered all records related to the breachfintiusion the infrastructure improvement project cybersecurity and decisions on implementing the recommendations made by the 016 1 As a result of documents produced by Cy t eeh and interviews with CyTeeh employees the Conunittce obtained evidence related to the efforts of other rms involved in the April 2015 incident response activities at 0PM including Cylanee SKA and meeratis Each of these companies was present throughout the incident response period and ultimately provided information useful in understanding the bigger picture of what unfolded before during and after the OPM data breaches The Committee investigated the role of Cylal lce Cylanee was first identified during a review of documents provided by CyTech In an April 24 2015 email an employee of Cylance Chris Coulter emailed C yTeeh s CEO to ask Would you be able to this file want to verify something In a September 28 Document Production Status Update Hearing Re ne the H Forum on Oversight d5 Gov 't Reform 1 1411 Cong Pan 7 lel at 1 113 3 Letter from the Hon Jason Chaffetz Chairman 11 Conun on Oversight 8t Govit Reform to the Hon Beth lCohert Din 1 1 5 Uf ee of Pets Mgmt Aug 21 21315 E-mail from Chris Coulter Managing Din Cylance to Benjamin Cotton ChiefL aec lUttieer CyTech Apr 24 2015 1 54 pan at 1 271r CyTech Production Aug 19 219 2015 brie ng to Committee staff UPtvI s Director oflT Security Operations Jeff Wagner told staff that Cylance executed the quarantine order on systems in April 2015 On December 3 2015 the sent a letter to Cylance inquiring about the activities it conducted at 0PM in April 2015 and requested related documents 12 Cylanee provided thousands of pages of documents on a rolling basis and in a timely manner and also made available to the Committee a virtual data room with additional pieces of information and evidence The Committee subsequently conducted transcribed interviews of two Cylance personnel The Committee conducted a transcribed interview with Cylance CEO Stuart McClure on February 4 21116 On February 12 2016 the Committee conducted a transcribed interview with Cylanee Managing Director of'Ineident Response and Forensies Chris Coultet Cortlter was heavily involved in providing assistance to 0PM with the deployment of Cylance tools The Committee investigated the role of SRA SKA International another 0PM contractor provided information that helped inform a more complete picture of the OPM data breach incidents identi ed in March 2014 and April 2015 12 The Committee was able to identify two key SRA employees who provided 0PM IT security operations contract support in 2014 and 21115 12 The BRA employees provided IT security operations center support under an BRA contract for IT management services and reported to Director of IT Security Operations Jeff Wagner The Committee contacted one of these SKA employees Brendan Saulsbury who responded to questions about his role in the 0PM data breach incident response in an informal interview in January 2016 Later on February 16 2016 Saulsbury participated in a transcribed interview 12m Saulsbury started with sea in early 2012 and by March 2012 began providinng security operations support to 0PM under an SRA contract Saulsbury adruinistered various IT security tools and played a key role in the 2014 and 2015 0PM data breach incident response and forensic investigation The other now former ERA employee identi ed through the Committee s investigation Jonathan Tonda began working for 0PM as a federal employee in the Fall of2015 As of May 2016 Saulsbur'y left ERA and is eruployed with another organization 'i i Letter from the Hon Jason Chaffetz Chairman and the Hon Elijah s Cummings Ranking Member n Comm on Oversight Guv't Reform to Stuart McClure Chief Exec D ieer Cylaltee Dee 3 2915 12' McClure Coulter Tr ms SIM International has combined with the North American Public Sector business of CSC to form SKA in the fall of 21 5 See CSC Press Release CSC to Combine Government Services Unit with Si Upon Sepoi orimr onr Combination Wit Create Lending Pure Fin Government LT Business in the US Aug 31 2015 E mail from Brendan Saulsbury Contractor for DPM Security Operations to left Wagner Dir Info Tech Sec Operations LLS Of ce of Pers Mgmt June 11 2015 11 44 pm Production Aug 19 2015 um Saulsbury Tr 220 The Committee Investigated IT Infrastructure Improvement Project and the Contract Awardee Imperatis On June 2 115 1G issued a Flash Audit Alert to then Director Katherine Archuleta regarding contract award to Imperatis for the IT Infrastructure Improvement project '22 This contract was awarded in one 2014 as part of response to the data breach discovered in March 2014 The Committee requested follow up information from the and raised further qzuestions about this contract based on the Flash Audit Alert during the June 24 2015 hearing '2 2 The Flash Audit Alert also led the Committee to review the Imperatis contract and its role in activities at 0PM in ApriltMay 2015 related to the data breach incident response As part of imperatis activities for the Tactical Phase 1 portion of the IT Infrastructure Improvement project Imperatis coordinated meetings with CyTech and 0PM and ultimately CyTechis demonstration of its tool at 0PM on April 21 2015 The CEO of CyTech identi ed key Imperatis personnel onsite for demonstration which assisted the investigation Chairman Chaffetz sent an August 13 2015 letter to imperatis requesting documents and communications related to CyTech and the 16 s Flash Audit Alertl i On September 1 2015 Imperatis responded to the Chairman s request and produced over 1 00 pages on the IT Infrastructure Improvement project contract including information on pro contract communications between 0PM and lrnperatis employees the security tools tested and deployed and contract performance 214 in addition hnperatis provided a brie ng to Committee staff on October 15 2015 explaining its role in scheduling and participating in the CyTech demonstration Finally Imperatis responded to supplemental requests by majority staff on contract developments and clarifications on its document production Document productions by Department of Homeland Security On August 19 2015 Chairman Chatfetz sent a letter to US-CERT requesting information and documents related to its role in assisting 0PM with incident response and the forensics investigation of the data breaches identi ed in March 2014 and Spring 2t 115 1225 was reluctant to provide documents directly and quickly because US-CERT expressed a preference that 0PM provide all US-CERT documents directly to the Committee due to its view that the documents were similar to a client s information Regardless of this view it is responsibility to fully respond in a timely manner to congressional information requests The Committee ultimately received a production of over 350 pages front US-CERT on December 11 21115 nearly four months alter the initial request mi The delay in receiving this information I32 our Flash Audit Alert June it sets '33 0PM Darn Breach For June 24 mi Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight 3t Gov't Reform to Major General rec Mastin President 3 ChiefExec Of cer Imperatis Aug lit 2015 Letter from Maj General trot Mastin Robeson President ti ChiefEaec Cll' eer Imperatis to the Hon Jason Chaffetz Chairman H Comm on Oversight St Gov t Reform Sept 1 20'15 133 Letter from the Hon Jason Chaffetz Chairman H Comm on Oversight Sr Gov t Reform to Ann Barrett- DiCamillo Din lComp Emergency Readiness Team LLB Dcp t of Homeland Sec Aug 19 2015 '35 Letter from M Tia Johnson Ass t Sec't for Legislative Affairs US Ilep t of Homeland See in the Hot-1 135011 I ChatTetz Chairman H Comm on Oversight 8c ch t Reform Doe 21315 221 could have been avoided had 0PM and US-CERT been more timely and responsive to Committee requests Unnecessary delays restrictions redactions and a congressional subpoena From July 2015 until early spring of 2016 0PM provided sluggish and incomplete responses to requests offering only tit camera review of certain documents and documents that were often riddled with redactions Further 0PM nally produced key documents with limited redactions to the Committee just a few days before the Committee conducted a transcribed interview with OPlvt s Director of Security Operations Jeff 1Wagner on February IE rim Of the multiple information requests sent to 0PM prior to the February 3 201a subpoena not a single one was answered completely within the requested timeframe This lack of cooperation slowed the Committee s investigation and resulted in the Committee having to make multiple requests to other stakeholders For example on August 13 2015 Chairman Chaffetz sent another letter to 0PM regarding the r stolen manuals issue and requested a response by September 1 2015 223 The letter referenced Tune 24 20 5 hearing testimony from thcn CIO Donna Seymour responding to the Chairman's questions about the exiiltration of security documents and manuals related to network 229 The letter requested documents and communications about the incident and the information that was stolen 123 When 0PM responded on September 18 2015 the reaponse contained signi cant redactions '23 in fact it was not until January 12 2016 nearly five months after the initial letter was sent and after a congressional hearing where Members of the Committee expressed frustration about the redactions that 0PM made the unredactcd documents available in camera 0PM nally produced these documents to the Committee without redactions on February 16 2016 The stolen manual production was critical to understanding more about the data breach diacovered in March 2014 The agency routinely provided the Committee with documents containing unnecessary redactions In addition to the aforementioned visitor logs that were redacted to the point of m Wagner Tr at 23 I'm Letter From the Hon Jason Chairs-ta Chairman H Comm on Oversight 8 tiov t Reform to the Hon Beth E%bcrt Acting Din US Of ce of Pers Mgint Aug IS 2015 use Letter from Jason Levine Dir Cong Legislative d Intergovemmental Affairs Of ce of Pets Mgmt to the lion Jason Cltaffetz Chairman ti Comm on Oversight Gov t Reforltt Sept 13 2015 222 1232 233 initially being useless the agency redacted the name of OPM press of cials in some cases There is no valid basis for OPM to redaet the name of its press of cials especially given their very public role in communicating with the press and public In another example 0PM redacted the name of the contracting of cer who was managing the first contract for the identity protection services for breach victims 1234 The agency redacted the name of the officer despite the fact that his name was publicly available on a new archived Fed BiaOps website page 235 Further the Committee requested the curriculum vitae of Jeff Wagner Director of Security Operations in its July 24 2015 letter to OPM 1235 When 0PM responded to the renuest ever a month letter OPM redacted Wagner s nan-ten MR LEV Director oftire J iee of Congressional A oirs Jason Levine testifies before the Committee 13' OPM redacted virtually every name on the visitor legs it provided the Committee pursuant to the July 24 2015 letter's second request E-mail from redacted to Jeff Wagner Dir Info Tech See Operations U S Of ce of Pets June 12 EDIE 1 511 at Production Feb 15 21316 '33 Winvaie Contract June 2 2915 at 023 OPM Production Aug 21 2015 35 Solicitation Number ormsaisreeis May 23 sets available at Letter from the Iien Jason Cha ffetz Chairman and the Hon L- lijah E Cummings Ranking Member H Comm on Oversight 3 Gev t Reform to the Hon Beth Colbert Acting Dir 11 5 Office of Fers Mgmt July 24 21315 Letter from Jason Levine Din Cong Legislative dc Intergovernmental Affairs LLB Of ce of Peru to the Hon Jason Chaffetz Chairman and the Hon Elijah E Cummings Ranking Member H Comm on Oversight Gov t Reform Aug 23 2015 OPM Pneduction Aug 23 2015 223 Subpoena issued to 0PM In a January 2 2016 hearing belore the Conunittec Jason Levine Director of the Of ce of Congressional Legislative and Intergovernmental Affairs at OPM testi ed that has worked tirelessly to respond to numerous congressional inquiries regarding the incidents and that has made every effort to work in good faith to respond to multiple congressional oversight requests including document Seven months after the Committee s rst request to OPM for information the Committee issued a subpoena on February 3 2016 to compel the agency to produce unredacted documents on a permanent basis mg As outlined above the Committee invested signi cant time and effort in attempting to extract documents and relevant information from 0PM in the months leading up to the February 3 2016 subpoena '24 1While OPM did eventually produce requested documents without redactions directly to the Committee it was only after multiple rounds of productions and signi cant time and effort to extract these documents from 0PM The fact is that OPle failed to fully cooperate with this investigation until a subpoena triggered greater cooperation In contrast to 0PM other relevant stakeholders contacted by the Committee were cooperative and responsive to the Committee s requests The Committee received documents from contractors and other relevant entities that it would receive from 0PM months later For example Ccheh provided documents to the Committee on August 19 2015 that included email conversations between Director of Security Operations Jeff Wagner and CyTech CEO Ben Cotton regarding the Wall Street Journal story on Cy l ceh The agency produced this same document in February 2016 alter the subpoena had been In another example CyTeeh produced an email in August 2015 that led the Committee to investigate Cylance s role in the incident response activities in April 2015 that 0PM only produced in Pebmary ms Document Production Status Update Hearing Before the H Comm on Overnight at Gov 't Reform 1 Idth Ceng Jan 2016 Statement ofJason K Levine Dir Office of CongJ Legislative and Intergovernmental Affairs U S Office of Pers MgmL Subpoena from the Hon Jason Chaffetz Chairman H Comm on Oversight 3r Gov t Reform to Beth Cohort ing Dir Of ce ofPersonnel Feb 3 2016 Id m Cotton T11 Ex 10 Email from Ben Cotton Chief Exec Of cer Cy l'ech to Jeff 1 i iiagner Dir Info Tech- See Operations US Of ce of Pers Mgmt June 12 2015 3 2 Email from Ben Cotton Chief Exec Officer CyTech to JeffWagner Dir Info Tech Sec Operations LLS Of ce ochrs Mgmt June 12 2015 1 05' at IIOGR020316-000205 Production Feb 16 2016 Cotton Tr Ex 5 Email from Chris Coulter Managing Dir Cylancc to Ben Cotton ChicfExec Of cer Apr- 24 2015 Email from Chris Coulter Managing Dir Cylanee to Ben Cotton Chief Exec Of cer Apr 24 2015 5 54 pm at Production Feb 16 2016 224 Conclusion The devastating consequences of OPM cyberattacks discovered in 2014 and 2015 will be felt by the country for decades to conre The key question now before the country is how will we respond Federal agencies including 0PM must remain vigilant in protecting the information of hundreds of millions of Americans and in an environment where a single vulnerability is all a sophisticated actor needs to steal or alter Americans information the identities of average Americans and profoundly damage the interests of U S national security The longstanding inability of OPM to adequately implement sometimes basic but necessary security measures despite years of warnings from its Inspector General represents a failure of culture and leadership not technology However the Committee remains hopeful that 0PM under the new leadership of Acting Director Beth Cobert is in the process of remedying decades of mismanagement In late June 2016 0PM reported to the Committee that over the past year has taken significant steps to enhance its cybersecurity posture protect individuals who had their data stolen in the incidents last summer and reestablish con dence in its ability to deliver on Fivi s core 0PM reports such steps include - Completing deployment of two factor Strong Authentication for all users which provides a strong barrier to networks from individuals that should not have access I implementing a continuous monitoring program for all IT systems - Creating and hiring a cybersecur ity adviser position that reports to the Director - Establishing an agency-wide centralized IT security workforce under a newly hired Chief In formation Security Of cer - Modifying the OPM network to limit remote access to exclusively government- owned computers - Deploying new cybersecurity tools including software that prevents malicious programs and viruses on our networks - Implementing a Data Loss Prevention System which automatically stops sensitive information such as social security numbers from leaving the network unless authorized and - Enhancing awareness training with emphasis on Phishing emails and other user based social engineering attacks 0PM also reports that it has taken steps to improve its cybersecurity capabilities many of which are part of the President s Cyber'security National Action Plan In particular 0PM reports being one ofthe rst agencies to fully implement Continuous Diagnostics and Mitigation CDM program and that it is targeted to complete its deployment by the end of summer 2016 0PM reports that CDM will allow 0PM to communicate with DHS more rapidly and effectively Email from Jason Levine Din Of ce of Cong Legislative Sr hrter'governmentai Affairs US Office of Pet's Mg th In H Comm on Oversight r3 Gov l Reform Staffilune 2 2015 15 54 pm on le with the Committee I14 Id 225 during cybersecurity incidents In addition 0PM has also completed the implementation of the latest release of Einstein Release 3a which is a DHS l'I defensive system that collects detects and prevents many cyber threats and potential eyber attacks before they can reach 0PM networks and its users 246 But questions remain as to the state and utility of new information technology infrasttucture How will the newly established National Background Investigations Bureau impact the new IT infrastructure that 0PM has built and that was designed for the Federal Investigative Service which will now belong to the DOD administered Such questions linger as 0PM continues to Spend tens of millions to maintain and operate both their existing legacy IT environment and the new IT infrastructure Only time will tell if 0PM is able to suf ciently respond to the call for the agency to address its information security shortcomings and IT challenges especially given the reality that federal CIOs have an average tenure ofonly two years n i As Representative Will Hard Chairman of the Information 'l echnology subcommittee stated during the first hearing the data breach at 0PM this is just another example of the undeniable fact that America is under constant attack It is not bombs dropping or missiles launching it is the constant stream of cyber weapons aimed at our data 1 4 0PM and all federal agencies must overcome the unique challenges that each faces with regard to their information environments Every American must have the con dence that the data they continue to entrust with the federal government will be protected- Agency leadership and their CIOs must be the ones to restore the public trust following the events that transpired at 0PM m Gov t Accountability Office 1-634 Fedora Chieffnformotion D ieers Opportunities Eris-t to Impact-e Role in In irmerioa Ehclmologv Management Oct Z l 1 ms 0PM Dem Hearing Before H Come on Ovenright and Gov Reform Cong June 16 HHS Statement of Rap Will I-lurd 226 Appendix Cyber security Spending at 0PM Fiscal Years 2012-2015 Table 1 Federal eyherseeurity spending by ageney in millions for nf lure quumrnL'nze nfEduealmn Depamnentaf nfiustiee ul'Lulmr Depanmem ui'StuIe ui'T Dcpumnem nI'VL-lemns Departmem nf'thc Interinr Depanmem ul'lhe Treasury Deparlmem of Defense 8 Human Services Depnnmenl of i nmeizmd See urih' Depm1memnt ilmeing a 1 Irhun Dev Em imnmemel l'releclien A Genera Services Administration Pr Elms Science l nundalmn National Aernnemlcsa Ermine Nuclear men - 5 me Pemnmel Small Business Adn tinistmiinn 5 Eswlul Sui ur Admlnislruliun 5i Tetnl Cyhersteuriiy Spending 513 1 1 mm rlul sum In 111 it'll Of ce ef Mgmt 51 Budget Exec Office of the President FF 2W5 Ammn Rep-3r to Congress F d r 'f ha n mmimr Security Management Act Mar 13 2016 doesf nal fv 20'15 sma repan cengress 18 2015 3 df 227 Table 2 Federal eyherseeurity spending by agency in millions for I lmrt Anal Shaping the igrnr Ialit inut f and Iiltgate l'yhrr-teettrit rthily- Inlruainm lint-imminent Department of Agriculltue 5-1 5th $3 Department of Commerce 556 $33 Sir-I $2l3 Depnnment of $1 '53 SI 531 Department 11' Energy SIDS $73 STI 5257 Department nl' $4 33 $44 55 Department nl Labor 5 3 53 $1 517 Department of State 55 $54 H Department of 542 $44 $5 $91 Depmntent nf' Veterans u ttTatrs SI 3 $9 $153 Department urthe 51 530 El 543 De partment nt'thc 'l'reasur 5 I22 353 i In $201 Department nt Defense $2 552 Biol 225 5 5 I73 53195 of 1E Human 554 y 525 SI 7 De trtmenl of lnntelnnd I Seepuntv $473 $122 Department of Inustng 8 Urban Sh 53 Development SCI 4 Environmental Protection Sb Agency Sf $1 General Sen tees 527 553 Pct-grants 5 54 $3 5 IE Science linundntinn 53 Se SI 5-1 5M3 i ll i if iwms 3 $35 3 5m Nuclear Regulnlm 1 53 I4 l'Jtl'tee nl Personnel Management 52 57' Sr 57 Small Hume-55 SI 54 in 5 5 Social Security S46 2 55 Total it'herscetlriry Spending 33392 $3 143 512 7115 Hi 1 Due tn retarding ext-pone may mt 311m lo the '25 Of ce of Er Budget Exec Of ce of the President 2m 43-111mm Report to Congress Federal Irg n'ttrun'mt Security Management Act 33 27 2015 W14 snta meat 12 228 3 Federal cybersacurity spending by agency in millions for 111' A icuilurc $63 5 13 of Education $22 of 5218 oI'IuSiicc 54-16 of Labor $23 ul Shnc $35 ul l' ion 3% AlTaIrz-i IE I D ot lhc $38 Dc ot'Ihn I reasur's- $263 Dc nl lkihnsa Sll Dept nt'llcallh 6 lluman Services 518 ufllomcland Securih' SI '09 Deal oi'llausmg 8 Urban Dev I Pmluclinn General Sen-ices International Asaislanca Pr aim N li l l l Science Foundation NASA Nuclear Regulalnq Ice at Administralmn Social Security Administration $40 Tutal Securiiy Spending Slim-H Of ce as Budgss Of ce nfihs President Report In Cungrm x Fnani-ni frg w mmimi Seaman Ac 65 May 1 _daca fy_2m 2014 pdf 229 Table 4 Federal cyberseeurity spending by agency in millions for FY 2012mm 51 44 - - I mr I - - 3 1 3 B 55 Ni 1 23 If pp qt 1F at Lil-F 'x c a Minn I 53 Office of Mgmt 3 Budget Ema Df ee of the President Year 30M Report to Congress on the In ammation of re Fedem Ilfammtfon Sec- rig ManagememAc Mar 2013 bfa etsfeguv_docsf r12_ sma_pdf 230 Table 5 0PM IT Budget and Spending OPM's IT Budget and Spending Over Time 13 12HMS Frzuzu rum mun rum-1 mats mails F'I'lel Bustier thetuals 115 15B 125 100 7 LI 1353 U S Of ce of Pers 0PM Cengressiauef Budget Just 'iemr'en Performance Budget VIEW 6 at 2 Feb 2315 Weheul us bud get performa I pdf Cyberseeurity is Dllt line item in total IT budget The ameunts requested fer IT spending everall and the amounts appropriated are shown in the Appendix In additien everall funding spikes in and 21333 are attributed tn 3 transfer from the Trust Fund Fer retirement modernization See 1 1 5 Of ce UFPers Mgmt 0PM Congressional Budget eri eation Perfemmnee Budget Feb 5 US Office if Pets Mgmt 0PM Congressional Budget Just ieatieu Performance Budget FY2008 Feb 5 2013 231