DOClD 4001121 i UP 'leltEY tfMIAA NSA and the Supercomputer Industry I b 3F PC86'36 NSA'S RELIANCE ON b 1 6 3 -50 USC 403 bX3 -P L 86-36 ---l HIGH PERFORMANCE PUTING TECHNOLOGY dependelltllP nhigh-performance 't' 'SCl NSA is heavily computing HPC technology heretofore known as liUpe'rcomputing technology particularly in the area o f cryptanalysis HPC is used forfundsmental cryptanalytic CA mathematical research diagnosis of unknown p-yptologics development of attacks and daily exploitation fot S1G1 production Often CA efforts which eventually fin their waxP to R · ' JJ el deVIces SP ls high-performance desktop computers juUl'their genesis in breakthroughs made on NSA l Upetcompu rs tiona y N SA - -' has- a smaller portion of its HPC assets devoted to signals processing and to the protection ofUnited States cryptographic systsms 01 _ __ _' As a consequence NSA has constructed the largest single·site singlemission supercomputer complex in the world This has allowed the Agency to gain unique HPC experience as well as leverage with HPC vendors A symbiotic relationship has evolved over time NSA shares problems and requirements with HPC vendors in exchange it gets systems that not only meet its needs better but also create improved products for the entire range of HPC customers Indeed NSA can cite multiple examples of supercomputer technology spanning several decades which was designed as a direct result ofour requests Of equal importance is technology related to the construction ofSPDs Without an extremely close relationship with HPC vendors many of NSA's current and future SPDs would not have been possible I 1 - b 1 b 3 -P L 86-36 ---' I HIGH-PERFORMANCE COMPUTING TECHNOLOGY TRENDS I te· e 1l Although vec A r supercomputers have commanded the bulk of the HPC marketplace until now a p radigm shift is under way to maesively paraliel processor MPP platforms MPPs tlieoretically offer more cost-effective hardware than vector I systems Unfortunately they suffer from the lack oflarge uniform access memory from high latencies li e • distribJted processor and memory communication times and from immature operation systems and software support tools MPPs can outperform vector machines by a wide margin f lr some types of problems However across the broad range of applications most programmers fall well short of achieving the theoretical peak I W i 1 PR8PRtM IW IIJPeRlO'ROJ'rJ pproved for Release by NSA on 9-26-2012 FOIA Case # 51546 fer 'EeRE' tltttlllA he opinions expressed in this article are hose of the author s and do not represent the IJfficial oninion of NSAfCSS OOClO 4001121 F8PSE€AR HMBM CRVPOOLOGIC QUARTERLY performance of MPPlI NSA has made extremely effective use onts MPP systems But it has come at the expense of the very best programmer talent which is a precious resource Furthermore for some CA users in particular the highly interactive researchers and those working in diagnosis the characteristics traditional vector supercomputers have to offer are still the best fit for the problems They provide an environment that fosters ingenuity and creativity while also providing rapid response to the researcher ttl 8 Ql The last point on software is multifaceted Vector supercomputers from Cray Research Incorporated CRI have well-developed and powerful operating systems and tools which are superior to those found on competing machines NSA has the world's largest pool of experience in making use of these platforms Programmers can spend their time concentrating on their difficult mission of cryptanalysis with minimal attention to the computing infrastructure Also the Agency of course has a large c01lection of legecy software estimated at close to billion of development effortlwhich can be easily reused on each succeediIig generation ofvector machines without the timely and expensive task of porting that code to a new architecture '3 U Pit8P'Nl The two largest U S vector supercomputer vendors have not neglected the technical and market trends Both CRI and Convex Computer Corporation have announced they have introduced their last pure vector supercomputers Both have MPP products available today Indeed in CRI's first year they went from 0 percent market share to become the world's leading MPP supplier in terms of revenue THE VENDORS Figure 1 shows the market share for vector supercomputers Vector supercomputers have long been the mainstay of CA processing and at least at the moment command the bulk of the HPC market for most other applications as well Fortunately for the U S CRI is the world leader in this area Convex is in the number three position a1l other vendors with any significant market share are Japanese • I compames I U One of NSA's meJor lncsrns in the HPC area is the declining marketplace at the high end Systems costing o er '10 million have been declining at almost a 50 percent rate per year as shown in figUre 2 This is due to a number of factors including erosion of the lower end of the market by high-performance reduced instruction set computers costs ofHPC in contrast to their sma1l niche in RISC the high research andI development the ovllrall computing marketplace a paradigm shift between vector processors and MPPs increasing computing performance over time for constant dollars and reduced governmental investment in HPC technology U S supercomputer vendors are also under pressure from export controls as we1l as significantly higher research and development spending by Japanese HPC multiproduct companies '8r 'EeR ' LIMB 2 I tteFfttlJ fAit l lfP6ftM1CTI 1f DOClD 4001121 NSA AND THE SUPERCOMPUTER INDUSTRY lGP EAEf YMIItlIt Hitachi Fujitsu Cray Research Inc Convex Computer Corporation Jl1g 1 Vector market sban by aompany U Ptt6PI AlI of the above have taken their toll on U S HPC vendors In the last few years sixteen domestic HPC vendors either have failed completely or have left the field of HPC ' Two more c6mpanies are in Chapter 11 ' Of the remaining companies many have never shown a profit e g Intel's supercomputer division has been kept afloat through subsidies by the larger chip manufacturing division Furthermore some have never even sold a machine e g Tera has been living off research grants and soon off investment capital alone Only four companies can simultaneously claim economic profitability significant market share systems which offer medium to high levels of performance and reasonable softwars tools including support for mass storage and highperformance networking Those four companies are CRI Convex IBM and Silicon Graphics Incorporated SGI 1 Alliant BBN Cray Computer Corp Control Dolo Corp Donoleo EIDi ETA Floating Point Systems Goodyear Multiflow Myriu Prin1e SCS Sequent Supercomputer Systems Inc in partnership with IBM and Symult taka Ametek 2 Kendall quan Research and Thinking Machines Corp 3 PR8PWBTilftlIUPtUtMJzTI8If TO' ll e U Y tJM11tJIt OOClO 4001121 -rep eRET liMB CRYPrOLOGIC QUARTERLY - 3 000 $10 Msystems $1-10 M systems $1 M systems 2 000 1 000 - o- ---------r-------- 1992 $2 9 BUlionl 1991 $' 6 SUlionl 1993 $2 3 BUlion FI I Worldwide HPe revenue by year both Voctors end MPPol Of those four companies only one em currently offers systems at the very extreme levels of high performance necessary for the demanding task of research diagnosis and exploitation of the most challenging target cryptologics Furthermore cm offers a completely balanced system which incorporates not only high performance processors but also large global memory high memory bandwidth and very fast YO It does little good to design high theoretical computational capacities if the processors are starved for data a lesson many vendors failed to learn This is of course one of the major reasons why we have been so dependent upon CRI for mainstream CA processing U Even while companies have been failing rapidly in the last few years and revenue from high-end systems has been declining CRl's market share has been steadily increasing The declining revenues have come by and large at the expense of CRl's competitors However that does not mean that cm is bealthy or that the future of HPC in the U S is secure Indeed CRl's revenues approximately $950 million have been flat for the past few years in an overall declining marketplace U Of the Japanese vendors Flijitsu has the largest market share followed by NEC and ilien Hitachi For some but not all standard industry benchmark sets the Flijitsu I lOP SECRE i UiiiaRA 4 rM'ltI l' itlty IIfF8ftMll1ll81l · OOClO 4001121 NSAANii THE SUPERCOMPUTER INDUSTRY TO' SEell1 'F IMIM and Hitachi machines have scored better than CRI so their hardware can be competitive for certain applications However the U S has a commanding lead in software development The Japanese currently seem to be concentrating on the migration to parallel processing systems with plans for both indigenous development as well as the acquisition of U S technology when appropriate THE FUTURE OF HIGH-PERFORMANCE COMPUTING The future can be divided into three phases now to 1998 1998 to 2000 and 2000 forward From now to 1998 there is little likelihood that anyone will be able to surpass CRI in performance NSA should depend on CRI for most of its CA needs during that period WOCO fR9WN Starting in 1998 things become more interesting CRI will introduce its scalable node architecture SNARK a hybrid system which will try to combine the best of both traditional vector processing and low-cost MPPs At the same time their plans are to discontinue introducing pure vector or pure MPP products thereby betting the company on the success of SNARK SGI hampered today by a bus-based architecture which prevents scaling up to a larger number of processors will be ready with a product they currently call LEGO which will use a true interconnection network SGI has the further advantage of customers in the growing multimedia and entertainment business which provides them with a volume market Convex has just announced an agreement to be purchased by Hewlett-Packard HP HP is further teamed with Intel Convex will introduce a sixty-four-bit MPP system based upon the merged HP arid Intel RlSC chips This alliance 'will be difficult to ignore as it will offer architecturally consistent hardware platforms from the desktop to the supercomputer all fully soRware compatible IBM is always difficult to count out however their SP systems are relatively high in latency and we have no indication that they plan to change this anytime soon IBM systems seem best suited for true embarrassingly parallel problems which are the most simple parallel proceSsing problems and form only a small fraction of the total problem set at NSA U It is also possible that some of the other smaller players e g MasPar NCube Tera a restructured Thinking Machines or a new start-up company might emerge as contenders in a few more years But unless they can break out of a small market niche it will be difficult for them to challenge the larger more well-established and more experienced HPC vendors Funding may be another very tough obstacle for the m to overcome as well in a very e pensive business U So from 1998 to 2000 it is likely that CRI will face serious competition from other vendors There will probably be a shakeout of the industry with not all players surviving at the high end Furthermore there will probably be no room for two vendors the current I size ofCRI 5 PRO id kin Uff8RM A181J YSP SEER tJMIRA DOClD 4001121 CRYPrOLOGIC QUARTERLY U From the year 2000 and beyond things become more difficult to predict in an industry changing as rapidly as computers However it is possible thet as RISC chips continue their phenomenal performance growth rate and as networking technology becomes better and less expensive we may begin to see yet another paradigm'shift Just as vector processors are in the process of yielding to MPPs development of MPP systems may yield to the implementation of clusters of distributed workstations in the business world Clustered RISC workstations have the same advantages and disadvantages as MPPs - only more so They are theoretically cheaper However they are currently tuned with the desktop applications in mind which means very high communications latencies poor memory bandwidth small memories and caches slow liD and immature software support Until business applications demand that these constraints be removed cluster computing using workstations will not be a viable alternative for our research and diagnostic problems However an additional force driving the industry in this direction is the ever-spiraling cost cif fabrication facilities Without the huge volumes of desktop and embedded systems onlY RISC processors will heve the capital investments necessary to sustain re arch and production will simply not exist Furthermore as a result of a consolidation in the sollware industry also brought about by the volume represented by desktopcomputinll there may be further market pressures to standardize on a fewer number of higher volume hardware platforms it'QU Jl If cm fails in the next few years NSA and the U S government would simply have no choice but to rely upon a less tems __ and Convex IBM and 8GI fo MPP system N8A would also be even more dependent than ever on Its c ip manu acturing technologies I P6tiQl Were that to happen it is important to keep several things in mind First the Agency would not be caught off guard NSA works very closely with CRI and would have sufficient advance notice Second it would probably be able to arrange for ongoing maintenance of its installed base of CRI hardware for the duration of its useful lifetime either through contracted maintenance or perhaps in-house support Third before CRI completely failed NSA would of course begin migrating its installed base of sollware to alternate systems in order to support mission requirements to the maximum extent possible I U NSA will continue i to closely monitor the volatile HPC technology and marketplace Planning should begin before 2000 to ensure thet the U S will continue to dominate HPC technology as e move into the next millennium I I RECOMMENDED COURSES OF ACTION The emphasis on courses of action should not be based on the premise that CRIwill be out of Iusiness within the next five years It is highly unlikely that they will NSA should target its emphasis toward two other goals First what can the Agency do 'to keep companies like CRI sufficiently profitable enough to be able to invest in the 'PeP 9EERET UP 'DR' 6 PiWF Eiidti UfF8R'I'TJON b 3 -PL 86-36 DOClD 4001121 NSA AND THE SUPERCOMPUTER INDUSTRY lap 5EEREf t1MIlItA development ofHPC hardware arid software to meet NSA's needs Second what can NSA do to encourage those throughout the HPC community not just the vendors · to develop state- lf-the-art software needed to take best advantage of emerging HPC technology NSA recommends the following courses ofaction be taken • Treat the supercomputing indusll'y as an aBBet critical to national defense OUQl The commercial viability of CRI and the rest of the supercomputing industry is critical not only to NSA but also to the entire Western world cryptanalytic community NSA relies on balanced high-performance computing products which currently are uniquely available from CRl and the Agency's meJor investment is in legacy applications software on CRI computers U NSA alone cannot sustain the healthofCRI or any of the other HPC vendors It needs the meJor invesiment assistance ofDoD and other governmental agencies at a minimum It is important to continue sustained annual DoD and other U S government investment in supercomputers Establishing a consortium of potential DoD users who would agree to earmark a specific sum of money for CRl or other vendors' purchases after ·consultation with the company could provide additional investment for the vendor and should be considered At this moment the bulk of this investment would'be made with the only viable company at the very high end of performance CRl U In order to counter the influence that the commercial niarket has on vendors' business strategies the U S government - not just the NSA nor the DoD - mould present a united front to the industry stating the importance of HPC to national security The interagency High-Performance Computing and Communications Program should place emphasis on projects that benefit companies interested in I scientific supercomputing I • Reduce export controls on supercomputer technology ssoeiated U The difficulties I with ihe export of high-end machines materially affect the viability of the supercomputer industry Increased market opportunities for CRI and others ca produce substantial gains in their financial health U Effective changes should be implemented in disclosure requirements within trade agreements in rder to allow American companies to compete with other foreign vendors on a balanced and fair playing field relative to export sales I I • Support expanded research funding in PC technologies U The U S govern ent commercial and academic organizations are critical to support fundamental esearch and development of PC technologies Each has a role to play in ensuring that vendors such as CRI continue to lead the world in HPC P8TJOl Research funding should continue for the Institute for Defense Analysis IDA Center for Computing Sciences and the Center for Communications Research These IDA centers should be able to acquire large research and development I I 7 SRORRII'fAWtUiPeflMA f10N fSP SEe tlM11tA DOCID 4001121 10 JfERRUMIP' CRVPi'OLOGIC QUARTERLV supercomputers· for their Vital work AI80 the interagency High-Performance Computing and Communications Program and joint research efforts such as Cooperative Research and Development Agreements and the gPD development projects should encourage continued progress in HPC technology via public lprivate partnersbipe Research funding should be supported 80 that small systems from other likely HPC companies can be brought in for evaluation and feedback Ongoing revenue even at a small level ill important for product development u U Increase staffing and training to organizations which have the expressed charter to develop or port all levels of 80ftware to MPP technology This inc Iudes traditional NSA organizatione ae well as Federally Funded Research and Development Centers Since industrial research tende to be short-terin long-term computer science reaearc h being done in academia ie important for future technology transfer to the marketplace U In those casee in which there is an insufficient commercial marketplace for HPC technologies the federal government may need to intervene One example of this is the limited demand for high capacity l'aat SRA ' I static random accoBS memory memories Although it is technically feasible to build these memories today the bulk of the production is going to slower DRAM dynamic random access memory memories used on desktop PCe which command a larger share of the commercial marketplace U lOP seacrrtfMIM 8 RROMlI 4fti l1UOKMXIION DOCID 4001121 NSA AND THE SUPERCOMPUTER INDUSTRY lOP ' ERET tJrw'BRA - - - - - - - - - - - - - - J Otr lis the Assistant Technical Director in J1 the Office of Cryptanalytic Processing He has been with NSA since 1987 serving as a computer analyst and computer scientist From 1987 to 1988 he was assigned to T3363 Supercomputing m nx Operating System Support From 19811t 1990 he was a T Systems Researchlntegree at the Institute for Defense Analysis Supercomputing Research Center and from 1990 to 1992 he was assi ned to T3353 b 6 lsamem ero 6 and N A ngmeering Society He received the author of several articles he is professio n al l ze a s a o m p 'u te r -r and is a Master in the Computer Systems Technical Track Derlved From Declassify On 9 f nUl Itl lTlift 2nTPQQ ' 0 TION NSAlCSSM 123-2 Dated September 1991 Source Marked OADR Date of source 3 Sep 91 1'8r 5EERE'TllUll po 6