INTERCONNECTIVITY AND VULNERA BI LITY REPORT December 1996 Office of the Manager National ommunicati ens System 701 South Courthouse Road Arlington VA 22204-2193 INTERCONNECTIVITY AND VULNERABILITY REPORT December 1996 Prepared by Booz Allen and Hamilton 8283 Greensboro Drive McLean VA 22102 Prepared for Office of the Manager National Communications System Under Contract DCA100-95-C-0113 Optional Task Orders EP Telecommunications Performance Analysis Task Network Security Support CDRL Item L001 and lnternet PSN lnterconnectivitjrr and Vulnerability Report Section 3 2 5 Contents Table of Contents 3 2 6 Figures and Tables List of Exhibits 3 2 9 Symbols Abbreviations and List of Acronyms Acronyms 3 3 1 - 3 3 6 Summary and Body ctions 1 - '3 References Reterences ill 1 lill Jill I I TABLE OF CONTENTS EXECUTIVE SUMMARY 1 INTRODUCTION 1 2 1 3 ORGANIZATION 2 HISTORY OF THE 3 INTERNET 3 1 INTERNET SERVICE PROVIDERS 3 1 1 National Service Providers 3 1 2 Regional Service Providers 3 1 3 Resellers 3 2 INTEREXCHANGE 3 2 1 IXP Functionality and Architecture 3 2 2 IXP Peering Agreements 3 2 3 National scope IXP Architecture Example 3 2 4 Metropolitan IXP Architecture Example 3 3 INTERNET ROUTING PROTOCOLS 3 3 1 Routing Information Protocol 3 3 2 Open Shortest Path First 3 3 3 Border Gateway ProtOCol Version 4 3 4 INTERNET ACCESS 3 4 2 Residential Access 4 INTERNET 4 1 INTERNET ANALYSIS TOOL FUNCTIONALITY 3-4 3-8 3-10 3-11 3-12 3-13 3-1-1 3-15 3-16 4 1 4 2 INTERNET ANALYSIS TOOL IMPLEMENTATION 4-3 4 3 INTERNET ANALYSIS RESULTS 4 3 1 Internet Analysis Methodology 4 3 2 Internet Analysis 5 VULNERABILITIES 4-4 5-1 5 1 INTERNET SERVICE PROVIDERS 5 1 1 Ill l Ill II all 5 1 2 Regional Service 5 1 3 Raellers - 5 2 INTEREXCHANGE INTERNET ACCESS aI-nunynunua-u APPENDIX A APPENDIX 3 LIST OF ACRONYMS REFERENCES iv 3 3 5 5 5 5 LIST OF EXHIBITS Exhibit 2-1 Internet 2-1 Exhibit 2-2 Original NSFNET Exhibit 2-3 NSFNET Three Tier Infrastructure 1986-1995 2 1 Exhibit 2-4 1938 T1 Backbone 2-5 Exhibit 2-5 1992 T3 NSFNET Backbone Exhibit 2-6 The National Science Foundation VBNS Network 2-9 Exhibit Countries and Networks Connected to NSFNET as of April 1995 2-1 Exhibit 3-1 REpresentative NSP Backbone Network 3-3 Exhibit 3-2 NorthWestNet Backbone Network 3-6 Exhibit 3-3 Backbone Network Exhibit 3-4 Selected Major IXP Locations 3-8 Exhibit 3-5 Typical National-scope IXP Configurations 3-9 Exhibit 3-6 PacBell San Francisco NAP Hybrid 3-12 Exhibit Analog Modern and ISDN Characteristics 3-18 Exhibit 3-8 Asymmetric Internet Access Characteristics 3-13 Exhibit Sample Output From the 4-2 Exhibit 4-3 IAT Site Locations 4-3 Exhibit 4-3 Internet Analysis Methodology 4-5 Exhibit 4-4 Status of IAT Traces 4-6 Exhibit 4-5 Categorization of Unsuccesstul IAT Exhibit 4-6 Average Round Trip Time Versus Time of 4-8 Exhibit 4-7 Typical Traffic Patterns at Exhibit 4-8 Typical Traffic Patterns at MAE-WEST Exhibit 4-9 Average Number of Hops Versus Time of Day Exhibit 4-10 Top 50 Routers' Normalized Frequency of Use 4-11 Exhibit 4-1 Normalized Frequency of ISP Network 4-12 Exhibit 4-12 Booz-Alien s Critical ISP Networks 4-13 Exhibit 4-13 Proxima s Critical ISP Networks i 4-13 Exhibit 4-14 Shared Critical 4-14 Exhibit 5-1 PN Three Tier Restoration Architecture 5-2 Exhibit 5-2 Internet Architecture Vulnerabilities 5-4 EXECUTIVE SU ARY Background The Office of the Manager National Communications System performs a broad range of activities in fulfilling its mission These activities include analyzing communications networks that support national seCurity and emergency preparedness EP communications As more businesses government organizations and the public use the Internet for their daily activities it has become more important for the OMNCS and its constituents to understand the operation of the Internet and its dependence on the existing conununications infrastructure The phenomenal growth of the Internet has been one of the most significant technological events of the last several years As an instrument for sharing and distributing information the Internet will be iudged one of the maior milestones of the latter part of the 20th century The exponential growth in lnternet traffic has fostered the concept of the Internet as the ubiquitous tool for sharing information However the accessibility and availability of the Internet depend on a physical infrastructure of software routErs and tranSmission media It is commonly pErceived that the Internet and the public telephone networks in the United States are two separate and distinct systems Although this is true to a certain extent most data networks including the Internet rely on the public networks PN to transport their traffic Internet Definition At the highest level the current Internet consists of multiple national and regional Internet Service Providers and interconnection points where the lSP's meet and exchange traffic This infrastructure is similar to that of the old National Science Foundation NSF network NSFN ET which consisted of a structure - Backbone network - Regional networks I Local campus networks The NSF NET was decommissioned in 1995 In its place are multiple nationwide networks similar to the original NSFNET backbone network Regional networks still aggregate their traffic and hand it off to the nationwide backbone nehvorks to which they are connected Interexchange points are located nationwide to facilitate the exchange of traffic between national and regional ISPs ationat Service Providers provide national backbone service This type of service provider owns or leases its own backbone network and has a nationwide customer base Additionally NSPs are generally cennected to all the major and vi 1 111 have peering agreements with other major NSPs at these exchange points Traffic originating with a customer on an that is destined for a customer on another NSF is transferred from the originating network to the terminating NSP's network at an Regional Service Providers are similar to the NSPs in that they own or lease their backbone network but they are much smaller in scale Their networks encompass a single region and usually have a regional customer base RSPs have peering agreements with NSPs to transfer traffic over the Internet RSPs either connect directly to the NSP or connect to an where they transfer traffic to the NSP network With the dissolution of the NSFNET backbone the NSF sponsored three primary and one secondary Network Access Points NAP The NSF's concern was that without the sponsorship of a core set of exchange points the commercial backbone providers would set up a conglomeration of bilateral connection pomts that would potentially result in routing chaos Each NAP operator provides the exchange facility while the that connects to the NAP establishes peering agreements with the other connecting to the same NAP The purpose of a peering agreement is to ensure that traffic from one ISP can reach all the customers on another by exchanging routing information between the two lSPs The current number of on the Internet far exceeds the original four NAPs sponsored by NSF The term is applied only to the NSF-sponsored lXPs whereas all IXPs provide the same functionality which is a common place for ISPs to exchange data Analysis The Internet is a very dynamic entity in that it is constantly evolving and growing Therefore it is impossible to accurately identify all components of the current Internet To develop the data for this report the Internet was analyzed to identify key components used to transmit network traffic across the Internet To achieve this purpose a software tool called the Internet Analysis Tool 1A1 was used to automatically trace the routes used to send traf c between two hosts on the lnternet The collects data from the set of routers an internet packet traverses on its path from one host to another The analysis of the routes identified by the yields traffic trends and identifies key components in the Intemet infrastructure vii For this analysis two IAT source sites were chosen I Booz'AIIen Hamilton McLean Virginia on the network - Prosima Inc McLean Virginia on the MCI Network The tool collected routes from each of these two sites to 105 other sites located across the United States The type of Web sites chosen for this analysis wEre the following 23 NCS Member Organizations Web sites 50 State Web sites Major university Web sites Popular commercial Web sites The output from an execution is the set of routers in the path between two hosts For each router three datagrams were sent at different times of the day and the round trip time from the originating host and the router was collected Analysis Results Traces performed throughout the test period indicated high success rates averaging between and 89 percent Of the unsuccessful trace attempts most resulted from an unreachable node a router or the destination server in the path that was probably either shut down or incompatible with the IAT software Internet use is highest during mid-to-late afternoon business hours Based on the round trip time for packets to traverse the network congestion peaks between the hours of 12 00 noon and 4 00 pm eastern time This analysis indicated that the number of router hops did not var l in accord with the time of day or the dag r of the week Thus the predictability of internet routing along with an increasing dependency on this conununications medium renders it vulnerable to targeted and intended network disruptions Routers appear to share a somewhat balanced traffic load within the backbone networks excluding those routers closest to the two sources As expected a high number of router Visits occurred in the initial hops of the traces These initial routers are critical to the sources however they are not necessarin critical to the entire Internet As the trace moved away from the source and into the backbone networks the number of visits per router stabilized Therefore a single critical router could not be identified however it could be determined which net'WOrks were more heavily traversed For this analysis MCl s network was traversed most frequently and was therefore critical to the success of the traces i ll Vulnerabilities The Internet can provide service in a volatile unreliable network environment But like the PM the Internet has vulnerabilities that can severely degrade its level of service Because the Internet relies on FM packet and circuit switched networks it is vulnerable to the same cable cuts and other damage that can affect the PN In addition some restoration techniques used by the PN carriers for circuit switched traffic cannot be used on the lnternet's packet switched traffic National are critical to the operation of the U5 portion of the Internet An IXP failure could greatly reduce the Internet s ability to transport traffic nationwide or even worldwide Congestion at these has also convinced ISPs that it is necessary to establish secondary means of interconnecting with one another Network routing protocols dictate how traffic is directed through the network in that they determine the paths that should be taken through the network to avoid congestion and network outages Some Internet routers are vulnerable to thrashing the optimal path through the network changes so frequently that the router spends more time computing these paths than actually routing users data In summary the initial analysis has determined that the Internets physical vulnerabilities are consistent with the vulnerabilities of other large communication networks most notably last mile issues and loss of backbone transport Additional vulnerabilities exist that are distinct to the Internet congestion exponential growth in traffic routing software and network server management issues 1 INTRODUCTION The National Communications System NCS is a federation of 23 federal departments agencies and organizations that are responsible for the survivability and interoperability of various components of government communications supporting national security and emergency preparedness activities The Office of the Manager National 'Communications System 0MNCS is the planning and operational element of the NCS The OMNCS pErforms a broad range of initiatives in fulfilling its mission including analyzing communications networks that support EP communications The analysis process utilizes a standard OMNCS modeling methodology that incorporates OMNCS and commercial-off-the-shelf models as well as public and proprietary data BACKGROUND The phenomenal growth of the Internet has been one of the most significant technological events of the last several years As a instrument for sharing and distributing information the Internet will be judged one of the major milestones ot the latter part of the 20th century The introduction of Web browsers dial-up communications protocols Point-to-Point Protocol Serial Line Interface Protocol SLIP and inSock and the increased efficiency of routers have made Internet access possible and cost effective even for small-business and at-home personal computer PC users The exponential increase in Internet traffic has fostered the concept of the Internet as the ubiquitous tool for sharing information However the accessibility and availability of the Internet depend on a physical infrastructure of software routers and transmission media As more businesses government organizations and the public use the Internet for their daily activities it becomes more important to understand the operation of the Internet and the reliance of the Internet on the existing communications infrastructure The infrastructure that supports the Internet has evolved from mainframes and large minicomputers using dedicated transmission lines to low-cost routers and dial-up access from modems on PCS Additionally a growing support industry is providing Internet services software and content As the Internet continues to evolve its users will increasingly be dependent on not only the physical infrastructure but also the supporting services that have allowed the Internet to become an unparalleled information sharing tool 1 2 SCOPE This report describes the Internet by tracing its growth and development over the last three decades It is difficult to provide a detailed definitive history of the Internet 1-1 Ill Jill I because much of its history has incorporated computer folklore and anecdotes However the major Internet milestones have been captured and serve as a baseline for its future growth In context of the current description of the Internet and the Public Networks PM this document addresses several lute r vulnerabilities These vulnerabilities are quantified using a simple route tracing tool that determines the physical path of Internet traffic The Internet routes are then overlaid onto the PN infrastructure to iilustrate the interdependence of the PM and the Internet 1 3 ORGANIZATION This document is organized into five sections Section I Introduction provides the background and scope of the Internet PEN lntercormectivity and Vulnerability Assessment Section 2 the History of the Internet provides a detailed description of the history of the Internet from its earliest inception in 1969 up to the dissolution of the National Science Foundation s NSF backbone network in 1995 Section 3 Internet Definition presents a breakdown of the different types of service providers a description of the Internet infrastructure at a high level and a discussion of the relationship of the Internet infrastructure to the PN infrastructure Section 4 Internet Analysis describes the Internet Analysis Tool IAT functionality and implementation This section also presents the analysis methodology and results from the IAT Finally Section 5 Vulnerabilities analyzes the current infrastructure of the Internet and discusses its major vulnerabilities 1-2 1 HISTORY OF THE INTERNET The Internet is a very complex entity of more than 10 million hosts connecting over 95 000 networks To fully describe what the Internet consists ot today it is necessary to look at how the internet began and evolved to its current state The roots of the technology employed by today's Internet are found by analyzing its evolution This section provides a detailed description of the history of the Internet beginning with the initial work performed by the Defense Advanced Research Projects Agency in 1969 to the recent commercialization of the Internet and the dissolution of the National Science Foundation Network NSFNET backbone in 1995 Exhibit 2 1 shows a timeline of the history of the Internet that this section will discuss in detail Exhibit Internet Timeline ARPANET - - arr Comma al - I lme 59 D 3 n NAP no and HILHET are - meat print Roule Servers pp imm- Eumf A tasF antral-ts Ht Irlem 9 Faro-1 lms for 1 All Han ns Pro-at T F '9 mg a radeh roe-t BVDWI 5 Est-Him I 39' Ca 1970 1980 1990 1995 I I i I inn-L and 73 p sub-shad I Standard APPANET NSF ssues Macaque Draiml fur Esubiuhl-d Flutrao Sol-Citations ta- Pacnl Swiss-nag DAR PA a ARPANET Using Prop-nus tor Tia- door NSF Ind DOD merrier VP HS le Arc-i Hume m Ere-W 'mnm Ul' CSHET Gateway In cu Em CSHET Ind temp- 11 Corinna Dun ARPAHFT m elm-i research Cmm il tra'fc TCPIIP Prom-Cd ISF tram now romeo thmuqh wIErmnnecte-a 2 5mmqu nth-on The inception of the Internet can be traced to 1969 when DARPA was commissioned by the United States Department of Defense DOD to develop a communications system that would be survivable in the face of enemy attacks including nuclear war In addition the network should allow military and academic researchers to collaborate on research projects and share computer processors across the coun In response to this 11 direction DARPA later renamed ARPA set up a network consisting of the following four nodes I- University of California at Los Angeles - Stanford Research Institute 0 University of California at Santa Barbara University of Utah ARPA used this four-node network referred to as ARPANET to experiment with the linkage to be used between and military research contractors In 19 0 ARPA began researching packet switched technology The goal of this technology was to decentralize the network by giving all nodes on the network equal authority to transmit and receive packets across the network The route each packet took to its destination was unimportant as long as it reached its destination Thus packet switching technology was effective when network connections were unreliable This packet switching technology employed by ARPA during the seventies was known as the Network Control Protocol NCP By the end of19 1 there were 15 nodes connecting 23 hosts to ARPANET in 1973 ARPA began the Internetting proiect The goal of this project was to develop a protocol that could seamlessly pass information between different networks This project culminated in 197 in a demonstration of networking through various media including satellite radio telephone and Ethernet The protocol developed in this project formed the basis for the Trans mission Centre Protocol and Internet Protocol If where IP handles the addressing of the individual packets while TCP coordinates the proper transmission of information By the end of 1982 ARPA established IP as the protocol suite for the ARPANET requiring that all nodes connecting to ARPAN ET use Additionally declared that l was to be its standard protocol The official cutover from NCP to IP was executed on January 1 1983 Aiding this transition was the incorporation of into Version 4 2 of Berkeley Standard Distribution of UNIX This version of the UNIX operating system was free to anyone who wanted it thus ensuring a wide deployment for The marriage of and UNIX began a long-standing affiliation between the Internet and the UNIX operating system that continues today Another major event in 1933 was the division of ARPAN ET into two networks ARPANET and MILNET MILNET was to be used for military specific communications whereas ARPAN ET was to continue its research and development in networking computers MILNET was integrated with the Defense Data Network created in 1982 The funding for ARPANET was provided by Defense Advanced 9 Id Research Projects Agency DARPA By 198-1 the number of hosts connecting to the ARPANET was more than 1 000 While the ARPANET was undergoing major changes another significant event in the history of the internet occurred In 197 representatives from DARPA and the NSF and computer scientists from several universities met to establish a Computer Science Department research computer network CSNE F One of the driving forces for the establishment of CSNET was the concern that computing facilities located at universities not connected to ARPANET did not have the same advantages in research and staff and student recruitment as those who were connected In 1981 CSNET was fully operational through money granted by NSF AlthOLigh designed initially to be a standalone network CSNET later incorporated a gateway connection to the ARPANET In the summer of 1980 a DARPA scientist proposed the interconnection of the not yet established CSNET and ARPANET using protocols that would provide services and the seamless transmission of information between users regardless of the type of network This set of protocols was IP The gateway connection between the two networks was established in 1933 In 1936 the NSF created the NSFN ET The purpose of this network was to provide hi gh- speed communications links between five major supercomputer centers located across the United States Although ARPANET was ourishing its S -Kbps backbone and network topology could not fulfill the demand for high-speed networking req uired by multiple research projects The goal of the NSFNET was to provide a reliable environment for the U5 research and education community and access to the major supercomputing centers The NSFNET essentially duplicated the functionality of the ARPANET NSF chose TCPX as the standard protocol for its new network This new network ultimately led to the downfall of ARPANET in 1990 ARPANET was formally retired The infrastructure of the NSFNET was a three-tier hierarchical structure 0 National backbone 0 Regional networlq - Local area networks LAN The original backbone of the NSFNET depicted in Exhibit 2-2 consisted of a So-Kbps nethrk This backbone network is Considered the basis of what is now called the Internet Regional networks hung off the backbone network and provided services to LANs at education and research facilities Universities and research associations combined to form the regional networks which in turn would aggrEgate their traffic and hand it off to the NSFN ET backbone Exhibit 2-3 depicts the three tier structure implemented in the NSFNET thmughout its existence Ill 1 Exhibit 2-2 Original NSFN ET Backbone 1935 mu m NSF Exhibit 2-3 NSFN ET Three Tier Infrastructure 1986 - 1995 rennin the Internal 5 Infrastructure Cone-sled oi a three-tiered oi smellm from nonmetal-lg upward tn linger 'm l'l Chm backbones of ever-mung bandwidth 45 COWHIE Research and cam Research ad campus Development Devebpn-Ient Educallonll Educatnnal Inst-tum Inshluie We Cmumam Because NSFNET's primary focus was for nonprofit research and development by universities and research groups NSF instituted an acceptable use policy that restricted the of the NSFN ET to noncommercial activities Additionally NSF offered financial help to those regional networks composed of university and research facility LANs who wished to connect to the NSFNET backbone By 1937 the NSFN ET outgrew its existing capacity NSF awarded a five year contract to Merit the Michigan state networking organization with MCI and IBM The purpose of this contract was to transition the NSFNET backbone to T1 links and provide several access points around the country Merit's role was to manage the backbone including routing whereas 18le provided the routing equipment and MCI provided the trunk lines- The transition to a T1 backbone was completed in 1988 By the end of the 19805 more than 100111 hosts from 1 countries worldwide were connecting to the NSFNET Exhibit 2-4 depicts the T1 backbone of the NSFNET in 1983 Exhibit 2-4 1933 T1 NSFNET Backbone l 988 Seame WA WHJ Pain Alto CA College Pam MD Sou rce NSF As the NSFNET grew some organizations realized that providing services and functionality similar to that of the NSFN ET without the access restrictions was a golden business opportunity These organizati0ns experienced in providing regional network operations seized the opportunity to set up their own natioHWide backbone networks Thus the first commercial Internet service providers were created These providers included Performance Systems PSlNet and Alternet which was generated from 1 Jill I I I Technologies The main focus of these networks was to provide the same functionality as the NSFN ET over their own networks but without any access restrictions In 1991 the fourth year of the five-year contract Merit IBM and MCI formed a new nonprofit corporation Advanced Networks and Services ANS which was given the operational responsibilities of the NSFNET In June 1991 ANS announced it would provide commercial access to the Internet thus nullifying the acceptable use policy By broadening access to the Internet ANS increased its efforts to expand connectivity and make the Internet a more powerful tool The new evolving private commercial networks were hindering research forcing researchers to spend time accessing several networks all in the name of science With expanded commercial providers on the lnternet there was a single common network that increased a researcher s ability to find any information needed and focus on the research at hand When NSF lifted its access restrictions in 1991 allowing commercial traffic on the NSFNET ANS formed a for- profit subsidiary ANS Commercial Research 6 Education to provide full commercial traffic across the backbone Once the acceptable use policy had been abolished Technologies and General Atomics created the Commercial Internet Exchange CIX CIX was a traffic exchange point between the NSFN ET and the commercial Internet service providers networks The other major event that occurred in 1991 was the transition of the NSFNET backbone from T1 links to T3 This transition like the initial transition from S -Kbps to T1 links in 1988 was because of the capacity of the backbone network could not meet the traffic loads Although this transition required new routing equipment and interfaces and at times proved to be technically challenging it was accompiished with relative ease This was due to the fact that the same organizations who were managing the old T1 backbone were responsible for implementing and overseeing the new T3 backbone network Additionally the T1 backbone still existed as a backup if the new network tailed Exhibit 2-5 depicts the T3 backbone as of 1992 In 1992 Vice President Al Gore drafted legislation that proposed a National Research and Education Network NREN This new network would consist of T3 links separate from those making up the NSFNEF backbone and would connect all schools libraries etc for a cost of over $2 billion Even though the legislation was passed no new network ever came into existence The REN effort did however succeed in sparking a greater interest in the Internet 9 Exhibit 2-5 1992 T3 NSFNET Backbone 1992 Seams WA ten MA Lincoln NE F an Ann NJ Plr l MD 3 Exterior Nod-es 0 Core inter-or Nodes rwource' NEI- The new public lnternet coincided with the release of the first Microsoft Windows version of Mosaic in 1993 Mosaic developed by the University of at Urbana- Champaign was an X-Windows interface to the World Wide Web The concept of the was started in 1989 in Switzerland as a means to easily share information among researchers in high-energy particle physics In 1991 the first server came into existence but without any client software The introduction of the first interface to included the capability to navigate through the Web via the mouse Today s Web browsers such as Netscape include File Transfer Protocol FTP E-rnail Telnet and many more capabilities The use of a graphical interface to access the Internet has played a significant role in the popularity growth of the network because it allowed access to the Internet without having knowledge or possession of the UN 1 operating system While the look and feel ot the Internet was undergoing changes NSF in 1992 began to question its role in the network NSF observed that its backbone network was operating in conjunction with several commercial nationwide backbone networks Essentially NSF was paying for users to access its network and thus the Internet whereas the other commewcial service providers were being paid for access to theirs Although in 1991 the NSF had notified the regional networks that they would have to become self-sustaining it was 1992 before the NSF took action The NSF began considering ways in which it could successfully pull out of the Internet arena with little Jill I disruption to the Internet while continuing its commitment to the education and research community With the five-year contract between the NSF and Merit drawing to a close Merit was granted an 18 month extension brayond the original October 1992 expiration date to allow the NSF time to work out how to transition its backbone network into a new structure This work culminated in a solicitation for proposals Solicitation 93-52 in the foilowing four areas that compose the new national Internet structure a Network Access Points NAP 0 Routing Arbiter In Regional network provider awards In A very high-speed Backbone Network Service The APs act as interconnection points where commercial Internet service providers can meet and exchange traffic The NSF believed that without such interconnect points backbone providers would likely establish their own independent bilateral connect points that would stifle the NSF's plan for full connectivity for the research and education community The NAP manager contracts were awarded to the following I Sprint for a New York NAP I Metropolitan Fiber Systems MFS Datanet for a Washington DC NAP - Bellcore and Ameritech for a Chicago NAP In Belicore and Pacific Bell for a California NAP The Routing Arbiter is an independent group that operates route servers at each NAP The transfer of traffic among the backbone providers that meet at the NAPs is facilitated by route databases contained in the route servers These databases contain routing information and policy requirements for each backbone provider and therefore indicate to which provider the incoming information should be sent This contract was awarded to Merit and the Information Sciences Institute 151 at the University of Southern California which together make up the Routing Arbiter group l vfith the dissolution of the NSFNET and the introduction of NAPS and commercial traffic access to the Internet by the NSF subsidized regional networks was no longer free The commercial backbone providers were now paying a fee to interconnect with the NAPs and passing these charges to their users the regional network providers Therefore the NSF decided to the regional network provider contracts to alleviate the regional networks initial shock of having to pay for Internet access The awards provided the regional networks with annual NSF funding with the funding declining to zero over a four-year period The regional network providers would use the subsidy to pay the commercial Internet providers who were in turn required to connect to the NAPs There were 17 contracts awarded to regional network providers for interregional connectivity 2-8 The NSF also proposed to sponsor a new backbone the operating at a minimum speed of OC-B 155 Mbps to link the following five NSF supercomputer centers - Cornell Theory Center a National Center for Atmospheric Research I National Center for Supercomputing Applications - Pittsburgh Supercomputing Center - San Diego Supercomputing Center Unlike the general purpose NSFNET infrastructure the functions as an advanced research laboratory allowing research development and integration of new networking requirements using technology beyond just IP routing There is a strict acceptable use policy the may only be used for meritorious high-band width research activities and it may not be used for general Internet traflic NSF entered into a fivesyear agreement with MCI to provide the During this five-year agreement MCI is expected to participate in the development and use ot advanced Internet routing teIChnologies At the end of the agreement it is anticipated that technology will exist that will increase the transmission speeds beyond 2 2 Cbps Additionally the will act as an experimental platform for the development and testing of broadband Internet services and equipment Exhibit 2-6 depicts the NSF's network Exhibit 2-6 The National Science Foundation Network Cornell Th a a - r Hatlonm Canter - motor I for Atmospheric Il' Research dl in I g Putt-Dumb 3 an Due-go Cornouter unmet T1 T3 NetStar IP Router 3 Clsoo IP Router ATM Sw ch li'l 2-9 I The result of NSF's solicitation for proposals was a new Internet structure In April 1995 the NSFNET backbone was formally retired At that time 93 countries and more than 50 003 networks were connected by the NSFNEF backbone Exhibit details the number of networks by co untry connected to the NSFNEF backbone by the end of the project NSF's original task lwas to improve the previous NSFNEI backbone push the technology to newer heights and implement it on a national level It was hoped that this would place a powerful tool in the hands of the research and education community and create innovative use and applications goals were accomplished the NSFNET backbone connected most of the higher research and education community to a robust and reliable high-speed network and it served as the sole player in making the Internet industry The NSF's task will continue to evolve in two directions 1 providing support tor the research and education community by guaranteeing the availability of services resOurces and tools to keep the Internet connected and 2 by continuing to push networking technology using the VBNS 2-10 Exhibit 2-7 Countries and Networks Connected to NSFNET as of April 1995 ve-Country -- 'Tota1 emu-Country Total Country Total u Networks Networks Networks Algeria 3 Greece 105 Norwav 214 Argentina 2 Guam 5 Panama 1 Armenia 3 Hong Kong 95 Peru 4-1 Australia 1375 Hungarv 164 Philippines 46 Austria 408 Iceland 31 ola nd 131 Belarus 1 India Portugal 92 Belgium 138 Indonesia 46 Puerto Rico 0 Bermuda 20 Ireland 168 Romania 26 Brazil 165 Israel 21 Russia 105 Bulgaria 9 ltalv 50o Senegal 11 Burkina Faso 2 Jamaica 1o Singapore 107' Cameroon 1 Japan 1847 Slovakia 69 Canada 4795 Kazakhstan 2 Slovenia 46 Chile 102 Kent a 1 South Africa 419 China 8 Korea South 4113 S in 25 Colombia 5 Kuwait 8 Swaziland 1 Costa Rica 5 Latvia 22 Sweden 415 Croatia 31 Lebanon 1 Switzerland 32-1 Cvprus 25 Liechtenstein 3 Taiwan 525 Czech Rep 459 Lithuania 1 Thailand 1117' Denmark 48 Luxembourg 59 Tunisia 19 Dominican Rep 1 Macao 1 Turkei' 3 Ecuador 85 Malaysia 6 Ukraine 60 Egypt 7' Mexico 126 Unit Arab 3 Emirates Estonia 49 Morocco 1 U K 143a Fiji 1 Mozambique United 28-1311 States i nla nd 0-13 Netherlands 406 rugpa 1 France 2003 New 1 Usbekistan 1 Caledonia French Polynesia 1 New 556 Venezuela 1 Zealand Germanr 1 7'50 Nicaragua 1 Vietnam 1 Ghana 1 Niger 1 Virgin 4 Islands Source Merit Network Inc 2-11 3 INTERNET DEFINITION At the highest level today's Internet consists of multiple national and regional Internet Service Providers ISP and interconnection points where the ISPs meet and exchange traffic This infrastructure is similar to that of the old NSFNET which consisted of a three-tier structure - Backbone network 0 Regional networks 1 Local campus networks On the NSFNET regional networks would aggregate their traffic and hand it off to the NSFNET backbone The regional networks comprised multiple local business and campus networks Although there were many regional and local networks there was only one backbone network As mentioned in Section 2 the NSFNET has been decommissioned In its place are multiple nationwide networks which are similar to the NSFNET backbone network Regional networks still aggregate their traffic and hand it off to the nationwide backbone network to which they are connected Interexchange points are located around the country whEre traffic is exchanged between national and regional lSPs Peering agreements are used between the connected at an IXP to determine how traffic is routed These service providers and interexchange centers are the main components of the US Internet This section will describe different elements of the Internet architecture and the different routing protocols used on today s lnternet 3 1 INTERNET SERVICE PROVIDERS ISPs are classified according to their network and customer base The network classification refers to whether or not the ISP owns or leases its network An 15 that does not own or lease its network is referred to as a reseller The customer base classification refers to an type of customers national or regional A particular 19 may have national and regional customers but generally it has more of one type than another There are three types of lSPs National Service Providers NSF - Regional Service Providers RSP Resellers The following sections provide further detail for each type of ISP 3 1 1 National Service Providers The first category of ISPs is NSF which provide national backbone service This type of service provider owns or leases its oWn backbone network and has a nationwide customer base Additionally NSPs are generally connected to all the major IXPs and have peering agreements with other major NSPs at these exchange points Traffic originating with a customer on an NSP that is destined for a customer on another NSF is transferred from the originating network to the terminating network at an IXP The NSPs network infrastructure consists of routers network layer and switches data link layer that are owned by the NSF The follow-ring are examples nt NSFs ANS BBN Sprint UUNEI Of the NSPs MCI and Sprint are the only two that own their entire network Other NSPs may own small parts of their networks but most of their networks consist of circuits leased from the PM providers Most of these circuits are leased item the large Intereuchange Carriers IEQI However some circuits are also leased from the Local Exchange Carriers LEC Bell Atlantic Competitive Access Providers CAP Metropolitan Fiber Systems and smaller lECs LDDS Exhibit 3-1 depicts a rePresentative backbone network for one of the NSPs mentioned above As shown in the exhibit like most NSPs has redundant connectivity between each switching node on its backbone network NSPs rarely sell directly to small consumers small businesses and residential customers because of the added customer handholding required by smaller less espe rieHCed users instead NSPs sell their services to large businesses and resellers Resellers in-turn resell Internet service to small business and residential customers It is important to note that not all NSPs resell their networks e g The architecture of an network may be separated into access and transport Access refers to the customer s connection to the NSF whereas transport refers to the backbone of the network Customers connect to NSPs via leased and dial-up lines Typical leased lines are 56-Kbps or T1 and usually terminate at an NSP's point of presence POP MCI advertises that 40% of all intemet traffic travels over MCI circuits This includes traihc on MCl's NSF and traffic on other NSF that use MCI leased lines 3 2 1 I 1 Jill I Exhibit 3-1 Representative NSP Backbone Network cumin-mu ill Hl Il For dial-up customers the NSP usually has digital and or analog modem banks terminating from its POP into the local central office using Tls Because NSPs have national presence and reach once a customer s traffic reaches an POP it has essentially reached the Internet The typical backbone of an NSP comprises routers and switches connected by T1 T3 or even OC-Jevel circuits These circuits may be leased from one or more IEC One NSF PSlNet leases backbone circuits from five different The NSP market has not escaped the notice of existing PN providers anxious to get involved in the growth of the Internet In the short term PN providers have chosen to partner with providers for internet backbone transport instead of deve10ping NSP expertise in-house For example GTE recently announCed a partnership with to provide internet access under the GTE name to customers in 46 US states Cross PN- NSP service agreements also exist behveen Pacific Bell and America On-Line which owns ANS and A I'dr'l and BBN 3 Washington Post luly 11 19% Page 19 3-3 The recently announced merger between and MFS may be a harbinger of tuture mergers between NSPs and PN providers PN providers own the data links necessary to run an NSF and have the marketing savvy to sell Internet service to business and residential customers NSPs on the other hand have the inshouse technical expertise to manage the switches routers and interconnection arrangements necessary to make the NSF backbone work Other future developments in the NSF market will include service differentiation to target selected customer markets For example MCI and BBN have announced services that provide a higher quality of service to business customers who subscribe to their NSF BBN provides priority treatment to business customers through Internet Protocol version 6 priority service protocols MCI provides a mparate network for its business subscribers' Internet traffic This separate network includes locally hosted mirror sites from popular Web sites on other NSP networks and in the future will include ll v6 priority treatment 3 1 2 Regional Service Providers The second category of ISPs are the RSPs These service providers are similar to the NSPs in that they own or lease their backbone network but are much smaller in scale Their networks encompass a single region and usually have a regional customer base RSl s have peering agreements with NSPs to transfer traffic over the Internet RSPs either connect directly to the NSP or connect to an where they transfer traffic to the NSP netvvork NorthWestNet is an example of an that connects directly to an NSF NorthWestNet which provides service to customers in Washington Oregon and idaho has direct connections to both and Sprint's NSF networks Lrols is an example of a network with a direct connection to an IXP Erols which provides service to customers in the metropolitan Washington DC area is connected to the Metropolitan Area Ethernet-East IXP where it can transfer traffic to most of the larger NSPs and several smaller RSPs RSP service is an attractive option for residential and small business customers Because of the small customer base can offer more hands-on assistance in the form of customer training and help desk operators trained to assist less knowledgeable users Like NSP networks the network architecture may be separated into access and transport portions though with different meanings In the scenario access refers not only to the customEr connecting to the RSP but also the RSP corumcting if at all to the Internet Transport refers to the backbone of the network As in the scenario customers connect to RSPS via leased and dial-up lines Typical leased lines are Sb-Kbps or T1 and usually terminate at an RSP's POP For dial-up customers the 1 111 i ll REP usually has digital andXor analog modem banks terminating from its POP into the local central office using Tls An RSP's backbone is typically restricted to a region as opposed to NSPs who have a national presence and whose backbone spans the entire United States Transport on an network or backbone comprises T1 and T3 circuits that connect their POPS and customers in a particular region These circuits are leased from LECs CAPs and IECs As noted above customers are primarily small business and residential Subscribers In the coming years new companies will enter this market Most notable are the Internet service offerings from the lECs and the Region Bell Operating Companies RBOC This increased competition may cause some consolidation of the REF market when smaller RSPs go out of business or are bought out by larger firms The remaining RSPs will survive by targeting market niches such as high volume residential users or businesses new to the Internet The exhibits below show two example RSP network backbones Exhibit 3-2 shows NorthWestNet s backbone and Exhibit 3-3 shows CERFnet's backbone Note that NorthWestNet has redundant connections to Sprint and MCI to transfer traffic whereas connects directly to NAPs to share traffic 3 1 3 Resellers Resellers are another member of the Internet provider family Resellers purchase service from NSPs or RSPs and resell this service to small business and residential customers Resellers are differentiated from RSP because resellers do not own or lease a network infrastructure Instead resellers typically operate out of a single site with a modem bank for customer access and a T1 connection to transfer traffic to the NSPX RSP network There are approximately 1 400 Internet resellers in the United States most of which base their business on subscriptions to Internet service As the Internet market matures Internet service is becoming a commodity This trend has been furthered by the entry of the RBOCs and IECs into the residential Internet service market Typically unlimited access is provided on a basis for a flat rate fee or a combination of flat-rate and usage-based pricing 3-5 Exhibit 3-2 NorthWestNet Backbone Network To International Internet Trar e To International Internet Traf c Sprint Exchange Centers to San 5 311 23 EthbsolnNse Franosco Chrcago New tori-t York and r and Washington KEY Global Internet Bad-room SIIE and Newark Operations Center - NorthwestNeI HUB erE Beau-non owes ho hi'hsl el Inc Boise ID source Inc Because Internet service has significant economies of scale the market favors the larger providers who can spread their fixed costs over a larger customer base Because of this many experts predict that the number of internet resellers will decrease dramatically in the next few years The Yankee Group predicts that there will only by 200 resellers left in business by 2000 The remaining resellers may survive by looking for market niches For example instead of providing Internet subscriptions resellers are already starting to provide value-added services such as Web page hosting Web page development security management and electronic commerce consulting In these areas a reseller may be able to provide better service to small businesses than a larger NSP or company 3 2 INTEREXCHA NGE POINTS With the dissolution of the NSFN ET backbone the NSF was concerned with maintaining connectivity between the commercial networks and the research and education community To address this i55ue the NSF sponSored three primary and one secondary NAPs Without the sponsorship of a core set of exchange points the NSF feared that the commercial backbone providErs would likely setup a hodgepodge of bilateral connect points potentially resulting in routing chaos 3-6 1 1111111111 1 111 11 Exhibit 3-3 Backbone Network I SHOE cut 34 San Jose 1m hm 1m All Ilpr 45 Hop-s 34 34 LDI l5 1013 Hour 100 Hops II 051 pm 05 45 bps 34 IIst Ear-Di atom a 9 mumps Bource Under the NSF model each NAP operator provides the exchange facility while the that connects to the NAP establishes the exchange agreements also known as peering agreements with the other ISPs connecting to the same NAP The purpose of a peering agreement is to ensure that traffic from one ISP can reach all the customers on another ISP by exchanging routing information of the two ISPs Today there are many more EXP centers on the Internet other than the original four sponsored by NSF The term NAP is applied only to the NSF sponsored IXPs whereas all provide the same functionality a common place for ISPs to exchange data Various cities and organizations have used different names for the exchange point NAP MAE CIX Federal Internet Exchange FIX Exhibit 3-4 presents a snapshot of several of the larger IXPs in the United States It is important to note that an does not have to serve the national lSPs There are metropolitan exchange points used today which are similar in structure to the Al s but service onlyr local and regional traffic This means that traffic originating and terminating in a single region would not traverse any of the national lSPs backbones thus removing some of the burden on these networks The remainder of this section describes the structure of an IXP and details the different types of peering agreements used by the at an IXP Exhibit 3-4 Selected M_ajor IXP Locations United States Ul'Ps MAE Chicago CD 0 CHI NAP FIN-AP 0 co SMDS 0 ms Lka MAE WEST MAE-LA MAE Dallas 0 MAE Houslm IXP N t NAP - Sprint hlAEs - Metropolitan Fiber in stem CH1 NAP - Arteritech C115 jointly operated by a Lonsortium SF NAP - Pacific ni 3 2 1 IXP Functionality and Architecture The large national-scope lXPs such as the NAPs or MAEs interconnect numerous national and may exchange data requiring large amounts of bandwidth The smaller regional or metropolitan le s will have fewer interconnects and require much less bandwidth The IXP structure is similar regardless of the size of the IXP or the technical architecture used to exchange the traffic IXP facilities generally consist of a high-speed LAN or metropolitan area network architecture capable of interconnecting various wide area network WAN technologies connect to the IXP LAN via either a high-speed router or an transfer mode ATM switch capable of connecting to the IXP architecture Each of the c0nnecting ISPs must negotiate bilateral or multilateral peering agreements with other interconnecting at the IXP The Routing Arbiter administers the traffic routing resulting from these peering agreements This traftic routing and addressing information is provided to each lSP's router by a route server within the IXP LAN Incoming packets are routed to the high-speed LAN ring where the route server indicates the pessible reutes available to the packet 3-3 1 The most common NAP architecture is a Fiber Distributed Data Interface FDDI dual ring backbone LAN running at 100 Mbps Routers for each ISP are horned to the dual ring bus in the various access configurations discussed below 1 The ISP provides and manages its own router collocated at the IXP facility The ISP Would have dedicated access to this router via its own dedicated line a T1 or T3 This option ma F not be available at all because of space limitations 2 The 15 leases an IXP provided router located at the IXP The ISP has dedicated access to the IMP router via its own dedicated line 3 The leases the dedicated connection and the router from the IXP 4 The ISP leases switched access service to the IXP facility from the IXP or another provider Switched access may include ATM Switched Multimegabit Data Service SMDS and frame relay These access coniigu rations are shown in Exhibit 3-5 For each of the access configurations all equipment is located in a single facility Exhibit 3-5 Typical National-scope IXP Configurations lhl Leased Dedicated Tl I I a if a EXP Leased I 0 I i c f 3 H2 Dedicated T3 Reuter a Leased Router 1 - g I Rework A-l - an Carrier Switched Service IXP Leased e 3 ENDS ltarrII rclayi Router i'pl'l l 3-9 Other IXP architectures that have been used include SMDS and ATM networks Lower bandwidth solutions such as may be more commonplace in regional or metropolitan IXPs All IXPs are privately owned and administered by lECs Incumbent Local Exchange Carriers Competitive Local Exchange Carriers CLEC or lSPs The four NSF- sponsored Al s are owned by Sprint MFS Pacific Bell and Ameritech Regional and metropolitan IXPs may also be owned by 151 s the SMDS Washington Area Bypass is operated by PSlNet The IXPs normally charge at interconnection tees and usage based fees to the interconnecting lSPs Large lECs and LECs can provide network management for their IXPs from their PM network management centers Most IXP operators will ensure reliability of service and mean time to repair and provide maintenance for collocated equipment The dual ring buses used in many large are also very robust to a single line fiber cut- A single dedicated connection from the ISP network to the IXP router will pose the greatest vulnerability in the IXP architecture Redundant connections to the IXP should be used by regional that do not have presence at multiple 3 2 2 IXP Peering Agreements The policies for data exchange at an are set forth by the parties involved Just because an ISP connects to a particular IXP does not guarantee that that can Exchange traffic with every other 51 connected to that exchange point Agreements that specify how traffic is carried and transferred and how billing is handled have to be established and maintained between the On an IXP Any ISP can connect to an as long as the agrees to the predefined policies Currently there are three different types of exchange policies Bilateral I Multilateral 4- Multi-party bilateral A bilateral agreEment is between only two ISPs at an exchange center A multilateral agreement is between many at an exchange center A multi-party bilateral agreement is between a small ISP and a large ISP to carry the small lSP's traffic to other ISPs The more a single ISP connects to the better the performance and reliability of the service Each IXP has its own procedures for establishing peering agreements among the lXP attached ISPs A peering agreement is defined as the advertising of routes via a routing protocol for customers of the IXP participants Specifically the is obligated to advertise all its customer s routes to all other participating and accept routes from the customer's 3-10 I ll 1 routes advertised by the SP are required to peer with the lXP's route server which facilitates the routing exchange between the routers The route sewer gathers the routing information from each router processes the information based on the routing policy requirements and passes the processed routing information to each of the IXP-attached lSPs Currently handles the work clone on the routing management system while Merit implements and maintains the route servers and route server databases 3 2 3 National-scope IXP Architecture Example Pacific Bell s NAP located in San Francisco California is fairly typical of national-scope lXPs PacBell s NAP is an FDDI hybrid LAN whereas other national scope IXPs may be straight FDDI design or an FE le Ethernet hybrid PacBell's use of ATM makes it one of the fastest IXPs capable of up to 139 for OC-3 access PacBell's FasTrak5M ATM Cell Relay Service offering is being rolled out in phases first utilizing Permanent Virtual Circuits PVC and in the future Switched Virtual Circuits SVC As the ATM technology matures and becomes more of an industry and user standard PacBell and other operators will migrate to fully switched ATM backbones SF NAP consists of ATM switching sites in the San Francisco area connected by OC-f v Optical Network SONET links Participants can access the NAP network using an ADC Kentrox ADSU and a Cisco 7000 or T010 router Access speeds reach 36 3 for DS-3 access and 139 for DOS access in addition to the ATM network the NAP includes an interconnected FDDI dual-ring LAN The FDDI LAN provides service to customers that require bandwidth less than 30 Mbps The FDDI LAN was added when PacBell tests indicated that the ATM network was dropping cells at speeds between 20 and 30 Mbps ISPs provide or lease dedicated T1 or T3 connections to PacBell DSUs and Cisco 7000 routers connected to the FDDI backbone Exhibit 3-6 depicts PacBell s San Francisco NAP hybrid network architeCtu re 3 2 3 1 Routing Each participating ISP must negotiate bilateral peering agreements with other before connecting with PacBell's San Francisco NAP Roo ng on the FDDI ring is accomplished via the route server database maintained by the Routing Arbiter On request PacBell will provide NAP clients with a PVC to the Routing Arbiter route server database to receive and provide routing updates Routing among peered may also be accomplished by direct PVC connections between the at the MAP without regards to the route server database 3-11 Exhibit 3-6 Pac Bell San Francisco NAP Hybrid Architecture Internet Service - - - - - - Network Serwce - Promotion and Regional Network Access palm Providers Network Providers 4 a r uh- Fi Em E mmg ij gw T i ir- L_r I Eff I ll rub-HUI 2 5 i -FDDJ Fm Ftoutlng Arbiter I - ma A fixem'u Em A I Clsoe 3 00 Router l2 353 ATM LII-ear Channel 053 I roou noun I m Hell 3 2 3 2 National ISP Clients The San Francisco NAP interconnects nuntcrou's national and regional ISPS National include ANS MCI and Sprint 3 2 4 Metropolitan IXP Architecture Example PSI Inc manages a metropolitan IXP in the Washington DC area 51 established the SWAB as an alternative IXP to the MAE-EAST NAP SWAB operates nearlyP identically to the national-scope lXPs requiring participating ISPS to negotiate peering agreements Unlike the NAPS the SWAB network is not facilities-based Instead each interconnecting ISP subscribes to Bell Atlantic s SMDS service over 1which the IP is routed 3-12 I Each participating must subscribe to Bell Atlantic's SMDS service at a specified access class speed SMDS may be accessed at up to 34 Mbps making it a lower bandh'ldtl l solution than FDDI or ATM The must supply its own dedicated access either T1 or T3 to the SMDS service To route over SMDS the must also provide an SMDS capable and an IP router that supports SMDS encapsulation at the SWAB interface SWAB provides broadcast capabilities by use of SMDS address groups The SWAB participants can have their SMDS address included in the SWAB SMDS address group for broadcast purposes 3 2 4 1 Routing The functionality of the Routing Arbiter's route server database is provided using SMDS address screening Address screening is used to filter out SMDS addresses from the SMDS connection analogous to how the 85 network can screen calls from a voice line An lSP's screen accepts packets from peered lSPs while refusing packets from other ISPs Each TSP must request that Bell Atlantic screen SMDS addresses from their SWAG interface 3 2 4 2 National 15 Ciients Currently and are the only national interconnected at the SWA B 3 3 INTERNET ROUTING PROTOCOLS The Internet as previously described is a collection of networks that allows conununications between research institutions universities and many other organizations worldwide These networks are connected by routers A router is connected to two or more networks appearing to each of these networks as a connected host Forwarding an datag-ram generally requires the router to choose the address of the next router in the path or for the final hop the destination host This choice called routing depends on a routing database located within the router The routing database is also known as a routing table or forwarding table The routing database should be maintained dynamically to re ect the current topology of the Internet A router normally accomplishes this by participating in distributed routing and least-cost routing algorithms with other routers Routers within the Internet are organized hierarchically Some routers are used to move information through one particular group of networks under the same administrative authority and control known as an autonomous system AS Routers used for this purpose are called interior routers and they use a variety of Interior 3-13 Gateway Protocols IGP Routers that move intormation between ASS are called esterior routers and they use Exterior Gateway Protocols EC P There is no standard protocol for either or EGP However there are three protocols that are used by the ISPs and at the on the Internet Generally ISPs use the Routing Information Protocol RIP or the Open Shortest Path First OSPF protocol Most use the Border Gateway Protocol Version -I as their routing protocol All three protocols are dynamic in that the routers interact with adjacent routers to learn which networks each router is currently connected The IGP protocols and 05PF are detailed in Section 3 3 1 and 3 3 2 respectiver is presented in ction 3 3 3 3 3 1 Routing Information Protocol RIP was developed by the Xerox Corporation in the early 19805 for use in Xerox Nehvork Systems XNS networks RIP is a dynamic protocol that continually updates its routing table based on intormation received from its adiacent routers RIP is a distance-vector protocol meaning that each router maintains a table of distances hop counts from itself to each other router in the system These routing tables are updated based on RIP messages from adjacent routers RIP performs five basic operations Ir Initialization - Request received - Response received a Regular roo ng updates Triggered updates On execution RIP determines which of the routers interfaces are up and sends a request packet out on each interface The purpose of this request packet is to ask each of its adjacent routers for their entire routing table A request received operation occurs when a router receives a packet from one of its adjacent routers asking for all or part of the router's muting table The router will process the request and reply by sending the requested data A response received Operation occurs when a router receives a response to its request tor all or part of its adjacent routers' roo ng table When a response is received the router must validate the response and update its routing table 3-H Regular routing updates every 30 seconds A router Sends either all or part of its routing table to all of its adjacent routers This ensures that each router on the network consistently has an accurate routing table Finally a triggered update occurs when a router notices that one of its routes has changed The router sends all routes from its routing table which are affected by the changed route which may or may not be the entire table Although RIP appears to be a very simple protocol it does have serious limitations First as shown by the series of operations in RI P the protocol propagates either all or part of a router's routing table every 30 seconds in addition to any triggered Updates Subsequently the protocol is very slow to stabilize when network failures or routing errors occur Second RIP limits the number of hops between any two hosts on the network to lo This means that hosts that are more than 15 hops apart within a single AS will not be able to communicate with one another As a result RIP is not well suited for large internetworks and works best in small environments Finally when faced with multiple routes between a router and a network RIP always chooses the path with smallest number of hops This choice does not consider other cost factors such as line speed and line utilization which are important when choosing a path between two nodes Although RIP is still a very popular protocol many companies are moving toward its replacement OSPF 3 3 2 Open Shortest Path First OSPF was developed by the Internet Engineering Task Force IETF as a replacement for RIP OSPF is designed to overcome the limitations of RIP and is supported by all major routing vendors OSPF uses IP and its own protocol and the transport layer not UDP or TCP OSPF is a dynamic link-state protocol unlike RIP which is a distance- vector protocol In a link-state protocol a router does not exchange distances with its neighbors Instead each router tests the status of its links with its neighbors and sends this information to each adjacent router Routers using OSPF are able to build an entire routing table based on the link-state information received from each of its neighbors In contrast to RIP OSPF does not make its routing decisions based on the number of hops to a destination Instead OSPF assigns a dimensionless cost to each of interfaces of the router This cost is not based on hop count but on throughput round trip time reliability etc When the router is faced with multiple paths for a particular route the routing decision is made using this cost If two routes exist with the same cost OSPF distributes the traffic equally among the routes Additionally OSPF allows multiple routes to a destination based on the type of service Telnet FTP SMTP This 3-15 means that a router can chose the best route for outgoing packets based on the type of traffic contained within the packet As described in Section RIP is not well suited for larger internetworks bacause of its functionality OSPF however is designed for larger networks and stabilizes much faster when network failures or routing errors occur OSPF also does not impose limitations on the number of hops between any two hosts because it does not use this metric when making routing decisions Although RIP is still very popular OSPF will ultimately replace RIP as the Internet grows 3 3 3 Border Gateway Protocol Version 4 The primary routing protocol used on the Internet is This protocol is used on Internet core high level routers to dynamically learn network reachability respond to outages and avoid routing loops in interconnected networks Although RIP and OSPF are lGPs is an EGP used to pass traffic between different autonomous systems uses the TCP protocol to communicate routing information with its peers Routers using classify traffic as either local traffic or transit traffic Local traffic is traffic that either originates or terminates in the router's AS All other traffic is classified as transit traffic The goal of is to reduce the amount of transit traffic on the Internet The system exchanges network reachability information with other systems This information includes the full path of autonomous systems that traffic must transit to reach the destination The network reachability information is used by the router to construct a graph of AS connectivity Once constructed routing Ioops can be removed from the AS connectivity graph and roo ng policy decisions can be enforced peers initially exchange their full reuting tables From then on incremental updates are sent as the routing tables change assigns a version number to the routing table and all adiacent routers will have the same version number for their routing tables This version number changes whenever the routing table is updated as a result of routing information changes To ensure that the each adjacent router is alive l-zeepalive3 packets are sent between peers whereas notification packets are sent in respowse to errors or other special conditions After a router using receives routing updates the protocol decides which paths to choose to reach a specific destination Like RIP is a distance-vector protocol that allows only a single path to a destination However BGP4 does not impose a limit on the number of hops between hvo hosts and stabilizes quickly after network failures or 5 The keepalive operation is independent from the TCP versitin of keep-alive 3-16 I ll telephone lines However ISBN is gaining popularity with residential users as ISDN equipment and service prices dr0p Both ISDN and analog modem connections use PN switched connections The characteristics of analog modem and ISDN connections are described in Exhibit 3a below The bandWidth allocation for ISDN and analog modems is symmetric meaning that there is an equal amount of inbound and outbound bandwidth Unfortunately many traffic applications are asymmetric whereby the user receives far more inbound traffic than he or she generates Examples of asymmetric applications include video-on- demand small request to access a movie results in many gigabits of high resolution video and Internet access small request to access a Web page re5uits in many megabits of text and images from the Web page Exhibit Analog Modern and ISDN Characteristics AhalogModem Speed 2 4 to 33 6 6-1 to 128 Equipment Cost $100 to $150 $300 to 5-100 Representative Flat Rate Plus Usage Service CosH 540 month 5100 month 5002 minute ILECs cable companies and direct satellite companies are testing and deploying several access technologies see Exhibit 3-3 below These technologies have up to 30 of inbound bandwidth and up to 2 of outbound bandwidth Exhibit 3-3 Asymmetric Internet Access Characteristics Characte stics Direct Broadcast I Cable Modems - Satellite Service DirecTV Satellite Cable Companies Provider Inbound Speed 400 1 544 to 6 10 to 30 Outbound 23 8 over 16 to 512 68 to 2 Speed analog phone lines Equipment 31300 $1 000 3500 Cost Service Cost 540 month 560 to $100 month 540 month Status Deployed In trial In trial 4 Includes the cost of service from the LEC and the cost of access from the ISP 3-13 routing errors occur- The decision process is based on different lactors including next hop path length route origin local preference always propagates the best path to its adjacent routers Currently is used by most IXPs on the Internet but is not defined as the standard EGP 3 4 INTERNET ACCESS he iast and in some ways the most vulnerable component of the Internet architecture is the link between the sarvice provider and customer This access connection is typically a single dedicated or switched line over PN facilities Because access is provided over a single PN line the connection is vulnerable to outages This situation is identical to the last mile vulnerability of the architecture Most other parts of the Internet architecture can use redundant lirth to route around outages However the access link is typically a single point of failure for an end u5er's connection to the Internet Internet access can be divided into two broad categories business access and residential access These categories are described separately below 3 4 1 Business Access Large and medium-size businesses use dedicated lines to connect their enterprise to the Internet These lines are either bundled with the ESP's service or leased separately by the company In either case the connection travels over PN tacilities- - Most large businesses use TI 1 544 Mbps or higher connection speeds Medium-size businesses use or fractional TI speeds 128 to 7'68 Kbps depending on their traffic requirements Small businesses 10 to 50 employee sites may be able to get by with a 56 leased line or a 123 Integrated Services Digital Network connection Leased line connections are available from ILECs and in metropolitan areas from CLECs Today CLEC companies include CAPs eg Metropolitan Fiber Systems Teleport Communications Group and in many cases lECs LDDS MCIMetro As legislation opens the local exchange to increased competition leased lines may be available from utility companies cable companies or other providers 3 4 2 Residential Access Residential access connects a single user's computer to an reseller or on-line provider Most residential access is through modem connections over a LEC analog 3-1 I ll 1 The direct broadcast satellite offering is the only one of the three that is currently in widespread distribution Direct broadcast satellite allows a user to receive inbound traffic over a 1-meter satellite dish and transmit outbound traffic over a standard analog modem line Asymmetric Digital Subsc ber Line ADSL is a technology developed by the to provide high bandwidth asymmetric connections over standard copper twisted pair wire ADSL was originally developed exclusively for the home entertainment market video on-demand interactive cable However as residential Internet access has grown in popularity the LECs have added Internet access to their ADSL marketing efforts ADSL is popular with LECs bewuse copper cable is the basis for almost every residential phone installation ADSL has a head start over its rival technologies because of the widespread deployment of copper wire which reaches 98 percent of US homes compared to 60 percent for cable However ADSL does have several drawbacks - Installation costs are high to upgrade existing copper cable to carry ADSL signals - Subscribers must be within 10 000 feet of the central office to reliably receive ADSL signals I Strong local AM stations can interfere with ADSL signals Ir The bandwidth available for communication is far less than the bandwidth available over cable modems Cable modems have the highest inbound and outbound bandwidth but also have the most obstacles to widespread deployment Cable modems depend on a two-Way communication path between the cable operator and the subscriber Almost every cable installation is designed to provide only a oneiway path for video To facilitate internet access over cable plant cable operators must upgrade their coaxial cable networks to two-way operation Once upgraded cable operators may have additional problems with the reliability of their plant cable wires are installed only several inches below ground level and are highly susceptible to outages due to unintentional cable cuts Once these issues are addressed cable modems may easily fill a niche in the new market of lnternet-enabled television Currently access for these devices is provided using analog modems over dial-up lines 3-19 4 INTERNET ANALYSIS As described in Section 3 the Internet can be viewed as an interconnection of national and regional networks end-users and organizations and intereschange points The Internet is a very dynamic entity that is constantly evolving and growing Therefore it is impossible to identify all of the components of today's Internet For this report the Internet is analyzed to identify key components used to transmit network traffic across the Internet To achieve this purpose a software tool referred to as the was used to automatically trace the routes used to send traffic between two hosts on the Internet The tool collects the set of rooters an IP packet traverses On its path from one host to another The analysis of these routes will identify traffic trends and key components in the Internet infrastructure This section provides an in-depth description of the IAT and analysis results Section 4 1 details the functionality of the IAT Section 4 2 details the implementation of the tool including the set of hosts that was analyzed- Section 4 3 presents the analysis methodology and the results of the analysis INTERNET ANALYSIS TOOL FUNCTIONALITY The purpose of the IAT is to collect the routes traveled by IP packets from one host to another Because it is impossible to collect and analyze routes between every host on the Internet a subset was chosen to provide an accurate sample of US Internet traffic Section 4 2 details the sites chosen for this analysis The utilizes a UNIX utility fmceronfe to record the different routers a packet traverses once it is sent from the originating host to the destination host The frat-aroma application is available with all UNIX and UNIX-variation operating systems traceronte use the Time To Live L field in the IP packet header to determine the routers in a particular path The purpose of the TTL field is to ensure that packets do not stay on the Internet for an infinite amount of time as a result of a routing loop Each router that receives an IP packet is required to decrement the TTL field in the 1P header by the number of seconds the router holds onto the datagram Because most routers process a datagram in less than one second the field effectively becomes a hop counter that is decremented by one by each router packets are usually transmitted with a TTL of 60 by the originating host When a router has an IP datagram with a TTL of one the router decrements the TTL to zero discards the packet and returns an error message to the originating host This error message is an Internet Control Message Protocol ICMP packet 4 1 I Ill ll that identifies the router that sent the error message and indicates that the time has been exceeded on the datagram The basic operation of the IAT is to send out tracer-mite IP datagrams beginning with a TIL of one then a TTL of two and so on until the entire route between two hosts is determined The router receiving the first 1P datagram with a TTL of one will decrement the TTL and return an ICMP message to the originating host This identifies the first router in the path The will then send out a second treccroittc 1P datagram with a TTL of two The first router decrements the TTL to one and sends the datagram to the next reuter in the path The second router will decrement the TH to zero and return the ICMP message This continues until enough datagrams have been sent to have one of them reach the destination host The destination will not discard the traceroute 1P datagram even though it will have a TTL of one because the datagram is addressed to that host For the LAT to detErmine that a datagram has reached its destination because it has not received the final ICMP message the IAT sends UDP datagrams to the destination host using a very high destination port number The destination host will not respond to incoming packets on this port number thus the destination host will send back an port unreachable error to the The IAT differentiates between the time exceeded and port unreachable errors to determine when the route has been full traced The output from an IAT exewtion is the set of routers in the path between two hosts For each router three datagrams are sent and the round trip time from the originating host and the router is collected Exhibit depicts the sample output from a source host to the destination unmadisamil Exhibit Sample Output From the Traci-route to 1 Cisco-AGS dcmetro bah com 15680 11 2ms 2ms 2 fr herndon va psi net 38 2 1041381219081219 45 ms 133ms 4 681115 541115 41 ms 5 137209 12 137209 12 Horns 168 ms 20-1 ms 6 1291115 132 ms 11-1 ms 7 164 112 2135 164117 213 13-1 ms 12 ms 13-4 ms 8 143ms 135 ms 125 ms 9 135 ms 1-17 ms 1 6 ms 4-2 4 2 INTERNET ANALYSIS TOOL IMPLEMENTATION This section details the implementation of the described in Section 4 1 For this analysis two sites were chosen as source sites 0 Boos-Allen t5 Hamilton McLean Virginia on the network - Proxima McLean Virginia on the Network The tool collected routes from each of these two sites to 105 other sites Deated across the United States The Web sites chosen for this analysis included the following 23 NCS Member Organizations Web sites i 50 State Web sites a Major university Web sites - Popular commercial Web sites Appendix A provides the entire list of Web sites used in this analysis Exhibit 4-2 also shows the geographic locations of these sites The IAT v hich collects the routes from the two source locations to all 105 sites is executed six times daily every four hours beginning at midnight This results in a sample of Internet traffic thrOughout the day The output from this tool is formatted and loaded into an Oracle database where the analysis on the collection of routes is performed Section 4 3 presents the analysis methodology followed by the study Exhibit 4-2 Site LOCations Web Sue Destinlhon Types 1 0 COME-J - Ed ttlt l-J l Fatima-annual Silt Dover- at 4 3 I I Jill I 1 ill ll 4 3 INTERNET ANALYSIS RESULTS The data collected using the IAT represents a general picture of Internet connectivity The destination Web sites used in the analysis were selected to provide both a United States and NCS specific view of the Internet's topology Internet Analysis Methodology An in-depth analysis of the physical topology of the Internet would be an incredibly complex and difficult task Because of the number of national backbones and regional distribution networks spanning multiple carriers the Internet's topology is an amalgamation of CLEC ILEC and IEC networks Determining the entire physical topology of the Internet may well be impossible without the cooperation of these PN carriers An analysis methodology was developed to provide the most complete and valuable view of the Internet and its topology The methodology defines the steps used to evaluate the data obtained from the IAT A description of the methodology is presented below I Identify Scope Although the internet is too large and complex to be handled in its entirety the scope of the analysis was selected to provide a representative view of the Internet The IAT is most useful in analyzing single specific routes not large network topologies Given enough representative routes the collective results of the IAT can provide a view of portions of the larger Internet By collecting data at various times of day from multiple routes the IAT provides a representative set of data The originating and destination sites selected for this analysis provide a distribution of sites across the United States The inclusion of the NCS Member Organizations provides a capability to capture and analyze data specifically f0r the NCS community I identify Pertinent Dale The data used in the analysis must provide a complete and accurate picture of how IP packet traffic will be routed over the Internet Variables such as the distance traveled number of networks traversed and the congestion of the network will affect how packets traverse the network The IAT provides a host of data that is used to analyze our representative Internet routes The data used in this analysis includes the following Origin to destination route - Physical distance of route in air-miles Time of day 4-4 - Round trip time Number of hops in route - Routers in route - Networks in route - Manna Valuable Results The purpose of the analysis is to identify the discriminating variables that affect the Internets performance Using the availabie data round trip time and number of hops in route the analysis shouid indicate differences in performance based on the following variables - Critical nodes Physical distance between hosts - Time of day congestion Number of networks traversed - Relative size of networks traversed NSF versus RSP These results will provide input to the analysis of the vulnerabilities of the Internet They may also identify how the OMNCS and NCS Member Organizations can improve Internet reliability by choosing certain lSPs mirroring important Web sites or performing in off-peak h0u rs Exhibit 4-3 illustrates the Internet analysis methodology Exhibit 4-3 Internet Analysis Methodology identify Scope of Analysis - Onginanng Sues I lestinatlon hues CON US coverage In 23 NCS Member Urganiuions 50 Sue neh sII-cs Hapr university ueb sues Popular commerclal ueh sites Identify Pertinent Data - Anal sis Constants Routes a Physical Distance - Anal sis Variables Rowers mule Networks in mun it 11 me in Round mourn-c in Time oi day 4 5 Results Identify Valuable I Collect Data and Complete Analyst Performance Vanables I- Cnlical nudes '3 of nonvolth mm at Nana-L size NSF ss Is 1111 ofDa lung Physical Distance 1 111 11 4 3 2 Internet Analysis Results The analysis focuses on identifying the path that is critical for transmitting data across the Internet s regional and national backbone networks The data provided by the IAT was analyzed to trace the paths through the Internet and to identify how Internet data traffic is affected by daily traffic surges and congestion and network outages This analysis is intended to provide an estimate of the performance characteristics of a portion of the U S -based lnternet However the results presented here cannot be assumed to represent the entire Internet or even the entire U S -based network This is because of the limited scope of the data and the sheer size of the Internet in terms of routers and hosts More thorough analyses of the entire Internet are planned as a follow- up to this initial analysis We chose to use two source hosts for this analysis one based on Booz- Alien Hamilton s network and the other on Proxima Inc's network Booz'Allen and Proxima Inc receive Internet service from two of the six NSPs and MCI respectively Therefore this analysis may primarily represent the characteristics of these two networks The data provided by the IAT traces is the basis of a statistical analysis of the number of hops and round trip time for the 210 source and destination pairs 2 sources and 105 destinations Traces were given a status of either successful or unsuccessful A successful trace was one in which the IAT packets generated reached the destination router address and an unsuccessful trace was one in which they did not Exhibit 4-4 shows the number of successful traces per source and the percentage of the total traces performed Exhibit 4 4 Status of IAT Traces Source Total Traces Success il Traces Unsiiccesstiil Traces 3mm Allen 5134 4468 ass 13 an Proxima Inc 9123 8098 it 1030 11 3 it A small percentage of the traces for both sources was determined unsuccessful An unsuccessful trace could typically be attributed to one of the followmg reasons 0 The destination name server entry could not be resolved and therefore the trace never began a An initial router of the ISP could not be reached - A router or gateway in the path of the trace was unreachable 1- The destination server was unreachable most liker due to it being shut down a The hosts network might use code that is incompatible with the testing protocol That might have resulted in a router not returning the ICMP messages required for the operation of the IAT Exhibit 4-5 illustrates an approximate categorization of reasons why traces were unsuccessful The percentages of those due to an unreachable path router an unreachable destination server or incompatible network code were combined A hop-by-hop analysis of all unsuccessful traces com prising nearly 45 000 hops would be required to determine the component percentages Exhibit 4-5 Categorization of Unsuccessful IAT Traces Booz' Allen Proaima Unresolved host name 2 6 a 0 u iSl unreachable 0 2 as 0 8 Do Router or destination machine unavailable 10 2 n 10 3 Total Unsuccessful 13 0 in 11 3 a The results described in the remainder of this analysis are solely based on successful traces 4 3 2 1 Traffic Congestion internet traffic encounters congestion due to surges in its use in daylight hours Traffic surges occur during working hours and most notany between noon and 6 00 pm Weekend traffic should not be as susceptible to Internet congestion because of the reduced number of business users Our analysis assumes the effects of congestion will become manifest in the response time for data traveling over the lnternet The collects the round trip time for a single datagram to travel to and from each of the destinations For each destination three datag'rams are sent and the total travel time is recorded for each The average travel time versus time of day for these datagram is shown in Exhibit 4-6 As Expected these results appear to coincide with traffic patterns for a typical east coast IXP MFS's MAE-EAST The additional traffic on the internet results in a proportional increase in the delay time Representative weekday and weekend data for MAE-EAST and MAE-WEST are shown in Exhibit 4-7 and Exhibit 4-3 respectively The traffic I ll 1111 increase between 12 00 noon and 4 00 p n1 shown in the MAE-EASI trattic profile is similar to that of our round trip time results Note that traffic on MAB bl located in lk'ashington DC and MAE-WEST located in San Jose CA are nearly identical for the time of day based on eastern standard time EST Because of the iarge amount of train traveling between the east and west coasts these two IXPs are interdependent the traffic generated on the east coast behreen 12 00 noon and 4 00 pm eastern time affects the west coast traffic patterns behveen 9 00 am and 1 00 p m Pacific time Exhibit 4-6 Average Round Trip Time Versus Time of Day 90000 _ so one moon some some i Round-Trip Trrne 40 000 I 3D DUG 1260 4GB 300 12Nocpn 400 60 Midnight a nu a Eastern Standard Tm Weekdays Baez-Allan EP Weekends oor-Allen BF Weekdays F'I-oxrre ISP Weekends Home ISP Fi 38 689 iti ii 63 ii 556 548 thlt'Scc 5' Exhibit Typical Traffic Patterns at MAE-EAST Nil Ht Inpurr'l ll 3 1'6'_ h r1 I - J Huh-In - uhIll I516 I-C II El 1'11 II-tort Source MI 3 Da la net Inc Exhibit 4-8 Typical Traffic Patterns at HIE urn - E1g us1cn II I 5 culuwilit 1 1 1 I i-urp Ill 11' - 5 1 l'r9--I- TI-I Poeltic Source- Datnnet11111 4 3 2 2 Network Outages Network outages are the most disruptive of the lnternet s vulnerabilities In the case of critical nodes a network outage can preclude access or egress from the network as in the case of an isolated regional or local network or severely hamper the flow of traffic as in the case of a NAP or IXP failure Network outages will occur with much less frequency than network congestion but they may result in a signi cant reduction of network capacity and availability depending on their severity We determined the number of hops in each successful trace from source to destination Exhibit 4-9 compares the average number of hops with the time of day for each source network It is clear that the number of hops does not depend on the time of day This indicates that the path taken from source to destination does not change freq uently due to outages or routing around network congestion Thl i is because lnternet routing tables are generally static Routing tables are meant to change during a disruption in service and In the event of network congestion Although some routing algorithms will route around link congestion th1s analyse indicates this is uncommon because the number of hops does not depend on the time of day while congestion does Creating large lnternet routing tables requires expensn'e processing power This process can result in more route thrashing than actual routing In fact routing tables will normally only be recreated when links become disrupted or when a network administrator manually replaces the routing table Exhibit 9 Average Number of HOPE Versus Time of Day -- I-ll I l-A_f Mon-gm Imam then 4 0me Ettme 4-10 We hypothesire that an outage in a critical nenvork node such as a national IXP would greatly reduce but not eliminate the ability of the Internet to route traffic quickly nationwide 4 3 2 3 Critical Network Nodes As explained in Section 3 2 the Internet relies primarily on the national IXPs to route and exchange traffic Exhibit 3-1 shows the locations of the major le s across the United States These high-speed LANs provide the majority of the routing among the backbone NSPs and the 151 s Additionally traffic is exchanged at private direct connects between networks Private direct connect exchange pomts of this kind are becoming more common due to congestion at the lXI s are establishing private direct connects to avoid congestion problems and improve routing redundancy The output provides the IP address of each of the routers traversed in the Internet traces Using this data we compiled lists of the most commonly visited routers for our two source networks Exhibit 4-10 shows the distribution of the normalized frequency of use for the top 50 routers for both sources The normalized frequency was obtained by dividing the number of hits on any router by the total number of hits recorded by the for that source This allows a direct comparison between the two sources The first second and third router in each trace is considered to be specific to the sou rce These three routers show a very high frequency of use for our sources and they are therefore critical to these sources but do not fairly represent the remainder of the Internet These three routers have been eliminated from the remainder of this analysis Exhibit 4-10 Top Routers Normalized Frequency Baez-Allan 115 Heart 15le 012 D1 003 005 00-4 002 1 3 5 l Honnaltzed Frequency of Use Top 50 Routers Non-denteali 4 11 I 1111 The network domain names provided by the IAT output identify the router's owner Using these domain names we identified the relative importance ot l iIJ networks to the two sources The normalized frequency of use tor each network is shown in Exhibit 4-11 Other networks were those networks that did not individually represent a large portion of the total frequency of use or that were not identified by a domain name Exhibit 4-1 Normalized Frequency of ISP Network Use 3 F'roxmaBPtlu a I Benz-Alien EP PSHatt 3 2 c 2 2 ED ED ISPNetwork The critical network nodes had the highest frequency of use Nehvork nodes were considered critical if 0 They had a high frequency of use - They were not too specific to the source routes the top three routers I 1 hey were not too specific to the destination routes All routers with normalized frequency greater than 0 004 were considered critical Each of the sources shows a dependence on multiple critical network routers to trace a path to the destinations Exhibits 4-12 and 4-13 show the critical ISP networks for the Booz' Allen and Proxima ISPs respectively 4 12 Exhibit 4-12 Booz- Allen s Critical Networks 31 Exhibit 4-13 Proxima s Critical Networks 33% ISBN Planet 31% Dapfll'Emil 2% 55% BBHHanet 15 lf Damn-14% - 5% mt TA I hbtcoI-n- 2% 2% Some of the critical nodes for one source were also critical to the other source These nodes become our most critical nodes which we can then identify as critical to the Internet based on our study Exhibit 4-14 shows the distribution of these critical nodes to networks 4-13 1 Exhibit 4-14 Shared Critical Nodes mm Numberet omem ML BEN Fla net 4 3 2 4 Conclusions Traces performed throughout the test period indicated high success rates averaging between 87 and percent Of the unsuccessful trace attempts most were due to an unreachable node a router or the destination server in the path that was probably either shutdown or incompatible with the IAT software Internet use is highest during mid todate afternoon business hours Based on the round trip time tor packets to traverse the network congestion peaks between the hours of12 0'0 noon and 4 00 p m eastern time However the dependence of businesses on the Internet could not be determined the analysis did not determine whether the Internet was used to conduct Critical business communications and research or simply for personal use This analysis indicated that the number of hops did not depend on the time of day or the day of the week Generally routing tables are rarely modified to route around network congestion Unlike switched traffic the routes of Internet connections were somewhat predictable Therefore the predictability of lnternet routing along with an increasing dependency on this conununications media renders it vulnerable to targeted and intended network disruptions Routers appear to share a somewhat balanced traffic load within the backbone networks excluding those routers closest to the two sources As expected a high number ot router ylsits occurred in the initial hops of the traces These initial routers are critical to the sources however they are not necessarily critical to the entire Internet As the trace moved away from the source and into the 4-14 backbone nehvorks the number of visits per router stabilized Therefore a Single critical router could not be identified however it could be determined which networks were more heavin traversed For this analysis MCI's network was traversed most frequentiy and was theretore critical to the success of the traces 4-15 5 VULNERABILITIES This section addresses connectivity vulnerabilities that are inherent in the architecture of the Internet These systemic vulnerabilities result from the utilization of the current PN infrastructure by the Internet composite networks The vulnerabilities include second order effects such as availability and reliability due to outages on critical links and routing database errors Security issues and vulnerabilities from outside in uences such as hackers are not addressed Internet vulnerabilities from hackers are addressed in the Electronic Threat liitnisien Report The vulnerabilities associated with the lSPs IXPs and Internet access connections are discussed below 5 1 INTERNET SERVICE PROVIDERS As introduced in previous sectiOns the provide the basic backbone architecture of the Internet The lSF s can be divided into three categories NSPs RSPs and resellers Internet vulnerabilities that are unique to each category are detailed in the following sections National Service Providers The majority of the NSP links travel over dedicated lines leased from the PN carriers PN dedicated lines travel in the same conduit as other switched PN lines Thus the NSP links have a physical reliability comparable to that of the carrier's network The IEC maintain their high reliability standards through a three tier restoration architecture This architecture is based on protocols physical diversity and switching algorithms Figure 5 1 details this tiered architecture The PM providers' current restoration techniques for cable cuts the most frequent cause of outages are not available for the dedicated lines used in the Internet The switched based mechanisms are not available because of the fundamental differences between switched voice and data communications The protocol- and physical-based restoration mechanisms however could be employed for dedicated line failures Each NSP needs to work closely with the PN providers to ensure that their dedicated lines are afforded these restoration tectmiques For example SONET rings are currently being deployed to increase the reliability of communications links Traffic on a SONET ring automatically reverses its direction as a result of a cable eut HOwever a PN provider may impose additional charges to add dedicated lines to a SONET ring if there are unused protected lines available Thus the primary alternate routing schemes used to ensure connectivity is dependent on the routers routing protocol and restoration plans 5-1 Exhibit 5-1 PN Three Tier Restoration Architecture Heohanisrn Basis Description i SDHET Protocol Based i i Digital Physical Pat Cross Based I I l l nects on Gaol-9P Digital i Smith Cross Connect Fagin swimh I I Direct Path Cable Dynamically Switching Controlled Based I Routing in Standish #n v 1 NSPs connect to multiple nationwide Typically an connection at each of these is non-redundant if this connection is lost the NSF will lose its connectivity to the IXP and the ability r to exchange traffic with the other interconnected NSPs However it the NSP has connections to other lXPs either regional or national the can still exchange traffic with the IXP-attached NSPs The loss of the connection between an NSF and an IXP is critical only if the NSP does not have connections to multiple lXPs netWorks are also susceptible to routing problems such as slow convergence and routing loops The three routing protocols discussed BOP-1 OSPF and RIP - can affect routing within and between 151 networks Because BGl -l is an external protocol it can affect routing between ISPs RIP and OSPF which are internal protocols will onl r affect an lSP s internal network RIP the oldest of the three routing protocols discussed has particular vulnerabilities that have been addressed by the newer protocols RIP is a distance rector protocol based on hop count to the destination node RIP routing tables contain nnii the single best route from origin to destination when a better route is present it replaces the old route When determining the best route available RIP only considers the hop count and not other important factors such as bandwidth and line utilization Additionally RIP is very slow to converge after a network failure or routing error has occurred If a link in the route path is disrupted RIP may not settle on the new best route for several minutes During those minutes service between those particular nodes is disrupted RIP is also susceptible to routing loops In the minutes that it takes RIP to converge after a failure routing loops may develop that will cause packets to route endlessly over the network until their TTL expires Although there are modifications to the implementation of the RIP protocol that will help to avoid routing loops they are subtle and may not be present in every network using RIP Finally because RIP propagates its routing table to each of its neighbors every 30 seconds RIP networks that are already congested by user traffic will be congested further by these routing tables The OSPF routing protocol overcomes shortfalls The link state vector characteristic of OSPF allows each router in the network to have complete routing tables with multiple paths to destination This greatly improves convergence time during a netwmk failure and eliminates the chance of routing loops OSPF routinely propagates route advertisements every half hour OSPF also uses lP's multicasting capability to reduce the bandwidth requirement for these advertisements This reduces the overall bandwidth overhead on the network attributed to the routing protocol In time will replace RIP as the standard internal routing protocol on the Internet Exhibit 5-2 summarizes the vulnerabilities of the NSF networks RSP networks lXPs and the access portion of the Internet architecture These vulnerabilities are described in greater detail in the following sections 5 1 2 Regional Service Providers RSPs have similar vulnerabilities to those of the NSPs These vulnerabilities may be compounded since smaller geographic scale limits the availability of physical diverse paths and their choice of a PM provider This increases the possibility of isolation of the RSPs RSPs usually have fewer connections to lXPs These connections may also be limited to the region that the REF services If one or more of an IXP connections is disrupted the service will suffer greater degradation than an RSP service could be seriously affected by a regional natural or man-made disaster 5-3 Exhibit 52 Internet Architecture Vulnerabilities Samar 'Herwot- Software Cable Hmling Sat-r 3 'Lasth'lve Configurabon Cuts Errors Lira 'Js'ee'ac- a National Networks 7' a i I I a lites Regional ISPs 9 - a lovernmontf ullnut I I LAN - Because of an smaller geographic coverage traffic will be carried over fewer links If a major link fails because of a cable cut it can have a large effect on the traf c within the RSP's network For example in NorthWestNet's backbone shown in Exhibit 3-2 the 35-3 circuit between Seattle WA and Portland OR is a critical high-bandwidth link If that link fails Portland s bandwidth to the national Internet connectivity provided at Seattle will fall from EDS-3015 Mbps to 2 3 088 Mbps a possible 3 percent drop in speed and bandwidth Since RSPs have smaller networks much of their traffic is transmitted over other lSP s networks Thus the effect of EGP routing the IXP connection and bilateral NSP network connection failures are more pronounced in an RSP's network The RSP's traffic will also encounter the vulnerabilities of the NSP network carrying its traffic including the reliability problems encountered due to routing errors 5 1 3 Resellers Resellers depend on their host network to provide reliable and responsive service Resellers typically r have a single dedicated connection between their distribution facilities and its 15 This connection typically travels over PN dedicated I ll 1 I I I lines A failure in the dedicated line will result in a loss of Service for the users homed to that distribution facility A reseller's network may become a congestion bottleneck when multiple customers access a single distribution facility with dedicated lines If a reseller has not engineered the network connection for sufficient bandwidth to support dedicated and dial-up users congestion may occur This problem may occur in some reseller networks more than others Network availability is also a concern for dial-up customers of reseller networks The ratio of customers to reseller modems may vary from 5 to more than 15 During high congestion periods customers may be unable to gain access to the Internet Higher ratio resellers have a greater potential for customer blocking 5 2 INTEREXCHAN GE POINTS The interexchange point is the central location where meet to exchange network traffic Recall from section 3 that all the necessary switching and routing equipment for all lXP-attached ISPs and for the are physically located within a single facility Subsequently any disruption or disaster encountered at that facility could result in the loss of service at the IXP For most NSPs the loss of one is not critical bemuse NSPs generally have connections to multiple IXPs nationwide However for RSPs the loss of an IXP is more critical specifically if the RSP is connected to a single IXP In addition to the physical vulnerabilities are susceptible to routing problems between the various interconnected ISPs Routing problems could come from EGP protocol faults or invalid IXP routing tables operators attempt to eliminate routing problems by requiring a single EGP protocol at the IXP 5 3 INTERNET ACCESS The Internet access connection is the most vulnerable aSpect of the internet with respect to business and residential end users Business connections are typically single non- redundant connections from the business' LAN to the ER Like all critical single lines it the connection is lost the company loses lnternet connectivity Large companies with advanced nationwide WANs GE IBM and Boeing may employ redundant connections to the Internet for reliability A business' Web page will also be vulnerable to a cut in the Internet access link However businesses may have their Web pages hosted on an ISP Web server instead of hosting them on their own network This practice reduces LAN traffic and provides those Web pages with the additional reliability provided by the ISP network Residential access to the Internet is provided almost exclusively through analog modem or ISDN dial-up access Both connections are over single connections and are a single point of failure for the residential connection However overall reliability of the PN remains very high Reliability will drop when users access the Internet using alternate schemes such as cable which are not built to telephone industry standards Flat-rate pricing for Internet service has also introduced new availability issues for LEC PN networks These networks' demand and pricing models were designed based on a 5-minute voice call whereas Internet data calls can last hours During times of crisis when voice and Internet traffic surge long dial-up data calls may reduce the availability of the voice network using the same end-office switching capacity Continued growth in the use of alternative access techniques such as cable modems and DirectF'C satellites should eventually reduce these switching issues in PM carrier networks Some lntemet users connect over direct broadcast satellite services such as DirecPC DirecPC uses an inbound satellite connection over a l-meter dish and an outbound connection over an analog modem If either leg of this connection fails the entire connection will be lost The reliability of the analog modem link will be the same as described above The reliability of the satellite link will depend on the satellite terminal at the residential location and the satellite company's downlink location ADSL will be comparable is reliability to other LEC access technologies analog modem and ISDN However ADSL has limitations to where it can be installed ADSL cannot be installed near a strong AM radio station because of AM frequency interference on the ADSL signal Additionally only homes within feet of the LEC central office may be serviced by In the short term cable modem reliability is close to that of the cable television provider Cable modem service poses special reliability concerns because the cable industry unlike the voice telephone industry has not been required or expected to have the degree of reliability of phone service because it is not considered essential to public weltare 911 emergency access Typically the cable has not been installed to telephone industry standards and has been installed in shallow trenches typically less than 6 inches deep Additionally cable providers do not employ the restoration mechanisms of the traditional carriers These factors make the cable facility and ultimately the cable modem connection very vulnerable to cable cuts and outage-s Ill 1 APPENDIX A INTERNET ANALYSIS TOOL SITES Organization Central Intelligence Agency Department of Commerce Department of Defense Department of Health and Human Services Department of Energy Department of the Interior Department of Justice Department of State Department of Transportation Department of the Treasury Department of Veteran Affairs Federal Communications Commission Federal Emergency Management Agencyr General Services Administration Joint Staff National Aeronautics and Space Administration National Communication System Nuclear Regulator r Commission United States Department of Agriculture United States Information Agency r United States Postal Service FedWorid Information Network Library of Congress Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware Florida Geargia Hawaii Idaho Illinois Indiana Iowa Web Site immdtiodlamil W dl1h5 got r mtdoegov madoigov one usdoj wuwstatetgov mneustreasgov maragov tmaviccgov Wiemagov mmdticdlamil Washnasagoi' madmamil nwnrogov wowusiagov W usps gov mewiedworldgov mmlocgov mmnasced wumnstateiaki us muwstateaaus innustatecaus mmstatecous numnstatectus mm state de us ww state us Whawaiigov mowstateinus Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode island South Carolina South Dakota Tennessee TExas Utah Vermont Virginia Washington West Virginia Wisconsin WyOming Alta Vista America Online Apple Computer Inc Computer Network cnet CNN CompuServe Digex lnterport ISP wwustatoksus wwstatekyus wumtstatelaus wuwstaternans nuwstatems us wm' state mo us nris mls mt gov wuwstatenxnus W state nh us mastatenmus uwstatehiuus Watatenc us nm statendus nwoklaos stateokus wuwetateorus uwstatesdus W' state m us Wiexasgov state ut us wuwstatewpus altavistadigitalcom com wwapplecom mm' cnn cum whwcompusen'ecorn Winterportcom 111111 I Lycos Inc Macro Computer Systems Inc ISP Microsoft Inc MTV NetCnm ISP Netscape Olympics Oracle Corporation I rimenet ESP Sun Microsystems Inc USA Today r WebCrawler Windows Home Page Word Magazine on line W orld Wide Web Consortium Yahoo Massachusetts institute of Technologyr Ohio State University Stanford Universityr University of California Los Angeles University of Illinois Urbana-Charnpaig'n University of Michigan University of North Carolina University of Texas A3 wuwv microsoft com com mnetscapecom mun- prirnenetcom mm-so n com m usatoday com nmwmebcrawlercorn umw windows95 com wwyahooxom mwmitedu wuwosuedu nanuclaedu wuwurnichedu manutexasedu r101 burder -lde I net 00191 SanFrancmqunm net net FdduO-O A1101 Hal burden 10111110 n41 bordur'l JddI- reensbmu nt'l atlanIaZ-chE not 13-2 was-dc-qwl nalcomnlt - r1p5-gw ncr2n net mae-eastdgexnel fdd1 maa-ea51 nalcamnal I r1111 r101 arIan1a3-0r1 net burden-I'der KansasCurr mm 1101 mndnet KanmC1w mannet SILOUIS SILou-sl IJO 91 not 1113 1110 the 1-191 borderuT-IddI-D WulowSpr r110 r101 nil taunt- 11992 1 12 r1111 net 5 15 H210 T3 sl 1-11 2 F010 lackson-cr l buds -fddr hmago mum mem 11110111101 053 Chlcaun m1 1 net Wast rangu mm net 0 5111 TCOI ALTEHNET bOdellz-fddl- 1 Seattlemm r101 net 001102 Seattla mcu net 14412813534 wa-hsu-SE-oslommu MI - sl-lw-E H3fU-T3 nal 1191' fdd15-0 ch1cagu clc n dgb-fddIE-D chucaau010 1101 DC 13 nn5 nel BPS 114 1 2213 21 1 51 1204 10 104 521 1204 10 41091 1192 221 253 221 1132 39 33131 1204 1'0 3 341 1123103702491 1204 10 00 11 '1204 70 31 51 11922212511 11011292201111 211 109 32 21 1132 41 122 1151 1192411222101 1140222991921 1204 701 211 1204 0 3 1141 119222125 2901 11521211211 1204 70 2 061 1204 1'0 411 El 1192 35 111 351 1129 112 111 2411 1204 10 104201 32 233 33 101 1192233149 2011 119223133 21 1199 92129 21 114422310 5111 11442242021 11922215121 1204102321 204 W124 61' 113139100231 l204 0 1 2101 1132 39 11221 1204 10 203 1191 1192 141 119 51 1204 20 203 1131 1204 110 4 33 1144223135941 1204 101 21 1144 23 30 11 1144223103151 1131 103 1 191 1144 2211 30151 1131 1031121 MCI MCI BEN Planet Allarne l MCI NC-REN MCI MCI BEN Planet Netcom NC FIEN Olga Human-1 ANS MCI MCI BEN Planet EEN P191101 MCI MCI Other O'Ihal MCI BEN Planet BEN Planet EEN Planr BEN Plane-I Spun Spur- 1 EEN Planet MCI MCI Allarnet MCI Alternet MCI HWNGI MCI MCI Spun MCI BpIInl Sprinl 1 11 14 1 11 11 '4115 I ADSL ANS A NS RE ARPA AS ATM BGPI CAP CIX CLEC DARPA EGP EST FDDI FIX FTP IAT ICMP IEC IETF 1GP ILEC ISDN 15 ISP IXP LAN LEC MAE MAN MFS MXP NAP NCP NCS NREN EP NSF NSFNET LIST OF ACRONYMS Asyn'lmetric Digital Subscriber Line Advanced Networks and Services ANS Commercial Research and Education Advanced Research Projects Agency Autonomous System Transfer Mode Border Gateway Protocol Version 4 Competitive Access Provider Commercial Internet Exchange Competitive Local Exchange Carrier Defense Advanced Research Projects Agency Department Of Defense Exterior Gateway Protocol Eastern Standard Time Fiber Distributed Data Interface Federal Internet Exchange File Transfer Protocol Internet Analysis Tool Internet Control Message Protocol Interexchange Carrier Internet Engineering Task Force Interior Gateway Protocol Incumbent Local Exchange Carrier 1nternet Protocol Version Six Integrated Services Digital Network Information Sciences Institute Internet Service Provider Interexchange Point Local Area Network Local Exchange CarriEr Metropolitan Area Ethernet Metropolitan Area Network Metropolitan Fiber Systems Metropolitan Exchange Point Network Access Point Network Control Protocol National Communication System National Research and Education Network National Security Emergency Preparedness National Science Foundation National Science Foundation Network NSP OMNCS OSPF PC PN POP PVC EEOC RIP REP SLIP SM D5 SONET SVC TTL WAN luqu XNS National Service Provider Office of the Manager NCS Open Shortest Path First Personal Computer Public Network Point of Presence Point to-Point Protocol Permanent Virtual Circuit Regional Bell Operating Company Routing Information Protocol Regional Service Provider Serial Line Interface Protocol Switched Multimegabit Data Service Optical Network Switched Virtual Circuit SMDS Washington Area By pass Transmission Control Protocolenternet Protocol Time To Live User Datagram Protocol Very High Speed Backbone Network Sen r ice Wide Area Network World Wide Web Xerox Network Systems 11 ll Jill I 1111 JJ Is 10 12 SECTION 2 REFERENCES Cerf Vinton 3 Computer Networking Global Infrastructure for the let Century World Wide Web networks htmL 1995 Network Service Provider Interconnections and Exchange Points World Wide Web Cooper Lane The Commercialization of the Internet Week April 1 1996 pp 135-139 Fazio Dennis Hang Onto Your Packets The Information Super Highway Heads to Valleytair or Building a High Performance Computer System Without Reading the Instructions World Wide Web March 14 1995 Frazer Karen D The Phenomenon World Wide Web nsfnetf final reportf phenom html Hard Henr r Edward Short History of the Net World Wide Web 1995 MCI Telecommunications Corporation The Network World Wide Web network_map html 1995 Merit Network Inc Transition to World Wide Web umnmeritedu nsfnet final report transition html Merit Network Inc Router Server Technical Overview World Wide Web RAX rs over1 'iew h trnl National Laboratory for Applied Network Research Background Information World Wide Web National Laboratory for Applied Network Research Collaboration on the Very High Speed Backbone Network Services World Wide Web Wenlanrnet VBNS National Laboratory for Applied Network Research The National Science Foundation Network World Wide Web November 23 1995 13 National Science Foundation 93-52 - Network Access Point Manager Routing Arbiter Regional Network Providers and Very High Speed Backbone Network Services Provider for NSFNET and the Program - Program Solicitation May 6 1993 1-1 Quarterman lohn What is the Internet Anywayl World Wide Web gopheriiecomf Bf matrix news v4 what 403 1994 15 Rietz Randy Lewis Will Hiser of the Internet World Wide Web July 1995 Sprint Network Access Point Handbook October 25 199-1 Sprint SprintLink Customer Handbook Sprint Document #5953-2 October 11 1995 18 Zakon Robert Hobbes Hobbes' Internet Timeline v2 4a World Wide Web into isoc org guest zakon lnternet History HIT html 19% 'b I SECTION 3 REFERENCES 1 Ameritech The Chicago World Wide Web tech comf products data I map TheJChica go_N AP html 1995 2 Associated Press Computer Network Weathers Big Jolt Internet Users Swap News Worries After Quake Hits Associated Press January 13 199- DJ Bickel Robert Building Intranets luteniet World March 1996 p 33 4 T3 Backbone and interconnectivity World Wide Web June 1996 U1 Cisco Systems Case Studies Tutorial Section World Wide Web 6 Cisco Systems Protocol Brief 199-1 Cortese Amy Here Comes the Intranet Business Week February 26 1996 p 76 B Coy Peter Judge Paul Limo Service for Cruising the Net MCI and BT Will Help Business Surfers Go First Class for a Price Business Week June 2-1 1996 p 46 9 Detroit MXP What is an World Wide Web inswmainet mxp detroit ll Eng Paul M War of the Web Commercial Online Service Providers Upstart Companies and Telecommunications Companies All Fighting for Internet Market Busmess Week March 4 1996 p Finneran Michael Cable Modem Madness Busmess March 1996 p 68 12 Holmes Allan Flood Data Rides Internet Wave Federal Computer Week February 5 1996 p 1 13 IITF Reliability and Vulnerability of the National Information Infrastructure Nil information infrastructure Task Force August 17 1995 1-1 Loeb Larry The Stage is SET The SET Agreement Between MasterCard and Visa Could Pave the Way for Widespread E-corrunerce inter-net lVorld August 1990 p 5-1- 16 18 19 21 23 2-1 29 Mac Kie-Mason Jeffrey Varian Hal R Pricing the Internet World Wide Web gopher econ lsa mich edu April 1993 MacKie Mason Jeffrey Varian Hal R Economic FAQs About the Internet World Wide Web gopherecon lsa umich edu August 21 199-1 MacKie Mason Jeffrey Varian Hal R Some FAQs About Usage-Based Pricing World Wide Web gopher econ lsa umich edu November 4 199-1 Mendes Gerald H Next-Generation Takes Shape Communications Review March 1996 p 19 Mills Mike Offers Customers Free Internet Access The l asliiuglou Post March 19 1996 p C1 Netscape Netscape Announces New Real-time Audio and Video Framework tor Internet Applications Netscape Press Release Januar r 31 1996 Pacific Bell Mold-Lateral Peering Agreements Pacific Bell Network Access Point World Wide Web mpacbellcom Products NAP mlpa html August 14 1995 Pacific Bell Pacific Bell Network Access Point World Wide Web wuwpacbellcom products business fastrak networking nap PC Week puts ADSL on trial PC Week June 1996 p 3 PSlNet PSlNet Technology and Infrastructure World Wide Web psi-tech psi-tech shmil 1995 PSlNet - SMDS Washington Area Bypass World Wide Web misc Typical POP Design World Wide Web pop hmt1 Reilly Patrick More Publishers Charging for Web Services l dall Flat-cl Journal May 8 1996 p BS Rigdon Joan E Blurring the Line New Technology Aims to Make the Web Look and Act More Like Television Wall Street Journal March 28 1996 p R5 Sandberg Jared Making the Sale The Allure of On-Line Commerce ts Proponents Argue Will Eventually Prove Ovenehelming l'Vall Street Journal June 17 1996 p R6 1111 I ill 1 30 Scott D F The Underground Internet Through the MBON E the Internet May 36 Become the World s Largest Broadcast Service Computer Shopper March 19% p 548 Sprint Network Access Point Handbook October 25 199-1 Stevens Richard IP Illustrated Volume 1 The Protocols Addison-Wesley Publishing 199-1 Chapter 10 pp 9 -110 Swisher Kara By the Sweat of Their Browser District Entrepreneurs Turn a Web Search Idea Into a $38 Million Deal The Washington Post June 4 19% C1 Vaughan-Nichols Steven J Radio Comes to CyberSpace Byte October 1995 p 46 Verity John W Invoice M hat's An Invoice Electronic Commerce Will Soon Radically Alter the Way Business Buys and Sells Business I v eek june 10 1996 p 110 Winglield Nick to Connect Virtual Private Networks IrrfoWorM Ianuar 15 1996 p UUN ET The UUNET Network Backbone World Wide Web Ziegler Bart Up and Running Why Did the Web Replace IntEractive TV as the New Mantra A Simple Reason It's Here Wall Street journal March 28 19% p R6 SECTION 4 REFERENCES Asif Federal and State Government Sites World Wide Web immilinksnetf -ace government htmlitsh Bruno Charles Internet Health Report Condition Serious Network World Septernber 16 1996 pp 1 104-111 InterNIC InterNIC Whois Service World Wide Web MFS Datanet East Statistics World Wide Web ext2 mfsdatanet com MFS Datanet West Statistics World Wide Web University of Illinois at Urbana-Champaign Host Name to World Wide Web slamm ileI June 19 1995 Stevens Richard Illustrated Volume 1 The Protocols Addison-Wesley Publishing 1994 Chapter 10 pp 9 7-110 I ll 1