Gigabit Campus Network Design—
Principles and Architecture
Introduction
The availability of multigigabit campus switches from Cisco presents customers the opportunity to build extremely high-performance networks
with high reliability. Gigabit Ethernet and Gigabit EtherChannel
® provide the high-capacity trunks needed to connect these gigabit switches.If the right network design approach is followed, performance and reliability are easy to achieve. Unfortunately, some alternative network
design approaches can result in a network with lower performance, reliability, and manageability. With so many features available, and with
so many permutations and combinations possible, it is easy to go astray. This paper is the result of Cisco’s experience with many different
customers and it represents a common sense approach to network design that will result in simple, reliable, manageable networks.
The conceptual approach followed in this paper has been used successfully in routed and switched networks around the world for many
years. This hierarchical approach is called the “multilayer design.” The multilayer design is modular and capacity scales as building blocks
are added. A multilayer campus intranet is highly deterministic, which makes it easy to troubleshoot as it scales. Intelligent Layer 3 services
reduce the scope of many typical problems caused by misconfigured or malfunctioning equipment. Intelligent Layer 3 routing protocols such
as Open Shortest Path First (OSPF) and Enhanced Interior Gateway Routing Protocol (EIGRP) handle load balancing and fast convergence.
The multilayer model makes migration easier because it preserves the existing addressing plan of campus networks based on routers and
hubs. Redundancy and fast convergence to the wiring closet are provided by Hot Standby Router Protocol (HSRP). Bandwidth scales from
Fast Ethernet to Fast EtherChannel and from Gigabit Ethernet to Gigabit EtherChannel. The model supports all common campus protocols.
The multilayer model will be described, along with two main scalability options appropriate for building-sized networks up to large
campus networks. Five different backbone designs with different performance and scalability are also presented. In this paper the term
backbone is used to represent the switches and links in the core of the network through which all traffic passes on its way from client to server.
Structured Design with Multilayer Switching
The development of Layer 2 switching in hardware several years ago led to network designs that emphasized Layer 2 switching. These designs
are characterized as “flat” because they avoid the logical, hierarchical structure and summarization provided by routers. Campus-wide virtual
LANs (VLANs) are also based on the flat design model.
Layer 3 switching provides the same advantages as routing in campus network design, with the added performance boost from packet
forwarding handled by specialized hardware. Putting Layer 3 switching in the distribution layer and backbone of the campus segments the
campus into smaller, more manageable pieces. Important multilayer services such as broadcast suppression and protocol filtering are used in
the Layer 2 switches at the access layer. The multilayer approach combines Layer 2 switching with Layer 3 switching to achieve robust, highly
available campus networks.
It is helpful to analyze campus network designs in the following ways:
Failure Domain
A group of Layer 2 switches connected together is called a Layer 2 switched domain. The Layer 2 switched domain can be considered as a
failure domain because a misconfigured or malfunctioning workstation can introduce errors that will impact or disable the entire domain. A
jabbering network interface card (NIC) may flood the entire domain with broadcasts. A workstation with the wrong IP address can become a
black hole for packets. Problems of this nature are difficult to localize.
The scope of the failure domain should be reduced by restricting it to a single Layer 2 switch in one wiring closet if possible. In order to
do this, the deployment of VLANs and VLAN trunking is restricted. Ideally one VLAN (IP subnet) is restricted to one wiring-closet switch.
The gigabit uplinks from each wiring-closet switch connect directly to routed interfaces on Layer 3 switches. One way to achieve load
balancing is to configure two such VLANs in the wiring-closet switch, which is shown later.
Broadcast Domain
Media Access Control (MAC)-layer broadcasts flood throughout the Layer 2 switched domain. Use Layer 3 switching in a structured design
to reduce the scope of broadcast domains. In addition, intelligent, protocol-aware features of Layer 3 switches will further contain broadcasts
such as Dynamic Host Configuration Protocol (DHCP) by converting them into directed unicasts. These protocol-aware features are a function
of the Cisco IOS
® software, which is common to Cisco Layer 3 switches and routers.Spanning-Tree Domain
Layer 2 switches run spanning-tree protocol to break loops in the Layer 2 topology. If loops are included in the Layer 2 design, then redundant
links are put in blocking mode and do not forward traffic. It is preferred to avoid Layer 2 loops by design and have the Layer 3 protocols handle
load balancing and redundancy, so that all links are used for traffic.
The spanning-tree domain should be kept as simple as possible and loops should be avoided. With loops in the Layer 2 topology,
spanning-tree protocol takes between 30 and 50 seconds to converge. So, avoiding loops is especially important in the mission- critical parts
of the network, such as the campus backbone. To prevent spanning-tree protocol convergence events in the campus backbone, ensure that all
links connecting backbone switches are routed links, not VLAN trunks. This will also constrain the broadcast and failure domains as explained
previously.
Use Layer 3 switching in a structured design to reduce the scope of spanning-tree domains. Let a Layer 3 routing protocol, such as
Enhanced IGRP or OSPF, handle load balancing, redundancy, and recovery in the backbone.
Virtual LAN
A VLAN is also an extended Layer 2 switched domain. If several VLANs coexist across a set of Layer 2 switches, each individual
VLAN has the same characteristics of a failure domain, broadcast domain, and spanning-tree domain, as described above. So, although
VLANs can be used to segment the campus network logically, deploying pervasive VLANs throughout the campus adds to the complexity.
Avoiding loops and restricting one VLAN to a single Layer 2 switch in one wiring closet will minimize the complexity.
One of the motivations in the development of VLAN technology was to take advantage of high-speed Layer 2 switching.With the advent
of high-performance Layer 3 switching in hardware, the use of VLANs is no longer related to performance. A VLAN can be used to logically
associate a workgroup with a common access policy as defined by access control lists (ACLs). Similarly, VLANs can be used within a server
farm to associate a group of servers with a common access policy as defined by ACLs.
IP Subnet
An IP subnet also maps to the Layer 2 switched domain; therefore, the IP subnet is the logical Layer 3 equivalent of the VLAN at Layer 2.
The IP subnet address is defined at the Layer 3 switch where the Layer 2 switch domain terminates. The advantage of subnetting is that Layer
3 switches exchange summarized reachability information, rather than learning the path to every host in the whole network. Summarization
is the key to the scalability benefits of routing protocols, such as Enhanced IGRP and OSPF.
In an ideal, highly structured design, one IP subnet maps to a single VLAN, which maps to a single switch in a wiring closet. This design
model is somewhat restrictive, but pays huge dividends in simplicity and ease of troubleshooting.
Policy Domain
Access policy is usually defined on the routers or Layer 3 switches in the campus intranet. A convenient way to define policy is with ACLs
that apply to an IP subnet. Thus, a group of servers with similar access policies can be conveniently grouped together in the same IP subnet
and the same VLAN. Other services, such as DHCP are defined on an IP subnet basis.
A useful new feature of the Catalyst
® 6000 family of products is the VLAN access control list (VACL). A Catalyst 6000 or Catalyst 6500can use conventional ACLs as well as VACLs. A VACL provides granular policy control applied between stations within a VLAN.