Server Connecivity Affects Embedded

The Fabric of the Cloud

Internet users and Internet-connected devices continue to place demands on the overall bandwidth and capacity of what has come to be called “The Cloud.” Advances in processing power, storage capacity and speed, along with local interconnect efficiency, are trying to keep up.


  • Page 1 of 1
    Bookmark and Share

Article Media

With the rapid growth of the Internet of Things, with the proliferation of machine-to-machine (M2M) systems and with the growing need for embedded devices to make vast amounts of data available for higher-level applications, the role of “The Cloud” is becoming pervasive. The Cloud is generally understood to be the Internet put to specific uses. But the growth of these uses puts even more demands on the host of data centers and server farms—and the vast numbers of server racks populating them—that constitute the physical presence of the Cloud.

With storage roughly doubling every two years, service providers are straining to keep up. And one of the big issues is latency among the components within the data center. So when it takes time to get to the page of a search, the delay is not only due to bandwidth limitations between the node and the service provider. It is also a result of latencies between components and networks within the data center, such as the need to translate protocols between the CPU and the network interface card (NIC) within the individual server and the external network switch and then to the network. In addition, most servers use two NICs—one for an Ethernet switch to the Internet and to other servers and another switch for Fibre Channel access to disk storage. While the latencies keep piling up, there is hope on the horizon for both increasing transfer speeds as well as simplification in the form of ever faster generations of PCI Express—currently with the arrival of Generation 3 at 8 Gtransfers/s and with Generation 4 at 16 Gtransfers/s about three years out.

A Transformation in Storage

Storage in the data center will be moving from hard disk drives (HDDs) to solid state drives (SSDs) as costs inevitably come down. The traditional data center accesses HDDs via a Fibre Channel network to which each server pizza box is connected via a Fibre Channel switch on top of the rack. This mostly requires a Fibre Channel NIC in each server in addition to the Ethernet NIC required for Internet access.

This does not mean that HDD storage will go away, but terabyte-class SSD storage will be coming to supplement it and provide for faster access in a multi-tiered caching scheme that will no doubt make use of PCIe as well. According to Akber Kazmi, senior marketing director for PLX Technology, the need for higher-speed transfer between CPU and the rest of the system is being driven by the increasing number of cores in the CPU and the need to get the data out. The recently introduced Romley CPUs from Intel with eight and more cores have also integrated multiple PCIe Gen3 lanes into the chip so the interface to the system is via PCIe to the NIC. The distance limit on the board is about 14 inches maximum without circuits such as retimers.

The next logical step would be to continue with that same speed between CPU and storage—and indeed, between CPU, storage and the Internet—an Internet that will be pushing to 10G, at least for PCIe Gen3. “If you look at the amount of data that is being sorted,” Kazmi notes, “to get the data in and out of the CPU, you need the PCIe bus. Now you have to go to the next step to meet the demand of the data interfaces.”

Another big difference is that the SSDs are more local to the server. If not within the box itself, they tend to be in or near the rack, making access with a PCIe over cable scheme practical and able to handle the higher data rates of which SSDs are capable. Previously the HDDs sit somewhere on the other side of the data center and communicate with the servers over Fibre Channel.

According to Kazmi, the use of SSDs in a tiered caching role in conjunction with HDDs is becoming quite popular. And the availability of large solid state memory arrays such as the 3000 Series from Violin Memory can now offer scalable storage from 500 Gbytes to 140 Terabytes (Figure 1). The cost of NAND flash is currently hovering around $1 per gigabyte and will eventually match that of HDD storage. Of course, care must be taken with the life cycle of SSDs in that they should only go into applications with relatively low write frequencies. In that respect, their role in the data center will complement that of the HDDs in greatly reducing latency for access to data.

Figure 1
The Violin 3000 series of solid state memory arrays can scale to more than 140 TB in a rack with performance of over two million IOPS.

Spreading the Fabric

If you look at the present day data center, Kazmi points out, “You have a 1G or 10G NIC in the server. Each server has two cards and each card connects to a top-of-the-rack switch (Ethernet and Fibre Channel) and each server has to bear the cost of the two adapters—heat and power” (Figure 2). In addition, of course, there are latencies due to electrical and protocol transitions between PCIe to NIC, NIC to switch and switch to network. PLX is advocating a simplification by using PCIe in what it is calling a converged ExpressFabic.

Figure 2
Today’s data center consists of rack-mounted servers with multiple network cards running different protocols to connect to the Ethernet (Internet), storage and other servers.

Since PCIe is now readily available in CPUs, Gen3 access to a PCIe NIC is now possible, and access to the top-of-the-rack switches and to other servers in the rack is handled by the Gen3 PCIe over cable technology developed by One Stop Systems. PLX has also developed a 48-lane fabric switch and is soon to introduce a 96-lane version. Such switches can be configured for x16 channels to interface with the Ethernet and Fibre Channel NICs and as x4 channels for access between servers in a rack using PCIe over cable. PLX has also announced that it is working with Avago on a low-cost optical connector that will enable 30- to 50-meter Gen3 access to other racks in the data center. In the near future there will be silicon that will do Ethernet on one side and PCIe on the other. The result could be a high-speed unified fabric such as the one in Figure 3.

Figure 3
A PCI Express-based server fabric will use Gen3 and, later, Gen4 PCIe fabric and switches throughout to minimize latency and access different parts of the data center including local solid state storage and GPGPU arrays.

In addition to local and massive SSD storage, Kazmi foresees Gen3 and eventually Gen4 PCIe access to arrays of general-purpose graphics processors (GPGPUs) for extremely intense floating point processing. Such arrays could act as cloud-based clusters for high-speed scientific processing. For example, installations such as a national laboratory like Lawrence Livermore could send jobs in the evening when traffic is down and pick up the results in the morning in situations where they happen to need more compute power than they have in house.

When Gen4?

Of course, now that PCIe Gen3 is starting to appear as actual products, everyone is clamoring to know when they can expect Gen4. Gen3 is attracting attention because Intel recently introduced its “Romley” Sandy Bridge-based server series of CPUs, with their 8 to 16 cores and that also integrate multiple PCIe Gen3 lanes, hence making them practical to use in schemes such as those discussed here.

We asked PLX’s Akber Kazmi, and his unofficial estimates for actual products on the market are approximately sometime in late 2014. The specification is still being defined and experiments are underway, and a completed spec may be expected sometime mid to late next year. There had been a debate over whether to implement Gen4 in copper or as an optical interface. The decision was finally made to go with copper one more time, although this is probably the limit. The limit on a run for copper is maybe 10 to 12 inches on a board after which additional electronics such as retimers and repeaters are necessary. Additionally, there will be market demand for cabling solutions, which will no doubt be optical as well. In addition, the prediction is that graphics processing will take a big advantage from the increased speed.

So the embedded developer might be forgiven for asking what in the world he or she might need with the 16 Gtransfers/s that will be available from PCIe Gen4. But that question appears to be limited to the local aspects of what are increasingly more global, cloud-oriented applications. If science and engineering can find ways to make things run faster, developers will definitely find ways to put that to use. 

PLX Technology
Sunnyvale, CA.
(408) 774-9060

One Stop Systems
Escondido, CA.
(877) 438-2724

Avago Technologies
San Jose, CA.
(408) 435-7400

Violin Memory
Mountain View, CA.
(650) 396-1500