High-speed networking did not always have to make the CPU handle every movement of data
Moving a large amount of information across a network requires more than a fast cable and a high-speed network adapter. The operating system normally has to process network traffic, move information through memory, and coordinate the work performed by applications and hardware.
At very high transfer rates, that processing becomes significant. A processor spending more time handling network activity has fewer resources available for virtual machines, databases, applications, and other workloads.
Remote Direct Memory Access, commonly called RDMA, provided a different approach. With compatible hardware and software, data could move between the memory of networked computers with substantially less involvement from the processors on those systems.
Remote Direct Memory Access is a networking technology designed to transfer data directly between the memory of connected systems while reducing the amount of operating-system and processor work normally required for network communication.
Fast Networking Could Create More Work for the CPU
A conventional network transfer passes through several layers of the computer. Data moves between an application, the operating system’s networking stack, system memory, drivers, and the network adapter.
Each layer performs useful work, but that work consumes processor time. As network speeds increase, the number of operations needed to keep data moving can become an important part of overall system load.
Requests or supplies the information being transferred across the network.
Processes the networking protocols and coordinates movement of the data.
Transmits and receives the actual network traffic through the physical connection.
A faster adapter therefore does not automatically eliminate processing overhead. Without suitable acceleration, increasing network throughput can also increase the amount of work performed elsewhere in the computer.
RDMA Changed the Route Through the System
RDMA-capable adapters were designed to reduce that overhead. Instead of requiring the processor and conventional networking stack to participate heavily in every stage of a large transfer, the hardware could place data more directly into memory.
This shortened the path between the network and the memory being used by the workload.
Conventional transfer
Network traffic moves through the normal networking stack, requiring repeated operating-system and processor involvement as information travels between the adapter and application.
RDMA transfer
Compatible network hardware can transfer information more directly between memory locations, reducing the amount of CPU processing needed to move the same data.
RDMA reduced processor involvement in the data path. The processor still controlled applications and the broader system, but it did not have to perform as much routine work for each transfer.
SMB Direct Connected File Sharing With RDMA
Windows used this capability through SMB Direct. SMB is the protocol commonly used for Windows file sharing, while SMB Direct allows suitable SMB traffic to use RDMA-capable network adapters.
When the required hardware and network configuration were available, SMB could take advantage of high throughput and low latency without placing the same processing burden on the host CPU.
The result was particularly useful when network storage needed to behave less like a slow remote resource and more like storage attached closely to the computer using it.
Remote Storage Could Feel Much Closer
Ordinary office file sharing rarely requires extraordinary network performance. Opening documents from a shared folder is very different from continuously moving large quantities of data for a database or a virtual machine.
Those demanding workloads can depend heavily on storage latency and throughput. If the storage is located on another server, the network becomes part of the storage path.
Data path
A fast storage server can still feel slow if the network path introduces excessive latency, consumes too much CPU time, or cannot move information quickly enough. SMB Direct addressed the network portion of that problem by combining SMB file access with RDMA-capable hardware.
This made the technology relevant to workloads such as Hyper-V and SQL Server, where large amounts of storage traffic could travel between machines and where both latency and processor efficiency mattered.
The Network Adapter Needed Special Capabilities
RDMA was not something that could be added to every network connection simply by changing a software setting. The network adapters themselves needed to support RDMA.
Several underlying technologies could provide that capability, including iWARP, InfiniBand, and RDMA over Converged Ethernet, commonly known as RoCE.
| Technology | General role |
|---|---|
| iWARP | Provides RDMA communication using an IP-based networking approach |
| InfiniBand | Provides a high-performance networking architecture commonly associated with demanding computing and storage workloads |
| RoCE | Carries RDMA communication over Ethernet infrastructure |
The hardware had to explicitly support an RDMA technology. Ordinary network adapters continued to use the conventional networking path even when their advertised link speed was high.
SMB Could Discover the Faster Path Automatically
SMB Multichannel worked alongside SMB Direct by identifying the capabilities of available network adapters. When SMB detected that suitable RDMA hardware was present, it could establish RDMA connections for the SMB session.
This relationship allowed the protocol to use the specialized path when it was available while retaining ordinary TCP/IP networking as part of the connection process.
The systems initially establish communication and determine the capabilities available to the session.
SMB Multichannel can recognize when the network interfaces support RDMA.
Compatible systems can move SMB traffic onto the accelerated data path.
Large amounts of information can move with less processor involvement than a comparable conventional transfer.
More Than One Connection Could Improve Resilience
SMB Multichannel could establish multiple network connections for a single SMB session. With suitable hardware, this provided benefits beyond raw speed.
- Multiple network paths could contribute additional throughput.
- RDMA-capable interfaces could provide low-latency data movement.
- Several interfaces could reduce dependence on one physical network path.
- A surviving connection could help preserve communication when another path failed.
Performance and fault tolerance therefore did not have to be completely separate goals. The same architecture that identified multiple capable interfaces could use those paths to make network storage more robust.
A high-speed network becomes more useful when moving the data requires less work from the computer that needs to process it.
Lower Processor Usage Could Matter as Much as Transfer Speed
Network performance is often reduced to a single number such as gigabits per second. That number describes only part of the system.
Two networking methods might move similar amounts of data while imposing very different loads on the processor. For a lightly used desktop computer, that difference may have little practical importance. On a server running many virtual machines or processing intensive workloads, it can be significant.
Reducing CPU overhead leaves more processor capacity available for the workload that the server was built to run. Network acceleration can therefore improve the efficiency of the entire system even when link speed itself does not change.
The Adapter Became Part of the Data Architecture
Network adapters were once easy to think of as simple gateways between a computer and a cable. Technologies such as RDMA made that description increasingly incomplete.
The adapter could participate directly in how efficiently information moved through the system. Its capabilities affected not only network speed but also latency, memory transfers, processor utilization, storage performance, and the behavior of server workloads.
SMB Direct demonstrated that faster networking was not only about transmitting more bits each second. By using RDMA-capable hardware, a network connection could move large amounts of data while demanding less routine processing from the CPU.
That made the network adapter more than a communication port. In high-performance systems, it had become an active part of the path between storage, memory, and the applications using the data.
