Accelerating Data Centers: FS Unveils Next-Gen 400G Solutions

As large-scale data centers transition to faster and more scalable infrastructures and with the rapid adoption of hyperscale cloud infrastructures and services, existing 100G networks fall short in meeting current demands. As the next-generation mainstream port technology, 400G significantly increases network bandwidth, enhances link utilization, and assists operators, OTT providers, and other clients in effectively managing unprecedented data traffic growth.

To meet the demand for higher data rates, FS has been actively developing a series of 400G products, including 400G switches, optical modules, cables, and network adapters.

FS 400G Switches

The emergence of 400G data center switches has facilitated the transition from 100G to 400G in data centers, providing flexibility for building large-scale leaf and spine designs while reducing the total number of network devices. This reduction can save costs and decrease power consumption. Whether it’s the powerful N9510-64D or the versatile N9550 series, FS 400G data center switches can deliver the performance and flexibility required for today’s data-intensive applications.

Of particular note is that, as open network switches, the N8550 and N9550 series switches can enhance flexibility by freely choosing preferred operating systems. They are designed to meet customer requirements by providing comprehensive support for L3 features, SONiC and Broadcom chips, and data center functionalities. Additionally, FS offers PicOS-based open network switch operating system solutions, which provide a more flexible, programmable, and scalable network operating system (NOS) at a lower total cost of ownership (TCO).

FS 400G Transceivers

FS offers two different types of packaging for its 400G transceivers: QSFP-DD and OSFP, developed to support 400G with performance as their hallmark. Additionally, FS provides CFP2 DCO transceivers for coherent transmission at various rates (100G/200G/400G) in DWDM applications. Moreover, FS has developed InfiniBand cables and transceivers to enhance the performance of HPC networks, meeting the requirements for high bandwidth, low latency, and highly reliable connections.

FS conducts rigorous testing on its 400G optical modules using advanced analytical equipment, including TX/RX testing, temperature measurement, rate testing, and spectrometer evaluation tests, to ensure the performance and compatibility of the optical modules.

FS 400G Cables

When planning 400G Ethernet cabling or connection schemes, it’s essential to choose devices with low insertion loss and good return loss to meet the performance requirements of high-density data center links. FS offers various wiring options, including DAC/AOC cables and breakout cables. FS DAC/AOC breakout cables provide three connection types to meet high-density requirements for standard and combination connector configurations: 4x100G, 2x200G, and 8x50G. Their low insertion loss and ultra-low crosstalk effectively enhance transmission performance, while their high bend flexibility offers cost-effective solutions for short links.

FS 400G Network Adapters

FS 400G network adapters utilize the industry-leading ConnectX-7 series cards. The ConnectX-7 VPI card offers a 400Gb/s port for InfiniBand, ultra-low latency, and delivers between 330 to 3.7 billion messages per second, enabling top performance and flexibility to meet the growing demands of data center applications. In addition to all existing innovative features from previous versions, the ConnectX-7 card also provides numerous enhanced functionalities to further boost performance and scalability.

FS 400G Networking Soluitons

To maximize the utilization of the 400G product series, FS offers comprehensive 400G network solutions, such as solutions tailored for upgrading from 100G to high-density 400G data centers. These solutions provide diverse and adaptable networking options customized for cloud data centers. They are designed to tackle the continuous increase in data center traffic and the growing need for high-bandwidth solutions in extensive 400G data center networks.

For more information about FS 400G products, please read FS 400G Product Family Introduction.

How FS Can Help

Register for an FS account now, choose from our range of 400G products and solutions tailored to your needs, and effortlessly upgrade your network.

Exploring FS 100G EDR InfiniBand Solutions: Powering HPC

In the realm of high-speed processing and complex workloads, InfiniBand is pivotal for HPC and hyperscale clouds. This article explores FS’s 100G EDR InfiniBand solution, emphasizing the deployment of QSFP28 EDR transceivers and cables to boost network performance.

What are the InfiniBand HDR 100G Cables and Transceivers

InfiniBand EDR 100G Active AOC Cables

The NVIDIA InfiniBand MFA1A00-E001, an active optical cable based on Class 1 FDA Laser, is designed for InfiniBand 100Gb/s EDR systems. With lengths ranging from 1m to 100m, these cables offer predictable latency, consuming a max of 3.5W, and enhancing airflow in high-speed HPC environments.

InfiniBand EDR 100G Passive Copper Cables

The NVIDIA InfiniBand MCP1600-E001E30 is available in lengths of 0.5m to 3m. With four high-speed copper pairs supporting up to 25Gb/s, it offers efficient short-haul connectivity. Featuring EEPROM on each QSFP28 port, it enhances host system communication, enabling higher port bandwidth, density, and configurability while reducing power demand in data centers.

InfiniBand EDR 100G Optical Modules

The 100Gb EDR optical modules, packaged in QSFP28 form factor with LC duplex or MTP/MPO-12 connectors, are suitable for both EDR InfiniBand and 100G Ethernet. They can be categorized into QSFP28 SR4, QSEP28 PSM4, QSFP28 CWDM4, and QSFP28 LR4 based on transmission distance requirements.

100Gb InfiniBand EDR System Scenario Applications

InfiniBand has gained widespread adoption in data centers and other domains, primarily employing the spine-leaf architecture. In data centers, transceivers and cables play a pivotal role in two key scenarios: Data Center to User and Data Center Interconnects.

For more on application scenarios, please read 100G InfiniBand EDR Solution.

Conclusion

Amidst the evolving landscape of 100G InfiniBand EDR, FS’s solution emerges as mature and robust. Offering high bandwidth, low latency, and reduced power consumption, it enables higher port density and configurability at a lower cost. Tailored for large-scale data centers, HPC, and future network expansion, customers can choose products based on application needs, transmission distance, and deployment. FS 100G EDR InfiniBand solution meets the escalating demands of modern computational workloads.

Revolutionizing Data Center Networking: From Traditional to Advanced Architectures

As businesses upgrade their data centers, they’re transitioning from traditional 2-layer network architectures to more advanced 3-layer routing frameworks. Protocols like OSPF and BGP are increasingly used to manage connectivity and maintain network reliability. However, certain applications, especially those related to virtualization, HPC, and storage, still rely on 2-layer network connectivity due to their specific requirements.

VXLAN Overlay Network Virtualization

In today’s fast-paced digital environment, applications are evolving to transcend physical hardware and networking constraints. An ideal networking solution offers scalability, seamless migration, and robust reliability within a 2-layer framework. VXLAN tunneling technology has emerged as a key enabler, constructing a virtual 2-layer network on top of the existing 3-layer infrastructure. Control plane protocols like EVPN synchronize network states and tables, fulfilling contemporary business networking requirements.

Network virtualization divides a single physical network into distinct virtual networks, optimizing resource use across data center infrastructure. VXLAN, utilizing standard overlay tunneling encapsulation, extends the control plane using the BGP protocol for better compatibility and flexibility. VXLAN provides a larger namespace for network isolation across the 3-layer network, supporting up to 16 million networks. EVPN disseminates layer 2 MAC and layer 3 IP information, enabling communication between VNIs and supporting both centralized and distributed deployment models.

For enhanced flexibility, this project utilizes a distributed gateway setup, supporting agile execution and deployment processes. Equal-Cost Multipath (ECMP) routing and other methodologies optimize resource utilization and offer protection from single node failures.

RoCE over EVPN-VXLAN

RoCE technology facilitates efficient data transfer between servers, reducing CPU overhead and network latency. Integrating RoCE with EVPN-VXLAN enables high-throughput, low-latency network transmission in high-performance data center environments, enhancing scalability. Network virtualization divides physical resources into virtual networks tailored to distinct business needs, allowing for agile resource management and rapid service deployment.

Simplified network planning, deployment, and operations are essential for managing large-scale networks efficiently. Unnumbered BGP eliminates the need for complex IP address schemes, improving efficiency and reducing operational risks. Real-time fault detection tools like WJH provide deep network insights, enabling quick resolution of network challenges.

Conclusion

Essentially, recent advancements in data center networking focus on simplifying network design, deployment, and management. Deploying technological solutions such as Unnumbered BGP eliminates the need for complex IP address schemes, reducing setup errors and boosting productivity. Tools like WJH enable immediate fault detection, providing valuable network insights and enabling quick resolution of network issues. The evolution of data center infrastructures is moving towards distributed and interconnected multi-data center configurations, requiring faster network connections and improving overall service quality for users.

For detailed information on EVPN-VXLAN and RoCE, you can read: Optimizing Data Center Networks: Harnessing the Power of EVPN-VXLAN, RoCE, and Advanced Routing Strategies.

HPC and Future Networks: Architectures, Technologies, and Innovations

High-Performance Computing (HPC) has become a crucial tool for solving complex problems and pushing the boundaries of scientific research, and various other applications. However, efficient operation of HPC systems requires specialized infrastructure and support. HPC has emerged as an indispensable tool across various domains, capable of addressing complex challenges and driving innovation in fields such as science, meteorology, finance, and healthcare.

Understanding the importance of data centers in supporting HPC is essential, as knowing the three fundamental components—compute, storage, and networking—that constitute high-performance computing systems is crucial.

Facilities in High-Performance Computing

Intensive computations in HPC environments generate substantial heat, necessitating advanced cooling solutions. Efficient cooling prevents overheating, ensuring system stability and prolonging hardware lifespan. Supporting HPC, data centers employ cutting-edge cooling facilities, including liquid cooling systems and precision air conditioning. Moreover, data center architects explore innovative cooling technologies like immersion cooling, submerging servers in special liquids for effective heat dissipation.

Success in HPC data centers relies on a range of specialized equipment tailored to meet the unique demands of high-performance computing. Key components include data center switches, server network cards, high-speed optical modules, DAC and AOC cables, and power supplies.

The Growing Demand for Network Infrastructure in High-Performance Computing

With revolutionary technologies like 5G, big data, and the Internet of Things (IoT) permeating various aspects of society, the trajectory towards an intelligent, digitized society over the next two to three decades is inevitable. Data center computing power has become a powerful driving force, shifting focus from resource scale to computational scale.

To meet the ever-growing demand for computing power, high-performance computing (HPC) has become a top priority, especially as computational cluster scales expand from the petascale to the exascale. This shift imposes increasingly higher demands on interconnect network performance, marking a clear trend of deep integration between computation and networking. HPC introduces different network performance requirements in three typical scenarios: loosely coupled computing scenarios, tightly coupled scenarios, and data-intensive computing scenarios.

In summary, high-performance computing (HPC) imposes stringent requirements on network throughput and latency. To meet these demands, the industry widely adopts Remote Direct Memory Access (RDMA) as an alternative to the TCP protocol to reduce latency and maximize CPU utilization on servers. Despite its advantages, the sensitivity of RDMA to network packet loss highlights the importance of lossless networks.

The Evolution of High-Performance Computing Networks

Traditional data center networks have historically adopted a multi-hop symmetric architecture based on Ethernet technology, relying on the TCP/IP protocol stack for transmission. However, despite over 30 years of development, Remote Direct Memory Access (RDMA) technology has gradually replaced TCP/IP, becoming the preferred protocol for HPC networks. Additionally, the choice of RDMA network layer protocols has evolved from expensive lossless networks based on the InfiniBand (IB) protocol to intelligent lossless networks based on Ethernet.

From TCP to RDMA

In traditional data centers, Ethernet technology and the TCP/IP protocol stack have been the norm for building multi-hop symmetric network architectures. However, due to two main limitations—latency issues and CPU utilization—the TCP/IP network is no longer sufficient to meet the demands of high-performance computing. To address these challenges, RDMA functionality has been introduced at the server side. RDMA is a direct memory access technology that enables data transfer directly between computer memories without involving the operating system, thus bypassing time-consuming processor operations. This approach achieves high bandwidth, low latency, and low resource utilization.

From IB to RoCE

RDMA enables direct data read and write between applications and network cards. RDMA’s zero-copy mechanism allows the receiving end to read data directly from the sending end’s memory, significantly reducing CPU burden and improving CPU efficiency. Currently, there are three choices for RDMA network layer protocols: InfiniBand, iWARP (Internet Wide Area RDMA Protocol), and RoCE (RDMA over Converged Ethernet). Although RoCE offers many advantages, its sensitivity to packet loss requires support from lossless Ethernet. This evolution of HPC networks reflects a continuous pursuit of enhanced performance, efficiency, and interoperability.

Enterprise Innovative Solution: Designing High-Performance Data Center Networks

The architecture of data center networks has evolved from the traditional core-aggregation-access model to the modern Spine-Leaf design. This approach fully utilizes network interconnection bandwidth, reduces multi-layer convergence rates, and is easy to scale. When traffic bottlenecks occur, horizontal expansion can be achieved by increasing uplink links and reducing convergence ratios, minimizing the impact on bandwidth expansion. Overlay networks utilize EVPN-VXLAN technology to achieve flexible network deployment and resource allocation.

This solution draws on the design experience of internet data center networks, adopting the Spine-Leaf architecture and EVPN-VXLAN technology to provide a versatile and scalable network infrastructure for upper-layer services. Production and office networks are isolated by domain firewalls and connected to office buildings, labs, and regional center exits. The core switches of the production network provide up to 1.6Tb/s of inter-POD communication bandwidth and 160G of high-speed network egress capacity, with each POD’s internal horizontal network capacity reaching 24Tb, ensuring minimal packet loss. The building wiring is planned based on the Spine-Leaf architecture, with each POD’s switches interconnected using 100G links and deployed in TOR mode. The overall network structure is more streamlined, improving cable deployment and management efficiency.

Future-Oriented Equipment Selection

When envisioning and building data center networks, careful consideration of technological advancements, industry trends, and operational costs over the next five years is crucial. The choice of network switches plays a vital role in the overall design of data center networks. Traditional large-scale network designs often opt for chassis-based equipment to enhance the overall capacity of the network system, but scalability is limited.

Therefore, for the network equipment selection of this project, NVIDIA strongly advocates for adopting a modular switch network architecture. This strategic approach facilitates rapid familiarization by maintenance teams. Additionally, it provides operational flexibility for future network architecture adjustments, equipment reuse, and maintenance replacements.

In response to the ongoing trend of business transformation and the surge in demand for big data, most data center network designs adopt the mature Spine-Leaf architecture, coupled with EVPN-VXLAN technology to achieve efficient network virtualization. This architectural approach ensures convenient high-bandwidth, low-latency network traffic, laying the foundation for scalability and flexibility.

How FS Can Help

FS is a professional provider of communication and high-speed network system solutions for network, data center, and telecommunications customers. Leveraging NVIDIA® InfiniBand switches, 100G/200G/400G/800G InfiniBand transceivers, and NVIDIA® InfiniBand adapters, FS offers customers a comprehensive set of solutions based on InfiniBand and lossless Ethernet (RoCE). These solutions meet diverse application requirements, enabling users to accelerate their businesses and enhance performance. For more information, please visit FS.COM.

Enhancing Data Center Networks with InfiniBand Solutions

With the rapid growth of data centres driven by expansive models, cloud computing, and big data analytics, there is an increasing demand for high-speed data transfer and low-latency communication. In this complex network ecosystem, InfiniBand (IB) technology has become a market leader, playing a vital role in addressing the challenges posed by the training and deployment of expansive models. Constructing high-speed networks within data centres requires essential components such as high-rate network cards, optical modules, switches, and advanced network interconnect technologies.

NVIDIA Quantum™-2 InfiniBand Switch

When selecting switches, NVIDIA’s QM9700 and QM9790 series stand out as the most advanced devices. Built on NVIDIA Quantum-2 architecture, they offer 64 NDR 400Gb/s InfiniBand ports within a standard 1U chassis. This breakthrough translates to an individual switch providing a total bidirectional bandwidth of 51.2 terabits per second (Tb/s), along with an unprecedented handling capacity exceeding 66.5 billion packets per second (BPPS).

The NVIDIA Quantum-2 InfiniBand switches extend beyond their NDR high-speed data transfer capabilities, incorporating extensive throughput, on-chip compute processing, advanced intelligent acceleration features, adaptability, and sturdy construction. These attributes establish them as the quintessential selections for sectors involving high-performance computing (HPC) and expansive cloud-based infrastructures.

Additionally, the integration of NDR switches helps minimise overall cost and complexity, thereby promoting the development of data centre network technology.

Also Check- Revolutionizing Data Center Networks: 800G Optical Modules and NDR Switches | FS Community

ConnectX®-7 InfiniBand Card

The NVIDIA ConnectX®-7 InfiniBand network card (HCA) ASIC delivers a staggering data throughput of 400Gb/s, supporting 16 lanes of PCIe 5.0 or PCIe 4.0 host interface. Utilising advanced SerDes technology with 100Gb/s per lane, the 400Gb/s InfiniBand is achieved through OSFP connectors on both the switch and HCA ports. The OSFP connector on the switch supports two 400Gb/s InfiniBand ports or 200Gb/s InfiniBand ports, while the network card HCA features one 400Gb/s InfiniBand port. The product range includes active and passive copper cables, transceivers, and MPO fibre cables. Notably, despite both using OSFP packaging, there are differences in physical dimensions, with the switch-side OSFP module equipped with heat fins for cooling.

OSFP 800G Optical Transceiver

The OSFP-800G SR8 Module is specifically crafted for utilization in 800Gb/s 2xNDR InfiniBand systems, offering throughput up to 30m over OM3 or 50m over OM4 multimode fibre (MMF) by utilising a wavelength of 850nm through dual MTP/MPO-12 connectors. Its dual-port configuration represents a significant advancement incorporating two internal transceiver engines, fully unlocking the switch’s potential.

This design allows the 32 physical interfaces to support up to 64 400G NDR interfaces. With its high-density and high-bandwidth design, this module enables data centres to seamlessly meet the escalating network demands of applications such as high-performance computing and cloud infrastructure.

Furthermore, the FS OSFP-800G SR8 Module provides outstanding performance and reliability, delivering robust optical interconnection choices for data centres. This module enables data centres to utilise the complete performance potential of the QM9700/9790 series switch, facilitating high-bandwidth and low-latency data transmission.

NDR Optical Connection Solution

Addressing the NDR optical connection challenge, the NDR switch ports utilize OSFP with eight channels per interface, each employing 100Gb/s SerDes. This allows for three mainstream connection speed options: 800G to 800G, 800G to 2X400G, and 800G to 4X200G. Additionally, each channel supports a downgrade from 100Gb/s to 50Gb/s, facilitating interoperability with previous-generation HDR devices.

The 400G NDR series cables and transceivers offer diverse product choices for configuring network switch and adapter systems, focusing on data centre lengths of up to 500 meters to accelerate HPC computing systems. The various connector types, including passive copper cables (DAC), active optical cables (AOC), and optical modules with jumpers, cater to different transmission distances and bandwidth requirements, ensuring low latency and an extremely low bit error rate for high-bandwidth HPC and accelerated computing applications.

Please see the article for deployment details Infiniband NDR OSFP Solution from the FS community.

Conclusion

In summary, InfiniBand (IB) technology offers unparalleled throughput, intelligent acceleration, and robust performance for HPC, and cloud infrastructures. FS OSFP-800G SR8 Module and NDR Optical Connection Solution further enhance data centre capabilities, enabling high-bandwidth, low-latency connectivity essential for modern computing applications.

Explore the full range of advanced networking solutions at FS.com and revolutionize your data centre network today!

Revolutionize High-Performance Computing with RDMA

To address the efficiency challenges of rapidly growing data storage and retrieval within data centers, the use of Ethernet-converged distributed storage networks is becoming increasingly popular. However, in storage networks where data flows are mainly characterized by large flows, packet loss caused by congestion will reduce data transmission efficiency and aggravate congestion. In order to solve this series of problems, RDMA technology emerged as the times require.

What is RDMA?

RDMA (Remote Direct Memory Access) is an advanced technology designed to reduce the latency associated with server-side data processing during network transfers. Allowing user-level applications to directly read from and write to remote memory without involving the CPU in multiple memory copies, RDMA bypasses the kernel and writes data directly to the network card. This achieves high throughput, ultra-low latency, and minimal CPU overhead. Presently, RDMA’s transport protocol over Ethernet is RoCEv2 (RDMA over Converged Ethernet v2). RoCEv2, a connectionless protocol based on UDP (User Datagram Protocol), is faster and consumes fewer CPU resources compared to the connection-oriented TCP (Transmission Control Protocol).

Building Lossless Network with RDMA

The RDMA networks achieve lossless transmission through the deployment of PFC and ECN functionalities. PFC technology controls RDMA-specific queue traffic on the link, applying backpressure to upstream devices during congestion at the switch’s ingress port. With ECN technology, end-to-end congestion control is achieved by marking packets during congestion at the egress port, prompting the sending end to reduce its transmission rate.

Optimal network performance is achieved by adjusting buffer thresholds for ECN and PFC, ensuring faster triggering of ECN than PFC. This allows the network to maintain full-speed data forwarding while actively reducing the server’s transmission rate to address congestion.

Accelerating Cluster Performance with GPU Direct-RDMA

The traditional TCP network heavily relies on CPU processing for packet management, often struggling to fully utilize available bandwidth. Therefore, in HPC environments, RDMA has become an indispensable network transfer technology, particularly during large-scale cluster training. It surpasses high-performance network transfers in user space data stored in CPU memory and contributes to GPU transfers within GPU clusters across multiple servers. And the Direct-RDMA technology is a key component in optimizing HPC performance, and NVIDIA enhances the performance of GPU clusters by supporting the function of GPU Direct-RDMA.

Streamlining RDMA Product Selection

In building high-performance RDMA networks, essential elements like RDMA adapters and powerful servers are necessary, but success also hinges on critical components such as high-speed optical modules, switches, and optical cables. As a leading provider of high-speed data transmission solutions, FS offers a diverse range of top-quality products, including high-performance switches, 200/400/800G optical modules, smart network cards, and more. These are precisely designed to meet the stringent requirements of low-latency and high-speed data transmission.

InfiniBand: Powering High-Performance Data Centers

Driven by the booming development of cloud computing and big data, InfiniBand has become a key technology and plays a vital role at the core of the data center. But what exactly is InfiniBand technology? What attributes contribute to its widespread adoption? The following guide will answer your questions.

What is InfiniBand?

InfiniBand is an open industrial standard that defines a high-speed network for interconnecting servers, storage devices, and more. It leverages point-to-point bidirectional links to enable seamless communication between processors located on different servers. It is compatible with various operating systems such as Linux, Windows, and ESXi.

InfiniBand Network Fabric

InfiniBand, built on a channel-based fabric, comprises key components like HCA (Host Channel Adapter), TCA (Target Channel Adapter), InfiniBand links (connecting channels, ranging from cables to fibers, and even on-board links), and InfiniBand switches and routers (integral for networking). Channel adapters, particularly HCA and TCA, are pivotal in forming InfiniBand channels, ensuring security and adherence to Quality of Service (QoS) levels for transmissions.

InfiniBand vs Ethernet

InfiniBand was developed to address data transmission bottlenecks in high-performance computing clusters. The primary differences with Ethernet lie in bandwidth, latency, network reliability, and more.

High Bandwidth and Low Latency

InfiniBand provides higher bandwidth and lower latency, meeting the performance demands of large-scale data transfer and real-time communication applications.

RDMA Support

InfiniBand supports Remote Direct Memory Access (RDMA), enabling direct data transfer between node memories. This reduces CPU overhead and improves transfer efficiency.

Scalability

InfiniBand Fabric allows for easy scalability by connecting a large number of nodes and supporting high-density server layouts. Additional InfiniBand switches and cables can expand network scale and bandwidth capacity.

High Reliability

InfiniBand Fabric incorporates redundant designs and fault isolation mechanisms, enhancing network availability and fault tolerance. Alternate paths maintain network connectivity in case of node or connection failures.

Conclusion

The InfiniBand network has undergone rapid iterations, progressing from SDR 10Gbps, DDR 20Gbps, QDR 40Gbps, FDR56Gbps, EDR 100Gbps, and now to HDR 200Gbps and NDR 400Gbps/800Gbps InfiniBand. For those considering the implementation of InfiniBand products in their high-performance data centers, further details are available from FS.com.