Why is PCIe 6.0 needed?
Since its official establishment in the early 21st century, the PCI Express (PCIe) interface has become the industry standard for chip-to-chip data transmission in data centers and computing applications. GPUs, FPGAs, and TPUs, among other specific acceleration cards, all use the PCIe interface to quickly transmit data with various computing nodes.
Different chips' business flow data is transmitted through the PCIe interface.
Since 2015, global data traffic has surged. As data centers transition to 100G Ethernet and beyond, PCIe interconnects in servers and network devices have become a bottleneck. The evolution trend of the PCIe standard is shown in the figure below. It can be seen that the transition from PCIe 2.0 to 3.0/4.0 took 5 and 7 years respectively. However, the transition from PCIe 4.0 to 5.0, and then to 6.0, took only 2 and 3 years respectively. Moreover, the data transmission rate of PCIe 6.0, which is 64GT/s, is eight times that of the PCIe 3.0 standard.
The performance acceleration curve of PCIe
PCIe 6.0 Meets the Bandwidth Requirements of 800G Data Centers
The PCIe 6.0 specification supports the transition of data centers to 800G Ethernet. The required bandwidth for 800 Gb/s is 100 GB/s. Under an x16 PCIe 6.0 link configuration, the unidirectional bandwidth can reach 128 GB/s. After aggregating multiple ports' links, it fully supports the bandwidth requirements of 1.6T Ethernet.
PCIe 6.0 Meets the Interconnection Needs of Multi-GPU Graphics Cards
NVLink focuses on data exchange between Nvidia's GPUs, meeting the bandwidth requirements for large-scale parallel processing and AI model training in high-performance computing tasks. PCIe, with its openness and extensive compatibility, quickly adapts to market changes and technological evolution.
PCIe 6.0 provides a unidirectional transmission rate of up to 64 GT/s, significantly increasing data throughput. In terms of energy efficiency, PCIe 6.0 adopts PAM-4 to optimize power consumption and signal integrity. Therefore, PCIe 6.0 has become the preferred choice for multi-GPU graphics card interconnection.
Memory Pool Sharing Compute Express Link (CXL)
CXL 3.0 is based on the PCIe 6.0 protocol. CXL memory pool sharing technology supports unified memory addressing across CPUs, GPUs, FPGAs, and other devices. It can aggregate scattered memory into a shared pool and dynamically allocate it to different tasks. Through CXL-connected memory expansion cards, the memory capacity of a single server can be increased from the TB level to the PB level, breaking through the memory limitations of large model training and reducing the purchase of expensive HBM.
NVMe
NVMe (Non-Volatile Memory Express) is a storage protocol designed for solid-state drives (SSDs), aiming to fully utilize the high-speed bandwidth of the PCIe bus to provide high-performance, low-latency storage solutions. PCIe 6.0 provides higher bandwidth and lower latency for NVMe storage devices, significantly improving storage performance and energy efficiency, and offering high-speed, low-latency storage solutions for data storage.
Technical Characteristics of PCIe 6.0
Signal Channels and PAM-4 Modulation
PCIe 5.0 uses the traditional NRZ (Non-Return-to-Zero) modulation method. When the data rate reaches 32GT/s, the Nyquist frequency also reaches 16GHz. The increased frequency brings increased losses, coupled with noise and crosstalk, making the PCIe 5.0 channel the most challenging NRZ channel to handle.
PCIe 6.0 uses PAM-4 modulation. PAM-4 uses four signal levels (0, 1, 2, 3), forming three "eyes," with each level corresponding to two bits of data. This multi-level modulation method significantly improves data transmission efficiency.
The eye diagram of PCIe 5.0 with a single eye compared to the three-eye diagram of PCIe 6.0
The Nyquist frequency for both PCIe 5.0 and PCIe 6.0 signals is 16GHz. However, PAM-4's four voltage levels use 2-bit encoding within one UI, while NRZ uses 1-bit encoding, thus doubling the data rate.
Since the overall voltage swing at the transmitter (TX) end has not increased, the available voltage for each eye of PAM-4 is only 1/3 of that of NRZ. Therefore, the signal is more sensitive to noise, and the requirements for signal integrity between TX and RX are higher. The transition from NRZ to PAM-4 signals requires a doubling of the signal jitter requirements and results in a 9.6dB degradation of the signal-to-noise ratio (SNR). Consequently, crosstalk and return loss in the packaging and PCB are higher than in PCIe 5.0.
Forward Error Correction (FEC)
The increased noise sensitivity means that the 1e-12 bit error rate (BER) used for PCIe is no longer feasible. Forward Error Correction (FEC) is required because the BER of PAM-4 signaling will be several orders of magnitude higher than 1e-12. The target for the first-bit error rate (FBER) is 1e-6. Forward Error Correction (FEC) compensates for this performance shortfall. The PCIe 6.0 specification limits the additional FEC delay to 2 nanoseconds or less. To keep the link retry probability at a low level, FEC and CRC (Cyclic Redundancy Check) need to work together.
FLIT Mode
Therefore, PCIe 6.0 adopts a 256-byte FLIT as the standard data transmission unit size. To support the fixed data packet size requirement of FEC, PCIe 6.0 includes encoding based on the flow control unit (FLIT). This modification simplifies the data management procedure by eliminating the need for data packet grouping at the physical layer, reducing latency and improving efficiency.
Low-Power Mode, L0p in FLIT Mode
PCIe 6.0 introduces a new low-power state called L0p, which allows PCIe 6.0 links to extend bandwidth utilization without interrupting data flow, thereby reducing power consumption. In previous generations, to change the link width, the entire link had to be retrained, causing traffic interruptions for several microseconds. However, L0p allows the link to shut down channels while always keeping at least one channel active, even as other channels are undergoing link training. It should be noted that this new low-power mode is only available in FLIT mode, while L0s supports non-FLIT mode.
This new low-power mode is symmetric, meaning that TX and RX scale together, and FLIT mode retimers that support this mode are also supported. The PHY power consumption of idle channels during L0p is expected to be similar to that when the channels are shut down.
PCB Material Selection for PCIe 6.0
Since PCIe 6.0 uses the PAM4 signal format, it cannot achieve a bit error rate requirement of 1E-06 under a 36 dB loss link. Therefore, the PCIe 6.0 specification had to reduce the total loss value of the entire physical channel to 32 dB.
Schematic diagram of PCIe signal transmission link
As shown in the typical diagram of the PCIe signal transmission link above, the PCIe card establishes a signal transmission channel with the chip on the motherboard through the CEM connector. Therefore, during signal transmission, losses such as PCB trace loss, packaging loss, connector loss, and via loss will occur. The PCIe 6.0 specification has the following loss requirements for each part:
The PCIe 6.0 specification has stringent requirements for transmission loss to ensure reliable high-speed data transmission.
As shown in the above figure of the PCIe 6.0 specification, the maximum loss caused by the PCB cannot exceed 13.5dB. Only when the PCB loss reaches 1.00dB/inch can the trace length reach the conventional 13.5-inch length.
At present, for the 16GT/s rate of PCIe 4.0, the motherboard already needs to use Megtron 4/Megtron 6 materials. Considering that the size of CPU chips is continuously increasing, in the era of PCIe 5.0 and PCIe 6.0, the motherboard PCB materials need to be further upgraded to Megtron 6/Megtron 7 levels. The copper foil type needs to be selected as HVLP (low roughness) or above to reduce the loss caused by the skin effect of the signal.
WeChat
