GPUs can now use PCIe-attached memory or SSDs to boost VRAM capacity —Panmnesia's CXL IP claims double-digit nanosecond latency

 9th July, 2024:  Auckland, New Zealand.  Blog posted by @davidlim, #davidlimnz #davidlim

Modern GPUs for AI and HPC applications come with a finite amount of high-bandwidth memory (HBM) built into the device, limiting their performance in AI and other workloads. However, new tech will allow companies to expand GPU memory capacity by slotting in more memory with devices connected to the PCIe bus instead of being limited to the memory built into the GPU — it even allows using SSDs for memory capacity expansion, too. Panmnesia, a company backed by South Korea's renowned KAIST research institute, has developed a low-latency CXL IP that could be used to expand GPU memory using CXL memory expanders.

The memory requirements of more advanced datasets for AI training are growing rapidly, which means that AI companies either have to buy new GPUs, use less sophisticated datasets, or use CPU memory at the cost of performance. Although CXL is a protocol that formally works on top of a PCIe link, thus enabling users to connect more memory to a system via the PCIe bus, the technology has to be recognized by an ASIC and its subsystem, so just adding a CXL controller is not enough to make the technology work, especially on a GPU.

Panmnesia faced challenges integrating CXL for GPU memory expansion due to the absence of a CXL logic fabric and subsystems that support DRAM and/or SSD endpoints in GPUs. In addition, GPU cache and memory subsystems do not recognize any expansions except unified virtual memory (UVM), which tends to be slow. 

To address this, Panmnesia developed a CXL 3.1-compliant root complex (RC) equipped with multiple root ports (RPs) that support external memory over PCIe) and a host bridge with a host-managed device memory (HDM) decoder that connects to the GPU's system bus. The HDM decoder, responsible for managing the address ranges of system memory, essentially makes the GPU's memory subsystem 'think' that it is dealing with system memory, but in reality, the subsystem uses PCIe-connected DRAM or NAND. That means either DDR5 or SSDs can be used to expand the GPU memory pool. 

The solution (based on a custom GPU and marked as CXL-Opt) underwent extensive testing, showing a two-digit nanosecond round-trip latency (compared to 250ns in the case of prototypes developed by Samsung and Meta, which is marked as CXL-Proto in the graphs below), including the time needed for protocol conversion between standard memory operations and CXL flit transmissions, according to Panmnesia. It has been successfully integrated into both memory expanders and GPU/CPU prototypes at the hardware RTL, demonstrating its compatibility with various computing hardware.

As tested by Panmnesia, UVM performs the worst among all tested GPU kernels due to overhead from host runtime intervention during page faults and transferring data at the page level, which often exceeds the GPU's needs. In contrast, CXL allows direct access to expanded storage via load/store instructions, eliminating these issues. 

Consequently, CXL-Proto's execution time is 1.94 times shorter than UVM. Panmnesia's CXL-Opt further reduces execution time by 1.66 times, with an optimized controller achieving two-digit nanosecond latency and minimizing read/write latency. This pattern is also evident in another figure, which displays IPC values recorded during GPU kernel execution. It reveals that Panmnesia's CXL-Opt achieves performance speeds 3.22 times and 1.65 times faster than UVM and CXL-Proto, respectively.

In general, CXL support can do a lot for AI/HPC GPUs, but performance is a big question. Additionally, whether companies like AMD and Nvidia will add CXL support to their GPUs remains to be seen. If the approach of using PCIe-attached memory for GPUs does gather steam, only time will tell if the industry heavyweights will use IP blocks from companies like Panmnesia or simply develop their own tech.  

Read the rest of the article herethe article here.

Please call us at 0800 429 429 to get a shipping cost quote before you buy mobile phone accessories, latest iPhone cases, heat dissipating covers, glass screen protector, selfie sticks, DJI Osmo SE Gimbal for vloggers and Tiktok influencers.  We are based in Takapuna, North Shore, Auckland.`

We have been serving the locals for Takapuna, North Shore, Auckland since 2011 for phone screen repair,  mobile phone and computer repair since 2011, give us a go and we promise not waste your time!  After all, we have "been keeping you in touch since 2011".  

Let us sort you out on crack screen repair for Galaxy Tab, battery banks or power banks, fast wireless charger  tablets, laptop,  smartphones; used Apple iPad Repair.

We cater for Samsung Galaxy phone unlock, liquid damage, impact damage, data recovery and insurance report. 

Are you on a tight budget and looking for a cost-effective way to acquire a refurbished iPhone, iPad, or laptop computer? We can assist you in maximizing the value you get for your budget by providing you with high-quality refurbished hardware.

If you help, support or information on AI (Artificial Intellegence) subject topics; we will be happ to sort you out. Please text us if you need support or online product queries:
Dr Mobiles Limited
1, Huron Street, Takapuna, Auckland 0622.  Toll: 0800 429 429
Email - Website - Blog - Facebook - Twitter - Map -

Keywords:
GPU, overclocking, VRAM, performance, PCIe, bandwidth, bottleneck, gaming, graphics card, best, high-performance GPU, GPU Boost technology, best ssd for gaming, NVMe ssd, PCIe ssd, GPU bottleneck, upgrade gaming PC, best graphics card for gaming, GPU Boost, VRAM, GPU clock speed, overclocking GPU, best VRAM for gaming, high VRAM games, GPU benchmark, best graphics card, best graphics card for the money,