Part 2 of this series covered the user-space tooling stack—cxl, ndctl, daxctl, numactl, lspci, and setpci—and walked the boot path from power-on through DRAM training, DVSEC and HDM reporting, decode programming, CDAT delivery, ACPI table handoff, and OS driver binding. It framed each stage as a validation gate so you can tell whether a failure is rooted in link training, capacity reporting, firmware tables, or policy—not only in application behavior.
Part 3 turns that framework into hands-on practice. You will learn how CXL memory may surface as system RAM or Device DAX, when to use daxctl and boot parameters such as efi=nosoftreserve, how to confirm expander memory as a distinct NUMA node, how to decode key lspci fields for link health and CXL.mem enablement, and how to drive targeted traffic with numactl and standard stress tools. So, you can separate transport defects from NUMA misconfiguration before closing bring-up or sign-off on CXL Type 3 device validation.
Integration modes: System RAM and Device DAX
CXL Type 3 host-managed memory may integrate in more than one way. The platform and kernel can expose it as conventional system RAM, or as persistent-memory-class capacity that surfaces as Device DAX character devices (for example /dev/dax0.0). The daxctl utility and libdaxctl can reconfigure those DAX instances; for example, switching a region to system-RAM mode so the same physical capacity behaves like normal DRAM instead of a raw DAX mapping, subject to firmware and driver support.
To reconfigure a device that appears as /dev/daxX.Y to system RAM:
sudo daxctl reconfigure-device –mode=system-ram daxX.Y
Firmware can mark some memory ranges in the system map as EFI “soft reserved.” Think of that map as the machine’s inventory of RAM, soft-reserved means, “this range exists, but do not treat it as ordinary free RAM yet.”
That pattern is common when capacity comes from persistent memory (PMEM) or CXL Type 3 expanders, because the platform often wants the OS to decide later whether that capacity should behave like normal DRAM or be exposed as a Device DAX mapping (a special character device such as/dev/dax0.0). By default, Linux honors those soft reservations, which leaves the memory set aside instead of freely handing it to applications as regular RAM.
The boot parameter efi=nosoftreserve changes that policy. It tells the kernel: do not keep soft-reserved areas reserved so the OS can bring that capacity online in the mode operators want (typically system RAM) when the firmware’s soft-reserve marking does not match the deployment goal.
Use efi=nosoftreserve only when your platform vendor or bring-up guidance says it’s appropriate. Applied without that context, it can change how capacity appears (RAM vs DAX), affect NUMA topology, and complicate debugging when firmware and OS expectations disagree.
Verifying NUMA topology
Use numactl to confirm the expander enumerates as its own NUMA node:
numactl -H
Figure 1 Sample numactl -H output on a two-socket system with 2 CXL devices, where each CPU is a NUMA node with native DRAM. Node 0 and node 1 show socket-local DRAM and the CXL devices appear as node 2 and node 3 with 128 GB of memory (each) and no local CPUs. This memory-only NUMA node pattern is common for Type 3 expanders and is the baseline for placement-aware testing. Source: Author
Initial sanity checks with lspci
lspci shows whether the CXL endpoint is present and reports vendor/device ID, class codes, PCIe link width and speed, and related DVSEC register status. It’s a fast first check before chasing firmware decode, driver bind, or memory-onlining issues.
lspci | grep -i cxl
lspci -s <BDF> -vvvv
Decoding key lspci fields
Device link width and speed
LnkCap and LnkSta should match expectations (for example speed 32GT/s, width x16). Unexpected degradation signals an unstable link and issues at the electrical layer. This must be resolved before proceeding further.
Figure 2 lspci output showing the negotiated PCIe link speed and width for a CXL Type 3 endpoint. Source: Author
CXL capabilities and status
CXL DVSEC blocks use Vendor ID 0x1E98 and a DVSEC ID identifying the structure type. A healthy device advertises both CXL.io and CXL.mem under CXLCap/CXLCtl. If CXL.mem is disabled (CXLCtl Mem-), possible causes include DRAM enumeration failure or the host failing to assign address space. HDMCount 0 is a red flag. When memory enumeration fails, verify DIMM compatibility and mounting, and use vendor SDKs to probe internal controller registers via out-of-band access where available.
Figure 3 CXL DVSEC capability and control fields showing CXL.io and CXL.mem enablement, HDM decoder configuration, and active memory ranges. Source: Author
Address range and Active bit
Device firmware can be set up to include the entire HDM in one range or split it into two. The range fields reflect the size and which range is active. Range fields should translate to the full HDM size with Valid+ and Active+ set. Otherwise, it indicates that DDR negotiation or mapping failed. For initial bring-up, set the decoder and mailbox timeout values to their maximum supported settings to avoid premature failures during DDR negotiation.
Driver and kernel modules
Kernel driver in use should be cxl_pci. To verify kernel version, kernel 6.3 and above are recommended. During link stability testing, unlink drivers as required by your test procedure.
The cxl CLI tool for CXL memory expander bring-up
The cxl command-line utility is the user-space front-end to libcxl, shipped with the ndctl project on most Linux distributions. It walks the kernel CXL sysfs hierarchy, which includes buses, ports, endpoints, memdevs, decoders, and regions and prints structured as JSON output.
It complements lspci, which shows transport and DVSEC state at the PCI layer, and daxctl or numactl, which show how onlined memory capacity is exposed to applications. Reach for cxl after dmesg if the firmware, CXL driver, and user-visible memory policy do not agree.
What cxl exposes
At a high level, cxl list reports the objects the Linux CXL core registers under /sys/bus/cxl/devices/; root buses, switch and root ports, endpoints tied to PCI functions, memory devices (mem0, mem1, …), host-managed device memory decoders, and regions that may span one or more expanders. For single-LD Type 3 cards, the first sanity check is usually whether a memdev appears with a non-zero ram_size, a host BDF, and decoders or regions in a committed decode_state after platform firmware has programmed HDM and asserted mem_enable.
Essential commands during bring-up and sanity testing
Initial discovery
Run these early in bring-up to confirm the kernel bound cxl_pci and registered at least one memdev:
cxl list -M
cxl list -M -u
Look for memdev entries with ram_size, serial, host (PCI BDF), and numa_node when memory is onlined. An empty list or zero-sized memdev often means the device is present on the bus but not yet consumable—trace back to HDM validity, decode programming, or driver bind before chasing application issues.
Topology and decoder verification
After link-up and driver bind, verify the decode path from root port through endpoint decoders:
cxl list -vvu
cxl list -D -d endpoint -u
cxl list -P -p switch,endpoint -m memX -u
Use -vvu for buses, ports, decoders, regions, and target mapping in one view. Filter by memX or by PCI BDF (cxl list -M -m 0000:bb:dd.f) when multiple expanders or a switch is present. Decoder listings should show plausible Host Physical Address (HPA) windows and a committed state before you treat CXL-attached DRAM as usable system memory.
Region and exposure mode checks
When the platform surfaces expander memory through a CXL region and DAX subsystem, confirm how capacity is configured before running daxctl or numactl tests:
cxl list -R -RXu
cxl list -r regionN -RXu
The daxregion section reports chardev names (for example dax0.0), mode (devdax versus system-ram), and memblock onlining progress. This tells you whether the next step is daxctl reconfigure-device, memory hotplug onlining, or NUMA verification with numactl -H.
Enable, disable, and health
Type 3 expanders usually auto-enable by default. These commands matter after manual disable, hot-reset recovery, or scripted regression gates:
cxl enable-memdev mem0
cxl disable-memdev mem0
cxl list -m mem0 -H -u
enable-memdev revalidates HDM decoders and CXL.mem enablement along the port hierarchy. The -H health listing exposes maintenance, media, and error counters when the device supports mailbox health reporting—useful during long stress runs alongside dmesg and RAS logs.
Suggested bring-up command sequence
A practical first-pass sanity script on a booting system:
lspci | grep -i cxl
lspci -s <BDF> -vvvv
cxl list -M -u
cxl list -vvu
cxl list -R -RXu
cxl list -m mem0 -H -u
numactl -H
Together, these commands separate device not seen (lspci) with details of DVSEC that can be parsed for information of interest, device seen but not registered (no memdev in cxl list), decode not committed (decoders/regions), and memory online but misconfigured (numactl shows wrong node or missing capacity). That layering matches the validation mindset used throughout the “Bring-up and testing of systems with CXL Type 3 memory expanders” series.
Keeping kernel pages off CXL memory
CXL expander memory should be treated as a migratable capacity tier, not as interchangeable DRAM. Slab caches, page tables, and other non-migratable kernel structures must stay on local socket memory because they cannot be demoted or migrated and would suffer higher latency and reliability risk on CXL.
Online CXL capacity in ZONE_MOVABLE
The primary remedy is to defer CXL expander capacity only into ZONE_MOVABLE on a discrete, CPU-less NUMA node. After confirming the device with cxl list and daxctl list, reconfigure it as system RAM with the default movable policy as we saw previously.
sudo daxctl reconfigure-device –mode=system-ram daxX.Y
daxctl onlines new blocks as movable by default; avoid –no-movable. Only move blocks assigned to CXL memory to ZONE_MOVABLE. This must be done carefully; determine the NUMA node assigned to a CXL device, use the sysfs interface, /sys/devices/system/node/node2/memory*, to determine which blocks are assigned to the CXL memory and only apply the ZONE_MOVABLE attribute to those blocks by looping over the respective CXL nodes. It’s important to note that CXL memory range onlined in ZONE_MOVABLE is not eligible for 1-GB Gigantic Page allocation.
Page temperature and tiered memory placement on CXL systems
Hot, warm, and cold pages
In CXL-enabled tiered memory systems, pages are commonly classified by access frequency over a sliding time window rather than by a fixed label. Hot pages are touched often enough to affect performance while cold pages are allocated but remain idle for minutes or longer.
Production studies report that a large share of allocated memory, often well over half, can be cold in short windows. Warm pages sit between those extremes: accessed occasionally or likely to be re-accessed soon, so they tolerate slower tiers better than hot data, but still benefit from promotion if they heat up.
TPP further notes that page type matters. Anonymous heap and stack pages tend to run hotter than file-backed cache and tmpfs, and temperature can flip quickly as pages are allocated and freed, so static placement fails.
Why page placement is critical
CXL expander memory is reachable with byte-addressable semantics but at higher latency and often lower effective bandwidth than socket-local DRAM. If the OS treats all NUMA nodes as equivalent, hot working sets can land on CXL, new allocations compete with cold data for fast DRAM, and capacity-bound workloads that should tolerate tiering still lose double-digit performance.
TPP reports up to roughly 18% performance improvement over default NUMA balancing on CXL-tiered platforms, and production analyses show sharp latency-sensitivity once the hot footprint exceeds what local DRAM can hold.
For CXL Type 3 bring-up, correctness may pass with memory online, but performance sign-off requires verifying not only that CXL capacity exists, but that hot, warm, and cold pages are landing on the intended tiers under the policies that the user fleet actually runs.
Default Linux policies
Default policies start simple: allocate preferentially from local DRAM and spill to the CXL NUMA node only when fast-tier capacity is exhausted. The kernel’s NUMA balancing (numa_balancing) enables and configures automatic, page-fault–based balancing. Setting it to NUMA_BALANCING_MEMORY_TIERING (value 2) tells the kernel to treat the NUMA nodes as tiers and promote frequently accessed (“hot”) pages into the fast tier.
echo 2 > /proc/sys/kernel/numa_balancing
Memory-tier demotion (numa_demotion_enabled) under reclaim pressure moves cold pages to the slow tier instead of swapping them to disk.
echo 1 > /sys/kernel/mm/numa/demotion_enabled
Zone reclaim mode is a Linux kernel parameter that controls how aggressively the system reclaims memory from a local NUMA node when that specific node runs out of memory, rather than allocating memory from other remote nodes. It accepts one of the specified policies or a combination. Typical production knob will enable all reclaim policies.
echo 7 > /proc/sys/vm/zone_reclaim_mode
The above setting should not be used in all conditions. It’s highly recommended that the user try combinations of the above settings with benchmarks that closely resemble the workloads and use the combination that works best.
Tools for traffic generation and performance validation
While a detailed discussion of each benchmark and the traffic type deserve a separate discussion, some common benchmarks and tools are mentioned below that should help the reader get started.
Memory structural test
Latency
Intel Memory Latency Checker (MLC) for x86 platforms
Bandwidth
Google Stressful Application Test (SAT/GSAT/stressapptest)
Google Multichase – pointer chasing benchmark
When expander memory appears as its own NUMA node, numactl+membind forces traffic onto CXL-attached DRAM.
numactl –cpubind=0-15 –membind=2 <command to run memory traffic>
The example below runs on node 0 CPUs but allocates memory on node 2. CXL memory (see Part 1, Figure 2); swapping –membind between 0 and 2 gives a direct local-DRAM versus CXL comparison. On single-socket systems, the CXL device may enumerate as NUMA node 1.
Matching CXL link width to memory bandwidth
When sizing a CXL memory expander, usable bandwidth is not simply the PCIe/CXL rate on the connector. Traffic crosses two stages: the host-facing PCIe/CXL link and the DDR channels behind the Type 3 controller. End-to-end bandwidth is therefore the lesser of the two.
Effective Bandwidth = min(PCIe/CXL link bandwidth, DDR bandwidth behind CXL)
Recycled DDR4 is often slower, or with fewer active channels than native DDR5. Subsequently, the DRAM side can limit throughput before a full-width link saturates. In that case, a x16 link may not add much, and an x8 (or narrower) link can already match what DDR4 sustains. This frees up lanes, cutting expander cost and power, and leaving host I/O for GPUs, NICs, and other devices. Size the link to the slower of the two bandwidths, not to the maximum PCIe generation width.
A cross-layer validation mindset
CXL Type 3 memory expanders offer a practical way to grow capacity and effective memory bandwidth for data- and memory-intensive workloads where traditional DDR scaling is constrained by I/O, cost, and signal integrity. But successful deployment is not only a silicon or link problem.
Host-visible expander memory remains physically and administratively distinct from socket-local DRAM, so discovery, NUMA topology, performance, and RAS must be validated as a cross-layer problem spanning CPU, firmware, kernel parameters, device firmware, and user-space policy.
This three-part series outlined system context and platform prerequisites; the boot timeline from power through DVSEC, decode/mem_enable, CDAT/DOE, ACPI tables, and driver attach; and user-space tooling plus transport-level checks for CXL.mem enablement and HDM validation.
Future work includes switched and multi-device topologies, CXL 3.x pooling models, formal compliance automation, and standardized regression coupling protocol evidence with OS topology and workload QoS.
Acknowledgment
The author thanks Linux CXL kernel developers (the detailed notes and exchanges in lwn.net are extremely valuable), the open-source CXL community, the CXL Consortium, and platform engineers at CXL memory expander vendors and hyperscalers. Any errors remain the author’s own.
Ameet Sanghavi works in post-silicon validation for PCIe and CXL at Nvidia with a focus on interface bring-up and validation on shipping products. He has worked on PCIe since 2005 (from PCIe 1.1 onward) and on CXL since 2020 (from CXL 1.1 onward).
Editor’s Note
The views and content of the article are the author’s own and not affiliated to any of his current or previous employers.
Related Content
- Stripped Down CXL Scales Memory Wall
- CXL Efforts Focus on Memory Expansion
- CXL Adds Port Bundling to Quench AI Thirst
- CXL: The key to memory capacity in next-gen data centers
- From GPUs to Memory Pools: Why AI Needs Compute Express Link (CXL)
The post Test, debug, and validation of CXL memory expanders appeared first on EDN.