Part 1 of this mini-series about PCIe 7.0 fundamentals explained ordering rules and the distinction between relaxed ordering and ID-based ordering. Part 2 elaborates why PCIe 7.0 bandwidth alone isn’t enough and how UIO addresses legacy ordering limitations in this version of high-speed serial interface specification.
As noted earlier, PCIe 7.0 doubles raw link bandwidth compared to PCIe 6.0, increasing full‑duplex throughput from 256 GB/s to 512 GB/s on an x16 link by raising the signaling rate to 128 GT/s in flit mode. However, raw bandwidth does not directly translate into sustained throughput in AI factories.
Large‑scale training and inference systems generate traffic patterns such as GPU collective operations, sharded parameter broadcasts, gradient reductions, and streaming access to disaggregated accelerator and memory resources. These patterns include many independent data streams that cross the PCIe fabric concurrently and continuously.
The legacy ordering model inherited from earlier PCIe generations, including strict ordering, relaxed ordering, and ID‑based ordering, was designed around a producer-consumer abstraction in which ordering conveys semantic meaning to software. Relaxed ordering and ID-based ordering loosen this model selectively.
Relaxed ordering allows certain transactions to bypass global ordering constraints, while still participating in fabric‑enforced ordering rules. ID-based ordering further scopes ordering guarantees to a requester or execution context, preserving program order within that scope. In both cases, the PCIe fabric requires tracking and enforcement of ordering relationships to ensure correctness.
However, fabric‑enforced ordering introduces head‑of‑the-line blocking, increases buffering pressure, and restricts the ability of switches and endpoints to exploit parallel paths. This is particularly the case for multi‑path and non‑tree topologies common in modern AI systems. These effects reduce effective link utilization even though physical bandwidth is available, making it difficult for highly parallel AI workloads to keep PCIe 7.0 links continuously busy.
Addressing legacy ordering limitations with UIO
The unordered I/O (UIO) engineering change notice (ECN) was introduced in the PCIe 6.1 specification and included in PCIe 7.0 to address the specific limitation noted above. UIO introduces a wire-level semantic that shifts producer-consumer ordering responsibility from the fabric to the endpoints. The UIO ECN declares that ordering may be irrelevant for certain traffic classes.
For AI factory workloads, where operations such as reductions, parameter streaming, and telemetry are independent or statistically aggregated and never consumed in program order, enforcing any form of ordering (even per‑ID ordering) adds overhead. UIO removes fabric‑enforced ordering, enabling true multi‑path parallelism and reducing buffering requirements.
This allows PCIe fabrics to sustain higher utilization for concurrent AI traffic. Since UIO enables independent transactions from different request originators to bypass one another safely, AI systems can optimize PCIe 7.0’s increased bandwidth to support rapidly growing model sizes and highly parallel GPU workloads.
UIO is especially effective at reducing read latency because multiple UIO read completions for a single UIO read request may be returned in any address order. This same flexibility applies to UIO write completions, with the additional capability that write completions for the same transaction ID may be coalesced. Since every UIO request has a corresponding completion, the request originator maintains the ordering of its own transactions. This allows the PCIe fabric to forward traffic along multiple paths without violating semantic correctness.
With its low latency, UIO transforms PCIe fabrics into high-throughput, highly parallel forwarding planes capable of accommodating modern AI workloads. Instead of relying on the fabric to manage per-flow sequencing, UIO shifts ordering control back to the source device that initiates the requests.
How UIO reduces latency and unlocks concurrency in AI applications
UIO’s command set and wire semantics reduce latency and boost performance for AI training and inference in several ways.
First, UIO mandates completions for all UIO requests. This gives GPU endpoints precise end-to-end flow control and prevents posted-write “fire and forget” bursts from clogging switch queues. It also cuts head-of-the-line blocking and shortens tail latency, speeding up requests by allowing different types of requests to bypass each other without applying any ordering rules within the PCIe fabric.
One of the classic head-of-the-line blocking examples in the baseline strict ordering rule is that current read requests are not permitted to bypass previous write requests. UIO eliminates this rule, allowing read and write requests to be processed in parallel and completed in any order, as shown in Figure 1.
Figure 1 UIO read and write requests are processed in parallel at the application layer. Source: Cadence Design Systems
In addition, UIO read requests reduce latency and buffering by allowing a completer to return read completions out of order. This enables data to be delivered as it becomes available, rather than delaying responses to preserve requests or address ordering. This improves overall efficiency by giving the device greater freedom to exploit internal data availability and minimizing completion queueing and reassembly overhead.
For example, Figure 2 and Figure 3 show the completion patterns for a single 512 MB MRD request for non-UIO (in-order) and UIO (out-of-order) cases, respectively.
Figure 2 Non-UIO completion responses must be in order for the same MRD request. Source: Cadence Design Systems
For non-UIO, Figure 2 illustrates that completions must arrive in order, starting at byte 0 and ending at byte 511. However, with UIO, the completion order can be random, as shown in Figure 3. The first two completions carry the last two chunks of MRD requests (256-383B and 384-511B) because they are already available in the local cache. After that, the application reads the remaining completion data from its local memory and sends the remaining two completions (0B-127B and 128B-255B).
Figure 3 UIO read and out-of-order completion responses are processed for the same request. Source: Cadence Design Systems
Second, because ordering is enforced at the source rather than at every intermediate hop, packets from unrelated GPU streams can be load-balanced across multiple parallel paths through the PCIe fabric without being serialized by switch-level producer-consumer rules. This increases effective throughput at a given link rate and stabilizes latency underload. In multi-path topologies, system architects often use a non-transparent bridge (NTB) to connect separate systems, enabling cross-system traffic within a larger fabric.
Third, UIO is available only in flit mode. Operating in fixed-size flits with UIO-specific VC3VC4 (via the streamlined virtual channel capability) isolates UIO traffic from legacy flows, minimizes delays, and improves switch buffer utilization.
Figure 4 The above diagram displays a multi-path application example. Source: Cadence Design Systems
Figure 4 shows two interconnected PCIe systems (System 0 and System 1), each with GPUs and local PCIe switches connected via multiple NTB links. The upper NTB link can operate with either UIO-enabled or non-UIO-enabled traffic, while the three diagonal and lower links operate with UIO-enabled NTB.
As a result, independent transactions can flow concurrently across switches SW0–SW3. This topology shows how UIO-based NTB paths improve GPU communication by enabling multipath routing, reducing latency, and increasing bandwidth in large-scale AI systems.
PCIe ordering: A traffic light analogy
A helpful way to think about PCIe ordering is traffic control in a city. Strict ordering is like running the entire city with a single traffic light, and every vehicle must wait its turn and proceed in sequence. While there is no ambiguity, congestion can quickly build up. Relaxed ordering allows certain vehicles to pass through intersections in specific emergency situations, provided it is safe to do so.
While this removes unnecessary traffic jams, it still assumes the traffic system is centrally managed. ID-based ordering further refines this model by assigning each neighborhood its own traffic lights. While cars within the same neighborhood must obey local ordering rules, traffic from different neighborhoods can flow independently. This improves parallelism without sacrificing local correctness.
UIO bypasses traffic light rules entirely. It is akin to routing traffic onto a freeway, where there are no intersections or signals at all, and vehicles move continuously as capacity allows. On a freeway, the infrastructure does not impose sequencing. Instead, the responsibility for safe merging and interpreting arrival order shifts to drivers.
Similarly, with UIO, the PCIe fabric no longer enforces producer‑consumer ordering or completion sequencing. The requester explicitly declares that ordering carries no semantic meaning, allowing the fabric and devices to deliver and complete transactions opportunistically. This maximizes parallelism while minimizing buffering and latency.
These four ordering schemes are a progression rather than a set of alternatives. Strict ordering prioritizes safety and simplicity, while relaxed ordering removes unnecessary global barriers. ID-based ordering preserves correctness within a context while enabling scale, and UIO explicitly abandons ordering when it has no value. This layered model allows PCIe to remain compatible with legacy software while scaling efficiently for modern accelerators, multi‑queue devices, and highly parallel workloads.
Turning PCIe bandwidth into system-level performance
Fully utilizing PCIe 7.0’s 128 GT/s link in today’s AI factories requires more than higher signaling rates. In an environment where thousands of GPUs, accelerators, and memory expanders operate as a single, distributed system, an ordering model that can scale with extreme parallelism is necessary.
Legacy relaxed ordering and ID-based ordering schemes retain implicit ordering constraints that limit their efficiency at PCIe 7.0 speeds, making them increasingly inadequate for AI factories operating at hyperscale.
UIO relaxes fabric‑enforced ordering and enables AI workloads to more effectively utilize multi‑path PCIe fabrics. By shifting ordering decisions to endpoints that already manage synchronization at the runtime and application levels, UIO reduces ordering-related head-of-the-line blocking issues.
Not only does this improve latency under bursty collective traffic, it also supports higher sustained link utilization across dense training and inference clusters. The result: Under AI workloads, PCIe 7.0 can be used more efficiently as a data plane, rather than simply serving as a peak‑bandwidth interconnect.
Vanessa Do is a senior product marketing manager for PCIe IP at Cadence with over 20 years of experience in PCIe design, system validation, and customer engagement. Her background spans PCIe protocol development, FPGA-based customer support, and leading cross‑functional teams to debug complex PCIe issues at the system level.
Editor’s Note
This is Part 2 of the article series about PCIe 7.0 fundamentals. Part 1 explained PCIe’s ordering rules and the distinction between relaxed ordering and ID-based ordering.
The post PCIe 7.0: Addressing legacy ordering limitations with UIO appeared first on EDN.