> What are they doing differently and why can't we do that?
You're (incorrectly) assuming they're doing Ethernet/IP in that test setup. They aren't (this is implied by the results section discussing various FEC, which is below even Ethernet framing), so it's just a petabit of raw layer 1 bandwidth.
It's also important to note that many optical links don't use ethernet as a protocol either (SDH/SONET are the common ones), although this is changing more and more.
Built, no, definitely not voluntarily¹, Ethernet is the only non-legacy thing surviving for new installations for anything more than short range (few kilometer) runs. InfiniBand, CPRI and SDI are dying too and getting replaced with various over-Ethernet things, even for low-layer line aggregation there's FlexE these days.
¹ some installations are the exception confirming the rules; but as a telco sinking more money into keeping an old SONET installation alive is totally the choice of last resort. You'll have problems getting hardware too.
Disclaimer: I don't know what military installations do.
It is a nice interconnect. It is a shame that the industry does not revisit the idea of connecting all components of a computer over infiniband. Nvlink fusion is a spiritual successor in that regard.
Yes, IB hasn't died… yet. But the writing's probably on the wall with Ultra Ethernet; the corporate development of Mellanox (now nVidia) is not a great sign either.
(Also you don't use IB for 1800km WAN links.)
FWIW I actually run IB and think it's a nice interconnect too :)
There is one difference, however. As far as I know, they did not make Ultra Ethernet because the existing Infiniband standard did not cover everyone’s use cases. They made Ultra Ethernet because Intel killed QLogic’s Infiniband business in an attempt to replace an open standard (Infiniband) with a proprietary one (Omni-Path) that they made out of QLogic’s infiniband business’ corpse in an attempt to have a monopoly (which failed spectacularly in true Intel fashion), NVIDIA purchased Mellanox becoming the dominant Infiniband vendor, this move turned out to be advantageous for AI training and everyone else wanted an industry association in which NVIDIA would not be the dominant vendor. The main reason people outside of HPC care about Infiniband level performance is AI training and Nvidia’s dominance is not going anywhere. Now that NVIDIA has joined the UEC, it is unclear to me what the point was. NVIDIA will be dominant in Ultra Ethernet as soon as it ships Ultra Ethernet hardware. Are Nvidia’s competitors going to make a third industry association once they realize that Nvidia is effectively in control of the UEC because nobody can sell anything if it is not compatible with Nvidia’s hardware?
Had they just used Infiniband, which they had spent billions of dollars developing just a few decades prior, they would have been further along in developing competing solutions. Reinventing the wheel with Ultra Ethernet was a huge gift to Nvidia. If they somehow succeed in switching people to Ultra Ethernet, what guarantee do we have that they will not repeat this cycle in a few decades after they have left the technology to become a single vendor solution due to myopic decisions and they decide to reinvent the wheel again? We already have been through this with Infiniband and I do not see much reason that anyone should follow them down this road again.
Mellanox already kind of controlled Infiniband so Intel/QLogic could either chase Mellanox or fork and IMO forking into Omni-Path wasn't necessarily a bad choice. Omni-Path got zero adoption but that could have been for any number of reasons.
So many customers have a mental block against anything that isn't called Ethernet so the industry cannot "just use Infiniband"; they really had no choice but to create UEC. I would predict that Broadcom ends up de facto controlling UEC since their market share is much higher than Nvidia and people now want anything but Nvidia to create competition.
I doubt anyone had a mental block against anything that was not Ethernet given that Infiniband had been taking market share from Ethernet since Nvidia acquired Mellanox in HPC if we go by the top 500 list:
Nvidia announced the acquisition of Mellanox on March 11, 2019. In June 2019, Ethernet had 54.4% market share to infiniband’s 24.6%. NVIDIA completed their acquisition of Mellanox on April 27, 2020. In June 2025, Ethernet had 33.4% marketshare to infiniband’s 54.2%. Note that I combined the Ethernet and Gigabit Ethernet fields when getting the data. It would be interesting if I could find broader market data comparing Ethernet and Infiniband, but that is hard to find since people stopped comparing them long ago when Infiniband famously retreated into HPC and Ethernet took over everywhere else. The only thing I could find was an advertisement of a paywalled analyst report, which says:
> When we first initiated our coverage of AI Back-end Networks in late 2023, the market was dominated by InfiniBand, holding over 80 percent share
They express optimism that Ethernet will overtake Infiniband, but the full rationale for that is not publicly accessible. It also does not say outside of the paywall to what extent that market share taken by Ethernet would be maintained by Nvidia. Another article I found gives a hint. It shows Nvidia is gaining market share rapidly in Ethernet switches:
The numbers on Ethernet switches are not showing “people now want anything but Nvidia to create competition”. Instead, they show that there is a competitive market, Nvidia is the current driver of competition in the market, and there is amazing demand for Ethernet switches from Nvidia over switches from the incumbents, who were behind UEC. Broadcom appears to be included under others there.
The public data that I can find largely does not support what you said.
I feel like the application of “the market wants alternatives to Nvidia” to networking could turn out similarly to how “people want a physical keyboard” turned out in smartphones.
In the networking market, Nvidia is the underdog. Anyone who wants alternatives there is going to Nvidia, not away from Nvidia. We will see how the market develops, but I do not think widely stated remarks about Nvidia in AI training apply to Nvidia in networking.
You're (incorrectly) assuming they're doing Ethernet/IP in that test setup. They aren't (this is implied by the results section discussing various FEC, which is below even Ethernet framing), so it's just a petabit of raw layer 1 bandwidth.