> If you switched all the CPU-only servers running AI and HPC worldwide to GPU-accelerated systems, you could save a whopping 11 trillion watt-hours of energy a year. That’s like saving the energy more than 1.5 million homes consume in a year.
That is a bold claim. It seems to me they simply assume all necessary code would trivially be converted to run on GPUs.
Which is extra important because, if someone is still running AI/ML code on the CPU rather than GPU in 2022, I suspect there must be very good reasons for doing so. So that makes it even less likely that this code can be ported in an economical way.
The reason for us is rental price. Customers don't notice the difference between 0.01s inference on GPU and 0.1s inference on CPU. Renting the necessary high-CPU servers from Hetzner is like $0.1 per hour. I'm not aware of any provider that would offer the equivalent of a 3090 for less than $2 per hour. So it's 20x cheaper to do inference on CPU.
Also, we tried to get into NVIDIA Inception and were rejected for the company being too small, so even if we wanted to do GPU deployment, we wouldn't be able to buy 3090s for testing. Scalpers / eBay are not an option in Germany because for company purchases, I need a proper tax invoice from another VAT-registered company.
I've had good luck getting A4000 cards from small vendors, at least as of last summer. ATLGaming was the shop we went with last. It's one person, and was able to source cards at MSRP. Very professional and excellent service. I don't know about EU and VAT, but they did handle US enterprise clients well.
At the time, the price of the A4000 was similar to scalper-prices for similarly-performing gaming cards, but:
- Had ECC RAM and was generally designed for ML/enterprise/data workloads
- Was more power-efficient
- Had 16GB RAM, which you won't get on anything short of an A3090
I'd taking a consumer card at MSRP over the A4000, but not at street price. For what I'm doing, an A3060 Ti would probably even have been fine. It has an MSRP of $330, but a street price of $500-$1000 depending on the month you look. The enterprise cards tend to sell at MSRP (which is just around a grand for the A4000).
Note that NVidia reuses part numbers. With "4000," there is also an older NVIDIA Quadro RTX 4000 (no A) selling for the same price on eBay / Amazon / etc. as the Ampere-based A4000.
If you can run your inference on CPU and it only takes 10x longer than on GPU, then it sounds to me like a 3090 will be vast overkill. Have you considered an inference-only GPU like the T4 or the A2? From what you're saying it seems to me like they're made for use cases exactly like yours.
We use the 3090 for testing and development, because it is "similar enough" to a V100. For deployment, we were evaluating A100 and V100. (You're not allowed to put a 3090 into a datacenter.) Since it's a 900 mio parameter model, T4 and A2 don't have enough RAM, I can barely train this thing on a 3090 ^_^
Once models are trained, it is not too difficult to compress them. Have you done any of that?
You can start with things like switching the weights and activations to fp16. A100 also supports BFloat16, which is even better and should work right out of the box.
It’s much much more complicated than that. They do typically use CPUs but there are many reasons for it. Starting with the fact that the path tracing algorithm is quite different when it runs on a CPU than on a GPU because thread divergence is a fundamental aspect of that formula. Second, the vast memory requirements are much easier to handle on a CPU than a GPU. There’s also the decades of man hours spent on the CPU code that is not so instantly ported to GPUs. Honestly this list goes on and on.
Well it's complicated when you consider it's taken many years of research to get where they are at currently, which is still far from where some have considered going.
I can very well imagine there is lots of non AI/ML code that smart people could make a lot faster on a GPU. I doubt there are anywhere near enough smart people to do that. I also imagine Nvidea might like to overstate the total potential.
The biggest hurdles are learning the dev tech stack and setting up the development environment.
The last time I tried out CUDA, it took me a couple of days to correctly setup the environment; just to spend an hour playing around parallelizing math operations.
The best way I could describe the first impression dev experience: It’s a huge hack, adding non-standard language extensions to C/C++
And so relying on their own compiler or linker (iirc), instead of the standard C++.
Couple this with NVidia not playing nice with OpenCL to force you into their playground, and I lost interest pretty quickly.
Besides that hack, getting your head around the way to optimize code for a GPU takes a while. Then the real challenge is taking code that is based on random access to gigabytes of RAM, and making that code work with any kind of performance on the GPU performance model.
There are cases where the random access to RAM are very deep in the algorithm. Consider e.g. sparse matrix multiplication.
“optimal” is a different goalpost than “a lot faster”.
Some easy-to-parallelize math and vector operations can yield impressive results without it being anywhere near optimal.
I do agree that optimal requires mastery of GPU programming; but I also think GPU programming can yield good results without said mastery, and be quite accessible
If anything it should be easier than writing raw multi-threading code, because GPUs aren’t (weren’t?) general purpose calculation units.
So more bare metal concerns, but less to reason about overall.
A lot of HPC code is by its very nature pretty parallelizable (given that supercomputers these days are often just a large cluster of typical datacenter CPUs/GPUs with very fast interconnects), so is theoretically portable to GPUs for some gain.
The hangups mostly seem to be in that a lot of scientific code hasn't been written with GPU style parallelization in mind, so updating the code takes time. That said, since a lot of newer supercomputers are including a lot of GPUs, there is a lot more incentive for developers to attempt to port at least the parts that can benefit from GPUs.
From my (admittedly limited) experience, the biggest hangup is convincing scientists that GPUs aren't just another tech fad and it's worth the effort to port. There also seems to be a bit of uncertainty over which platform to trust, OpenCL support has been shaky on most vendors and is fairly primitive in terms of features, CUDA is tied to one company and AMD doesn't have a great record of supporting HIP across their product stack, so choosing any one is a big decision (comparable to choosing to use C/C++ or Python, which are effectively guaranteed to be supported on every supercomputer for the next decade or two).
I work as a computational mathematician. Part of the problem on my end is that while the libraries have gotten better, they have not contained the operations that I need for good algorithms in my field. As an example, when GPUs first came onto the field, cublas computed a subset of the BLAS routines, which are pretty fundamental for putting together known, good algorithms. Eventually, that got better, but then the operations needed to fit onto a single GPU. Eventually, cublas-xt came onto the market and that allowed parallelization across GPUs, which was great. However, it still didn't have all of the needed matrix operations. At the moment, I'm not sure where it currently is.
Of course, matrix algebra was only one of the problems. BLAS supports the writing of LAPACK routines, which tend to deal more with dense factorizations and eigenvalue problems. I believe more factorizations have been implemented recently, but I'm not currently up to date. Nevertheless, that was absolutely a bottle neck for the longest time. Yes, iterative methods don't need factorizations, but a good fraction of the preconditioners do.
Then, of course, this speaks nothing of the sparse linear algebra problem. There's multiple ways to do it, but many of the good sparse factorization routines need dense factorization routines, so these needed to come onto the market first and that took time.
And, to be clear, I know that there are multiple, good teams working on this. It takes time. And, there's a huge number of operations. If you're bored, go look at the manual for Intel's MKL and see the number and variety of operations that it provides. Those operations are there because people like me need them to do our job. I'll also agree that the kinds of operations we use to do our job will evolve over time with hardware. However, matrix algebra, factorizations, and eigenvalue problems lie at the very core of applied mathematics and expecting the mathematics to rework the last several hundred years of practice to accommodate the lack of tooling from the GPU manufacturers isn't realistic either.
Anyway, if someone knows the current state of what's possible, I'd love to hear. Selfishly, what it really boils down to is what operations (algebra, factorization, or eigenvalue), how big (how much memory or on multiple GPUs), and dense or sparse.
Honestly, a combination of time, money, and expertise. Getting a good, low level factorization that works well in parallel is hard. I can write them. Others write them better. However, outside of that, I don't have any financial support to write them and there's no way that I can convince my clients to fund that effort.
Personally, I don't really care where the libraries come from, but I will contend that the hardware manufacturers are generally the best place for this work to occur. For many years, each of the chip makers published their own high performance BLAS and LAPACK routines, which worked really well. Intel had MKL, AMD had AMCL, Sun had sunperf, IBM had ESSL. NVIDIA does the same, and I'm hugely grateful for that, but it's not complete.
Really, though, the comment is more to answer why more mathematicians don't use GPUs. My contention is not that I or my colleagues view it as a fad, but more that there's a lack of routines that we depend on.
I currently work for myself, but I'll also mention that the manufacturers did go to management at places I've worked and management did put pressure on staff to just rewrite everything in CUDA. As staff does, they said, ok, fine, but it's going to cost you labor. Management didn't have the money, or didn't want to spend it, and some mild office conflict occurred.
Anyway, mostly that's to say that I will gladly spend thousands on GPUs and recommend it to my clients as soon as I don't have to write all of the low level routines myself.
the biggest hangup is convincing scientists that GPUs aren't just another tech fad
CUDA is 15 years old. All top supercomputers use GPUs and have been for a while. Mature software tools like cupy exist to make GPU programming easier. I don’t think anyone thinks today “GPUs are a fad”. The problem is a lot of code is simply hard to parallelize, and would need to be rewritten from scratch, likely in another language. For large old codebases this is a massive effort.
CUDA is 15 years old, but in terms of significant presence in supercomputers, it's just a generation or two in. For example, NERSC only made its first system that's "GPU-first" available to all users two weeks ago, with the previous system (which was #5 in the world when commissioned in 2016) only having gotten some GPU nodes in 2020 to help smooth the transition.
This was what led to the hesitance from the people I worked with. They were unsure if in another generation or two they'd have to look at another large effort to support another new programming paradigm. There was also hesitance in relying on CUDA since it locks them into a vendor.
From an engineer's point of view, yeah, GPUs obviously aren't a fad, but a lot of this software is written by scientists, where the computer is simply a means of getting their result and not something they keep up with as they would with their own field and yet the software needs to remain relatively stable to be reliable for research. Thus anyone asking for a large modification of the code is going to be viewed with extreme skepticism.
For reference, I had to spend around 2 months of weekly meetings and presentations showing test results and discussing the risk/reward tradeoffs in detail to convince a group that a limited port of the code would be worth going ahead with and that CUDA was effectively the only reliable option for now. They've only gotten serious about a more in-depth port from seeing the large speedups without breaking compatibility which we managed to get after a few iterations.
Most of the code ran in large datacenters isn't classic HPC. It's cloud workloads serving live traffic or running database backends or some analytical pipelines/queries running over the stored data.
Deep ML is about the only thing that's both large and "classic parallelizable HPC".
I was talking about scientific code, as that's what I have experience with. There a ton of work is large and parallelizable. Molecular dynamics, physical optics, magnetostatics, fluid simulations etc all benefit a lot from large scale parallelization/GPU support.
I really like the Noctua cards they're collaborating on, it's not really necessary to go water-cooled once you've got a 4ish slot card/heatsink with quiet 120mm fans!
Sure, but especially in a data center that’s a huge waste of density for compute. Only a single card in the space for four. Even less, really, because that card ideally has a gap for the non-blower cooler
Water cooling is just a way to transfer the heat to a larger heatsink somewhere else. It doesn't magically absorb the heat but just adds complexity and overhead. If there's a way to make a direct heatsink work, then that's the more convenient option every time.
For datacenters it would make more sense to make it all water cooled though, since you can likely have one central loop connected to an HVAC unit or something.
Water cooling is more than that. Heatpipes have much lower transfer rates than pipes with water flowing. Pound for pound you dissipate a lot more heat with water being pumped through the rad.
Ignoring that, putting the radiator anywhere other than the precious motherboard real estate has a lot of value. It's such a waste to hang a 10 pound piece of copper over all of those fast, unused PCIe lanes.
I think there are some cases with a daughterboard containing the GPU slot in a separate “heat compartment” from the rest of the case. The PCIe is extended over to that board so everything else should still be standard. It probably adds some small latency, though.
It also removes a big chunk of metal and fans taking up space on your motherboard, covering other slots. It adds complexity, not really sure what the "overhead" here is that you're referring to.
- Ever since 2063, we simply drop a giant ice cube into the ocean every now and then. Of course, since the greenhouse gases are still building up it takes more and more ice each time. Thus, solving the problem once and for all.
I would say that "ever increasing power and cooling requirements somehow equate" to bad hardware development. Why optimise when you can cool it and the client pays the electricity bill.
Reminds me of the old IBM mainframes (until the 1990s) that were liquid cooled. They had an ingenious approach called TCM (Thermal Conduction Module) with the silicon chips directly attached to a cooled substrate.
This makes very little sense. How exactly does water cooling contribute to power efficiency? Once you get that heat off the chip it has to go somewhere and what is the complexity of running water cooling hoses all over the datacenter. How efficient is it if you get a leak even 1% of the time?
Transporting X Joules of heat around by pumping water is definitely a lot more efficient than transporting the same heat energy around by blowing hot air. So pumping losses are much smaller.
Water cooling is hugely problematic because the cooling system will eventually spring a leak, and having water spraying/dripping all over your expensive server hardware is not anyone's idea of fun. That's why data centers use air cooling and a separate HVAC.
Sure but notably car water cooling, as robust as it is, does eventually spring leaks.
I think it's a question of whether the efficiency tradeoff is worth the maintenance tradeoff (which we probably won't know since there's not a ton of these kinds of solutions in use). With air cooling it's easier since the entire medium around the servers is the fluid. With liquid cooling, short of doing a bath setup (which doesn't appear to be what's proposed here) I'm not convinced it's going to work well; Just imagine 2-4 pipes running to every 2U server on a rack.
IMO, we shouldn't be against trying an idea again if it seems like on paper it could make a lot of sense. There's a lot to like about the potential of this solution; improved efficiency, density, etc. Heck, you might even make a fun side project of trying to reclaim that waste heat and turn it into electrical generation.
I just think the plumbing alone is likely to make it cost ineffective compared to existing air systems and then add quite a bit of potential maintenance headache.
> notably car water cooling, as robust as it is, does eventually spring leaks.
Internal combustion engine cooling is not the same application as industrial computing. Differences include
- ICE Cooling relies on pressure (boiling if released to atmospheric pressure
- Maintenance (Schedules and Leak Detection); Most modern cars go a lifetime, if not 100k miles with no cooling system maintenance; if preventative maintenance is performed (replacement of radiator, pressure tank and/or rubber hosing regularly)and the system is monitored for level, repairs can be pre-empted. (very rarely does the leak happen in the block))
- Vibration and Environment (including cost or effort of maintenance)
“Water” cooling in datacenter environments is almost always done with specialized liquids which are definitely not water, so a leak isn’t really an issue. Fluorinert is one of the more commonly used options.
It's the extra things in the water that make it both awesome and terrible. The really funny part - since we're approaching it all from an environmental perspective - is that it has huge global warming potential, so you want to be really careful to keep the stuff contained.
They also take forever to break down and are not fantastic for human health.
Like Sowell said "there are no solutions, there are only trade-offs".
I want to be pedantic and point out that water as a substance is not the same thing as water as an ingredient in another substance. It is not water, it is something else which contains water.
“Water” cooling is most often literally distilled water or possibly distilled water with a fungicide. This is not what is used in datacenters. They use Fluorinated fluids, PAOs, or PFOAs which all have unique properties different from treated waters.
> I want to be pedantic and point out that water as a substance is not the same thing as water as an ingredient in another substance. It is not water, it is something else which contains water.
It's possible to take that pedantry a step too far. Is seawater not water because it contains a lot of salt, other minerals, fish poo, and whatnot?
> “Water” cooling is most often literally distilled water or possibly distilled water with a fungicide. This is not what is used in datacenters.
I've worked with datacenter equipment cooled by (AFAIK non-distilled) water-glycol, so, well, it does exist and is used.
I mean a leak isn't an issue as far as equipment damage, which is the general concern with "water cooling" leaking. These specialized fluids are non-conductive.
In most of the facilities I've been in that used these special fluids they had specially designed floors that acted as drains and capture in case any leaks occurred or for incidental spillage as part of replacing components in immersion cooled systems. All of the technicians were properly trained for safe handling, and of course these fluids are not something you want to be exposed to without the proper PPE. How they were plumbed and contained, a leak would not result in direct contact of the fluid to a person, nor would it result in it escaping to the environment outside the containment area.
Yes, I'm personally not very happy about another large scale usage of flourocarbons. Seems the history of these compounds is basically "look, we found this new compound with some very exciting and useful properties" followed a few decades later by "Oh shit". Rinse and repeat.
That being said, there are alternatives. Like polyalphaolefin (PAO), which is apparently used for electronics cooling in aerospace applications and for some immersion cooling datacenter products.
Water distribution is absolutely ubiquitous on the planet with leak rates that are quite low. When there are leaks it is because of an incident of damage, decades without maintenance, or poor implementation. Adding heat changes nothing. Many residential and commercial use water for heat exchange. Humans are good at this.
> Many residential and commercial use water for heat exchange.
Stupendous amounts of water are lost to leaks all over the place. The difference is that it does most of the time no damage, as it just goes to the ground.
It still ends up in residential buildings every now and then, and it usually isn’t pretty.
That depends on what your water cooling system does. I can totally believe that circulating water through cooling pipes from the hvac to the GPUs is more energy efficient than moving huge amounts of air, and gives you better heat differentials.
But if the datacenter doesn't have that and you build a more consumer-typical setup with a small pump and a radiator you're no more efficient than air cooling, you just moved the fans by a couple centimeters from the GPU to the radiator.
Exactly this. Makes no sense for a single desktop computer, but when you have a huge room filled with servers, it's a lot more efficient to pump cooling water around than blowing a huge amount of air around. Also air cooling a rack full of equipment is getting more and more difficult as power densities increase, which is one of the main motivations of going to liquid cooling.
Article says closed loop cooling so I imagine it’s just a big scale version of pc cooling but the radiators vent outside. But I wonder if you could pump sea water in to a datacenter and cool using an open loop. Perhaps salt and dirt would become too much of a problem.
Very geography dependent, but e.g. in the Nordics waste heat is a resource. This is why we burn our combustible trash instead of putting it in landfills.
Unfortunately, for most of the year the same wouldn’t we viable in California, and I guess the investment costs make the whole thing unprofitable.
Here’s an article [1] from last year about the transition of Google’s NL datacenter to use surface water to cool their servers.
It mentions the DC uses both open and closed loop cooling, and you can see a picture of the heat exchanger, so it seems you are correct (at least for this DC).
I really hope we end up with a low-grade widespread heat net (say 30 degrees celsius) using this kind of waste-heat, combined with heat-pumps to actually bring hot enough temperatures.
That’s how it is done in, for example, boats as well as other places where you want to extract heat to a lake or to a sea.
You have your typical nice and clean coolant circuit that circulates through the engine, and then through a heat exchanger, you give that heat out to the sea water. This way the heat exchanger becomes somewhat sacrificial and you replace only that if it gets destroyed. And you don’t have to harden every single surface for sea water and all the creatures that come with it.
Boat engines with direct seawater cooling are definitely a thing. Usually only on the cheapest and nastiest engines, better ones definitely have a separate heat exchanger.
Data centers need this for space constraints, but if you're just the average Joe gamer I encourage you to think twice before considering liquid cooling (especially a custom loop).
Why do I say this? Because...my goodness, it gets expensive, and you're really not going to get much more performance or even noise reduction out of it (and like...don't you already wear a headset while you game?)
I've never understood the market for CPU AIOs especially. They're just so expensive compared to fitting a giant heatsink and fan on there, which will be quite quiet if it's large enough.
I've built systems with air cooling, with water cooling, and I've also bought an air-cooled prebuilt. My goal, in each case, was to minimize noise.
What I've taken away from that is:
- It's perfectly possible to build a quiet, air-cooled, relatively cheap system.
- But you probably don't have the skill for that. It's a complex task, and requires either good intuition or an entire engineering degree.
- Building a quiet water-cooled desktop is relatively easy, because you can oversize the cooling massively.
So would I recommend water-cooling? My goodness no. It's a lot of work, it's expensive, and in the end the pre-built was marginally quieter. But if the comparison is against a self-built air-cooled one, then water cooling can indeed come out on top.
I’ll add on top of this: some of the air cooled solutions are stupidly cheap. The Cooler Master Hyper 212 Evo has been on the market forever and it’s dirt cheap, good quiet performance, upgradable to any fan you damn please.
It’s really as simple as a big-ass hunk of metal, it’s hard to go wrong.
As far as graphics card noise you can usually buy something off the shelf and undervolt it in MSI Afterburner…or just buy a very large three fan mid-range graphics card where the cooling package is designed for something that draws way more power.
Water cooling rigs aren't literally inaudible, either. They still have a pump and fans for the radiator.
I'm talking about noise performance that is "more than good enough for most people."
The price to dB reduction ratio of most water cooling setups, whether it's a cheap AIO or full blown custom loop, just doesn't make any sense when compared to air. These are seriously diminishing returns on investment.
The reason people install custom loops is because they look cool, that's 80% of it. It's a hot rodding culture, and that can be fun and appreciated for what it is. But if all you want is a system that's quiet enough for the game audio on a relatively low volume to drown out the sound of the PC, air is all you need.
I agree in general, liquid cooling loops are a lot of work for usually very little benefit for average users. Even for people who need to put a couple of GPUs together, it turns out that mining rig frames with 16x extenders are pretty effective, since unlike a datacenter, you don't really need to optimize for density.
But, I think CPU AIOs are the one exception since giant heatsinks often interfere with other parts (overhanging ram slots or the first M.2 slot) and add a lot of weight to be hanging off the motherboard.
I still really don't understand CPU AIOs from a value perspective.
- I've gotten more than acceptable noise performance out of very small mini-ITX cases by using ITX-sized heatsinks with high quality fans/designs/materials (e.g., Noctua). You don't even really need a giant heatsink to get good noise performance in many instances, and in terms of price it’s very easy to beat an AIO to get a higher quality air setup.
- I haven't seen many heatsink/board designs made in recent years that don't take RAM and slots into consideration, and I don't really care if I have to take a heatsink off to remove an M.2 card – by the time I want to do that, it's probably about time to re-apply thermal paste anyway.
- In what world does "weight hanging off the motherboard" matter? I've never seen or heard of any motherboard issue caused by this. The Cooler Master Hyper 212 is massive, and it has been a budget staple for over a decade and nobody's had this problem with it to my knowledge.
- Most gamers don't need anything above mid-range for their CPU (e.g., AMD 5600X at 65 watts), but you'll see YouTubers building rigs with crazy high end stuff that doesn't significantly impact gaming performance and requires better cooling. Unless you're trying to go from 300 to 350FPS in an eSports game at 1080p, there's no reason to go above mid-range, your money is better spent on the GPU (i.e., most people game in a way that's GPU-bound).
- Case volume and well-engineered case thermal design is "free." If all you care about is reasonable cost, low noise, and performance per dollar, using an AIO setup is just about the worst way to spend your money when you can just get a larger case with good airflow design by perusing reviews.
- If you're doing any kind of socializing while gaming (e.g., Discord), you should be on headphones anyway because otherwise you'll be feeding back into your microphone. As long as your computer doesn't sound like a PS4 Pro I'd say it's fine.
Minor, that should be 66% fewer in the chart, not "less".
So question, how do these compare in terms of cost against the non liquid variants? I've been buying the Evga Hybrid variants of Nvidia GPUs, and they are not cheap at all, more on the premium side. That's the only cost reference I've got.
Nvidia GPUs have a couple of microcontrollers for various management and ancillary tasks, you could probably take over one of those and take keyboard input from there? (search [1] for Falcon, those are what Nvidia is now moving to RISC-V for). Not sure what use that would be though
That is a bold claim. It seems to me they simply assume all necessary code would trivially be converted to run on GPUs.