There is evidence that the performance of the models scale linearly with size, so moores law scaling is likely to get us some “free” improvement even if no one ever invents a better ML technique.
Wow, somehow I missed this one with all the whirlwind of image models recently. It’s very illuminating how the capability keeps scaling in their examples.
If you read through the research papers, a big section always deals with benchmarks. That’s because the output needs to be quantified in some way in order to improve the models. Several benchmarks have been proposed for text to image models.
That makes sense, but that would imply that there's a limit, right? Once the image is pixel-perfect and outputs the optimal image, what does increasing the model size do? Who and how can decide: "yes, this is more Picasso looking than that one", or "this one indeed looks more energetic", or "this image does make me sadder than this one". How do you benchmark this?
Yes you are on the right track. Once you get really close to a perfect score on your benchmark you can no longer improve so you need to develop a better benchmark with more headroom. And you have the right idea of how you go about benchmarking subjective quality. A bunch of humans produce output-scoring pairings and the model is judged against that. To train an AI you need a very measurable goal and in this case the measure is “humans like it.”
If you are noticing that this seems to fundamentally limit model performance on certain tasks to aggregate human capability, you are noticing correctly.
To give you some idea of what these benchmarks look like, here’s the prompt list from DrawBench which Google created as part of training their Imagen model.
Also, after a point the differences will be more given by the specific individual that views the image, not by what the AI can generate, so the AI would have to optimize it's output per individual and would need to have a deep understanding of them.
I realize we are not done with it yet. There are new process node launches planned for the next few years and each processor generation continues to improve density, power consumption and price per transistor.
I’ll hold off declaring it dead till it is well and truly dead. And even then we could expect cost improvements as the great wheel of investment into the next node would no longer need to turn and the last node would become a final standard.
As to physical limits, there are plenty of weird quantum particle effects to explore so that seems overstated. We are still just flipping on and off electromagnetic charge. Haven’t even gotten to the quarks yet!
You can draw a straight line right through this log scale plot that goes to 2020. Not sure what definition of Moores Law you are using, but it doesn’t seem to match the one on Wikipedia.
