Thanks for your kind words and feedback. I love seeing all these links to your scribbles.
I'm an engineer at Replicate, which is a place to run ML models in the cloud. [0] We built Scribble Diffusion as an open-source app [1] to demonstrate how to use Replicate.
This is all built on ControlNet [2], a brilliant technique by Lvmin Zhang and Maneesh Agrawala [3] for conditioning diffusion models on additional sources, in this case human scribbles. It also allows for controlling Stable Diffusion using other inputs like pose estimation, edge detection, and depth maps.
ControlNet has only existed for three weeks, but people are already doing all kinds of cool stuff with it, like an app [4] that lets you pose a stick figure and generate DreamBooth images that match the pose. There are already a bunch of models [5] on Replicate that build on it.
I see a few bits of feedback here about issues with the Scribble Diffusion UI, and I'm tracking them on the GitHub repo. If you want to help out, please feel free to open an issue or pull request.
What a simple, but excellent concept! I kind of expected it to crash after being posted to HN, but surprisingly it is still going strong.
What does it cost to host something like that? What does it cost to generate each scribble image?
If you already have your own images, you can use the Replicate model directly: https://replicate.com/jagilley/controlnet-scribble -- you can upload your image using the Replicate web UI or do it programmatically using the model's HTTP API.
I run a custom Stable Diffusion bot for a small community that has generated many tens of thousands of images. The community wanted to know what it was costing me, so I dug into it a bit and have a fairly "literal" answer to that question (generally). With the RTX3060 I'm running it on, using a kill-a-wat, I very roughly calculated that generating a 512x640 image consumed ~ +170 watts for about 6 seconds (on top of the baseline power consumption of the PC when idle). This comes out to a little over 1,000 Watt-seconds or 0.000295 kWh per image. I'll leave it as an exercise for the reader to look up their current cost per kWh and work out what that comes out to. This is extremely, extremely rough, but helped me wrap my head around the amount of energy we were using to generate all these images and roughly how much of my power bill I had to thank my friends for, haha.
Of course running on rented/hosted GPU's, it's a simpler, but much more expensive story — basically however much you're paying for GPU instances to run Stable Diffusion divided by how many images you generate. :)
Sorry about the trouble. The Firefox incompatibility is the result of a bug in the underlying npm package we're using to render the drawing tool and canvas.
Maybe put a banner notifying users about this? That was a pretty bad experience while I struggled to make it work & thought the author/you had shipped something totally broken lol.
Thanks for your kind words and feedback. I love seeing all these links to your scribbles.
I'm an engineer at Replicate, which is a place to run ML models in the cloud. [0] We built Scribble Diffusion as an open-source app [1] to demonstrate how to use Replicate.
This is all built on ControlNet [2], a brilliant technique by Lvmin Zhang and Maneesh Agrawala [3] for conditioning diffusion models on additional sources, in this case human scribbles. It also allows for controlling Stable Diffusion using other inputs like pose estimation, edge detection, and depth maps.
ControlNet has only existed for three weeks, but people are already doing all kinds of cool stuff with it, like an app [4] that lets you pose a stick figure and generate DreamBooth images that match the pose. There are already a bunch of models [5] on Replicate that build on it.
I see a few bits of feedback here about issues with the Scribble Diffusion UI, and I'm tracking them on the GitHub repo. If you want to help out, please feel free to open an issue or pull request.
[0] https://replicate.com
[1] https://github.com/replicate/scribble-diffusion
[2] https://github.com/lllyasviel/ControlNet
[3] https://arxiv.org/abs/2302.05543
[4] https://twitter.com/dannypostmaa/status/1630442372206133248
[5] https://replicate.com/explore?query=controlnet