If you like this, I suggest checking out PixiVisor. It's an experimental tool by NightRadio (Alexander Zolotov), who also made the amazing modular synth + tracker "SunVox".
PixiVisor takes a short video loop (~animated-GIF or sequence of images), modulates the frames (progressive-scan) into audio. A 2nd instance of the app (usually on another device) demodulates the audio back into similar video frames automagically.
"Similar"? Well... after picking up the multi-path room echoes and the "colorful" transfer functions of the DAC, amp, speaker, mic, ADC, etc, there is substantial distortion. This is intentional, as you can vary the signal considerably and the receiver app still reliably demodulates it back into video.
This hit the "wait, isn't that impossible?!" stage for me in this demo video:
Using two mono channels of an audio mixer to mix video live is already insane. Then the effect insert is turned up, and various types of reverb get added to the video frames in a way that perfectly matches what you hear.
Also, this video, of live room echoes and hand waves that clearly result in changing reflections/blocking across many frequencies. Multi-path is also very visible.
> by NightRadio (Alexander Zolotov), who also made the amazing modular synth + tracker "SunVox".
Heh, first thing I thought when reading the article was SunVox. PixiVisor is closer in effect to what the article describes, but I was reminded of the "SunVox: Sampler eats JPEG images" video:
Side note: You actually need permission to use pictures of the Eiffel Tower if it is illuminated. In other words "it is no longer legal to publish contemporary photographs of the tower at night without permission in France and some other countries."
https://en.wikipedia.org/wiki/Eiffel_Tower#Image_copyright_c...
This happens quite a bit. I took a nice photo of the Asahi building[0] in Tokyo and tried to add it to Wikipedia, only to be told that the gold embellishment on top was considered sculpture under Japanese law and therefore subject to copyright and not permissible on Wikimedia.
In some countries even just regular buildings are considered copyrighted if they have at all nontrivial decoration or design, and photos of them require permission of the person/company that holds the design copyright on the facade. The keyword to find each country's laws is "freedom of panorama", an exception to copyright law that allows photographing public places, even if the photograph includes something copyrighted. Some countries give a blanket exception; others give an exception only for buildings (but not sculptures or other artwork that appears in public, which can complicate photographing a building if it also has sculptures); and those differ on whether it includes only building exteriors or also photographs of the interiors of public lobbies; still others have freedom of panorama but only for non-commercial use; etc.
Thank you for enlightening me on this issue. I had no idea this kind of thing was even desirable for sovereign territories, let alone actually in place legally. The lengths some people will go...
The effects are pretty cool, but I can't help but wonder how much better it would be if done with a bit of intention and knowledge of what these algorithms are actually doing.
"We found that while the raw data makes no music, the waveforms share patterns"
Its not that surprising, since music is frequency based while pictures are spatially based. I expect you could get something that sounded like music from a picture if you did some Fourier-transform filtering before with different interval domains (different sizes of 'chunks' being converted to frequencies).
These days there are a number of audio synthesizers which do basically what you said, such as the Beepmap plugin that comes stock with FL Studio.
Among the first to use the technique artistically, Aphex Twin famously drew his own face in the spectrum of his track "Equation" (https://www.youtube.com/watch?v=wSYAZnQmffg - skip to 5:20 for the face).
On a similar note, the images to sounds thing has been used by musicians / producers for a while. One of the more high profile users of this is Aphex Twin.
For all the naysayers, not understanding DSP is the whole point. The naive abuse of tools designed for other tasks is a powerful thing, its a totally different way of working than deliberate application of reasoned technique.
I think it would've been more interesting to apply a fourier transformation to the image, convert that to audio, and apply the wah-wah (which is essentially just a low-pass filter) to that.
The filters used in the article are meant to be applied on linear sample streams. A RAW image is not a linear sample stream. Re-encoding the image into a linear sample stream, applying the effect and then decoding the resulting signal to a RAW image again could be truer to the idea of "Paris with an echo".
Oh jesus I can't believe this made the washington post. It's basically a bunch of people not-really-understanding DSP. It's a highly known and highly explored field.
It's not Paris with an echo as much as it is the raw data of a specific digital encoding of a picture of Paris fed through a digital filter suited for linear streams of samples.
The effect is a direct result of the encoding more than anything.
I suspect that better sounds could be made from images if the data was treated as if it were in frequency domain instead of time domain, because then some filtering could make more sense. This could open up some more possibilities, such as even transforming color distribution data into musical notes (I wonder if nice-looking color chords can be transformed into nice-sounding musical chords).
This is very interesting, and I love both coding and music, so I guess I'll get on making something like that :)
I'm not sure I understand your comment, but I think that's what is already happening. Consider this: WAV is a sequence of samples - BMP is sequence of RGB values. So you can more or less treat it the same.
If you would try to read JPEG (i.e. frequency domain data) as WAV it would be a mess. The "appropriate" format would then be something like MP3 but you would run into issues - MP3 contains more metadata in the streams, JPEG encodes 8x8px 2D images while MP3 encodes ~576 1D samples...
If you like this, I suggest checking out PixiVisor. It's an experimental tool by NightRadio (Alexander Zolotov), who also made the amazing modular synth + tracker "SunVox".
PixiVisor takes a short video loop (~animated-GIF or sequence of images), modulates the frames (progressive-scan) into audio. A 2nd instance of the app (usually on another device) demodulates the audio back into similar video frames automagically.
"Similar"? Well... after picking up the multi-path room echoes and the "colorful" transfer functions of the DAC, amp, speaker, mic, ADC, etc, there is substantial distortion. This is intentional, as you can vary the signal considerably and the receiver app still reliably demodulates it back into video.
This hit the "wait, isn't that impossible?!" stage for me in this demo video:
https://www.youtube.com/watch?v=HkdLAWPsONQ
Using two mono channels of an audio mixer to mix video live is already insane. Then the effect insert is turned up, and various types of reverb get added to the video frames in a way that perfectly matches what you hear.
Also, this video, of live room echoes and hand waves that clearly result in changing reflections/blocking across many frequencies. Multi-path is also very visible.
https://www.youtube.com/watch?v=mF5C93cDocA
It's even multi-platform! (Windows, Linux, OSX, Android, iOS)
/* sigh, now I'm going to be goofing off with SunVox instead of writing RSpec */