Back in the day there was AviSynth to do this kind of thing. Unfortunately the version 3 rewrite which was supposed to use GStreamer and Ruby never went anywhere. Is there finally something similar for Linux?
VapourSynth[1] is what the AviSynth rewrite was supposed to be from what I have gathered. Hence the name ;). OTOH, it is still for the Windows platform, so not really what you were after but I thought I'd mention it anyway.
