> You can literally pipe binary data into it at one end and receive that data into a program at the other end

That is the user perspective, not what has to be in a ssh program to work.