I whole-heartedly agree with you. In fact, I have the impression that there's a rather interesting variation of the Dunning-Kruger effect in play on this issue. Usually, mentioning DK implies a derogatory connotation (akin "dumb people don't notice that they're dumb"), but in this case, many actually very bright people can't imagine the upsides of touch typing because they never experienced them themselves. There also seems to be a similar discussion whenever Vim and Emacs bindings come up. Yes, you don't technically need them, but they sure help you manipulating code at a pace where you're not constantly loosing your train of thought because of some small change you're throwing in.
I have no problem with people who decide that the learning curve of both is not for them. The thing I personally take issue with is when someone then tries to explain to me that those upsides are effectively nil, because "you're not really typing that much all day anyway". I call BS on that. If your code is non-trivial and you're trying to make it readable, you are going to iterate on it a couple of times. No amount of scribbling and diagrams is going to prevent that - even with the perfect data layout, the actual logic will need tweaking until it's suitable. So, the only argument you're then making from my perspective is that you actually don't care about readability, at which point there's a much bigger elephant in the room than your typing style.
I have no problem with people who decide that the learning curve of both is not for them. The thing I personally take issue with is when someone then tries to explain to me that those upsides are effectively nil, because "you're not really typing that much all day anyway". I call BS on that. If your code is non-trivial and you're trying to make it readable, you are going to iterate on it a couple of times. No amount of scribbling and diagrams is going to prevent that - even with the perfect data layout, the actual logic will need tweaking until it's suitable. So, the only argument you're then making from my perspective is that you actually don't care about readability, at which point there's a much bigger elephant in the room than your typing style.