Not sure why it happens though, but it led to all sorts of strange intermittent issues with broken connections.
Once I replaced this logic with passing files using GetFile().Fd() instead it started working fine, so beware of this. I still wonder why it happens though.
EDIT: added author