If you use Ctrl+N (or whatever the shortcuts for 'Switch to Desktop N' are set to in the settings) instead of swiping or using Ctrl+Left/Right it's much faster, and input is re-enabled for the desktop you're switching to basically instantly