Your server has NTP. Your client likely, too, if it has enough network to connect to the server. If it does not use NTP, it's easy to set the time within the minute or so required manually.
Hardware clocks are famously unreliable and inaccurate. NTP has failure modes that can result in servers being wildly out of sync with reality. Letting either one of those hose your ability to log in and diagnose/recover the system is a mistake.
It really depends on what you need. Yes, hardware clocks and NTP have failure modes[1], but how common are they? Hosing your bootloader or its ability to find the kernel is also a thing that happens where you definitely need some lower level access, which would also help with a mismatched server clock and TOTP. It depends on a lot of factors whether that is more or less common than NTP failure modes.
Would I employ this for a critical commercial server? No. Would I do so for my private server? Absolutely, without hesitation. If it really loses track of time so badly (hasn't happened yet to my knowledge), I can just log in through virtual console.
[1] I'm excluding drift and inaccuracy, since from my anecdotal experience they are usually not that bad.
I disagree that requiring the time to be correct precise to 30 seconds is "a mistake" in the general case, but it is a good point that this might happen in some weird edge case (where also changing the local clock +/-60s is not enough to fix it). Having a fallback that lets you in via a mechanism that does not depend on any external factors or uninterrupted power supply does seem like a good idea for important physical infrastructure. (For virtual stuff, in such a weird case you can always just modify the disk image to get ntp from a working host and boot again.)
