Thats clever. I used the same approach with lua_sethook to run a hook every X opcodes and check if time or memory constraints have been exceeded. But instead of yielding I used longjmp to terminate execution (basically like raising an exception and catching it a few levels up in the stack).
It's not bulletproof though and the script can still block e.g. when calling into native code.
Yeah, native code can block, and in those cases requires additional checks in that exposed native code. That runs into all the usual multithreading headaches, and depends on exactly which functions have been exposed to the lua interpreter.
The advantage to this method over longjmp is that the function call is still valid and can be resumed at any point.
It's not bulletproof though and the script can still block e.g. when calling into native code.