Perhaps allowing people to call via a static IPv6 address routed to their phone would resolve the issue with spoofing. A single range would be reserved for VOIP services such as 2002::/16 and subscribers would be allocated addresses that could be shortened and made to look to like traditional phone numbers.

I feel like any other solution adds yet more complexity and I would argue the only thing that should be necessary to make calls is an internet connection.

Some VOIP applications already accept calling via IPv4/IPv6 addresses.