There are techniques to mitigate this. You can reuse containers instead of creating a new one each time. You can mount in directories (like ~/.claude) from your local machine so you dont have to set claude up each time.
I use agents in a container and persist their config like you suggest. After seeing some interest I shared my setup at https://github.com/asfaload/agents_container
It works fine for me on Linux.
