It’s already possible. See the Stagecraft studio they built for the production of TV series The Mandalorian.
> shooting the series on a stage surrounded by massive LED walls displaying dynamic digital sets, with the ability to react to and manipulate this digital content in real time during live production
> The StageCraft process involves shooting live-action actors and sets surrounded by large, very high-definition LED video walls. These walls display computer-generated imagery backdrops, once traditionally composited primarily in post-production after shooting with chroma key screens. These facilities are known as "volumes". When shooting, the production team is able to realign the background instantly based on moving camera positions. The entire CGI background can be manipulated in real-time.
It’s already possible. See the Stagecraft studio they built for the production of TV series The Mandalorian.
> shooting the series on a stage surrounded by massive LED walls displaying dynamic digital sets, with the ability to react to and manipulate this digital content in real time during live production
https://www.unrealengine.com/fr/blog/forging-new-paths-for-f...
> The StageCraft process involves shooting live-action actors and sets surrounded by large, very high-definition LED video walls. These walls display computer-generated imagery backdrops, once traditionally composited primarily in post-production after shooting with chroma key screens. These facilities are known as "volumes". When shooting, the production team is able to realign the background instantly based on moving camera positions. The entire CGI background can be manipulated in real-time.
https://en.wikipedia.org/wiki/StageCraft