A team unfamiliar with a code base demoed asking questions to an LLM about it. The answers genuinely excited some. But anyone who had spent a short time in the code base knew the answers were wrong. Oh well.
That is one anecdote, but it doesn't really have any information in it. To debug the process we'd need to know which LLM, the developer's backgrounds, what prompts they used etc.
I've used a variety of LLMs to ask questions about probably dozens of unfamiliar code bases many of which are very complicated technically.
At this point I think LLMs are indispensable to understanding unfamiliar code bases. Nearly always much better than documentation and search engines combined.
