I find these retro blog posts from people building LLM solutions super useful for helping me try out new prompting, eval, etc. techniques. Which blog posts have you all found most useful? Would love a link.
I've spent the past few years teaching business analysts at large financial institutions how to automate their Excel workflows with Python. My main takeaway are:
1. Business analysts can spend up to weeks per month updating existing month-end Excel reports with new data.
2. Python automations can reduce the updating process to a matter of minutes.
3. BUT actually understanding the Excel file & writing the Python automation code can take several months.
4. Tools like ChatGPT are hugely useful when given a specific translation task — eg: convert the formula =SUM(A1, B1) to Python — but get completely lost on more complicated translations.
So I'm trying to build an AI-powered tool that can automatically convert Excel workbooks to Python. The key to doing so is 1) using plain old programming to parse the Excel file and build a translation plan (finding all of the unique tables in the Excel file, identifying repeated logic, determining the correct order to output the code), and 2) to prompt an LLM to execute each piece of that translation plan one at a time.
> or that they break from earlier stated requirements as new requirements are requested.
This nails one of the biggest issues that we're trying to help Mito users with -- code that meets __changing__ requirements. ie: the input data column headers change.
The best approach we have right now is giving users flexibility to parametrize their script (ie: tell the script the right column header or reference columns by position instead of name)
It turns out that teams that have moved their data to Snowflake tend to avoid these issues primarily because the schema of their data changes much less frequently.
> Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
I love the idea of giving users feedback on how to get better at prompting the LLM. I think the key to using this approach within Mito is giving users guidance at the right time -- sometimes shorter prompts get the job done, and they're always easier to write :)
A really sweet integration of this approach could be: when the LLM generated code errors or when we notice that the user undoes their previous prompt, we offer the user help in converting non-working prompts into ones that follow best practices of breaking complex tasks down into small steps.
> On the Design of AI-powered Code Assistants for Notebooks - uses Mito as part of their case study
Andrew McNutt, one of the authors presented this paper here: https://www.youtube.com/watch?v=g0prh8mE3bI Their different classifications of notebook code-gen tools has actually been super helpful in my own thinking. Thanks for the help, Andrew if you're a HNer
That's an interesting idea. When converting an Excel workbook to Python, checking your work as you go is the hardest thing for us to convince people to do. (See here: https://blog.trymito.io/automating-spreadsheets-with-python-...). So tools to make that double checking process easier are really interesting to us.
I've personally never seen a Mito users with a detailed enough spec of their report that an LLM would be able to use it to check compliance -- but maybe if we built the functionality users would create them...
Hey, I'm Aaron, co-founder of Mito. Funnily enough, doing "diff detection" in spreadsheets is like the first thing we made when building Mito. We built Git for Excel to enable better collaboration around Excel models -- turns out Excel power users would rather play in single player mode. So it's funny to be exploring spreadsheet difference detection again a few years later. This time, thinking about it purely in single player mode to understand the impact of LLM generated code on your data.
This reminds me of an Upton Sinclair quote: “was one to believe that there was nowhere a god of hogs, to whine this hog personality was precious. To whome these hog squeals and agonies had meaning?”
