I’m an independent consultant building agent systems. I kept running into the same problem where my clients and I tried to use LLMs when regular code was the better tool. The last straw for me was a prospecting agent that ruined 10-20% of the records it inserted into my CRM, and I had to review all 800+ entries by hand. I thought I’d write up what I’ve learned from this experience - I landed on a six-question framework that has helped me greatly when deciding if a solution could benefit from an LLM. I’d be interested to hear what boundary questions others use.
Exactly! This is a huge thing I preach in development workflows, if you're going to need to do something more than once (or have your agent do something more than once), make it repeatable and reliable. What kinds of things have you seen given to LLMs that should have been code?
I’m an independent consultant building agent systems. I kept running into the same problem where my clients and I tried to use LLMs when regular code was the better tool. The last straw for me was a prospecting agent that ruined 10-20% of the records it inserted into my CRM, and I had to review all 800+ entries by hand. I thought I’d write up what I’ve learned from this experience - I landed on a six-question framework that has helped me greatly when deciding if a solution could benefit from an LLM. I’d be interested to hear what boundary questions others use.
This really resonates with me.
So many tasks given to LLMs can be performed more reliably and efficiently with deterministic code.
In those cases the LLM can help you write the code. But doesn’t need to be invoked every time the task executes.
Exactly! This is a huge thing I preach in development workflows, if you're going to need to do something more than once (or have your agent do something more than once), make it repeatable and reliable. What kinds of things have you seen given to LLMs that should have been code?