It makes sense to take cost per task more important than cost per token, but for long running tasks, are more appropriate indicator could be "cost per accepted outcome". Because for a task, it is hard to say:
What is the boundary of the task?
Do we count failure and retry as the same task?
What if the result "looks" good but rejected by the user?
How do we measure the extra human effort for verifying and resuming when using the cheap model?
In my opinion, "cost" means more than the direct token usage of a finished task.
Agreed. While “Cost per task” is a much better unit than cost per token, it’s still an intermediate metric.
The metric we actually care about is closer to "cost per accepted outcome" which would roughly be "total inference + retries + failures + the human effort required to verify, correct, or resume the work."
The first few are easy to instrument.
But "human effort verification" and “accepted” are much harder to define consistently across different agent workflows.
For our code review product we have a decent proxy because we can measure user reactions and whether findings are actionable and if it's merged/accepted.
For more open-ended agent tasks, I don’t think the industry has a particularly good answer yet... I would love a point if there is one?
The cheaper model isn’t actually cheaper if a human has to spend the savings babysitting it...
That's something I have been working on because I want to decouple work from the agent. In my opinion, work should be treated as a first-class object.
A long running work could have many attempts, failures and retries should be aggregated into the same result.
An acceptance of a work result should not be inferred as "the model said it is done", but be recorded as a human approval.
To calculate the cost of the whole work, the outcome and verified result of each attempt must all be saved.
I don't have a final method to measure human effort, right now I start by recording recovery, review and explicit acceptance events, the at least we have data to analyze.
Finally an evaluation that go beyond benchmarks. Thanks for sharing details and takeaways.
I wonder how you come you opted for GLM; how did the selection process look like?
It makes sense to take cost per task more important than cost per token, but for long running tasks, are more appropriate indicator could be "cost per accepted outcome". Because for a task, it is hard to say:
What is the boundary of the task?
Do we count failure and retry as the same task?
What if the result "looks" good but rejected by the user?
How do we measure the extra human effort for verifying and resuming when using the cheap model?
In my opinion, "cost" means more than the direct token usage of a finished task.
Agreed. While “Cost per task” is a much better unit than cost per token, it’s still an intermediate metric.
The metric we actually care about is closer to "cost per accepted outcome" which would roughly be "total inference + retries + failures + the human effort required to verify, correct, or resume the work."
The first few are easy to instrument.
But "human effort verification" and “accepted” are much harder to define consistently across different agent workflows.
For our code review product we have a decent proxy because we can measure user reactions and whether findings are actionable and if it's merged/accepted.
For more open-ended agent tasks, I don’t think the industry has a particularly good answer yet... I would love a point if there is one?
The cheaper model isn’t actually cheaper if a human has to spend the savings babysitting it...
That's something I have been working on because I want to decouple work from the agent. In my opinion, work should be treated as a first-class object.
A long running work could have many attempts, failures and retries should be aggregated into the same result.
An acceptance of a work result should not be inferred as "the model said it is done", but be recorded as a human approval.
To calculate the cost of the whole work, the outcome and verified result of each attempt must all be saved.
I don't have a final method to measure human effort, right now I start by recording recovery, review and explicit acceptance events, the at least we have data to analyze.
I'm exploring this in a work-runtime project: https://github.com/kungfu-systems/kungfu