A command-line utility to process OpenInference spans generated from a single agent instance and output:
- the number of tokens for each sub-agent separated by category
- each step sorted by the monetary price
- any context block that was re-sent between steps along with the cost of each re-send
Requires Python 3.10+ and no dependencies. The repository comes with a sample trace for testing purposes.
Author here. This is kind of my first project, so I would appreciate any kind of feedback, critical or not. Claude Code wrote the majority of this. But what I really need from you guys is to run it on a live trace and report back what it got wrong.
here's the quick summary: kind of.
LangSmith and Langfuse allow ranking on cost per step, but as dashboard widgets aggregating data over many runs over a period of time. Helicone allows doing so if you label your call and have to write your own SQL queries. Phoenix provides cost per span and per project but no middle ground. So if you need to know what made this particular run cost the way it costs, you will still need to read the trace tree manually. None of them detect repeated contexts in subsequent steps, which, in my experiments, is the higher figure.
limitations because you will discover them anyway:
Repeat detection works with exact matching and prefix sharing. Mismatched strings in between aren't detected.
The token counts in repeat detection are len(text) // 4, not true tokenization. Marked as such in the output.
Two model price examples are hard coded. All other input produces n/a instead of guessing.
This was tested on the included fixtures and one live trace run, the three node LangGraph case. The outputs of Phoenix, Langfuse, CrewAI, and AutoGen have not been processed by it yet.
A command-line utility to process OpenInference spans generated from a single agent instance and output: - the number of tokens for each sub-agent separated by category - each step sorted by the monetary price - any context block that was re-sent between steps along with the cost of each re-send
Requires Python 3.10+ and no dependencies. The repository comes with a sample trace for testing purposes.
Author here. This is kind of my first project, so I would appreciate any kind of feedback, critical or not. Claude Code wrote the majority of this. But what I really need from you guys is to run it on a live trace and report back what it got wrong. here's the quick summary: kind of.
LangSmith and Langfuse allow ranking on cost per step, but as dashboard widgets aggregating data over many runs over a period of time. Helicone allows doing so if you label your call and have to write your own SQL queries. Phoenix provides cost per span and per project but no middle ground. So if you need to know what made this particular run cost the way it costs, you will still need to read the trace tree manually. None of them detect repeated contexts in subsequent steps, which, in my experiments, is the higher figure. limitations because you will discover them anyway:
Repeat detection works with exact matching and prefix sharing. Mismatched strings in between aren't detected. The token counts in repeat detection are len(text) // 4, not true tokenization. Marked as such in the output. Two model price examples are hard coded. All other input produces n/a instead of guessing. This was tested on the included fixtures and one live trace run, the three node LangGraph case. The outputs of Phoenix, Langfuse, CrewAI, and AutoGen have not been processed by it yet.