It would be great for this to actually explain what sorts of metrics are being computed here beyond what you get for free by instrumenting the requests library.
From looking at the screenshots, it looks like it can monitor number of tokens, which seems useful, but I'm not clear why that needed a whole big project.
I feel like the stuff you actually want to monitor in prod for ML that you don't get from infra monitoring are things that are not trivial to drop in because you want a sense for how well the ML components are working, which is generally pretty application specific. Having a general framework for that seems useful, but not really what we have here, at least for the moment.
Also, it just seems a bit weird for this to have it's own UI. Part of the point of OTEL is so that you can send all your metrics to one place. Not totally possible all the time and turning metrics into dashboards takes time, but the point of OTEL seems to be to separate these concerns.
Hey! On the metrics front, we do basic metrics like requests, tokens and cost for now as all of the usefull information about llm is included directly in the Span Attributes of the Trace.
So what a lot of users have told me is the want to track their usage(cost/tokens) and want to keep an eye on the user interactions like prompts and responses, We are looking to add more accuracy based metrics too which is very highly asked to us.
We do Otel metrics for the main things needed in dashboarding
1. Requests (Counter)
2. Tokens (Counter) (Seperate metric for Prompt Tokens, Completion and Total Tokens)
3. Cost (Histogram)
Since Otel doesn't yet support Synchronous gauge + Users wanted a trace of the LLM RAG application, We opted to Traces which is quite standard now in LLM Observability tool.
Lemme know if you had something more in mind, Love getting feedback from amazing folks like you!
Yup thats correct, Love as more tools come up! We though tried to follow the actual OpenTelemetry's Semantic Convention for GenAI, the others dont.
Additionally none of these libraries "then"(Can check again) seemed to send the cost attribute which was what most of our user's mentioned, So we added that.
Langtrace core maintainer here. We capture input/output/total tokens for both streaming and non streaming across various LLM providers. For streaming we calculate using tiktoken library as OpenAI does not show that with their outputs. Cost is calculated on the client side as it may keep changing. But all you need is the cost table for the llm vendor and you can get the cost for token usage
Does the dashboard/UI support traces? I would love a tool in which to view opentelemetry traces, that can neatly display full prompt and response for the spans that represent LLM queries. I'm planning to add opentelemetry instrumentation to magentic [1] and looking for a UI that is easy to run locally that makes it easy to see what an agent is doing (via OTEL traces). I have more of my thoughts on the github issue: https://github.com/jackmpcollins/magentic/issues/136
Shameless plug. We are building exactly this. Fully open source and open telemetry standard tracing with a visualization client that’s optimized for LLM application observability. Check it out here
I appreciate you sharing your project with me. It's great to see others working on solutions in this space. While our offerings may have some similarities, I'm sure there are unique aspects to what you've built that could be valuable to users. I encourage you to continue innovating and pushing the boundaries of what's possible. Healthy competition ultimately benefits the entire community, as it drives us all to create better products. I wish you the best of luck with your project.
How does it do that for ruby for example? (which is in the link you provided). OTEL instrumentation for HTTP doesn’t instrument the body so you won’t be able to see token usage, prompts and completions. Or am I missing something?
So, do I understand this right. But is this supposed to be a centralized (in a way) application for collecting all OpenTelemtry data related to LLM's? And it's supposed to support various of related services and/or services today?
I see how this can be really useful. But at the same time, how do you guys see your self differentiate your self from cloud hosted equivalents? (e.g dashboard-like services in Azure and similar).
Anyhow, interesting project. I'll keep an eye on it for future use
Yup, You are on point!
We do see the UI/dashnoarding layer to find other usage a bit more as we add more tools(Compare/Evaluate) to help the overall LLM Developer Experience.
Also, I totally get the existing tools like Grafana are great, which is why we built our SDK in a very vendor neutral way using OTEL, So that yiu can send these traces and meyrics to our choice of OTLP backend.
I browsed through the readme, and don’t quite understand how LLM or “GenAI” is supposed to help me with observability.
Pro tip: if you truly want to build an open source community around this. Don’t build your community using proprietary chat platforms (ie, slack). Slack in particular only keeps a small amount of history.
I don’t think it was ambiguous at all, perhaps you just don’t have the problem they describe?
They have an observability platform for LLMs and other generative AI services. Just like you can use observability tools on services like databases or APIs, they’ve made an observability tool for the aforementioned AI services and the applications built with them. They use OpenTelemetry, a pretty standard observability protocol, to interoperate with other tools that ingest observability-oriented data.
It would be great for this to actually explain what sorts of metrics are being computed here beyond what you get for free by instrumenting the requests library.
From looking at the screenshots, it looks like it can monitor number of tokens, which seems useful, but I'm not clear why that needed a whole big project.
I feel like the stuff you actually want to monitor in prod for ML that you don't get from infra monitoring are things that are not trivial to drop in because you want a sense for how well the ML components are working, which is generally pretty application specific. Having a general framework for that seems useful, but not really what we have here, at least for the moment.
Also, it just seems a bit weird for this to have it's own UI. Part of the point of OTEL is so that you can send all your metrics to one place. Not totally possible all the time and turning metrics into dashboards takes time, but the point of OTEL seems to be to separate these concerns.
Hey! On the metrics front, we do basic metrics like requests, tokens and cost for now as all of the usefull information about llm is included directly in the Span Attributes of the Trace.
So what a lot of users have told me is the want to track their usage(cost/tokens) and want to keep an eye on the user interactions like prompts and responses, We are looking to add more accuracy based metrics too which is very highly asked to us.
Also re on UI: UI is optional, You can use the sdk to directly send otel traces and meyrics to your preferred destination. https://docs.openlit.io/latest/connections/intro
On the span is an interesting attribute to view, but had you considered using otel meters/metrics for that instead?
I think cardinality in span attributes can be a problem, and meters are better for aggregating and graphing
We do Otel metrics for the main things needed in dashboarding 1. Requests (Counter) 2. Tokens (Counter) (Seperate metric for Prompt Tokens, Completion and Total Tokens) 3. Cost (Histogram)
I did attach a Grafana dashboard too (Works for Grafana Cloud, Ill get something for OSS this week) https://docs.openlit.io/latest/connections/grafanacloud
Since Otel doesn't yet support Synchronous gauge + Users wanted a trace of the LLM RAG application, We opted to Traces which is quite standard now in LLM Observability tool.
Lemme know if you had something more in mind, Love getting feedback from amazing folks like you!
Very cool. QQ: How does this differ from Langtrace (https://www.langtrace.ai) / (https://github.com/Scale3-Labs/langtrace)?
Some others have mentioned before, but there are actually several projects working on exactly this:
https://github.com/Arize-ai/openinference
https://github.com/traceloop/openllmetry
https://github.com/Scale3-Labs/langtrace
Yup thats correct, Love as more tools come up! We though tried to follow the actual OpenTelemetry's Semantic Convention for GenAI, the others dont.
Additionally none of these libraries "then"(Can check again) seemed to send the cost attribute which was what most of our user's mentioned, So we added that.
Langtrace core maintainer here. We capture input/output/total tokens for both streaming and non streaming across various LLM providers. For streaming we calculate using tiktoken library as OpenAI does not show that with their outputs. Cost is calculated on the client side as it may keep changing. But all you need is the cost table for the llm vendor and you can get the cost for token usage
Does the dashboard/UI support traces? I would love a tool in which to view opentelemetry traces, that can neatly display full prompt and response for the spans that represent LLM queries. I'm planning to add opentelemetry instrumentation to magentic [1] and looking for a UI that is easy to run locally that makes it easy to see what an agent is doing (via OTEL traces). I have more of my thoughts on the github issue: https://github.com/jackmpcollins/magentic/issues/136
[1] https://github.com/jackmpcollins/magentic
Shameless plug. We are building exactly this. Fully open source and open telemetry standard tracing with a visualization client that’s optimized for LLM application observability. Check it out here
- https://github.com/Scale3-Labs/langtrace - https://langtrace.ai/
I appreciate you sharing your project with me. It's great to see others working on solutions in this space. While our offerings may have some similarities, I'm sure there are unique aspects to what you've built that could be valuable to users. I encourage you to continue innovating and pushing the boundaries of what's possible. Healthy competition ultimately benefits the entire community, as it drives us all to create better products. I wish you the best of luck with your project.
Yup, The Dashboard is entirely powered by OTEL traces and yeah, you can see full prompts and responses including cost and request metadata.
The UI is pretty easy to run, Its one single image.
Lemme know if you need any help on instrumentation, If you have a PR, Lemme know, Ill try to assist on this
You can add whatever span attributes you like to otel traces, and then show those attributes in whatever UI you have (I use Grafana).
How does this differ from Traceloop’s Openllmetry? https://github.com/traceloop/openllmetry
From what I can see, Openllmetry asks you to manually call tracer for non-default OpenAI libraries (i.e. not Python/Node.js) [1]
OpenLIT might be easier to integrate with any language that supports OTEL for HTTP clients — you just trace your HTTP calls to OpenAI.
[1] https://www.traceloop.com/docs/openllmetry/getting-started-r...
How does it do that for ruby for example? (which is in the link you provided). OTEL instrumentation for HTTP doesn’t instrument the body so you won’t be able to see token usage, prompts and completions. Or am I missing something?
So, do I understand this right. But is this supposed to be a centralized (in a way) application for collecting all OpenTelemtry data related to LLM's? And it's supposed to support various of related services and/or services today?
I see how this can be really useful. But at the same time, how do you guys see your self differentiate your self from cloud hosted equivalents? (e.g dashboard-like services in Azure and similar).
Anyhow, interesting project. I'll keep an eye on it for future use
Yup, You are on point! We do see the UI/dashnoarding layer to find other usage a bit more as we add more tools(Compare/Evaluate) to help the overall LLM Developer Experience.
Also, I totally get the existing tools like Grafana are great, which is why we built our SDK in a very vendor neutral way using OTEL, So that yiu can send these traces and meyrics to our choice of OTLP backend.
Thanks!
I browsed through the readme, and don’t quite understand how LLM or “GenAI” is supposed to help me with observability.
Pro tip: if you truly want to build an open source community around this. Don’t build your community using proprietary chat platforms (ie, slack). Slack in particular only keeps a small amount of history.
I don’t think it was ambiguous at all, perhaps you just don’t have the problem they describe?
They have an observability platform for LLMs and other generative AI services. Just like you can use observability tools on services like databases or APIs, they’ve made an observability tool for the aforementioned AI services and the applications built with them. They use OpenTelemetry, a pretty standard observability protocol, to interoperate with other tools that ingest observability-oriented data.
Yeah, 100%, I couldn't agree more on this!
Though if you'd like to see something more, Lemme know!
Thanks!
@xyst
https://discord.gg/RbNPvG54
Thanks for the suggestion on community (Not Slack). I Created a Discord server now, Feel free to spam questions, suggestions! Thanks for the feedback!