The LLM approach is simpler and more flexible since it works with every library and language out of the box.
Looking up official documentation would require shipping sophisticated parsers for each language, plus a way to map tokens to their corresponding docs. I'd also need to maintain those mappings as libraries evolve and documentation moves. Some ecosystems make this easier (Rust with docs.rs), but others would be much harder (AWS documentation, for example).
I also want explanations visible directly in hover hints rather than linking out to external docs. So even with official documentation, I'd need to extract and present the relevant content anyway.
Beyond that, the LLM approach adapts to context in ways static docs can't. It generates explanations for code within the documentation you're reading, and it works on code that doesn't compile, like snippets with stubs or incomplete examples.
It could be interesting in the future to look into doing some type of hybrid approach where an LLM goes out and searches up the documentation, that way it's a little bit more flexible. But that would also be a bit slower and more costly.
> Looking up official documentation would require shipping sophisticated parsers for each language,
You could just token match (use tree-sitter or something similar) and fetch the official docs. Keep it dead-simple so there's no way you can provide false positives (unlike what's happening rn where hallucinations will creep in).
> It generates explanation
Again, I don't want that. It's not a feature, it's a limitation that right now gives you fake information.
For whom? The whole reason I want to consult docs is to get the official documentation on a given topic. How could I trust anything it says, and what’s to say any earned trust is durable over time?
what is the name for this kind of pointless, lazy, selective, quoting that willfully misconstrues what's being quoted? the answer to this question is incredibly clear: for the developer that created this tool. if that makes you unhappy enough to malign them then maybe you should just not use it?
> pointless, lazy, selective, quoting that willfully misconstrues what's being quoted
They quoted the part they were replying to. The point was to show what they were asking about. If your question pertains to only a part of the text, it only makes sense to be selective. That's not wilfully misconstruing anything; that’s communicating in a clear, easy-to-follow way. The context is still right up there for reading, for anyone who needs to review it.
> the answer to this question is incredibly clear: for the developer that created this tool
Questions aren’t only ever asked out of pure curiosity; sometimes they’re asked to make the other person give them more consideration. The question you quote was accompanied by an explanation of how the commenter found the approach less simple for them as a user, suggesting that perhaps they think the developer would have done better to consider that a higher priority. (I might add that you, too, chose to selectively omit this context from your quoting—which I personally don’t see as problematic on its own, but the context does require consideration, too.)
> if that makes you unhappy enough to malign them then maybe you should just not use it?
The author of the extension chose to share what they made for others to use. They asked for feedback on user experience and expressed doubt about their design decisions. If someone finds they might not want to use it because of what they consider fundamentally flawed design, why couldn’t they tell the author? It’s not like they were rude or accused them of any wrong-doing (other than possibly making poor design choices).
lol thank you, I was just going to respond to them. One thing I should mention too is that if it were at all practical to build without using generative AI, someone would have built something similar years ago before LLMs.
If there’s any amount of irony in your comment, I’m missing it - and I apologize for that.
That said, people have built this without LLMs years, even decades, ago. But UX has fallen by the wayside for quite some time in the companies that used to build IDEs. Then some fresher devs come along and begin a project without the benefit of experience in a codebase with a given feature … and after some time someone writes a plugin for VSCode to provide documentation tooltips generated by LLM because “there is just no other way it can be done.”
We have language servers for most programming languages. Those language servers provide the tokens one needs to use when referencing the documentation. And it would be so much faster than waiting for an LLM to get back to you.
TBH, if anyone’s excuse is “an LLM is the only way to implement feature Q,” then they’re definitely in need of some experience in software creation.
I don't think you're wrong, but question: it's the weekend, you have an idea for something like this that you want to crank out. Is it really better for you to never ship because it takes a long time to build, or is it better to be able to ship using something like an LLM?
In my opinion the shipped product is better than the unshipped product. While of course I would prefer the version that you have designed, I sure don't have time to build it, and I'm guessing you don't either.
If this was our day jobs and we were being paid for it, it would be a much different story, but this is a hobby project made open source for the world.
Please post here once you have something working, or what you find if you struggle. I would be interested to see if you could get something working and would love to be wrong, as it would be nice to have something similar that does not use an LLM. It would be really cool if something more useful comes out of my experiment. I don't think you could reuse much from my codebase, but feel free to take anything from it that you want.
Some things that might be useful to know to speed you up:
1. Most code blocks on the internet are easy to find in a webpage. They generally are surrounded by `<code/>` tags. You can query for these using the method in my extension. Then you will need to filter out any code block that has a `<span/>` count <= 1, as code blocks are used to highlight arbitrary stuff on the internet and you would really just want to find the blocks of code.
2. You will need a method to identify when to generate documentation for a code block even with your implementation, as some documentation websites are one really long page. You can do this with two types of observers that I use in my codebase. One will identify when a code block is in view and the other will keep track of mutations to code blocks. You need to keep track of mutations because sites like ChatGPT continuously edit a code block while streaming a response. You want to generate documentation once it's done, as that's when the code is well formed. I have a janky example of how to do this in my extension. Claude should be able to find the code.
3. LLMs were useful for building this, but they struggled with design decisions, especially around UX. This project seems out of distribution for them. Claude probably won't suggest the right solution a lot of the time, but if you have it list out multiple options, it can usually identify which one is best.
I will probably check back here in a couple weeks if I don't hear anything from you. I would be really impressed if you can get something working in a weekend that is *not just hard coded for a specific use case, but scales well.*
Oh, one more thing: I should mention that you’ll probably want to fetch documentation only after a code block has been in view for n milliseconds (like what I do in my extension). Otherwise, you risk throttling if a user scrolls very quickly through a webpage. It should only fetch documentation once the user pauses on a block.
I agree that parsing codebases and linking code to documentation is a solved problem. I think @ramon156's suggestion to use tree-sitter or something similar to parse an abstract syntax tree makes sense.
To clarify my earlier point, I wasn't suggesting this is impossible, just that it's not *practical* to build a universal LSP that works with every language and framework out of the box without anything local to index. I don't think an reusing an LSP would be a great fit here either, since LSPs rely on having full project context, dependencies, and type information. These aren't available when analyzing code snippets on arbitrary webpages.
Parsing was never my major concern though. It's the "map tokens to URLs" part. A universal mapping for every token to every piece of documentation on the internet is *impractical* and difficult to maintain. To achieve parity without LLMs, I'd need to write and maintain parsers for every documentation website, and that assumes documentation even exists for most tokens (which it doesn't).
I think kristopolous's suggestion of grounding the LLM with data sources that keep a serialized database of documentation from many different places makes the most sense. That way, the LLM is just extracting and presenting key information from real documentation rather than generating from scratch.
There are probably ways to make this easier. Maybe an offline job that uses LLMs to keep mappings up to date. The project could also be scoped down to a single ecosystem like Rust where documentation is centralized, though that falls apart once you try to scale beyond one language as mentioned above. Maybe I could use raw definition on GitHub combined with an LSP to generate information?
Open to other suggestions on how to bridge this gap.
Nice idea, but unless it's using up-to-date docs it's likely to introduce a lot of confusion, especially if you're reading docs to find out what changed between versions. You really need to be sending the code and the version of the software that the person reading the page is looking at.
LLMs get interfaces wrong all the time, they make up methods that don't exist, invent parameters, etc. This needs to be 100% accurate or it is not useful.
> the extension detects tokens and sends the code to an LLM
> The LLM generates documentation
so, not documentation? Why not write your own engine and detect the official docs? e.g. docs.rs would do this wonderfully
The LLM approach is simpler and more flexible since it works with every library and language out of the box.
Looking up official documentation would require shipping sophisticated parsers for each language, plus a way to map tokens to their corresponding docs. I'd also need to maintain those mappings as libraries evolve and documentation moves. Some ecosystems make this easier (Rust with docs.rs), but others would be much harder (AWS documentation, for example).
I also want explanations visible directly in hover hints rather than linking out to external docs. So even with official documentation, I'd need to extract and present the relevant content anyway.
Beyond that, the LLM approach adapts to context in ways static docs can't. It generates explanations for code within the documentation you're reading, and it works on code that doesn't compile, like snippets with stubs or incomplete examples.
It could be interesting in the future to look into doing some type of hybrid approach where an LLM goes out and searches up the documentation, that way it's a little bit more flexible. But that would also be a bit slower and more costly.
> Looking up official documentation would require shipping sophisticated parsers for each language,
You could just token match (use tree-sitter or something similar) and fetch the official docs. Keep it dead-simple so there's no way you can provide false positives (unlike what's happening rn where hallucinations will creep in).
> It generates explanation
Again, I don't want that. It's not a feature, it's a limitation that right now gives you fake information.
> The LLM approach is simpler
For whom? The whole reason I want to consult docs is to get the official documentation on a given topic. How could I trust anything it says, and what’s to say any earned trust is durable over time?
> For whom?
what is the name for this kind of pointless, lazy, selective, quoting that willfully misconstrues what's being quoted? the answer to this question is incredibly clear: for the developer that created this tool. if that makes you unhappy enough to malign them then maybe you should just not use it?
> pointless, lazy, selective, quoting that willfully misconstrues what's being quoted
They quoted the part they were replying to. The point was to show what they were asking about. If your question pertains to only a part of the text, it only makes sense to be selective. That's not wilfully misconstruing anything; that’s communicating in a clear, easy-to-follow way. The context is still right up there for reading, for anyone who needs to review it.
> the answer to this question is incredibly clear: for the developer that created this tool
Questions aren’t only ever asked out of pure curiosity; sometimes they’re asked to make the other person give them more consideration. The question you quote was accompanied by an explanation of how the commenter found the approach less simple for them as a user, suggesting that perhaps they think the developer would have done better to consider that a higher priority. (I might add that you, too, chose to selectively omit this context from your quoting—which I personally don’t see as problematic on its own, but the context does require consideration, too.)
> if that makes you unhappy enough to malign them then maybe you should just not use it?
The author of the extension chose to share what they made for others to use. They asked for feedback on user experience and expressed doubt about their design decisions. If someone finds they might not want to use it because of what they consider fundamentally flawed design, why couldn’t they tell the author? It’s not like they were rude or accused them of any wrong-doing (other than possibly making poor design choices).
lol thank you, I was just going to respond to them. One thing I should mention too is that if it were at all practical to build without using generative AI, someone would have built something similar years ago before LLMs.
If there’s any amount of irony in your comment, I’m missing it - and I apologize for that.
That said, people have built this without LLMs years, even decades, ago. But UX has fallen by the wayside for quite some time in the companies that used to build IDEs. Then some fresher devs come along and begin a project without the benefit of experience in a codebase with a given feature … and after some time someone writes a plugin for VSCode to provide documentation tooltips generated by LLM because “there is just no other way it can be done.”
We have language servers for most programming languages. Those language servers provide the tokens one needs to use when referencing the documentation. And it would be so much faster than waiting for an LLM to get back to you.
TBH, if anyone’s excuse is “an LLM is the only way to implement feature Q,” then they’re definitely in need of some experience in software creation.
I don't think you're wrong, but question: it's the weekend, you have an idea for something like this that you want to crank out. Is it really better for you to never ship because it takes a long time to build, or is it better to be able to ship using something like an LLM?
In my opinion the shipped product is better than the unshipped product. While of course I would prefer the version that you have designed, I sure don't have time to build it, and I'm guessing you don't either.
If this was our day jobs and we were being paid for it, it would be a much different story, but this is a hobby project made open source for the world.
I’m going to get the LLM to assist me in building it. The shipped product is not going to rely on the LLM. That’s how I get it done over a weekend.
Please post here once you have something working, or what you find if you struggle. I would be interested to see if you could get something working and would love to be wrong, as it would be nice to have something similar that does not use an LLM. It would be really cool if something more useful comes out of my experiment. I don't think you could reuse much from my codebase, but feel free to take anything from it that you want.
Some things that might be useful to know to speed you up:
1. Most code blocks on the internet are easy to find in a webpage. They generally are surrounded by `<code/>` tags. You can query for these using the method in my extension. Then you will need to filter out any code block that has a `<span/>` count <= 1, as code blocks are used to highlight arbitrary stuff on the internet and you would really just want to find the blocks of code.
2. You will need a method to identify when to generate documentation for a code block even with your implementation, as some documentation websites are one really long page. You can do this with two types of observers that I use in my codebase. One will identify when a code block is in view and the other will keep track of mutations to code blocks. You need to keep track of mutations because sites like ChatGPT continuously edit a code block while streaming a response. You want to generate documentation once it's done, as that's when the code is well formed. I have a janky example of how to do this in my extension. Claude should be able to find the code.
3. LLMs were useful for building this, but they struggled with design decisions, especially around UX. This project seems out of distribution for them. Claude probably won't suggest the right solution a lot of the time, but if you have it list out multiple options, it can usually identify which one is best.
I will probably check back here in a couple weeks if I don't hear anything from you. I would be really impressed if you can get something working in a weekend that is *not just hard coded for a specific use case, but scales well.*
Oh, one more thing: I should mention that you’ll probably want to fetch documentation only after a code block has been in view for n milliseconds (like what I do in my extension). Otherwise, you risk throttling if a user scrolls very quickly through a webpage. It should only fetch documentation once the user pauses on a block.
I agree that parsing codebases and linking code to documentation is a solved problem. I think @ramon156's suggestion to use tree-sitter or something similar to parse an abstract syntax tree makes sense.
To clarify my earlier point, I wasn't suggesting this is impossible, just that it's not *practical* to build a universal LSP that works with every language and framework out of the box without anything local to index. I don't think an reusing an LSP would be a great fit here either, since LSPs rely on having full project context, dependencies, and type information. These aren't available when analyzing code snippets on arbitrary webpages.
Parsing was never my major concern though. It's the "map tokens to URLs" part. A universal mapping for every token to every piece of documentation on the internet is *impractical* and difficult to maintain. To achieve parity without LLMs, I'd need to write and maintain parsers for every documentation website, and that assumes documentation even exists for most tokens (which it doesn't).
I think kristopolous's suggestion of grounding the LLM with data sources that keep a serialized database of documentation from many different places makes the most sense. That way, the LLM is just extracting and presenting key information from real documentation rather than generating from scratch.
There are probably ways to make this easier. Maybe an offline job that uses LLMs to keep mappings up to date. The project could also be scoped down to a single ecosystem like Rust where documentation is centralized, though that falls apart once you try to scale beyond one language as mentioned above. Maybe I could use raw definition on GitHub combined with an LSP to generate information?
Open to other suggestions on how to bridge this gap.
Oh…I guess this isn’t the documentation-surfacing droids I’m looking for.
I could see getting actual docs being useful. Spitting out the delusions of an LLM is pretty well covered already, at least in my stack.
Very little “engine” required - use an existing language sever, map tokens to URLs, done.
Since these are called "tooltips" you might want to use the term more prominently.
Nice idea, but unless it's using up-to-date docs it's likely to introduce a lot of confusion, especially if you're reading docs to find out what changed between versions. You really need to be sending the code and the version of the software that the person reading the page is looking at.
Good point, thank you
interesting idea for a 1.0. Using https://context7.com/ might be the right next move here.
Also look into https://cht.sh/
Remember: incorrect (misleading) documentation is worse than no documentation.
What this might be better for is use-cases that don't require extreme precision. Imagine it for learning language or reading sophisticated academic literature. For example, https://archive.org/details/pdfy-TJ7HxrAly-MtUP4B/page/n111/...
Stuff like that is hard and every tool to make the complicated more legible I'd embrace.
Thank you! I will look into these! Yeah, it could be useful for providing explanations of key terms in papers or something similar.
Those are great links.
cht.sh is a fairly unknown community i think ... it's really great. Try
curl cht.sh
it's a wiki system.
Firefox has built-in link previews with AI. Hold mouse1 on a link.
Great idea with a terrible implementation.
Can you elaborate more. It's still early and I would love feedback
LLMs get interfaces wrong all the time, they make up methods that don't exist, invent parameters, etc. This needs to be 100% accurate or it is not useful.
It uses LLM under the hood. This is the bad part.