I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
[0]: https://github.com/dtcxzyw/llvm-hackme
Fantastic article. Thanks for sharing. I love the honest take on LLM-based fuzzing. It’s exactly something I would do with a similar problem.
Writing code with an LLM seems like a deal with the devil but debugging logical errors or type coercion bugs is hell.