Hey HN! We build a document parsing API, and if you have experience training models, it's unsurprising that tables are the biggest latency bottleneck. A dense table can take thousands of tokens to represent in HTML, and the model has to generate every cell value and every <td> tag.
We knew we needed to use speculative decoding techniques but thought Eagle and MTP didn't offer enough upside to justify training (speedup saturates at high concurrency)
Instead, we started with n-gram speculation. Table HTML repeats itself, so we could use earlier output to guess what came next without training another model. That worked well when the table repeated itself but not all of them follow a predictable pattern.
Then we tried an off-the-shelf DFlash drafter. It proposes a block of tokens in one pass, and the table model checks how much of that block it can keep. We expected to need a drafter trained for our exact checkpoint since DFlash reads the target model’s hidden states and fine-tuning can change those states. So it really surprised us when a drafter trained for the base model nearly halved latency on both of our fine-tuned table parsing models.
Our guess is that fine-tuning changed the hidden states just enough to teach the model table HTML while leaving enough of the signal the drafter relied on. That suggests a cheap first experiment for other fine-tuned models: try the base model’s drafter before spending time training one. In our case, the stock result gave us a reason to see how much further we could go by training the drafter on the HTML our table models actually generate.
On one H100, with one request in flight and real-world tables averaging about 100 cells, here’s what we saw for median latency with speculation off, stock DFlash, and fine-tuned DFlash:
Light Table parsing went from 3.35s → 1.76s → 0.76s.
Performance Table parsing went from 3.85s → 2.01s → 0.96s. After fine-tuning, each verification pass advanced the output by 12.48 tokens on average, up from 6.73 with the stock drafter.
this was a great read, appreciated the discussion around dflash stock header. wouldn't have guessed!
Hey HN! We build a document parsing API, and if you have experience training models, it's unsurprising that tables are the biggest latency bottleneck. A dense table can take thousands of tokens to represent in HTML, and the model has to generate every cell value and every <td> tag.
We knew we needed to use speculative decoding techniques but thought Eagle and MTP didn't offer enough upside to justify training (speedup saturates at high concurrency)
Instead, we started with n-gram speculation. Table HTML repeats itself, so we could use earlier output to guess what came next without training another model. That worked well when the table repeated itself but not all of them follow a predictable pattern.
Then we tried an off-the-shelf DFlash drafter. It proposes a block of tokens in one pass, and the table model checks how much of that block it can keep. We expected to need a drafter trained for our exact checkpoint since DFlash reads the target model’s hidden states and fine-tuning can change those states. So it really surprised us when a drafter trained for the base model nearly halved latency on both of our fine-tuned table parsing models.
Our guess is that fine-tuning changed the hidden states just enough to teach the model table HTML while leaving enough of the signal the drafter relied on. That suggests a cheap first experiment for other fine-tuned models: try the base model’s drafter before spending time training one. In our case, the stock result gave us a reason to see how much further we could go by training the drafter on the HTML our table models actually generate.
On one H100, with one request in flight and real-world tables averaging about 100 cells, here’s what we saw for median latency with speculation off, stock DFlash, and fine-tuned DFlash:
Light Table parsing went from 3.35s → 1.76s → 0.76s. Performance Table parsing went from 3.85s → 2.01s → 0.96s. After fine-tuning, each verification pass advanced the output by 12.48 tokens on average, up from 6.73 with the stock drafter.
We wrote up what we tried and how the results changed with more requests in flight: https://www.extend.ai/resources/table-parsing-with-dflash
If you’ve tried speculative decoding for structured outputs, I’d love to hear what worked for you. Happy to answer questions about the setup.