1. The test setup is divergent from the real world (single node k8s cluster)
2. Their product is #1 in every metric they measured, but is also missing features from those compared against. Will those features change the results?
> The test setup is divergent from the real world (single node k8s cluster)
The setup was chosen to simplify the suite, so it can be easily run anywhere.
In real world, log collectors are mostly deployed as deamonsets and this is what was tested during the benchmark. The vlagent was initially developed to run as a deployment, though. So I don't think changing setup will affect its performance.
> Their product is #1 in every metric they measured, but is also missing features from those compared against. Will those features change the results?
Depending on what features will be involved into the testing. In the benchmark, all collectors are doing the same job: collecting logs, parsing JSONs, shipping log records. So they are even in used features.
Of course, the #1 product is missing features for log transformations, but these features aren't used during testing and shouldn't affect performance of other log collectors.
The bottom line of the post has "Should I switch" section explaining what's missing yet in product #1 for transparency.
The benchmark suite is available here: https://github.com/VictoriaMetrics/log-collectors-benchmark
https://victoriametrics.com/blog/log-collectors-benchmark-20...
1. The test setup is divergent from the real world (single node k8s cluster)
2. Their product is #1 in every metric they measured, but is also missing features from those compared against. Will those features change the results?
Disclaimer: I am affiliated with VictoriaMetrics.
> The test setup is divergent from the real world (single node k8s cluster)
The setup was chosen to simplify the suite, so it can be easily run anywhere. In real world, log collectors are mostly deployed as deamonsets and this is what was tested during the benchmark. The vlagent was initially developed to run as a deployment, though. So I don't think changing setup will affect its performance.
> Their product is #1 in every metric they measured, but is also missing features from those compared against. Will those features change the results?
Depending on what features will be involved into the testing. In the benchmark, all collectors are doing the same job: collecting logs, parsing JSONs, shipping log records. So they are even in used features. Of course, the #1 product is missing features for log transformations, but these features aren't used during testing and shouldn't affect performance of other log collectors.
The bottom line of the post has "Should I switch" section explaining what's missing yet in product #1 for transparency.
Using a single node cluster is not representative, full stop.