I prompted VLM Run’s visual agent Orion to segment all of the blue bouldering holds, and it did a good job! It is interesting that now we can prompt VLMs to segment all of the holds, rather than creating a new dataset from scratch to train a model.
With holds detection + pose estimation, I can show how each hold gets activated as a hand or foot uses it. Once we touch the final hold with both hands, the route is completed, and I show the overall path of my torso midpoint.
A tool like this could help climbers understand their movement better. I’m still very much a beginner at bouldering, so it would be great to get quantitative feedback.
There are definitely things to improve, but overall I’m encouraged by this first demo.
I prompted VLM Run’s visual agent Orion to segment all of the blue bouldering holds, and it did a good job! It is interesting that now we can prompt VLMs to segment all of the holds, rather than creating a new dataset from scratch to train a model.
With holds detection + pose estimation, I can show how each hold gets activated as a hand or foot uses it. Once we touch the final hold with both hands, the route is completed, and I show the overall path of my torso midpoint.
A tool like this could help climbers understand their movement better. I’m still very much a beginner at bouldering, so it would be great to get quantitative feedback.
There are definitely things to improve, but overall I’m encouraged by this first demo.
Models used:
- VLM Run’s Orion for segmentation (https://www.vlm.run/)
- ViTPose+ Huge for pose estimation (via Hugging Face)
- RT-DETR for person detection (via Hugging Face)
Shoutout to Daniel Reiff and his bouldering + computer vision project for the inspiration!
Link to Daniel Reiff's bouldering + computer vision blog: https://blog.roboflow.com/bouldering/