I see a huge accessibility opportunity for this. Gaze + voice running inside the app (with actual React state access) is way more reliable than screen-reader bolt-ons for hands-free use. Curious if you've thought about other nonverbal inputs, head nods for confirm/cancel, blink patterns, facial expressions since you already have the webcam feed.
Accessibility wasn’t the starting point, but the more I work on this the more it feels like a natural fit.
On nonverbal inputs, I’ve focused on gaze and gestures so far. I’ve thought about things like head nods or blink patterns for simple confirm/cancel, but not explored them deeply yet.
Right now the main challenge is keeping everything reliable without adding too much complexity.
Hey HN, I’ve been working on a side project called Exocor.
It’s a React SDK that lets you control your app with voice, gaze and hand gestures, no mouse or keyboard.
The idea is simple: instead of figuring out where to click, you can just look at something and say what you want.
Example:
look at a row → “open this” | say “navigate to equipment” | say “create a ticket” → it builds it
Some interactions are instant (navigation, selection), while more complex ones use an LLM and take a few seconds.
What makes it different from most “AI agents” is that it runs inside the app, not outside of it.
It has access to:
React state, routing, visible UI
So it doesn’t rely on screenshots or DOM guessing, it actually understands what’s on screen.
I originally started this thinking about environments where mouse/keyboard don’t work well (gloves, operating rooms, field work), but it’s also interesting for internal tools and dashboards.
This is v0.1 still rough in places, but the core flow works.
I see a huge accessibility opportunity for this. Gaze + voice running inside the app (with actual React state access) is way more reliable than screen-reader bolt-ons for hands-free use. Curious if you've thought about other nonverbal inputs, head nods for confirm/cancel, blink patterns, facial expressions since you already have the webcam feed.
That’s a really interesting angle.
Accessibility wasn’t the starting point, but the more I work on this the more it feels like a natural fit.
On nonverbal inputs, I’ve focused on gaze and gestures so far. I’ve thought about things like head nods or blink patterns for simple confirm/cancel, but not explored them deeply yet.
Right now the main challenge is keeping everything reliable without adding too much complexity.
Curious how you’d see this used in practice?
Hey HN, I’ve been working on a side project called Exocor.
It’s a React SDK that lets you control your app with voice, gaze and hand gestures, no mouse or keyboard.
The idea is simple: instead of figuring out where to click, you can just look at something and say what you want.
Example: look at a row → “open this” | say “navigate to equipment” | say “create a ticket” → it builds it
Some interactions are instant (navigation, selection), while more complex ones use an LLM and take a few seconds.
What makes it different from most “AI agents” is that it runs inside the app, not outside of it.
It has access to: React state, routing, visible UI
So it doesn’t rely on screenshots or DOM guessing, it actually understands what’s on screen.
I originally started this thinking about environments where mouse/keyboard don’t work well (gloves, operating rooms, field work), but it’s also interesting for internal tools and dashboards.
This is v0.1 still rough in places, but the core flow works.
GitHub: https://github.com/haelo-labs/exocor npm: https://www.npmjs.com/package/exocor
Curious what you think, especially if this feels useful or just like a gimmick.
nice job , bro
Appreciate it! Curious what you’d try this on?