Researchers used AI to turn ordinary smartwatches into devices for precise hand movement measurement

New smartwatch assistants from Cornell University and KAIST

Scientists from Cornell University (USA) and the Korea Advanced Institute of Science and Technology (KAIST) have created the WatchHand system, which turns ordinary smartwatches into full-fledged hand‑movement trackers. The system uses only the watch’s built‑in speaker and microphone, requiring no external sensors.

How WatchHand Works
1. Acoustic signal – the watch emits inaudible sound waves.
2. Echo reception – the microphone records reflected signals and creates an echo profile.
3. On‑device processing – a locally running machine‑learning algorithm converts the profile into three‑axis hand coordinates.

Thus, data is processed directly on the watch, ensuring privacy and minimal latency.

Benefits
WatchHand Indicator Traditional solutions Does not require external sensors ✔❌ (cameras, IMU sensors) Suitable for everyday use ✔❌ Scalable to millions of devices ✔❌ Local data processing ✔❌

Testing
- Participants – 40 people.
- Data – about 36 hours of gestures collected in four independent experiments.
- Platforms – several smartwatch models.
- Scenarios – various hand positions and noisy conditions.

Results showed reliable tracking of finger movements and wrist rotation even under challenging conditions.

Applications Prospects
* Controlling computers and applications via gestures.
* Interaction with AR/VR systems.
* Assisting people with limited mobility or speech.

KAIST graduate student Jivan Kim notes: “If a device has a speaker and microphone, our approach is applicable.”

Limitations and Future
* Currently works only on Android devices.
* Accuracy decreases when the user moves (e.g., while walking).

The team is actively developing improvements for mobile scenarios.

Research findings will be presented at ACM CHI 2026 on human factors in computing systems.

Key Idea

Associate Professor at Cornell University Cheng Zhang summed up: “WatchHand demonstrates how a simple software update can turn millions of existing watches into intelligent platforms for behavior tracking. This opens the way to new ways of interacting with digital devices without keyboards and mice.”