Prototype | Gesture Recognition for Video Conferencing (Zoom and Teams)

With so many virtual meetings during this global pandemic that causes us all to work remotely, I built a Snap Lens with the goal of solving some common challenges with webcam communication.

Since this is a Snap Lens and running in the background during a video call, the interaction with the Lens is set of Gestures.

  1. Settings Menu

The menu controls global settings like meeting duration/end-time, and whether or not other advanced gestures have been enabled.

Caption: Gesture Based Menu shows up on top of the camera feed.

Menu navigation is performed via the “Index Finger Gesture”.

Caption: Index Finger Gesutre

Menu selection is performed via the “Thumb Gesture“.

Caption: Thumb Gesutre

Menu Enable/Disable is toggled by the “Victory Gesture” (peace sign).

Caption: Victory Gesutre

Learning: One of the things I quickly discovered is the importance of not using gestures that aren’t accidently used during meetings. For example, I originally used the “Open Gesture” for enabling the menu. However, I found that waving hello/goodbye to meeting participants would inadvertently activate the menu.

Caption: Open Gesutre

Consequently, the index_finger gesture work great for menus because it is distinctive, memorable, and can be disabled when the menu is deactivated to prevent accidently use.

2. Non-menu Gestures

Stonks/Not Stocks – The goal here was to quickly show agreement/disagreement, without meeting interrupting the conversation. With a “Thumbs Up” for agreement (Stonks) and a “Horns” for disagreement (Not Stonks), the background image changes.

When not gesturing, the background runs through a series of images that visually represent the duration of the meeting (sunrise for start of the meeting throuhg sunset for end of meeting), giving a visual cue as to how long the meeting has been going.

In addition, a counter in the center of the screen indicates the time remaining, with the goal of preventing meeting from running over time.

3. Be Right Back

And finally, for those times when you have to step away from your monitor to get a cup of coffee, you can enable the “Be Right Back” card.

Be Right Back Image
Caption: Be Right Back

To see it all in action, you can check out this video recording made through the Teams meeting capture.

Conclusion: It was a fun prototype, but a bit clunky. But if you want to try it yourself, just grab Snap Camera and you can find the Lens here.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.