Shawn Zhao and Brian A. Barsky

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2024-179

August 12, 2024

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-179.pdf

Upon its introduction in 1968, the computer mouse arrived as a novel input device for controlling a computer pointer on a display. Even now, over 50 years later, the mouse and keyboard remain the ubiquitous combination of input devices for operating a computer. In those years, computers have undergone drastic changes to become increasingly more advanced and convenient for users, and while mice and keyboards themselves have undergone significant advancements in design, comfort, and sensing, the basic mechanical method for their usage as input devices remains the same. With the increasing prevalence of computers, there has likewise been increasing research into alternative input methods, including gestures. While gesture-based cursor control and keyboard input have been previously explored, we leverage a common hardware peripheral, the camera, and combine its capabilities with vision processing techniques to create a more natural handed input system.

We further develop an existing two-handed gesture control system to replace the functionality of a traditional mouse and keyboard for those who may find difficulty in operating those devices. The system uses a computer’s webcam to detect hand movements and recognize gestures that correspond to cursor movements and other mouse and keyboard actions without the need for additional hardware. We add further features such as in-air writing and multi-frame gestures. Through evaluation of the accuracy of our gesture and writing recognition after the incorporation of trained datasets, we confirm that our system is a functional alternative control system to the traditional mouse-and-keyboard input methods, and have created a bundled executable for users to download and further test.

Advisors: Brian A. Barsky

---

BibTeX citation:

@mastersthesis{Zhao:EECS-2024-179,
    Author= {Zhao, Shawn and Barsky, Brian A.},
    Title= {Advancing Assistive Mouse and Keyboard Control Scheme Using Two-Handed Gesture Recognition for Human-Computer Interaction},
    School= {EECS Department, University of California, Berkeley},
    Year= {2024},
    Month= {Aug},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-179.html},
    Number= {UCB/EECS-2024-179},
    Abstract= {Upon its introduction in 1968, the computer mouse arrived as a novel input device for controlling a computer pointer on a display. Even now, over 50 years later, the mouse and keyboard remain the ubiquitous combination of input devices for operating a computer. In those years, computers have undergone drastic changes to become increasingly more advanced and convenient for users, and while mice and keyboards themselves have undergone significant advancements in design, comfort, and sensing, the basic mechanical method for their usage as input devices remains the same. With the increasing prevalence of computers, there has likewise been increasing research into alternative input methods, including gestures. While gesture-based cursor control and keyboard input have been previously explored, we leverage a common hardware peripheral, the camera, and combine its capabilities with vision processing techniques to create a more natural handed input system.

We further develop an existing two-handed gesture control system to replace the functionality of a traditional mouse and keyboard for those who may find difficulty in operating those devices. The system uses a computer’s webcam to detect hand movements and recognize gestures that correspond to cursor movements and other mouse and keyboard actions without the need for additional hardware. We add further features such as in-air writing and multi-frame gestures. Through evaluation of the accuracy of our gesture and writing recognition after the incorporation of trained datasets, we confirm that our system is a functional alternative control system to the traditional mouse-and-keyboard input methods, and have created a bundled executable for users to download and further test.},
}

EndNote citation:

%0 Thesis
%A Zhao, Shawn 
%A Barsky, Brian A. 
%T Advancing Assistive Mouse and Keyboard Control Scheme Using Two-Handed Gesture Recognition for Human-Computer Interaction
%I EECS Department, University of California, Berkeley
%D 2024
%8 August 12
%@ UCB/EECS-2024-179
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-179.html
%F Zhao:EECS-2024-179