Towards hand gesture-based control of virtual keyboards for effective communication

Abstract Individuals with disabilities are often restricted from performing certain daily life activities that the non-disabled persons would normally perform. One of such restrictions is regarding daily life communication that would require listening, speaking, and sometimes movement of one’s body part often referred to as a gesture. Thus, disabled persons without fingers and cannot speak fluently finds it challenging to communicate effectively during daily life activities. Therefore, a virtual Graphical User Interface (GUI)-based keyboard communication tool is developed in this study to aid such individuals in their daily life activities. The developed virtual GUI-based keyboard can support the different directional movement of the hand including up, down, left, and right, that were coded as different communication signs. The choice “select” can be done by turning the wrist up and “erasing” by turning the wrist down. Every hand movement is read and translated by a leap motion sensor. There are two kinds of GUI used in this research, the first GUI with the Alphabet key is equipped with a pictorial word while the second GUI Qwerty comes with a pictorial word. Experimental results concerning the effectiveness of the typing effectiveness show the alphabet GUI has a faster typing time than the Qwerty GUI with an average difference of fewer than 15 seconds.