A GUI-interaction aiding system for cut-and-paste operation based on image processing for the visually impaired

Abstract

In this paper we propose a method to aid a visually impaired person in the operation of a computer running a graphical user interface (GUI). It is based on image processing techniques, using images taken by a color camera placed over a Braille display. The shape of the user's hand is extracted from the image by analyzing the hue and saturation histograms. The orientation of the hand, given by an angle 0 with the vertical axis, is calculated based on central moments. The image of the hand is then rotated to a normalized position. The number of pixels in each column of the normalized image is counted, and the result is put in a histogram. By analyzing the coefficient of asymmetry of this histogram, it can be determined whether the thumb is positioned along the pointing finger, or whether it is far from the other fingers. These two positions define two states that correspond to a mouse button up or down. In this way, by rotating the hand and moving the thumb, we can emulate the acts of moving a scroll bar and depressing a mouse button, respectively. These operations can be used to perform tasks in a GUI, such as cut-and-paste, for example. Experimental results show that this method is fast and efficient for the proposed application.