Add the following snippet to your HTML:
An attempt to develop a Gaming Controller for Persons with Disabilities, that works with movement of face/head to control movements in games Polyester Capacitor
Read up about this project on
An attempt to develop a Gaming Controller for Persons with Disabilities, that works with movement of face/head to control movements in games
Gaming can be challenging for people with disabilities. Even playing simple games can be difficult for them, as most gaming controller are not designed with their needs in mind.
In this project, I will attempt to develop a gaming controller which can be used with face/head movement, it's handsfree design, minimal input for playing simple online games/in browser games.
In a nutshell, here is how this device works:
Person Sensor is a camera module with a pre programmed microcontroller with computer vision face recognition model.
The prototype is build according to the following schematic
This is the actual hardware prototype
Hardware Description (Bill of Materials)
Person Sensor Useful Sensor: Person sensor as the name suggests, is a person's face detection sensor using computer vision. It is preprogrammed with face detection algorithm, can track face movement in it's field of view. Person sensor does not allow access to raw image data, instead it returns information over I2C interface about face detected or not, number of faces, face position etc.
That's why it's easy to use with microcontrollers without knowing details about computer vision or face recognition AI.
White LED Lamps with Diffusers: Person sensor requires enough lighting on the user's face to work properly. That's why I added two white LEDs with diffusers. They are driven with a PWM pin to adjust the brightness.
Make sure there is enough light on the face of the person or the camera will lose face tracking
Beetle Board (Atmega32U4 mcu):This is a very small size Arduino Leonardo equivalent board with Atmega32U4 microcontroller. It is used to drive the OLED display1306, read Person Sensor data over I2C interface, drive the servo motor angle, drive the LED lamps brightness, sense the Photo Transistor beam interruption with ADC, drive the vibration motor, emulate HID USB keyboard on host PC and send press key commands based on face position in front of person sensor. Basically this mcu runs the whole device.
OLED Display: A small, low cost, I2C based OLED display for user interface with 1306 display controller. This display is used to show the face position in the field of view, if no face is detected it will also inform user about that.
I found this display very useful tool for debugging the code while working with person sensor.
Dot Laser Diodes:These are tiny LED based semiconductor laser diodes that can produce monochrome beam of Red light. Laser beam is used here to shine on Photo Transistors (LDR or Diode will work too). The plan is to develop two "Zero Pressing Force" Air Buttons. Air buttons are just interruption of Laser beams with finger or any object to sense user input.
Photo Transistor: Two photo transistors are used in this device to make to Air Tap buttons (basically interrupt the beam to push switch). These photo transistors are connected to two ADC inputs of the microcontroller. When the laser beam is interrupted, then the transistors are nearly turn off causing ADC to sense Gnd potential. This change is registered as user 'button press'
Vibration Motors: I have added two vibration motors for haptic feedback, since the optical interrupt buttons (the Laser beam and Photo Transistor getting interrupted by finger or object) don't have any force feeling, haptic will give the feeling of something happening.
Servo Motor: The purpose of this servo motor is to align the camera initially with the face in center position. While the hardware is connected, I have not implemented this feature yet in the firmware.
Download the latest Arduino IDE from here. IDE version 2.1 and latest are recommended for better compatibility but any version from 1.8+ or above will work.
Tools > Board > Arduino AVR Board > Leonardo
Add Libraries to Arduino IDE
Following libraries are necessary for this project, please make sure you have included them in Arduino IDE. Details already available on GitHub links
U8G2 Library for Display Driver
Useful Sensor Library for Person Sensor
Next, compile and upload the code *attached below) into Beetle board and get ready to test the concept.
Here are some of the testing videos, yep, you guessed it ! There is a bit of lag because of 7 Hz only polling rate of the camera on face tracking !
Still, the idea is what I am trying to demo here, implementing this idea on a single board computer with at faster camera can solve the lag !
This project is far from complete. I am rushing this at the last moment. The firmware is still under development. Instead of considering this a finished project, think of it as a prototype concept.
I want to make more complex gaming controller that can take other forms of input from user like making certain sounds with mouth (and detect them with sound recognition) as game controlling input. Developing gaming controller for people with disabilities is not easy, as everyone's need is different.
Let's keep exploring new ideas and try them out, something wonderful might happen and we will have the ultimate gaming controller that everyone can use
Rj 45 Connectors Hackster.io, an Avnet Community © 2024