HomeProjectsInformation and Communication TechnologiesThe World First Multi-language Online Sign Language Game with Automatic Sign Language Recognition

The World First Multi-language Online Sign Language Game with Automatic Sign Language Recognition

Project Quick Facts

Principal Investigator

  • Prof. TANG Wai Lan Gladys

    Department of Linguistics and Modern Languages

  • Funding

    The Nippon Foundation

  • Collaboration

    The Nippon Foundation, Google, The Sign Language Research Center of Kwansei Gakuin University

SignTown is a web-based, real-time sign recognition game, where users take control of an avatar to communicate with other characters in sign language to get things done. The game employed an end-to-end open source platform for machine learning, TensorFlowTM, to train the sign language recognition model. It enables users to learn and express themselves in sign language and to get feedback about their signing simply with a computer and a webcam. SignTown is designed around various common categories and themes. It also provides information about deaf culture in various regions. Users can enjoy the experience of signing into Hong Kong sign language and Japanese sign language in the game.

The opening screen of the SignTown
Users collect different items by selecting and performing the sign to the camera.
SignTown provides information about deaf culture in various regions.
Multiple machine learning models are used to track the phonetic features of sign language.

Uniqueness and Competitive Advantages:

  • Can track facial expressions and non-manual features without the need for special equipment or specific settings
  • Can switch between HKSL and JSL in the game
  • More accurate results on sign recognition

Do you like our project?

MORE TO EXPLORE