Making Progress...
Our team of engineers has placed many milestones while trying to make the world a better place for hearing impaired people. We have completed all engineering aspects of our product with full integrity and hard work.
Speech-to-Text
Apphear converts speech to text via cloud computing using Google’s Cloud Speech API. It provides streaming audio speech conversion using its streaming mode translation, which is the mode that is used for this project. To be able to use the API in this mode we used gRPC protocol. gRPC is an open source remote protocol call system developed by Google. The API is used in Winform Application, written in C# with the following three steps: Configuration, Streaming, Printing.
Natural Language Processing
Our software uses open source libraries as well as commercially available API’s. The source we use for natural language processing is zemberek-nlp which is available as an open source for developers. The library has been worked on for several years and is currently being developed by Ahmet A. Akın where the code is available at his GitHub page. Zemberek-NLP is an offline API, which is desirable. The code is written in Java programming language. By using the methods provided by zemberek NLP API, our team of engineers were able to analyse the morphological structure of sentences hence, increasing the accuracy of the 3D animations.
3D Animation
Everything processed by our software is delivered to the user screen by using animations. Every gesture in the animations is generated from scratch. For each word, an actor is scanned with a IR camera while doing the gestures and the footage is used to play an animation figure. Any mismatch between the movements of the animation figure and the actor is adjusted manually. Sign language is a delicate language thus requires precision.