I've been thinking about creating an app or website that uses a camera to recognize sign language, convert it into text on the screen, and read the result aloud automatically. I'm imagining something similar to real-time visual translation tools, but focused on signed communication and possibly using motion tracking or computer vision.
My programming experience is limited to Python, so I'm not sure what technologies, frameworks, or learning steps I should begin with. I'm also wondering whether this idea is realistic for a beginner, how different sign languages would affect the project, and whether I would need to collect and train a large dataset. I'm willing to start with a small prototype and build it gradually, but I'd appreciate advice on a sensible first version.
4 Answers
This is a very ambitious project, but you can make it manageable by starting with a tiny prototype. Use Python and OpenCV to access the camera, then try recognizing only a few static hand signs. Once that works, add text output, and later experiment with text-to-speech. Don’t aim for full real-time conversation right away—prove each piece separately first.
The recognition part will probably require machine learning. You could investigate hand-tracking tools and use the detected hand landmarks as input to a classifier. OpenCV can help with the camera and image processing, but the difficult part is creating or finding a large, varied dataset that includes different people, lighting conditions, hand sizes, and signing styles.
Before building too much, decide what version of the project you want: a desktop program, a mobile app, or a web app. Also choose one specific sign language, since sign languages differ between countries and aren’t just universal hand gestures. For a first project, recognizing a small vocabulary from recorded clips is much more realistic than translating continuous signing in real time.
You should also research existing projects and tutorials first. Similar systems have already been demonstrated, so you may be able to learn from their approach or use an existing hand-tracking or speech library instead of creating every component yourself. Even if the final result is rough, building a limited version would teach you a lot.
I didn’t realize there were already examples of this. I’ll look at them to understand how the problem is usually approached.
That makes sense. I’ll start with just a few signs instead of trying to build the complete app immediately.