The aim of the project is to try and convert any computer screen into a touch screen using computer vision algorithms. With remote communication becoming more popular such a technology helps remove barriers to express ideas freely on a virtual whiteboard without the need for a mouse. A regular pen or any pointer object can be used to draw on your laptop screen, without any ink! The algorithm tracks the position of your pointer and sends signals to the browser to mimic a mouse move.
Below is a basic screencast showing how the custom detection model detects the whiteboard (compute screen) and a marker on a set of images completely in browser using tensorflow-js. It can be reproduced on your machine by hosting this version of the project using a simple HTTP server.
For a live demo of the latest app & model visit https://invisible-pen.vercel.app/
- Visit https://bit.ly/invisible-pen on your phone and give permission to access your webcam. The phone tracks your hand movements and sends them to your laptop wirelessly via websockets.
- Install conda and create an environment, say
invipen, using commands listed here - Install uWSGI for speedy socket based wireless communication between the phone and laptop.
brew install uwsgiworks on Mac as described at http://macappstore.org/uwsgi/ - Open a new terminal session for the changes to kick in
- Install gevent for high performance async computing with uWSGI. It depends on libraries like
libevent, which can be installed viabrew install libeventon Mac - Open a new terminal session for the changes to kick in
- Generate certificates and place the .crt, .pem, .key files in a new folder called app/server/certs/trusted-openssl
- Start server
- This website is eventually launched on a mobile device, which tracks the user's hand position
npm run devinapp/clientfolder which starts a react server on http://0.0.0.0:3000
- Install ngrok and forward react server's port using
ngrok http 3000 - Access the pasted HTTPS from the URL pasted by ngrok. You may have to turn off any VPN gateway to access the site. HTTPS URL is needed to access webcam on phone and ngrok helps with this tunneling, and is convinient while development as it also supports hot reload facility
conda activate invipen- Run
python app/index.pyto start the mouse controller server inside the activated virtual environment - Note: remember to remove gevent or eventlet to start the server in development mode as shared at https://flask-socketio.readthedocs.io/en/latest/#embedded-server