screen-complete

Screen complete is a proof of concept universal screenshot-based text completion tool. Inspired by tools like cursor and github copilot, it allows you to fill in arbitrary selected text on your screen using a hot key.

Quick Start (Windows/OSX)

1. Download the latest version from the releases page
2. Create a config file screen_complete.yml with openai_api_key: ... (in the same directory)
3. Run screen-complete (OSX may require enabling screen recording permissions)

The UI/UX is extremely minimal, on a page you want to fill in text:

1. Place your cursor where you want to type text (or select text you want to replace)
2. Move your mouse (without clicking) the top left corner of the applicable window
3. Hold down Ctrl+Q
4. Move your cursor to the bottom right corner of the applicable window
5. Release Ctrl+Q

Examples

Description	Image
Writing text in a google doc
Filling in the title of a GitHub issue
Drafting a reddit comment
Drafting a slack reply

Configuration

This tool currently supports OpenAI and Azure OpenAI. Only fields for Azure OR OpenAI are required.

Via Environment Variables

AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=...
AZURE_OPENAI_DEPLOYMENT=...
OPENAI_API_KEY=...
OPENAI_MODEL=... (optional)

Via screen_complete.yml

azure_openai_api_key: ...
azure_openai_endpoint: ...
azure_openai_deployment: ...
openai_api_key: ...
openai_model: ... (optional)

Building

Windows

1. Download the Mingw, then set system environment variables C:\mingw64\bin to the Path
2. go build -o screen-complete.exe cmd\screen_complete\main.go

MacOS

1. xcode-select --install
2. go build -o screen-complete cmd\screen_complete\main.go

Privacy

Your screen is only captured and sent to OpenAI when you release the hot key.