Home
Prefab helps you reverse engineer graphical interfaces from their pixels. This repository describes how you can use Prefab to analyze screenshots and output a tree structure, kind of like a DOM for a website. To see the research papers that describe this, please visit http://homes.cs.washington.edu/~mdixon/research/prefab/
This image illustrates the basic mechanism behind Prefab, wherein (1) a bitmap of an existing source window is captured, (2) the contents of the source are interpreted, (3) a modified interface is presented in a target window (with the source potentially hidden using virtual desktop methods), (4) input in the target is mapped back the source, which then (5) generates new output that is captured and used to update the target.
When Prefab interprets the contents of a screenshot, it outputs a hierarchy, where each node in the hierarchy represents an identified widget, and the root node represents the captured screenshot. This is similar to how a web page is represented with a DOM, except Prefab's hierarchies are based on pixels.
You can work on top of Prefab by operating on this hierarchy. The image above shows a portion of a hierarchy written as json, the hierarchy was recovered from the screenshot of the "Run Dialog" to the left. Prefab only sees widgets, but it does not understand what they are or how they are relevant to your application. So adding semantic information, such as naming a node as "ok button", or setting a node as "is_text", is requires additional interpretation. Thus we extensible provide tools for modifying this hierarchy and enhancing it with higher-level semantic information.
For more details, check out our papers and videos at http://homes.cs.washington.edu/~mdixon/research/prefab/index.html.
Read about how to download and install Prefab here Setting up Prefab
And check out our basic tutorial here Hello World Screenshot Editor Tutorial. This set of tutorials is based on the tasks in our UIST 2014 paper, and we will continue to update them with more instructions on setting up the software.