readme.longer.md

Napkinmatic Spatial Computing Platform

Napkinmatic is a robust spatial computing platform connecting the real world to AI and back again in AR. The name derives from how the platform can turn your proverbial napkin sketch wireframe into a working website or app (GPT4 multi-modal), with several design sets to choose from (ControlNet), all augmented back on the napkin, so that you can interface with the app "in real time, in reality". The platform features an opensource framework which allows anyone to swap out the model with the latest SOTA by updating endpoints, as well as a built-in javascript engine for AppMode.

Examples

SOTA Endpoint Usage and Chaining

Controlnet-Scribble to turn sketches into masterpieces

graph LR
I1[Prompt]-->B[**ControlNet**]
I3[Image]-->B-->C[vivid napkin design!]

Loading

Prompt2Prompt for Quick Design Iterations

graph LR
I1[Prompt]-->I2[**Prompt2Prompt**]-->B[**ControlNet**]
I3[Image]-->B-->C[vivid napkin design!]

Loading

TEXTurePaper for automatic texturing of 3D Meshes

graph LR
J[AppMode invoke]-->K[3D Sticky Notes Mode]
K-->L[3D model creation rectangle]
K-->M[Texture creation buttons]

M-->P1[Prompt]-->Q[Model Creation Method **FacedSculpt**]
M-->P3[Image]-->Q-->C 

L-->I1[Prompt]-->B[**TEXTurePaper**]
L-->I3[3D Model]-->B-->C[original textured mesh!]

Loading

Using the built-in JS engine to add interactivity

Connect UI with 3D models

Raycasting to Napkin Surface

We raycast paintball splatters on "napkin" in our GDC23 AIGames Hackathon entry PaperSet Multiplayer and Instagram short here and it won 2 prizes, enough to cover SF rent for a month!