Skip to content

Latest commit

 

History

History
758 lines (623 loc) · 72.2 KB

README.md

File metadata and controls

758 lines (623 loc) · 72.2 KB
Raw

Awesome-Sketch-Based-Applications

A collection of sketch based applications.

Feel free to create a PR or an issue.

Outline

0. Survey

Paper Source Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey TPAMI 2022 [code]

1. Sketch Based Image Synthesis

1.1 Automatic Synthesis

  • Natural Image or Object
Paper Source Code/Project Link
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis CVPR 2018 [code]
Image Generation from Sketch Constraint Using Contextual GAN ECCV 2018 [code] [project]
Multi-Instance Sketch to Image Synthesis With Progressive Generative Adversarial Networks IEEE Access 2019
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation ICCV 2019 [code] [project]
SketchyCOCO: Image Generation from Freehand Scene Sketches CVPR 2020 [code]
Unsupervised Sketch-to-Photo Synthesis ECCV 2020 [code] [project]
Sketch Your Own GAN ICCV 2021 [code] [webpage]
Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis WACV 2022 [code]
Customizing GAN Using Few-Shot Sketches ACM MM 2022
DiffSketching: Sketch Control Image Synthesis with Diffusion Models BMVC 2022 [code]
MaskSketch: Unpaired Structure-guided Masked Image Generation CVPR 2023 [project] [code]
Picture that Sketch: Photorealistic Image Generation from Abstract Sketches CVPR 2023 [project] [code]
  • Human Face / Portrait / Body
Paper Source Code/Project Link
LinesToFacePhoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network ACM MM 2019
DeepFaceDrawing: Deep Generation of Face Images from Sketches SIGGRAPH 2020 [project] [code]
DeepFacePencil: Creating Face Images from Freehand Sketches ACM MM 2020 [project] [code]
Controllable Sketch-to-Image Translation for Robust Face Synthesis TIP 2021
DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN TVCG 2022
DeepPortraitDrawing: Generating Human Body Images from Freehand Sketches C&G 2023
Semantics-Preserving Sketch Embedding for Face Generation TMM 2023 [project] [code]
Parsing-Conditioned Anime Translation: A New Dataset and Method TOG 2023 [code]
Sketch2Human: Deep Human Generation with Disentangled Geometry and Appearance Constraints TVCG 2024
  • 3D image
Paper Source Code/Project Link
3D-aware Conditional Image Synthesis CVPR 2023 [project] [code]

1.2 Style-conditioned

Paper Source Code/Project Link
TextureGAN: Controlling Deep Image Synthesis with Texture Patches CVPR 2018 [code]
Multimodal Unsupervised Image-to-Image Translation ECCV 2018 [code]
SketchPatch: Sketch Stylization via Seamless Patch-level Synthesis SIGGRAPH Asia 2020
Self-Supervised Sketch-to-Image Synthesis AAAI 2021 [code]
CoGS: Controllable Generation and Search from Sketch and Style ECCV 2022
Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model WACV 2023 [project] [code]
DemoCaricature: Democratising Caricature Generation with a Rough Sketch CVPR 2024 [project] [code]
SMFS-GAN: Style-Guided Multi-class Freehand Sketch-to-Image Synthesis CGF 2024

1.3 Text-conditioned

Paper Source Code/Project Link
Sketch-Guided Text-to-Image Diffusion Models SIGGRAPH 2023 [project]
Adding Conditional Control to Text-to-Image Diffusion Models ICCV 2023 [code]
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild NeurIPS 2023 [code] [project]
Stable Doodle (ClipDrop) [demo]
CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing PG 2024
Block and Detail: Scaffolding Sketch-to-Image Generation UIST 2024
One-Step Image Translation with Text-to-Image Models arxiv 24.03 [code]
It's All About Your Sketch: Democratising Sketch Control in Diffusion Models CVPR 2024 [code]
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models AAAI 2024 [code]
Training-Free Sketch-Guided Diffusion with Latent Optimization arxiv 24.09
Scribble-Guided Diffusion for Training-free Text-to-Image Generation arxiv 24.09 [code]

2. Sketch Based Image Editing

  • Arbitrary Image
Paper Source Code/Project Link
Free-Form Image Inpainting with Gated Convolution (DeepFill v2) ICCV 2019 [project] [code]
Fashion Editing with Multi-scale Attention Normalization 1906.00884
DeFLOCNet: Deep Image Editing via Flexible Low-level Controls CVPR 2021 [code]
DeepSIM: Image Shape Manipulation from a Single Augmented Training Sample ICCV 2021 [code] [project]
SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches CVPR 2022 [code] [project]
Draw2Edit: Mask-Free Sketch-Guided Image Manipulation ACM MM 2023 [code]
Towards Interactive Image Inpainting via Robust Sketch Refinement TMM 2024 [code] [project]
  • Human Face / Portrait / Hair
Paper Source Code/Project Link
FaceShop: Deep Sketch-based Face Image Editing SIGGRAPH 2018 [project]
CaricatureShop: Personalized and Photorealistic Caricature Sketching TVCG 2018
Sparse, Smart Contours to Represent and Edit Images CVPR 2018 [project]
Example-Guided Style-Consistent Image Synthesis from Semantic Labeling CVPR 2019 [code]
SC-FEGAN: Face Editing Generative Adversarial Network With User's Sketch and Color ICCV 2019 [code]
Intuitive, Interactive Beard and Hair Synthesis with Generative Models CVPR 2020
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches ECCV 2020 [code] [project]
DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control SIGGRAPH 2021 [code] [project]
SketchHairSalon: Deep Sketch-based Hair Image Synthesis SIGGRAPH Asia 2021 [project]
Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing ECCV 2022 [code] [project]
NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields SIGGRAPH Asia 2022 [project]
SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields SIGGRAPH 2023
  • Anime Editing
Paper Source Code/Project Link
AniFaceDrawing: Anime Portrait Exploration during Your Sketching SIGGRAPH 2023 [project]
  • Video Editing
Paper Source Code/Project Link
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos SIGGRAPH 2022 [project]

3. Sketch Based Image Retrieval (SBIR)

  • Object-level
Paper Source Code/Project Link
Sketch Me That Shoe CVPR 2016 [code-caffe] [code-tf] [project]
Deep Multi-task Attribute-driven Ranking for Fine-grained Sketch-based Image Retrieval BMVC 2016
Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval CVPR 2017 [code]
Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval ICCV 2017 [project]
Zero-Shot Sketch-Image Hashing CVPR 2018
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval CVPR 2018
Generative Domain-Migration Hashing for Sketch-to-Image Retrieval ECCV 2018 [code]
A Zero-Shot Framework for Sketch Based Image Retrieval ECCV 2018 [code]
Generalising Fine-Grained Sketch-Based Image Retrieval CVPR 2019
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval CVPR 2019 [project] [code]
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search CVPR 2019
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval CVPR 2019 [code]
Learning Structural Representations via Dynamic Object Landmarks Discovery for Sketch Recognition and Retrieval TIP 2019
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval ICCV 2019
TC-Net for iSBIR: Triplet Classification Network for Instance-level Sketch Based Image Retrieval ACM MM 2019
Sketch-Based Image Retrieval With Multi-Clustering Re-Ranking TCSVT 2019
Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval TCSVT 2019
Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network AAAI 2020
Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval CVPR 2020 [code]
Solving Mixed-modal Jigsaw Puzzle for Fine-Grained Sketch-Based Image Retrieval CVPR 2020
Fine-Grained Instance-Level Sketch-Based Image Retrieval IJCV 2020
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval CVPR 2021
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch-Based Image Retrieval CVPR 2021 [code]
DLA-Net for FG-SBIR: Dynamic Local Aligned Network for Fine-Grained Sketch-Based Image Retrieval ACM MM 2021
Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax ACM MM 2021 [code]
Relationship-Preserving Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval ACM MM 2021
Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval TPAMI 2021 [project]
TVT: Three-Way Vision Transformer through Multi-Modal Hypersphere Learning for Zero-Shot Sketch-Based Image Retrieval AAAI 2022
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval CVPR 2022 [code]
Sketch3T: Test-time Training for Zero-Shot SBIR CVPR 2022
Augmented Multi-Modality Fusion for Generalized Zero-Shot Sketch-based Visual Retrieval TIP 2022 [code]
Adaptive Fine-Grained Sketch-Based Image Retrieval ECCV 2022 [code]
Conditional Stroke Recovery for Fine-Grained Sketch-Based Image Retrieval ECCV 2022 [code]
A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch ECCV 2022 [code] [project]
Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval ACM MM 2022 [code]
Prototype-based Selective Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval ACM MM 2022
DLI-Net: Dual Local Interaction Network for Fine-Grained Sketch-Based Image Retrieval TCSVT 2022 [code]
Data-Free Sketch-Based Image Retrieval CVPR 2023 [code]
Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style CVPR 2023 [code]
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR CVPR 2023 [project]
CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not CVPR 2023 [project]
Cross-Domain Alignment for Zero-Shot Sketch-Based Image Retrieval TCSVT 2023
Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval AAAI 2023
Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers CVPR 2024 [project]
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval CVPR 2024 [project]
How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval? CVPR 2024 [project]
Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval AAAI 2024
Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions AAAI 2024 [project]
Freeview Sketching: View-Aware Fine-Grained Sketch-Based Image Retrieval ECCV 2024
Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning ECCV 2024 [code]
Zero-Shot Sketch Based Image Retrieval via Modality Capacity Guidance IJCAI-24 [code]
  • Scene-level
Paper Source Code/Project Link
SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches ECCV 2020
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input CVPR 2022
SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval using Adaptive GCNs TIP 2022
FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context ECCV 2022 [code] [Dataset]
Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision AAAI 2023
FreestyleRet: Retrieving Images from Style-Diversified Queries ECCV 2024 [code]
  • Video Retrieval
Paper Source Code/Project Link
Fine-Grained Instance-Level Sketch-Based Video Retrieval TCSVT 2020
Fine-Grained Video Retrieval with Scene Sketches TIP 2023 [project]

4. Sketch Based 3D Shape Retrieval

Paper Source Code/Project Link
Sketch-based 3D Shape Retrieval using Convolutional Neural Networks CVPR 2015
Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval AAAI 2016
Deep Correlated Metric Learning for Sketch-based 3D Shape Retrieval AAAI 2017
Learning Barycentric Representations of 3D Shapes for Sketch-based 3D Shape Retrieval CVPR 2017
Deep Cross-modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-based 3D Shape Retrieval ECCV 2018
Unsupervised Learning of 3D Model Reconstruction from Hand-Drawn Sketches ACMMM 2018
Towards 3D VR-Sketch to 3D Shape Retrieval 3DV 2020 [code] [project]
Uncertainty Learning for Noise Resistant Sketch-Based 3D Shape Retrieval TIP 2021
Towards Fine-Grained Sketch-Based 3D Shape Retrieval TIP 2021
Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval AAAI 2022
Retrieval-Specific View Learning for Sketch-to-Shape Retrieval TMM 2023
Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval AAAI 2023 [code]
Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting ICCV 2023

5. Sketch Based 3D Shape Modeling

  • Free-hand sketch
Paper Source Code/Project Link
3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks 3DV 2017 [code] [project]
Shape Synthesis from Sketches via Procedural Models and Convolutional Networks TVCG 2017
DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling SIGGRAPH 2017 [project] [code]
BendSketch: Modeling Freeform Surfaces Through 2D Sketching SIGGRAPH 2017 [project]
Robust Flow-Guided Neural Prediction for Sketch-Based Freeform Surface Modeling SIGGRAPH Asia 2018 [code] [project]
DeepSketchHair: Deep Sketch-based 3D Hair Modeling TVCG 2019
Interactive Liquid Splash Modeling by User Sketches SIGGRAPH Asia 2020
Monster Mash: A Single-View Approach to Casual 3D Modeling and Animation SIGGRAPH Asia 2020 [project] [code] [demo]
SAniHead: Sketching Animal-like 3D Character Heads Using a View-surface Collaborative Mesh Generative Network TVCG 2020
Towards Practical Sketch-Based 3D Shape Generation: The Role of Professional Sketches TCSVT 2020
Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches CVPR 2021
Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches ICCV 2021
Real-time Skeletonization for Sketch-based Modeling SMI 2021 [code]
Sketch2Pose: Estimating a 3D Character Pose from a Bitmap Sketch SIGGRAPH 2022 [project] [code]
Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch TVCG 2022
Structure-aware Editable Morphable Model for 3D Facial Detail Animation and Manipulation ECCV 2022 [code]
SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling ECCV 2022 [code]
Deep Reconstruction of 3D Smoke Densities from Artist Sketches EG 2022 [project] [code]
A Diffusion-ReFinement Model for Sketch-to-Point Modeling ACCV 2022 [code]
RaBit: Parametric Modeling of 3D Biped Cartoon Characters with a Topological-consistent Dataset CVPR 2023 [project]
SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling TVCG 2023 [project] [code]
Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches TMM 2023
GA-Sketching: Shape Modeling from Multi-View Sketching with Geometry-Aligned Deep Implicit Functions PG 2023 [code]
Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes CVPR 2024 [project] [code]
SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling EG 2024 [code] [project]
3D Reconstruction from a Single Sketch via View-dependent Depth Sampling TPAMI 2024 [code]
S2TD-Face: Reconstruct a Detailed 3D Face with Controllable Texture from a Single Sketch ACM MM 2024 [code]
Sketch2Vox: Learning 3D Reconstruction from a Single Monocular Sketch ECCV 2024
  • Scene Generation
Paper Source Code/Project Link
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches arxiv 24.08 [webpage]
GroundUp: Rapid Sketch-Based 3D City Massing ECCV 2024 [code]
External Knowledge Enhanced 3D Scene Generation from Sketch ECCV 2024
  • NeRF
Paper Source Code/Project Link
Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation arxiv 24.01
SKED: Sketch-guided Text-based 3D Editing ICCV 2023 [project] [code]
SketchDream: Sketch-based Text-to-3D Generation and Editing SIGGRAPH 2024 [project]
  • CAD sketch
Paper Source Code/Project Link
Lifting Freehand Concept Sketches into 3D SIGGRAPH Asia 2020 [project] [code]
Sketch2CAD: Sequential CAD Modeling by Sketching in Context SIGGRAPH Asia 2020 [project] [code]
Free2CAD: Parsing Freehand Drawings into CAD Commands SIGGRAPH 2022 [project] [code]
Symmetry-driven 3D Reconstruction From Concept Sketches SIGGRAPH 2022 [project]
Piecewise-smooth Surface Fitting Onto Unstructured 3D Sketches SIGGRAPH 2022 [project]
Reconstruction of Machine-Made Shapes from Bitmap Sketches SIGGRAPH Asia 2023 [project]
CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention CVPR 2024
SfmCAD: Unsupervised CAD Reconstruction by Learning Sketch-based Feature Modeling Operations CVPR 2024 [code]

6. Sketch Based Garment Design

Paper Source Code/Project Link
Sketching Folds: Developable Surfaces from Non-Planar Silhouettes TOG 2015 [project]
FoldSketch: Enriching Garments with Physically Reproducible Folds SIGGRAPH 2018 [project]
Learning a Shared Shape Space for Multimodal Garment Design SIGGRAPH Asia 2018 [project]
Garment Ideation: Iterative view-aware sketch-based garment modeling 3DV 2022 [code]
Controllable Visual-Tactile Synthesis ICCV 2023 [project] [code]
Controllable Garment Image Synthesis Integrated with Frequency Domain Features PG 2023
Toward Intelligent Interactive Design: A Generation Framework Based on Cross-domain Fashion Elements ACM MM 2023
FashionDiff: A Controllable Diffusion Model Using Pairwise Fashion Elements for Intelligent Design ACM MM 2023
TexControl: Sketch-Based Two-Stage Fashion Image Generation Using Diffusion Model NICOGRAPH 2024

7. Sketch Completion

  • Sketch completion
Paper Source Code/Project Link
SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network CVPR 2019
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches BMVC 2020 [code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt CVPR 2020 [code]
Generative Sketch Healing IJCV 2022
  • Sketch gap / connectivity detection
Paper Source Code/Project Link
Joint Gap Detection and Inpainting of Line Drawings CVPR 2017 [project] [code]
Detecting Viewer-Perceived Intended Vector Sketch Connectivity SIGGRAPH 2022 [project]

8. Sketch Restoration, Retargeting and Inpainting

  • Manga
Paper Source Code/Project Link
Exploiting Aliasing for Manga Restoration CVPR 2021 [webpage] [code]
Seamless Manga Inpainting with Semantics Awareness SIGGRAPH 2021 [webpage] [code]
Screentone-Preserved Manga Retargeting arxiv 22.03
Manga Rescreening with Interpretable Screentone Representation arxiv 23.06
  • Hand-Drawn Drawings
Paper Source Code/Project Link
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator CVPR 2023

9. Sketch and Shading

Paper Source Code/Project Link
Learning to Shadow Hand-drawn Sketches CVPR 2020 [project] [code]
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings ICCV 2021 [project]

10. Sketch Enhancement / Beautification

  • Sketch Enhancement
Paper Source Code/Project Link
SketchMan: Learning to Create Professional Sketches ACM MM 2020 [code]
  • Sketch Beautification
Paper Source Code/Project Link
Sketch Beautification: Learning Part Beautification and Structure Refinement for Sketches of Man-made Objects TVCG 2024
PICASSO: A Feed-Forward Framework for Parametric Inference of CAD Sketches via Rendering Self-Supervision arxiv 24.07
  • Super Resolution
Paper Source Code/Project Link
APISR: Anime Production Inspired Real-World Anime Super-Resolution CVPR 2024 [code]

11. Sketch-Guided Object Localization

Paper Source Code/Project Link
Sketch-Guided Object Localization in Natural Images ECCV 2020 [project] [code]
Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild arxiv 21.09 [code]
What Can Human Sketches Do for Object Detection? CVPR 2023 [project]
Sketch-based Video Object Segmentation: Benchmark and Analysis BMVC 2023 [code]
Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch WACV 2024 [project] [code]

12. Sketch-Guided Video Synthesis

Paper Source Code/Project Link
Deep Sketch-guided Cartoon Video Inbetweening TVCG 2021 [code]
ToonCrafter: Generative Cartoon Interpolation SIGGRAPH Asia 2024 [code] [webpage]
LVCD: Reference-based Lineart Video Colorization with Diffusion Models SIGGRAPH Asia 2024 [webpage]

13. Sketch Recognition

Paper Source Code/Project Link
Sketch-a-Net that Beats Humans BMVC 2015 [code]
Sketch-a-Net: A Deep Neural Network that Beats Humans IJCV 2017 [code]
Deep Self-Supervised Representation Learning for Free-Hand Sketch TCSVT 2020 [code]
S3Net:Graph Representational Network For Sketch Recognition ICME 2020 [code]
Sketchformer: Transformer-based Representation for Sketched Structure CVPR 2020 [code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt CVPR 2020 [code]
Multi-Graph Transformer for Free-Hand Sketch Recognition TNNLS 2021 [code]
Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition TVCG 2021 [code]
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting CVPR 2021 [code]
SketchAA: Abstract Representation for Abstract Sketches ICCV 2021
SketchLattice: Latticed Representation for Sketch Manipulation ICCV 2021 [code]
Multi-Stage Spatio-Temporal Networks for Robust Sketch Recognition TIP 2022

14. Sketch Segmentation and Perceptual Grouping

  • Semantic / Instance Segmentation
Type Paper Source Code/Project Link
Pixelwise SPFusionNet: Sketch Segmentation Using Multi-modal Data Fusion ICME 2019
SketchyScene: Richly-Annotated Scene Sketches (scene-level) ECCV 2018 [code]
Exploring Local Detail Perception for Scene Sketch Semantic Segmentation (scene-level) TIP 2022
Open Vocabulary Semantic Scene Sketch Understanding (scene-level) CVPR 2024 [code] [project]
Stroke-level Free Hand-Drawn Sketch Segmentation ECCV 2012
Data-driven Segmentation and Labeling of Freehand Sketches SIGGRAPH Asia 2014 [code] [project] [dataset]
Example-Based Sketch Segmentation and Labeling Using CRFs TOG 2016
SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation IEEE Access 2019
Fast Sketch Segmentation and Labeling With Deep Learning CGA 2019
Stroke-based sketched symbol reconstruction and segmentation CGA 2020
SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks TOG 2021 [code]
One Sketch for All: One-Shot Personalized Sketch Segmentation TIP 2022
Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches TIP 2023 [code]
CreativeSeg: Semantic Segmentation of Creative Sketches TIP 2024 [code]
ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention CVPR 2024 [webpage]
  • Perceptual Grouping
Type Paper Source Code/Project Link
Stroke-level Sketching by perceptual grouping ICIP 2013
Making Better Use of Edges via Perceptual Grouping CVPR 2015
Universal Sketch Perceptual Grouping / Toward Deep Universal Sketch Perceptual Grouper ECCV 2018 / TIP 2019 [code]
  • FloorPlan
Type Paper Source Code/Project Link
Vector Recognizing Vector Graphics without Rasterization NeurIPS 2021 [code]
FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting ICCV 2021 [project]
GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings CVPR 2022
CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings CVPR 2022 [code]
VectorFloorSeg: Two-Stream Graph Attention Network for Vectorized Roughcast Floorplan Segmentation CVPR 2023 [code]
Hierarchical Recognizing Vector Graphics and A New Chart-based Vector Graphics Dataset TPAMI 2024 [code]

15. Sketch Representation Learning

  • Stroke order importance/saliency, sketch abstraction
Paper Source Code/Project Link Data Manner Related Applications
Learning Deep Sketch Abstraction CVPR 2018 Vector/stroke-level FG-SBIR
Goal-Driven Sequential Data Abstraction ICCV 2019 Vector/stroke-level Sketch recognition
Pixelor: a competitive sketching AI agent. So you think you can sketch? SIGGRAPH Asia 2020 [Project] [Code] Vector/stroke-level Sketch synthesis and recognition
SketchAA: Abstract Representation for Abstract Sketches ICCV 2021 Vector/stroke-level Sketch recognition, FB-SBIR, sketch healing
CLIPasso: Semantically-Aware Object Sketching SIGGRAPH 2022 [project] [code] Vector/stroke-level
Abstracting Sketches through Simple Primitives ECCV 2022 [code] Vector/stroke-level Sketch recognition, FG-SBIR
Learning Geometry-aware Representations by Sketching CVPR 2023 [code] Vector/stroke-level object attribute classification, domain transfer, stroke-based generation, FG-SBIR
SketchXAI: A First Look at Explainability for Human Sketches CVPR 2023 [project] Vector/stroke-level sketch recognition
Prediction with Visual Evidence: Sketch Classification Explanation via Stroke-Level Attributions TIP 2023 Vector/stroke-level
What Sketch Explainability Really Means for Downstream Tasks CVPR 2024 Vector/stroke-level
Do Generalised Classifiers really work on Human Drawn Sketches? ECCV 2024 Pixel-level
  • Conventional Representation Learning
Paper Source Code/Project Link Data Manner Related Applications
A Neural Representation of Sketch Drawings (Sketch-RNN) ICLR 2018 [code] Vector/stroke-level Reconstruction and interpolation
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches BMVC 2020 [code] Vector/stroke-level Sketch recognition, retrieval, completion and analogy
Sketchformer: Transformer-based Representation for Sketched Structure CVPR 2020 [code] Vector/stroke-level Sketch classification, SBIR, reconstruction and interpolation
CoSE: Compositional Stroke Embeddings NeurIPS 2020 [Code] Vector/stroke-level Auto-completing diagrams
SketchLattice: Latticed Representation for Sketch Manipulation ICCV 2021 [code] Lattice graph Sketch healing and image-to-sketch synthesis
SketchODE: Learning neural sketch representation in continuous time ICLR 2022 [Project] Vector/Stroke-level Continuous-time representation, reconstruction & interpolation
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation AAAI 2023 [code] Vector/Stroke-level Sketch synthesis and sketch healing
SketchKnitter: Vectorized Sketch Generation with Diffusion Models ICLR 2023 [code] Vector/Stroke-level vectorized sketch generation
ChiroDiff: Modelling chirographic data with Diffusion Models ICLR 2023 [Project] Vector/Stroke-level vectorization, de-noising/healing, abstraction
Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing AAAI 2024 [code] Vector/Stroke-level recognition and segmentation
Modelling complex vector drawings with stroke-clouds ICLR 2024 [code] Vector/Stroke-level reconstruction and generation
Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation CVM 2024 Vector/Stroke-level reconstruction, recognition and semantic segmentation
Paper Source Code/Project Link Data Manner Related Applications
A Learned Representation for Scalable Vector Graphics ICCV 2019 [code] SVG Font design
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation NeurIPS 2020 [Code] [Project] SVG Vector Graphics Animation, reconstruction and interpolation
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers CVPR 2023 SVG Reconstruction, classification, interpolation, retrieval
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis arxiv 24.01 SVG generation
  • Self-supervised or few/zero-shot learning
Paper Source Code/Project Link Data Manner Related Applications
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation TCSVT 2019 few-shot classification
On Learning Semantic Representations for Large-Scale Abstract Sketches TCSVT 2020 [Code] Bitmap and Vector Retrieval and recognition
Deep Self-Supervised Representation Learning for Free-Hand Sketch TCSVT 2020 [Code] Vector/stroke-level Retrieval and recognition
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt CVPR 2020 [Code] Vector/stroke-level Sketch recognition, retrieval, and gestalt
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting CVPR 2021 [Code] Both Vector and Raster-level Recognition
Self-Supervised Learning for Enhancing Spatial Awareness in Free-Hand Sketches IJCAI 2024 [Code] Vector-level Synthesis
  • Scene-level
Paper Source Code/Project Link
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text CVPR 2023
  • Implicit Neural Representations
Paper Source Code/Project Link
SketchINR: A First Look into Sketches as Implicit Neural Representations CVPR 2024

16. Sketch and Visual Correspondence

  • Datasets
Name Paper Source Code/Project Link
CreativeFlow+ Creative Flow+ Dataset CVPR 2019 [code]
ATD-12K Deep animation video interpolation in the wild CVPR 2021 [code]
AnimeRun AnimeRun: 2D Animation Correspondence from Open Source 3D Movies NeurIPS 2022 [code]
PaintBucket-Character Learning Inclusion Matching for Animation Paint Bucket Colorization CVPR 2024 [code]
  • Methods
Matching Type Paper Source Code/Project Link
Region Globally optimal toon tracking SIGGRAPH 2016 [project]
Region The Animation Transformer: Visual Correspondence via Segment Matching ICCV 2021
Region Learning Inclusion Matching for Animation Paint Bucket Colorization CVPR 2024 [code] [project]
Point SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence TCSVT 2020
Point SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings CGF 2021 [project]
Stroke Differential Operators on Sketches via Alpha Contours SIGGRAPH 2023 [code]

17. Sketch Animation/Inbetweening

Please refer to repository MarkMoHR/Awesome-2D-Animation for details.

18. Sketch and AR/VR

Paper Source Code/Project Link
SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image UIST 2017 [video]
Model-Guided 3D Sketching TVCG 2018 [video]
Mobi3DSketch: 3D Sketching in Mobile AR CHI 2019 [video]
Interactive Liquid Splash Modeling by User Sketches SIGGRAPH Asia 2020 [video]
Towards 3D VR-Sketch to 3D Shape Retrieval 3DV 2020 [code] [project]
3D Curve Creation on and around Physical Objects with Mobile AR TVCG 2021 [video]
HandPainter - 3D Sketching in VR with Hand-based Physical Proxy CHI 2021 [video]
Fine-Grained VR Sketching: Dataset and Insights 3DV 2021 [code]
Structure-Aware 3D VR Sketch to 3D Shape Retrieval 3DV 2022 [code]
GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand TVCG 2023
3D VR Sketch Guided 3D Shape Prototyping and Exploration ICCV 2023 [code]

19. Sketch Based Incremental Learning

Paper Source Code/Project Link
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches CVPR 2022 [code]

20. Sketch Quality Measurement

Paper Source Code/Project Link
Annotation-Free Human Sketch Quality Assessment IJCV 2024 [code]
Finding Badly Drawn Bunnies CVPR 2022 [code]

21. Cloud Augmentation with Sketches

Paper Source Code/Project Link
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches ACM MM 2022 [project]

22. Sketch and Re-identification

Paper Source Code/Project Link
Cross-Compatible Embedding and Semantic Consistent Feature Construction for Sketch Re-identification ACM MM 2022 [code]
SketchTrans: Disentangled Prototype Learning with Transformer for Sketch-Photo Recognition TPAMI 2023 [code]
Differentiable Auxiliary Learning for Sketch Re-Identification AAAI 2024

23. Sketch-based Salient Object Detection

Paper Source Code/Project Link
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings CVPR 2023

24. Study on Strokes

Paper Source Code/Project Link
Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools SIGGRAPH Asia 2022 [project]
Ciallo: GPU-Accelerated Rendering of Vector Brush Strokes SIGGRAPH 2024 [code]