Repository for the survey on stroke-based painterly rendering algorithms: Stroke-based Rendering: From Heuristics to Deep Learning
Note: Not all GitHub links are the official implementations!
Edge alignment: Align stroke direction with edges and contours of image.
-
Low-level: e.g. edges from Sobel filter or Canny edge detection
-
High-level: e.g. semantic / salient edges and contours of objects
Region approximation: Replace image segments with strokes.
-
Image moments: Soft / fuzzy image segmentation algorithms
-
imgage segmentation "Normal" image segmentation algorithms (hard edges)
| Name | Algorithm | Notes / Description |
|---|---|---|
| Paint by numbers: Abstract image representations | Low-level edge alignment | The first painterly rendering paper! |
| Processing images and video for an impressionist effect | Low-level edge alignment | |
| Painterly rendering with curved brush strokes of multiple sizes | Low-level edge alignment | First algorithm with curved strokes. [Java] [Python (Unofficial)] |
| Image and video based painterly animation | Low-level edge alignment | |
| Painterly rendering controlled by multiscale image features | Low-level edge alignment | |
| Interactive painterly stylization of images, videos and 3D animations | Low-level edge alignment | |
| Contour-driven Sumi-e rendering of real photos | Low-level edge alignment | Connected to Artist Agent |
| Painterly rendering with content-dependent natural paint strokes | Low-level edge alignment | |
| Painterly rendering using image salience | High-level edge alignment | |
| Abstracted painterly renderings using eye-tracking data | High-level edge alignment | |
| From image parsing to painterly rendering | High-level edge alignment | |
| Video painting with space-time-varying style parameters | High-level edge alignment | |
| Artistic composition for painterly rendering | High-level edge alignment | |
| An algorithm for automatic painterly rendering based on local source image approximation | Image moments | |
| Multiscale moment-based painterly rendering | Image moments | |
| Artistic vision: painterly rendering using computer vision techniques | Image segmentation | |
| Empathic painting: interactive stylization through observed emotional state | Image segmentation | |
| Emotionally aware automated portrait painting | Image segmentation | |
| Abstract art by shape classification | Image segmentation | |
| A generic framework for the structured abstraction of images | Image segmentation | [Code] |
Brute force: Systematically try out many different paramters until a good result is achieved
- Stroke database: Choose best stroke from database
Randomized search: Randomly perturb stroke parameters to find good painting. Different stochastic optimization algorithms are possible.
- Hill climbing
- Simulated annealing
- Genetic / evolutionary optimization
Gradient descent: Change parameters in negative gradient direction. Good and fast results, but usually requires differentiable rendering.
| Name | Algorithm | Notes / Description |
|---|---|---|
| Paint by Numbers | Hill climbing | The first painterly rendering paper! |
| Paint by relaxation | Brute force | |
| Anipaint: Interactive painterly animation from video | Brute force | |
| Interactive painterly rendering with artistic error correction | Stroke database | |
| A painterly rendering based on stroke profile and database | Stroke database | |
| Stroke based painterly rendering with mass data through auto warping generation | Stroke database | |
| Portrait painting using active templates | Stroke database | |
| Random paintbrush transformation | Hill climbing | |
| Optimization of paintbrush rendering of images by dynamic mcmc methods | Hill climbing | |
| Genetic paint: A search for salient paintings | Genetic / evolutionary optimization | |
| A unified scheme for adaptive stroke-based rendering | Simulated annealing | |
| Image-based painterly rendering by evolutionary algorithm | Genetic / evolutionary optimization | |
| Evolved art with transparent, overlapping, and geometric shapes | Genetic / evolutionary optimization | [Code] |
| Paintings, polygons and plant propagation | Genetic / evolutionary optimization, Simulated annealing, Hill climbing | [Code] |
| Automatic and interactive evolution of vector graphics images with genetic algorithms | Genetic / evolutionary optimization | |
| Incremental Evolution of Stylized Images | Genetic / evolutionary optimization | [Code] |
| Generative art using neural visual grammars and dual encoders | Genetic / evolutionary optimization | [Code] |
| Modern evolution strategies for creativity: Fitting concrete images and abstract concepts | Genetic / evolutionary optimization | [Code] |
| Customizing painterly rendering styles using stroke processes | Reaction-diffusion process | [Code] |
| Neural painters: A learned differentiable constraint for generating brushstroke paintings | Gradient descent | [Code] |
| Differentiable vector graphics rasterization for editing and learning | Gradient descent | [Code] |
| Stylized neural painting | Gradient descent | [Code] |
| From Objects to a Whole Painting | Gradient descent | |
| Rethinking style transfer: From pixels to parameterized brushstrokes | Gradient descent | [Code] |
| Differentiable drawing and sketching | Gradient descent | [Code] |
| CLIPDraw: Exploring text-to-drawing synthesis through language-image encoders | Gradient descent | [Code] |
| StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation | Gradient descent | [Code] |
| CLIP-CLOP: CLIP-Guided Collage and Photomontage | Gradient descent | [Code] |
Supervised Learning: Predicts sequence of stroke parameters from pixel canvas by using differentiable rendering
Deep Reinforcement Learning (DRL): Agent predicts next stroke to paint, given canvas and target.
- Model-free DRL: Does not use differentiable rendering
- Model-based DRL: Uses differentiable rendering
| Name | Architecture | Notes / Description |
|---|---|---|
| Neural painters: A learned differentiable constraint for generating brushstroke paintings | Supervised Learning | [Code 1] [Code 2] |
| Unsupervised image to sequence translation with canvas-drawer networks | Supervised learning | [Unofficial code] |
| Strokenet: A neural painting environment | Supervised learning | [Code] |
| Paint transformer: Feed forward neural painting with stroke prediction | Supervised learning | [Code] |
| Synthesizing programs for images using reinforced adversarial learning ("SPIRAL") | Model-free DRL | [Code] |
| Unsupervised doodling and painting with improved spiral ("SPIRAL++") | Model-free DRL | [Unofficial code] |
| Paintbot: A reinforcement learning approach for natural media painting | Model-free DRL | |
| LpaintB: Learning to paint from self-supervision | Model-free DRL | |
| Learning to paint with model-based deep reinforcement learning | Model-based DRL | [Code] |
| Demystify Painting with RL | Model-based DRL | |
| Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent | Model-based DRL | [Code] |
| Combining semantic guidance and deep reinforcement learning for generating human level paintings | Model-based DRL | [Code] |
| Intelli-Paint: Towards Developing Human-like Painting Agents | Model-based DRL |
Different techniques for extending painterly rendering algorithms to handle video.
| Name | GitHub | Notes / Description |
|---|---|---|
| Primitive Pictures | [Code] | Ready to use application / GitHub for video painting |
| AniPainter | [Code] | Digitally painted and robot painted videos |
| Processing images and video for an impressionist effect | ||
| Painterly rendering for video and interaction | ||
| Image and video based painterly animation | ||
| Video painting with space-time-varying style parameters | ||
| Motion map generation for maintaining the temporalcoherence of brush strokes | ||
| Interactive painterly stylization of images, videos and 3D animations | ||
| Painterly animation using video semantics and feature correspondence | ||
| Anipaint: Interactive painterly animation from video |