LayerInbetween: Occlusion-Aware Stroke Correspondence and Inbetweening with Automatic Layering - SIGGRAPH 2026 (TOG)

[Paper] | [Project Page]

This project can be used to produce automatic inbetweening/2D animations from clean line drawings, rough sketches, multi-character line art, complex scenes, abstract drawings, or multi-keyframe sequences. Our vector-based approach can facilitate convenient inbetweening editing.

   

Environment

Create a Conda environment named layerinbetween and install the main dependencies from requirements.txt:

conda create -n layerinbetween python=3.10 -y
conda activate layerinbetween
pip install -r requirements.txt

If some packages are missing from requirements.txt, their exact versions can be found in requirements-full.txt.

To avoid environment conflicts with SAM2, create a separate Conda environment named sam and install SAM2 using its instructions.

Getting Started

Download Checkpoints

Our project relies on external models, including optical flow estimation, depth estimation, and SAM2 video tracking. Please download the corresponding models and place them according to the following file structure. Our models can be downloaded here.

optical_flow/
  AnimeRun/
    checkpoints/
      20000_gma-animerun-v2-ft.pth
      20000_raft-animerun-v2-ft_again.pth

depth_estimation/
  Depth-Anything-V2/
    checkpoints/
      depth_anything_v2_vitl.pth

video_tracking/
  sam2/
    models/
      SAM2.1/
        sam2.1_hiera_large.pt

outputs/
  ctrlpoint/
    snapshot/
      FAD3-CP46-sep-dist/
        sketch_ctrlpoint_30000.pkl
  endpoint/
    snapshot/
      FAD3-EPO35-1.5x-min=64/
        sketch_endpoint_30000.pkl
  transform/
    snapshot/
      FAD3-T12-2.0x-51-min=64/
        sketch_transform_30000.pkl
      FAD3-T13-2.0x-51/
        sketch_transform_local_30000.pkl

vgg_utils/
  quickdraw-perceptual.pth

Input Preparation

Prepare raster keyframes and the vector image for the first keyframe (following the tutorial here to create SVGs). Then, place them to test_examples/raster_black/ and test_examples/svg/ folders.

Please name them X_ref.png, X_tar.png, and X.svg using the same index X.

If the raster keyframes are rough sketches, please place them in the test_examples/raster_black/rough_raw/ folder first, and then run the following command for binarization and image squaring. The resulting clean keyframes will be saved to the test_examples/raster_black/ folder:

python data_preprocessing/image_preprocess.py --image_id X

If the raster keyframes are not square, use the script and the command above after commenting out the binarize(img) and darken(img) in the script.

Step 1: Forward Prediction

Go to configs/example_configs.py, and set the test_img_id to X. If you want to use the provided cases directly, please set the test_img_id to the corresponding image index. Then, run the following command:

sh run_forward.sh

This script includes several sub-processes. Refer to it for details. Finally, the output results of vector stroke correspondence are placed in the outputs/stroke_correspondence_results folder.

Step 2: Inverse Prediction

1. Prepare data for the inverse prediction:

PYTHONPATH=. python data_preprocessing/prepare_inverse_data.py

This will create a test_examples/[0inv]/ folder, which stores raster images (i.e., alternating the original reference and target keyframes) and the SVG (i.e., the predicted vector image for the original target keyframe).

2. Make a vector image X.svg for the missing strokes following the tutorial here.

3. Place the X.svg to the test_examples/[0inv]/svg/ folder.

4. Set do_inv = True in configs/example_configs.py to indicate the prediction direction. Then, run the following command:

sh run_inverse.sh

This script also includes several sub-processes. Finally, the output results of vector stroke correspondence for the complementary strokes are placed in the outputs/stroke_correspondence_results/[0inv] folder.

Step 3: Combine Two Directions

Combine the results of the two directions using the following command:

PYTHONPATH=. python data_preprocessing/combine_two_directions.py

Afterward, the results are saved to the outputs/stroke_correspondence_results/[1comb] folder.

Multi-frame Prediction

Input Preparation

1. We first define the K-th generation: K=1 denotes the second generation (3 keyframes); K=2 denotes the third generation (4 keyframes); etc.

2. Place raster keyframes to test_examples-GenK/raster_black/ (replace K with 1, 2, ...). Note that the X_ref.png should be the X_tar.png in the K-1-th generation.

3. Copy the output vector parameters (e.g., outputs/stroke_correspondence_results/[1comb]/vector-params/X_tar.jsonl) to test_examples-GenK/vector-params/ folder, and rename it to X_ref.jsonl. Note that if inverse prediction was not done in the K-1-th generation, the Step 3: Combine Two Directions should also be executed.

Run

Go to configs/example_configs.py, and set the gen_time to K. Then, perform the Step 1 to Step 3 above again.

TODOs

• Occlusion resolving after interpolation

Citation

If you use the code and models, please cite:

@article{mo2026layerinbetween,
  title={LayerInbetween: Occlusion-Aware Stroke Correspondence and Inbetweening with Automatic Layering},
  author={Mo, Haoran and Guan, Zhongyue and Hu, Yixin and Wang, Zeyu},
  journal={ACM Transactions on Graphics (TOG)},
  volume={45},
  number={4},
  pages={1--18},
  year={2026},
  publisher={ACM New York, NY, USA}
}