Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 12 additions & 12 deletions docs/source/en/_toctree.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,26 +12,24 @@
sections:
- isExpanded: false
sections:
- local: using-diffusers/weighted_prompts
title: Prompting
- local: using-diffusers/loading
title: DiffusionPipeline
- local: tutorials/autopipeline
title: AutoPipeline
- local: using-diffusers/custom_pipeline_overview
title: Community pipelines and components
- local: using-diffusers/callback
title: Pipeline callbacks
- local: using-diffusers/reusing_seeds
title: Reproducibility
- local: using-diffusers/schedulers
title: Schedulers
- local: using-diffusers/guiders
title: Guiders
- local: using-diffusers/automodel
title: AutoModel
- local: using-diffusers/other-formats
title: Model formats
- local: using-diffusers/schedulers
title: Schedulers
- local: using-diffusers/weighted_prompts
title: Prompting
- local: using-diffusers/reusing_seeds
title: Reproducibility
- local: using-diffusers/callback
title: Pipeline callbacks
- local: using-diffusers/custom_pipeline_overview
title: Community pipelines and components
- local: using-diffusers/push_to_hub
title: Sharing pipelines and models
title: Using diffusion pipelines
Expand Down Expand Up @@ -155,6 +153,8 @@
title: AutoPipelineBlocks
- local: modular_diffusers/modular_pipeline
title: ModularPipeline
- local: using-diffusers/guiders
title: Guiders
- local: modular_diffusers/components_manager
title: ComponentsManager
- local: modular_diffusers/auto_docstring
Expand Down
54 changes: 39 additions & 15 deletions docs/source/en/tutorials/autopipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,21 @@ specific language governing permissions and limitations under the License.

# AutoPipeline

[AutoPipeline](../api/models/auto_model) is a *task-and-model* pipeline that automatically selects the correct pipeline subclass based on the task. It handles the complexity of loading different pipeline subclasses without needing to know the specific pipeline subclass name.
[AutoPipeline](../api/pipelines/auto_pipeline) is a *task-and-model* pipeline that automatically selects the correct pipeline subclass based on the task. It handles the complexity of loading different pipeline subclasses without needing to know the specific pipeline subclass name.

This is unlike [`DiffusionPipeline`], a *model-only* pipeline that automatically selects the pipeline subclass based on the model.

[`AutoPipelineForImage2Image`] returns a specific pipeline subclass, (for example, [`StableDiffusionXLImg2ImgPipeline`]), which can only be used for image-to-image tasks.
```text
AutoPipelineForImage2Image.from_pretrained(model_id)
|
+-- read model_index.json (e.g. StableDiffusionXLPipeline)
+-- task mapping (image-to-image)
|
v
StableDiffusionXLImg2ImgPipeline // returned instance
```

[`AutoPipelineForImage2Image`] returns the task-specific subclass (for example, [`StableDiffusionXLImg2ImgPipeline`]), which can only be used for image-to-image tasks.

```py
import torch
Expand All @@ -26,13 +36,13 @@ pipeline = AutoPipelineForImage2Image.from_pretrained(
"RunDiffusion/Juggernaut-XL-v9", dtype=torch.bfloat16, device_map="cuda", # or "mps", "xpu", "cpu"
)
print(pipeline)
"StableDiffusionXLImg2ImgPipeline {
"_class_name": "StableDiffusionXLImg2ImgPipeline",
...
"
# StableDiffusionXLImg2ImgPipeline {
# "_class_name": "StableDiffusionXLImg2ImgPipeline",
# ...
# }
```

Loading the same model with [`DiffusionPipeline`] returns the [`StableDiffusionXLPipeline`] subclass. It can be used for text-to-image, image-to-image, or inpainting tasks depending on the inputs.
Loading the same model with [`DiffusionPipeline`] returns the default text-to-image subclass, [`StableDiffusionXLPipeline`]. That pipeline is for text-to-image. For image-to-image or inpainting, load a task AutoPipeline such as [`AutoPipelineForImage2Image`] or [`AutoPipelineForInpainting`], or the matching task-specific subclass.

```py
import torch
Expand All @@ -42,15 +52,29 @@ pipeline = DiffusionPipeline.from_pretrained(
"RunDiffusion/Juggernaut-XL-v9", dtype=torch.bfloat16, device_map="cuda", # or "mps", "xpu", "cpu"
)
print(pipeline)
"StableDiffusionXLPipeline {
"_class_name": "StableDiffusionXLPipeline",
...
"
# StableDiffusionXLPipeline {
# "_class_name": "StableDiffusionXLPipeline",
# ...
# }
```

## Switch tasks with from_pipe

Load a task AutoPipeline once, then switch tasks with [`~AutoPipelineForImage2Image.from_pipe`] without downloading the weights again. Components are reused from the source pipeline.

```py
import torch
from diffusers import AutoPipelineForText2Image, AutoPipelineForImage2Image

pipeline_t2i = AutoPipelineForText2Image.from_pretrained(
"RunDiffusion/Juggernaut-XL-v9", dtype=torch.bfloat16, device_map="cuda", # or "mps", "xpu", "cpu"
)
pipeline_i2i = AutoPipelineForImage2Image.from_pipe(pipeline_t2i)
```

Check the [mappings](https://github.com/huggingface/diffusers/blob/130fd8df54f24ffb006d84787b598d8adc899f23/src/diffusers/pipelines/auto_pipeline.py#L114) to see whether a model is supported or not.
See [Reusing models in multiple pipelines](../using-diffusers/loading#reusing-models-in-multiple-pipelines) for more details.

Trying to load an unsupported model returns an error.
Check the [mappings](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/auto_pipeline.py) to see whether a model is supported or not. Trying to load an unsupported model returns an error.

```py
import torch
Expand All @@ -62,13 +86,13 @@ pipeline = AutoPipelineForImage2Image.from_pretrained(
"ValueError: AutoPipeline can't find a pipeline linked to ShapEImg2ImgPipeline for None"
```

There are four types of [AutoPipeline](../api/models/auto_model) classes:
There are four types of [AutoPipeline](../api/pipelines/auto_pipeline) classes:

- [`AutoPipelineForText2Image`]
- [`AutoPipelineForImage2Image`]
- [`AutoPipelineForInpainting`]
- [`AutoPipelineForText2Audio`]

Each of these classes have a predefined mapping, linking a pipeline to their task-specific subclass.
Each of these classes has a predefined mapping, linking a pipeline to their task-specific subclass.

When [`~AutoPipelineForText2Image.from_pretrained`] is called, it extracts the class name from the `model_index.json` file and selects the appropriate pipeline subclass for the task based on the mapping.
7 changes: 4 additions & 3 deletions docs/source/en/using-diffusers/automodel.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ If the custom model inherits from the [`ModelMixin`] class, it gets access to th
> )
> ```

### Saving custom models
## Saving custom models

Use [`~ConfigMixin.register_for_auto_class`] to add the `auto_map` entry to `config.json` automatically when saving. This avoids having to manually edit the config file.

Expand All @@ -124,5 +124,6 @@ The saved `config.json` will include the `auto_map` field.
}
```

> [!NOTE]
> Learn more about implementing custom models in the [Community components](../using-diffusers/custom_pipeline_overview#community-components) guide.
## Next steps

Learn more about implementing custom models in the [Community components](../using-diffusers/custom_pipeline_overview#community-components) guide.
47 changes: 25 additions & 22 deletions docs/source/en/using-diffusers/callback.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,20 @@ specific language governing permissions and limitations under the License.

# Pipeline callbacks

A callback is a function that modifies [`DiffusionPipeline`] behavior and it is executed at the end of a denoising step. The changes are propagated to subsequent steps in the denoising process. It is useful for adjusting pipeline attributes or tensor variables to support new features without rewriting the underlying pipeline code.
A callback runs at the end of a denoising step and can change pipeline state or tensors for later steps. Use it to adjust attributes or tensor variables for new behavior without rewriting the pipeline.

These callbacks apply to classic [`DiffusionPipeline`] loops. In [Modular Diffusers](../modular_diffusers/overview), you can build and add custom pipeline blocks instead of `callback_on_step_end`.

Diffusers provides several callbacks in the pipeline [overview](../api/pipelines/overview#diffusers.callbacks.PipelineCallback).

To enable a callback, configure when the callback is executed after a certain number of denoising steps with one of the following arguments.

- `cutoff_step_ratio` specifies when a callback is activated as a percentage of the total denoising steps.
- `cutoff_step_index` specifies the exact step number a callback is activated.
- `cutoff_step_ratio` specifies when a callback is activated as a percentage of the total denoising steps. Use when the cutoff should scale with `num_inference_steps` (for example, drop CFG after 40% of run).
- `cutoff_step_index` specifies the exact step number a callback is activated. Use when you care about an absolute step (for example, step `10` on a fixed 25-step schedule).

The example below uses `cutoff_step_ratio=0.4`, which means the callback is activated once denoising reaches 40% of the total inference steps. [`~callbacks.SDXLCFGCutoffCallback`] disables classifier-free guidance (CFG) after a certain number of steps, which can help save compute without significantly affecting performance.

Define a callback with either of the `cutoff` arguments and pass it to the `callback_on_step_end` parameter in the pipeline.
Define a callback with one of the `cutoff` arguments and pass it to the `callback_on_step_end` parameter in the pipeline.

```py
import torch
Expand All @@ -41,52 +43,57 @@ pipeline = StableDiffusionXLPipeline.from_pretrained(
)
pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, use_karras_sigmas=True)

prompt = "a sports car at the road, best quality, high quality, high detail, 8k resolution"
prompt = "a sports car on the road, best quality, high quality, high detail, 8k resolution"
output = pipeline(
prompt=prompt,
negative_prompt="",
guidance_scale=6.5,
num_inference_steps=25,
generator=generator,
callback_on_step_end=callback,
)
```

Official callbacks set their own tensor inputs. For a custom function, pass `callback_on_step_end_tensor_inputs` as in [Display intermediate images](#display-intermediate-images).

If you want to add a new official callback, feel free to open a [feature request](https://github.com/huggingface/diffusers/issues/new/choose) or [submit a PR](https://huggingface.co/docs/diffusers/main/en/conceptual/contribution#how-to-open-a-pr). Otherwise, you can also create your own callback as shown below.

## Early stopping

Early stopping is useful if you aren't happy with the intermediate results during generation. This callback sets a hardcoded stop point after which the pipeline terminates by setting the `_interrupt` attribute to `True`.
Early stopping is useful if you aren't happy with the intermediate results during generation. This callback sets a hardcoded stop point by setting the `_interrupt` attribute to `True`, which makes the denoising loop skip the remaining steps.

```py
from diffusers import StableDiffusionXLPipeline
import torch
from diffusers import DiffusionPipeline

def interrupt_callback(pipeline, i, t, callback_kwargs):
stop_idx = 10
if i == stop_idx:
pipeline._interrupt = True

return callback_kwargs

pipeline = StableDiffusionXLPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5"
pipeline = DiffusionPipeline.from_pretrained(
"Qwen/Qwen-Image",
dtype=torch.bfloat16,
device_map="cuda", # or "mps", "xpu", "cpu"
)
num_inference_steps = 50

pipeline(
"A photo of a cat",
num_inference_steps=num_inference_steps,
prompt="A photo of a cat",
num_inference_steps=50,
callback_on_step_end=interrupt_callback,
)
```

## Display intermediate images

Visualizing the intermediate images is useful for progress monitoring and assessing the quality of the generated content. This callback decodes the latent tensors at each step and converts them to images.
Visualizing intermediate images is useful for progress monitoring. The preview below is SDXL-only. It maps SDXL latents to RGB with a linear transform for a quick look during denoising. Those weights do not transfer to other models. For Qwen-Image and similar checkpoints, decode with the model VAE instead of this helper.

[Convert](https://huggingface.co/blog/TimothyAlexisVass/explaining-the-sdxl-latent-space) the Stable Diffusion XL latents from latents (4 channels) to RGB tensors (3 tensors).
[Convert](https://huggingface.co/blog/TimothyAlexisVass/explaining-the-sdxl-latent-space) Stable Diffusion XL latents (4 channels) to RGB tensors (3 channels).

```py
import torch
from PIL import Image
from diffusers import AutoPipelineForText2Image

def latents_to_rgb(latents):
weights = (
(60, -60, 25, -70),
Expand Down Expand Up @@ -114,13 +121,9 @@ def decode_tensors(pipe, step, timestep, callback_kwargs):
return callback_kwargs
```

Use the `callback_on_step_end_tensor_inputs` parameter to specify what input type to modify, which in this case, are the latents.
Use `callback_on_step_end_tensor_inputs` to choose which tensors the callback receives, which in this case, are the latents.

```py
import torch
from PIL import Image
from diffusers import AutoPipelineForText2Image

pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16,
Expand Down
Loading
Loading