diff --git a/docs/source/en/_toctree.yml b/docs/source/en/_toctree.yml index 525734a2e4cb..9c1fd994a4c5 100644 --- a/docs/source/en/_toctree.yml +++ b/docs/source/en/_toctree.yml @@ -12,26 +12,24 @@ sections: - isExpanded: false sections: - - local: using-diffusers/weighted_prompts - title: Prompting - local: using-diffusers/loading title: DiffusionPipeline - local: tutorials/autopipeline title: AutoPipeline - - local: using-diffusers/custom_pipeline_overview - title: Community pipelines and components - - local: using-diffusers/callback - title: Pipeline callbacks - - local: using-diffusers/reusing_seeds - title: Reproducibility - - local: using-diffusers/schedulers - title: Schedulers - - local: using-diffusers/guiders - title: Guiders - local: using-diffusers/automodel title: AutoModel - local: using-diffusers/other-formats title: Model formats + - local: using-diffusers/schedulers + title: Schedulers + - local: using-diffusers/weighted_prompts + title: Prompting + - local: using-diffusers/reusing_seeds + title: Reproducibility + - local: using-diffusers/callback + title: Pipeline callbacks + - local: using-diffusers/custom_pipeline_overview + title: Community pipelines and components - local: using-diffusers/push_to_hub title: Sharing pipelines and models title: Using diffusion pipelines @@ -155,6 +153,8 @@ title: AutoPipelineBlocks - local: modular_diffusers/modular_pipeline title: ModularPipeline + - local: using-diffusers/guiders + title: Guiders - local: modular_diffusers/components_manager title: ComponentsManager - local: modular_diffusers/auto_docstring diff --git a/docs/source/en/tutorials/autopipeline.md b/docs/source/en/tutorials/autopipeline.md index d3f829aa6ae4..df53a9d6dc6f 100644 --- a/docs/source/en/tutorials/autopipeline.md +++ b/docs/source/en/tutorials/autopipeline.md @@ -12,11 +12,21 @@ specific language governing permissions and limitations under the License. # AutoPipeline -[AutoPipeline](../api/models/auto_model) is a *task-and-model* pipeline that automatically selects the correct pipeline subclass based on the task. It handles the complexity of loading different pipeline subclasses without needing to know the specific pipeline subclass name. +[AutoPipeline](../api/pipelines/auto_pipeline) is a *task-and-model* pipeline that automatically selects the correct pipeline subclass based on the task. It handles the complexity of loading different pipeline subclasses without needing to know the specific pipeline subclass name. This is unlike [`DiffusionPipeline`], a *model-only* pipeline that automatically selects the pipeline subclass based on the model. -[`AutoPipelineForImage2Image`] returns a specific pipeline subclass, (for example, [`StableDiffusionXLImg2ImgPipeline`]), which can only be used for image-to-image tasks. +```text +AutoPipelineForImage2Image.from_pretrained(model_id) + | + +-- read model_index.json (e.g. StableDiffusionXLPipeline) + +-- task mapping (image-to-image) + | + v +StableDiffusionXLImg2ImgPipeline // returned instance +``` + +[`AutoPipelineForImage2Image`] returns the task-specific subclass (for example, [`StableDiffusionXLImg2ImgPipeline`]), which can only be used for image-to-image tasks. ```py import torch @@ -26,13 +36,13 @@ pipeline = AutoPipelineForImage2Image.from_pretrained( "RunDiffusion/Juggernaut-XL-v9", dtype=torch.bfloat16, device_map="cuda", # or "mps", "xpu", "cpu" ) print(pipeline) -"StableDiffusionXLImg2ImgPipeline { - "_class_name": "StableDiffusionXLImg2ImgPipeline", - ... -" +# StableDiffusionXLImg2ImgPipeline { +# "_class_name": "StableDiffusionXLImg2ImgPipeline", +# ... +# } ``` -Loading the same model with [`DiffusionPipeline`] returns the [`StableDiffusionXLPipeline`] subclass. It can be used for text-to-image, image-to-image, or inpainting tasks depending on the inputs. +Loading the same model with [`DiffusionPipeline`] returns the default text-to-image subclass, [`StableDiffusionXLPipeline`]. That pipeline is for text-to-image. For image-to-image or inpainting, load a task AutoPipeline such as [`AutoPipelineForImage2Image`] or [`AutoPipelineForInpainting`], or the matching task-specific subclass. ```py import torch @@ -42,15 +52,29 @@ pipeline = DiffusionPipeline.from_pretrained( "RunDiffusion/Juggernaut-XL-v9", dtype=torch.bfloat16, device_map="cuda", # or "mps", "xpu", "cpu" ) print(pipeline) -"StableDiffusionXLPipeline { - "_class_name": "StableDiffusionXLPipeline", - ... -" +# StableDiffusionXLPipeline { +# "_class_name": "StableDiffusionXLPipeline", +# ... +# } +``` + +## Switch tasks with from_pipe + +Load a task AutoPipeline once, then switch tasks with [`~AutoPipelineForImage2Image.from_pipe`] without downloading the weights again. Components are reused from the source pipeline. + +```py +import torch +from diffusers import AutoPipelineForText2Image, AutoPipelineForImage2Image + +pipeline_t2i = AutoPipelineForText2Image.from_pretrained( + "RunDiffusion/Juggernaut-XL-v9", dtype=torch.bfloat16, device_map="cuda", # or "mps", "xpu", "cpu" +) +pipeline_i2i = AutoPipelineForImage2Image.from_pipe(pipeline_t2i) ``` -Check the [mappings](https://github.com/huggingface/diffusers/blob/130fd8df54f24ffb006d84787b598d8adc899f23/src/diffusers/pipelines/auto_pipeline.py#L114) to see whether a model is supported or not. +See [Reusing models in multiple pipelines](../using-diffusers/loading#reusing-models-in-multiple-pipelines) for more details. -Trying to load an unsupported model returns an error. +Check the [mappings](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/auto_pipeline.py) to see whether a model is supported or not. Trying to load an unsupported model returns an error. ```py import torch @@ -62,13 +86,13 @@ pipeline = AutoPipelineForImage2Image.from_pretrained( "ValueError: AutoPipeline can't find a pipeline linked to ShapEImg2ImgPipeline for None" ``` -There are four types of [AutoPipeline](../api/models/auto_model) classes: +There are four types of [AutoPipeline](../api/pipelines/auto_pipeline) classes: - [`AutoPipelineForText2Image`] - [`AutoPipelineForImage2Image`] - [`AutoPipelineForInpainting`] - [`AutoPipelineForText2Audio`] -Each of these classes have a predefined mapping, linking a pipeline to their task-specific subclass. +Each of these classes has a predefined mapping, linking a pipeline to their task-specific subclass. When [`~AutoPipelineForText2Image.from_pretrained`] is called, it extracts the class name from the `model_index.json` file and selects the appropriate pipeline subclass for the task based on the mapping. \ No newline at end of file diff --git a/docs/source/en/using-diffusers/automodel.md b/docs/source/en/using-diffusers/automodel.md index 38d30539973e..997e922057bd 100644 --- a/docs/source/en/using-diffusers/automodel.md +++ b/docs/source/en/using-diffusers/automodel.md @@ -97,7 +97,7 @@ If the custom model inherits from the [`ModelMixin`] class, it gets access to th > ) > ``` -### Saving custom models +## Saving custom models Use [`~ConfigMixin.register_for_auto_class`] to add the `auto_map` entry to `config.json` automatically when saving. This avoids having to manually edit the config file. @@ -124,5 +124,6 @@ The saved `config.json` will include the `auto_map` field. } ``` -> [!NOTE] -> Learn more about implementing custom models in the [Community components](../using-diffusers/custom_pipeline_overview#community-components) guide. \ No newline at end of file +## Next steps + +Learn more about implementing custom models in the [Community components](../using-diffusers/custom_pipeline_overview#community-components) guide. \ No newline at end of file diff --git a/docs/source/en/using-diffusers/callback.md b/docs/source/en/using-diffusers/callback.md index cf088fc731de..8a44f5e825a4 100644 --- a/docs/source/en/using-diffusers/callback.md +++ b/docs/source/en/using-diffusers/callback.md @@ -12,18 +12,20 @@ specific language governing permissions and limitations under the License. # Pipeline callbacks -A callback is a function that modifies [`DiffusionPipeline`] behavior and it is executed at the end of a denoising step. The changes are propagated to subsequent steps in the denoising process. It is useful for adjusting pipeline attributes or tensor variables to support new features without rewriting the underlying pipeline code. +A callback runs at the end of a denoising step and can change pipeline state or tensors for later steps. Use it to adjust attributes or tensor variables for new behavior without rewriting the pipeline. + +These callbacks apply to classic [`DiffusionPipeline`] loops. In [Modular Diffusers](../modular_diffusers/overview), you can build and add custom pipeline blocks instead of `callback_on_step_end`. Diffusers provides several callbacks in the pipeline [overview](../api/pipelines/overview#diffusers.callbacks.PipelineCallback). To enable a callback, configure when the callback is executed after a certain number of denoising steps with one of the following arguments. -- `cutoff_step_ratio` specifies when a callback is activated as a percentage of the total denoising steps. -- `cutoff_step_index` specifies the exact step number a callback is activated. +- `cutoff_step_ratio` specifies when a callback is activated as a percentage of the total denoising steps. Use when the cutoff should scale with `num_inference_steps` (for example, drop CFG after 40% of run). +- `cutoff_step_index` specifies the exact step number a callback is activated. Use when you care about an absolute step (for example, step `10` on a fixed 25-step schedule). The example below uses `cutoff_step_ratio=0.4`, which means the callback is activated once denoising reaches 40% of the total inference steps. [`~callbacks.SDXLCFGCutoffCallback`] disables classifier-free guidance (CFG) after a certain number of steps, which can help save compute without significantly affecting performance. -Define a callback with either of the `cutoff` arguments and pass it to the `callback_on_step_end` parameter in the pipeline. +Define a callback with one of the `cutoff` arguments and pass it to the `callback_on_step_end` parameter in the pipeline. ```py import torch @@ -41,52 +43,57 @@ pipeline = StableDiffusionXLPipeline.from_pretrained( ) pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config, use_karras_sigmas=True) -prompt = "a sports car at the road, best quality, high quality, high detail, 8k resolution" +prompt = "a sports car on the road, best quality, high quality, high detail, 8k resolution" output = pipeline( prompt=prompt, negative_prompt="", guidance_scale=6.5, num_inference_steps=25, - generator=generator, callback_on_step_end=callback, ) ``` +Official callbacks set their own tensor inputs. For a custom function, pass `callback_on_step_end_tensor_inputs` as in [Display intermediate images](#display-intermediate-images). + If you want to add a new official callback, feel free to open a [feature request](https://github.com/huggingface/diffusers/issues/new/choose) or [submit a PR](https://huggingface.co/docs/diffusers/main/en/conceptual/contribution#how-to-open-a-pr). Otherwise, you can also create your own callback as shown below. ## Early stopping -Early stopping is useful if you aren't happy with the intermediate results during generation. This callback sets a hardcoded stop point after which the pipeline terminates by setting the `_interrupt` attribute to `True`. +Early stopping is useful if you aren't happy with the intermediate results during generation. This callback sets a hardcoded stop point by setting the `_interrupt` attribute to `True`, which makes the denoising loop skip the remaining steps. ```py -from diffusers import StableDiffusionXLPipeline +import torch +from diffusers import DiffusionPipeline def interrupt_callback(pipeline, i, t, callback_kwargs): stop_idx = 10 if i == stop_idx: pipeline._interrupt = True - return callback_kwargs -pipeline = StableDiffusionXLPipeline.from_pretrained( - "stable-diffusion-v1-5/stable-diffusion-v1-5" +pipeline = DiffusionPipeline.from_pretrained( + "Qwen/Qwen-Image", + dtype=torch.bfloat16, + device_map="cuda", # or "mps", "xpu", "cpu" ) -num_inference_steps = 50 - pipeline( - "A photo of a cat", - num_inference_steps=num_inference_steps, + prompt="A photo of a cat", + num_inference_steps=50, callback_on_step_end=interrupt_callback, ) ``` ## Display intermediate images -Visualizing the intermediate images is useful for progress monitoring and assessing the quality of the generated content. This callback decodes the latent tensors at each step and converts them to images. +Visualizing intermediate images is useful for progress monitoring. The preview below is SDXL-only. It maps SDXL latents to RGB with a linear transform for a quick look during denoising. Those weights do not transfer to other models. For Qwen-Image and similar checkpoints, decode with the model VAE instead of this helper. -[Convert](https://huggingface.co/blog/TimothyAlexisVass/explaining-the-sdxl-latent-space) the Stable Diffusion XL latents from latents (4 channels) to RGB tensors (3 tensors). +[Convert](https://huggingface.co/blog/TimothyAlexisVass/explaining-the-sdxl-latent-space) Stable Diffusion XL latents (4 channels) to RGB tensors (3 channels). ```py +import torch +from PIL import Image +from diffusers import AutoPipelineForText2Image + def latents_to_rgb(latents): weights = ( (60, -60, 25, -70), @@ -114,13 +121,9 @@ def decode_tensors(pipe, step, timestep, callback_kwargs): return callback_kwargs ``` -Use the `callback_on_step_end_tensor_inputs` parameter to specify what input type to modify, which in this case, are the latents. +Use `callback_on_step_end_tensor_inputs` to choose which tensors the callback receives, which in this case, are the latents. ```py -import torch -from PIL import Image -from diffusers import AutoPipelineForText2Image - pipeline = AutoPipelineForText2Image.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", dtype=torch.float16, diff --git a/docs/source/en/using-diffusers/custom_pipeline_overview.md b/docs/source/en/using-diffusers/custom_pipeline_overview.md index 0bc002c83dc6..3b03bf863452 100644 --- a/docs/source/en/using-diffusers/custom_pipeline_overview.md +++ b/docs/source/en/using-diffusers/custom_pipeline_overview.md @@ -10,8 +10,6 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -[[open-in-colab]] - # Community pipelines and components Community pipelines are [`DiffusionPipeline`] classes that are different from the original paper implementation. They provide additional functionality or extend the original pipeline implementation. @@ -19,17 +17,17 @@ Community pipelines are [`DiffusionPipeline`] classes that are different from th > [!TIP] > Check out the community pipelines in [diffusers/examples/community](https://github.com/huggingface/diffusers/tree/main/examples/community) with inference and training examples for how to use them. -Community pipelines are either stored on the Hub or the Diffusers' GitHub repository. Hub pipelines are completely customizable (scheduler, models, pipeline code, etc.) while GitHub pipelines are limited to only the custom pipeline code. Further compare the two community pipeline types in the table below. +Community pipelines are either stored on the Hub or the Diffusers' GitHub repository. Hub pipelines are completely customizable (scheduler, models, pipeline code, etc.) while GitHub pipelines are limited to only the custom pipeline code. Compare the two pipeline types in the table below. | | GitHub | Hub | |---|---|---| -| Usage | Same. | Same. | +| Usage | `custom_pipeline=""` | Hub repo id or `trust_remote_code=True` when `_class_name` points at custom code | | Review process | Open a Pull Request on GitHub and undergo a review process from the Diffusers team before merging. This option is slower. | Upload directly to a Hub repository without a review. This is the fastest option. | | Visibility | Included in the official Diffusers repository and docs. | Included on your Hub profile and relies on your own usage and promotion to gain visibility. | -## custom_pipeline +## Loading a community pipeline -Load either community pipeline types by passing the `custom_pipeline` argument to [`~DiffusionPipeline.from_pretrained`]. +Load either type of community pipeline by passing `custom_pipeline` to [`~DiffusionPipeline.from_pretrained`]. ```py import torch @@ -52,7 +50,7 @@ from diffusers import DiffusionPipeline pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-3-medium-diffusers", custom_pipeline="pipeline_stable_diffusion_3_instruct_pix2pix", - custom_revision="main" + custom_revision="main", dtype=torch.float16, device_map="cuda" # or "mps", "xpu", "cpu" ) @@ -72,6 +70,7 @@ There are a few ways to load a community pipeline. pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-3-medium-diffusers", custom_pipeline="path/to/pipeline_directory", + trust_remote_code=True, dtype=torch.float16, device_map="cuda" # or "mps", "xpu", "cpu" ) @@ -83,9 +82,11 @@ There are a few ways to load a community pipeline. import torch from diffusers import DiffusionPipeline - pipeline_sd = DiffusionPipeline.from_pretrained("emilianJR/CyberRealistic_V3", dtype=torch.float16, device_map="cuda") # or "mps", "xpu", "cpu" + pipeline_sd = DiffusionPipeline.from_pretrained( + "emilianJR/CyberRealistic_V3", dtype=torch.float16, device_map="cuda" # or "mps", "xpu", "cpu" + ) pipeline_lpw = DiffusionPipeline.from_pipe( - pipeline_sd, custom_pipeline="lpw_stable_diffusion", device_map="cuda" + pipeline_sd, custom_pipeline="lpw_stable_diffusion" ) ``` @@ -93,7 +94,8 @@ There are a few ways to load a community pipeline. ## Community components -Community components let users build pipelines with custom transformers, UNets, VAEs, and schedulers not supported by Diffusers. These components require Python module implementations. +Community components let you build pipelines with custom transformers, UNets, VAEs, and schedulers not supported by Diffusers. These components need their own Python modules. + This section shows how users can use community components to build a community pipeline using [showlab/show-1-base](https://huggingface.co/showlab/show-1-base) as an example. @@ -106,8 +108,8 @@ from diffusers import DPMSolverMultistepScheduler pipeline_id = "showlab/show-1-base" tokenizer = T5Tokenizer.from_pretrained(pipeline_id, subfolder="tokenizer") text_encoder = T5EncoderModel.from_pretrained(pipeline_id, subfolder="text_encoder") -scheduler = DPMSolverMultistepScheduler.from_pretrained(pipe_id, subfolder="scheduler") -feature_extractor = CLIPImageProcessor.from_pretrained(pipe_id, subfolder="feature_extractor") +scheduler = DPMSolverMultistepScheduler.from_pretrained(pipeline_id, subfolder="scheduler") +feature_extractor = CLIPImageProcessor.from_pretrained(pipeline_id, subfolder="feature_extractor") ``` > [!WARNING] @@ -121,7 +123,7 @@ from showone_unet_3d_condition import ShowOneUNet3DConditionModel unet = ShowOneUNet3DConditionModel.from_pretrained(pipeline_id, subfolder="unet") ``` -3. Load the custom pipeline code (already implemented in [pipeline_t2v_base_pixel.py](https://huggingface.co/sayakpaul/show-1-base-with-code/blob/main/pipeline_t2v_base_pixel.py)). This script contains a custom `TextToVideoIFPipeline` class for generating videos from text. Like the custom UNet, any code required for `TextToVideIFPipeline` should be placed in `pipeline_t2v_base_pixel.py`. +3. Load the custom pipeline code (already implemented in [pipeline_t2v_base_pixel.py](https://huggingface.co/sayakpaul/show-1-base-with-code/blob/main/pipeline_t2v_base_pixel.py)). This script contains a custom `TextToVideoIFPipeline` class for generating videos from text. Like the custom UNet, any code required for `TextToVideoIFPipeline` should be placed in `pipeline_t2v_base_pixel.py`. Initialize `TextToVideoIFPipeline` with `ShowOneUNet3DConditionModel`. @@ -135,9 +137,8 @@ pipeline = TextToVideoIFPipeline( tokenizer=tokenizer, scheduler=scheduler, feature_extractor=feature_extractor, - device_map="cuda", # or "mps", "xpu", "cpu" - dtype=torch.float16 ) +pipeline = pipeline.to("cuda") # or "mps", "xpu", "cpu" ``` 4. Push the pipeline to the Hub to share with the community. @@ -148,7 +149,7 @@ pipeline.push_to_hub("custom-t2v-pipeline") After the pipeline is successfully pushed, make the following changes. -- Change the `_class_name` attribute in [model_index.json](https://huggingface.co/sayakpaul/show-1-base-with-code/blob/main/model_index.json#L2) to `"pipeline_t2v_base_pixel"` and `"TextToVideoIFPipeline"`. +- Change the `_class_name` attribute in [model_index.json](https://huggingface.co/sayakpaul/show-1-base-with-code/blob/main/model_index.json#L2) to `["pipeline_t2v_base_pixel", "TextToVideoIFPipeline"]`. - Upload `showone_unet_3d_condition.py` to the [unet](https://huggingface.co/sayakpaul/show-1-base-with-code/blob/main/unet/showone_unet_3d_condition.py) subfolder. - Upload `pipeline_t2v_base_pixel.py` to the pipeline [repository](https://huggingface.co/sayakpaul/show-1-base-with-code/tree/main). @@ -164,9 +165,9 @@ pipeline = DiffusionPipeline.from_pretrained( ``` > [!WARNING] -> As an additional precaution with `trust_remote_code=True`, we strongly encourage passing a commit hash to the `revision` argument in [`~DiffusionPipeline.from_pretrained`] to make sure the code hasn't been updated with new malicious code (unless you fully trust the model owners). +> As an additional precaution with `trust_remote_code=True`, pass a commit hash to the `revision` argument in [`~DiffusionPipeline.from_pretrained`] to make sure the code hasn't been updated with new malicious code (unless you fully trust the model owners). -## Resources +## Next steps - Take a look at Issue [#841](https://github.com/huggingface/diffusers/issues/841) for more context about why we're adding community pipelines to help everyone easily share their work without being slowed down. - Check out the [stabilityai/japanese-stable-diffusion-xl](https://huggingface.co/stabilityai/japanese-stable-diffusion-xl/) repository for an additional example of a community pipeline that also uses the `trust_remote_code` feature. \ No newline at end of file diff --git a/docs/source/en/using-diffusers/loading.md b/docs/source/en/using-diffusers/loading.md index b98e7e25b2d0..c59a3b0ebb9a 100644 --- a/docs/source/en/using-diffusers/loading.md +++ b/docs/source/en/using-diffusers/loading.md @@ -10,11 +10,9 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -[[open-in-colab]] - # DiffusionPipeline -Diffusion models consists of multiple components like UNets or diffusion transformers (DiTs), text encoders, variational autoencoders (VAEs), and schedulers. The [`DiffusionPipeline`] wraps all of these components into a single easy-to-use API without giving up the flexibility to modify it's components. +Diffusion models consist of multiple components like UNets or diffusion transformers (DiTs), text encoders, variational autoencoders (VAEs), and schedulers. The [`DiffusionPipeline`] wraps all of these components into a single easy-to-use API without giving up the flexibility to modify its components. This guide will show you how to load a [`DiffusionPipeline`]. @@ -33,7 +31,7 @@ pipeline = DiffusionPipeline.from_pretrained( ) ``` -Every model has a specific pipeline subclass that inherits from [`DiffusionPipeline`]. A subclass usually has a narrow focus and are task-specific. See the table below for an example. +Every model has a specific pipeline subclass that inherits from [`DiffusionPipeline`]. A subclass usually has a narrow focus and is task-specific. See the table below for an example. | pipeline subclass | task | |---|---| @@ -62,17 +60,17 @@ Pipelines can also be run locally. Use [`~huggingface_hub.snapshot_download`] to ```py from huggingface_hub import snapshot_download -snapshot_download(repo_id="Qwen/Qwen-Image") +local_dir = snapshot_download(repo_id="Qwen/Qwen-Image") ``` -The model is downloaded to your [cache](../installation#cache). Pass the folder path to [`~QwenImagePipeline.from_pretrained`] to load it. +Pass that path to [`~QwenImagePipeline.from_pretrained`] to load it. ```py import torch from diffusers import QwenImagePipeline pipeline = QwenImagePipeline.from_pretrained( - "path/to/your/cache", dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu" + local_dir, dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu" ) ``` @@ -109,14 +107,13 @@ print(pipeline.transformer.dtype, pipeline.vae.dtype) ## Device placement -The `device_map` argument determines individual model or pipeline placement on an accelerator like a GPU. It is especially helpful when there are multiple GPUs. - -A pipeline supports two options for `device_map`, `"cuda"` and `"balanced"`. Refer to the table below to compare the placement strategies. +The `device_map` argument places a pipeline on devices. At runtime, Diffusers accepts `"balanced"`, `"cpu"`, and the accelerator string for the machine you are on (typically `"cuda"`, `"mps"`, or `"xpu"`). | parameter | description | |---|---| -| `"cuda"` | places pipeline on a supported accelerator device like CUDA | -| `"balanced"` | evenly distributes pipeline on all GPUs | +| accelerator string | Places the pipeline on that device. Pass the accelerator your machine supports, such as `"cuda"`, `"mps"`, or `"xpu"`. | +| `"balanced"` | Spreads pipeline components across visible GPUs. Use with `max_memory` when you need per-device caps. | +| `"cpu"` | Places the pipeline on CPU. | Use the `max_memory` argument in [`~DiffusionPipeline.from_pretrained`] to allocate a maximum amount of memory to use on each device. By default, Diffusers uses the maximum amount available. @@ -126,9 +123,10 @@ from diffusers import DiffusionPipeline max_memory = {0: "16GB", 1: "16GB"} pipeline = DiffusionPipeline.from_pretrained( - "Qwen/Qwen-Image", + "Qwen/Qwen-Image", dtype=torch.bfloat16, - device_map="cuda", # or "mps", "xpu", "cpu" + device_map="balanced", + max_memory=max_memory, ) ``` @@ -136,7 +134,7 @@ The `hf_device_map` attribute allows you to access and view the `device_map`. ```py print(pipeline.hf_device_map) -# {'unet': 1, 'vae': 1, 'safety_checker': 0, 'text_encoder': 0} +# {'transformer': 1, 'vae': 1, 'text_encoder': 0} ``` Reset a pipeline's `device_map` with the [`~DiffusionPipeline.reset_device_map`] method. This is necessary if you want to use methods such as `.to()`, [`~DiffusionPipeline.enable_sequential_cpu_offload`], and [`~DiffusionPipeline.enable_model_cpu_offload`]. @@ -167,7 +165,7 @@ pipeline = DiffusionPipeline.from_pretrained( ## Replacing models in a pipeline -[`DiffusionPipeline`] is flexible and accommodates loading different models or schedulers. You can experiment with different schedulers to optimize for generation speed or quality, and you can replace models with more performant ones. +[`DiffusionPipeline`] is flexible and accommodates loading other models or schedulers. You can experiment with different schedulers to optimize for generation speed or quality, and you can replace models with more performant ones. The example below uses a more stable VAE version. @@ -236,7 +234,7 @@ Some methods may not work correctly on pipelines created with [`~DiffusionPipeli ## Safety checker -Diffusers provides a [safety checker](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) for older Stable Diffusion models to prevent generating harmful content. It screens the generated output against a set of hardcoded harmful concepts. +Diffusers provides a [safety checker](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/safety_checker.py) for older Stable Diffusion checkpoints to prevent generating harmful content. It screens the generated output against a set of hardcoded harmful concepts. Newer models such as Qwen-Image do not have this checker. If you want to disable the safety checker, pass `safety_checker=None` in [`~DiffusionPipeline.from_pretrained`] as shown below. @@ -250,3 +248,12 @@ pipeline = DiffusionPipeline.from_pretrained( You have disabled the safety checker for by passing `safety_checker=None`. Ensure that you abide by the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend keeping the safety filter enabled in all public-facing circumstances, disabling it only for use cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 . """ ``` + +## Next steps + +Once a pipeline loads, you usually tune the denoising schedule, swap weight formats, or attach adapters. + +- [Schedulers](./schedulers) covers swapping and configuring the denoising algorithm. +- [Model formats](./other-formats) covers GGUF, single-file checkpoints, and other weight layouts. +- [Reduce memory usage](../optimization/memory) covers offloading and other memory tools. +- [LoRA](../tutorials/using_peft_for_inference) covers loading adapters on a pipeline. diff --git a/docs/source/en/using-diffusers/other-formats.md b/docs/source/en/using-diffusers/other-formats.md index a4a7620d587b..c3e3c99abc29 100644 --- a/docs/source/en/using-diffusers/other-formats.md +++ b/docs/source/en/using-diffusers/other-formats.md @@ -10,14 +10,22 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -[[open-in-colab]] - # Model formats Diffusion models are typically stored in the Diffusers format or single-file format. Model files can be stored in various file types such as safetensors or ckpt. > [!TIP] -> Format refers to whether the weights are stored in a directory structure and file refers to the file type. +> Format refers to whether the weights are stored in a directory structure. File type refers to how those weights are serialized, such as safetensors or ckpt. + +```text +Diffusers format Single-file format +model/ model.safetensors (or .ckpt) +├─ model_index.json └─ all components in one file +├─ unet/ (or transformer/) +├─ text_encoder/ +├─ vae/ +└─ scheduler/ +``` This guide will show you how to load pipelines and models from these formats and files. @@ -32,7 +40,7 @@ The Diffusers format stores each model (UNet, transformer, text encoder) in a se ## Single file format -A single-file format stores *all* the model (UNet, transformer, text encoder) weights in a single file. Benefits of single-file formats include the following. +A single-file format stores *all* the model weights (UNet, transformer, text encoder) in a single file. Benefits of single-file formats include: - Greater compatibility with [ComfyUI](https://github.com/comfyanonymous/ComfyUI) or [Automatic1111](https://github.com/AUTOMATIC1111/stable-diffusion-webui). - Easier to download and share a single file. @@ -81,10 +89,13 @@ ckpt_path = "https://huggingface.co/segmind/SSD-1B/blob/main/SSD-1B.safetensors" pipeline = StableDiffusionXLPipeline.from_single_file(ckpt_path, config="segmind/SSD-1B") ``` -Diffusers attempts to infer the pipeline components based on the signature types of the pipeline class when using `original_config` with `local_files_only=True`. It won't download the config files from a Hub repository to avoid backward breaking changes when you can't connect to the internet. This method isn't as reliable as providing a path to a local model with the `config` argument and may lead to errors. You should run the pipeline with `local_files_only=False` to download the config files to the local cache to avoid errors. +When you pass `original_config` with `local_files_only=True`, Diffusers infers pipeline components from the pipeline class signature. It does not download config files from the Hub, which avoids breaking changes when you are offline. That path is less reliable than giving `config` a local model path, and it can error. Run once with `local_files_only=False` so the configs land in the local cache if you need them offline later. Override default configs by passing the arguments directly to [`~loaders.FromSingleFileMixin.from_single_file`]. The examples below demonstrate how to override the configs in a pipeline or model. + + + ```py from diffusers import StableDiffusionXLInstructPix2PixPipeline @@ -94,6 +105,9 @@ pipeline = StableDiffusionXLInstructPix2PixPipeline.from_single_file( ) ``` + + + ```py from diffusers import UNet2DConditionModel @@ -101,6 +115,9 @@ ckpt_path = "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blo model = UNet2DConditionModel.from_single_file(ckpt_path, upcast_attention=True) ``` + + + ### Local files The [`~loaders.FromSingleFileMixin.from_single_file`] method attempts to configure a pipeline or model by inferring the model type from the keys in the checkpoint file. For example, any single file checkpoint based on the Stable Diffusion XL base model is configured from [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0). @@ -128,7 +145,7 @@ pipeline = StableDiffusionXLPipeline.from_single_file( ### Symlink -If you're working with a file system that does not support symlinking, download the checkpoint file to a local directory first with the `local_dir` parameter. Using the `local_dir` parameter automatically disables symlinks. +If you're working with a file system that doesn't support symlinking, download the checkpoint file to a local directory first with the `local_dir` parameter. Using the `local_dir` parameter automatically disables symlinks. ```py from huggingface_hub import hf_hub_download, snapshot_download @@ -136,7 +153,7 @@ from diffusers import StableDiffusionXLPipeline my_local_checkpoint_path = hf_hub_download( repo_id="segmind/SSD-1B", - filename="SSD-1B.safetensors" + filename="SSD-1B.safetensors", local_dir="my_local_checkpoints", ) print("My local checkpoint: ", my_local_checkpoint_path) @@ -170,35 +187,39 @@ Use [`~DiffusionPipeline.from_pretrained`] or [`~loaders.FromSingleFileMixin.fro ```py import torch -from diffusers import DiffusionPipeline +from diffusers import DiffusionPipeline, StableDiffusionXLPipeline pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", - torch.dtype=torch.float16, - device_map="cuda" # or "mps", "xpu", "cpu" + dtype=torch.float16, + device_map="cuda", # or "mps", "xpu", "cpu" ) -pipeline = DiffusionPipeline.from_single_file( +pipeline = StableDiffusionXLPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", dtype=torch.float16, ) ``` -If you're using a checkpoint trained with a Diffusers training script, metadata such as the LoRA configuration, is automatically saved. When the file is loaded, the metadata is parsed to correctly configure the LoRA and avoid missing or incorrect LoRA configs. Inspect the metadata of a safetensors file by clicking on the safetensors logo logo next to the file on the Hub. +If you're using a checkpoint trained with a Diffusers training script, metadata such as the LoRA configuration is automatically saved. When the file is loaded, that metadata is parsed so the LoRA is configured correctly. Inspect the metadata of a safetensors file by clicking on the safetensors logo icon next to the file on the Hub. -Save the metadata for LoRAs that aren't trained with Diffusers with either `transformer_lora_adapter_metadata` or `unet_lora_adapter_metadata` depending on your model. For the text encoder, use the `text_encoder_lora_adapter_metadata` and `text_encoder_2_lora_adapter_metadata` arguments in [`~loaders.FluxLoraLoaderMixin.save_lora_weights`]. This is only supported for safetensors files. +Save LoRA adapter metadata for checkpoints that aren't trained with Diffusers by passing it to [`~loaders.FluxLoraLoaderMixin.save_lora_weights`]. Use `transformer_lora_adapter_metadata` for the transformer and `text_encoder_lora_adapter_metadata` for the text encoder. You must also pass `save_directory` and at least one of `transformer_lora_layers` or `text_encoder_lora_layers`. This path is only supported for safetensors files. ```py import torch +from peft.utils import get_peft_model_state_dict from diffusers import FluxPipeline pipeline = FluxPipeline.from_pretrained( - "black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16 -).to("cuda") # or "mps", "xpu", "cpu" + "black-forest-labs/FLUX.1-dev", dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu" +) pipeline.load_lora_weights("linoyts/yarn_art_Flux_LoRA") + +transformer_lora_layers = get_peft_model_state_dict(pipeline.transformer) pipeline.save_lora_weights( - text_encoder_lora_adapter_metadata={"r": 8, "lora_alpha": 8}, - text_encoder_2_lora_adapter_metadata={"r": 8, "lora_alpha": 8} + save_directory="path/to/lora", + transformer_lora_layers=transformer_lora_layers, + transformer_lora_adapter_metadata={"r": 8, "lora_alpha": 8}, ) ``` @@ -211,18 +232,34 @@ Pickled files may be unsafe because they can be exploited to execute malicious c Use [`~loaders.FromSingleFileMixin.from_single_file`] to load a ckpt file. ```py -from diffusers import DiffusionPipeline +from diffusers import StableDiffusionPipeline -pipeline = DiffusionPipeline.from_single_file( +pipeline = StableDiffusionPipeline.from_single_file( "https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/blob/main/v1-5-pruned.ckpt" ) ``` +### GGUF + +GGUF stores prequantized weights in a single file. Diffusers loads GGUF through model [`~loaders.FromSingleFileMixin.from_single_file`] with [`GGUFQuantizationConfig`]. Pipeline-level GGUF loading is not supported. + +See [GGUF](../quantization/gguf) for install steps and full examples. + +```py +import torch +from diffusers import FluxTransformer2DModel, GGUFQuantizationConfig + +transformer = FluxTransformer2DModel.from_single_file( + "https://huggingface.co/city96/FLUX.1-dev-gguf/blob/main/flux1-dev-Q2_K.gguf", + quantization_config=GGUFQuantizationConfig(compute_dtype=torch.bfloat16), +) +``` + ## Converting formats and files -Diffusers provides scripts and methods to convert format and files to enable broader support across the diffusion ecosystem. +Diffusers provides scripts and methods to convert formats and files so more tools in the diffusion ecosystem can use them. -Take a look at the [diffusers/scripts](https://github.com/huggingface/diffusers/tree/main/scripts) folder to find a conversion script. Scripts with `"to_diffusers` appended at the end converts a model to the Diffusers format. Each script has a specific set of arguments for configuring the conversion. Make sure you check what arguments are available. +Take a look at the [diffusers/scripts](https://github.com/huggingface/diffusers/tree/main/scripts) folder to find a conversion script. Scripts with `"to_diffusers"` appended at the end convert a model to the Diffusers format. Each script has a specific set of arguments for configuring the conversion. Make sure you check what arguments are available. The example below converts a model stored in Diffusers format to a single-file format. Provide the path to the model to convert and where to save the converted model. You can optionally specify what file type and data type to save the model as. @@ -233,16 +270,14 @@ python convert_diffusers_to_original_sdxl.py --model_path path/to/model/to/conve The [`~DiffusionPipeline.save_pretrained`] method also saves a model in Diffusers format and takes care of creating subfolders for each model. It saves the files as safetensor files by default. ```py -from diffusers import DiffusionPipeline +from diffusers import StableDiffusionXLPipeline -pipeline = DiffusionPipeline.from_single_file( +pipeline = StableDiffusionXLPipeline.from_single_file( "https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0.safetensors", ) -pipeline.save_pretrained() +pipeline.save_pretrained("path/to/save/model") ``` -Finally, you can use a Space like [SD To Diffusers](https://hf.co/spaces/diffusers/sd-to-diffusers) or [SD-XL To Diffusers](https://hf.co/spaces/diffusers/sdxl-to-diffusers) to convert models to the Diffusers format. It'll open a PR on your model repository with the converted files. This is the easiest way to convert a model, but it may fail for more complicated models. Using a conversion script is more reliable. - -## Resources +## Next steps - Learn more about the design decisions and why safetensor files are preferred for saving and loading model weights in the [Safetensors audited as really safe and becoming the default](https://blog.eleuther.ai/safetensors-security-audit/) blog post. diff --git a/docs/source/en/using-diffusers/push_to_hub.md b/docs/source/en/using-diffusers/push_to_hub.md index 4319f620a915..7efa026b3cf4 100644 --- a/docs/source/en/using-diffusers/push_to_hub.md +++ b/docs/source/en/using-diffusers/push_to_hub.md @@ -10,17 +10,13 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -[[open-in-colab]] - # Sharing pipelines and models -Share your pipeline or models and schedulers on the Hub with the [`~diffusers.utils.PushToHubMixin`] class. This class: +Share your pipelines, models, and schedulers on the Hub with [`~utils.PushToHubMixin`]. This mixin: 1. creates a repository on the Hub 2. saves your model, scheduler, or pipeline files so they can be reloaded later -3. uploads folder containing these files to the Hub - -This guide will show you how to upload your files to the Hub with the [`~diffusers.utils.PushToHubMixin`] class. +3. uploads the folder containing these files to the Hub Log in to your Hugging Face account with your access [token](https://huggingface.co/settings/tokens). @@ -43,9 +39,11 @@ hf auth login +Push to your user namespace with a short id (`"my-controlnet-model"`) or to an org with `"your-org/my-controlnet-model"`. + ## Models -To push a model to the Hub, call [`~diffusers.utils.PushToHubMixin.push_to_hub`] and specify the repository id of the model. +To push a model to the Hub, call [`~utils.PushToHubMixin.push_to_hub`] and specify the repository id of the model. ```py from diffusers import ControlNetModel @@ -61,9 +59,9 @@ controlnet = ControlNetModel( controlnet.push_to_hub("my-controlnet-model") ``` -The [`~diffusers.utils.PushToHubMixin.push_to_hub`] method saves the model's `config.json` file and the weights are automatically saved as safetensors files. +The [`~utils.PushToHubMixin.push_to_hub`] method saves the model's `config.json` file and the weights are automatically saved as [safetensors files](./other-formats#safetensors). -Load the model again with [`~DiffusionPipeline.from_pretrained`]. +Load the model again with [`ControlNetModel.from_pretrained`]. ```py model = ControlNetModel.from_pretrained("your-namespace/my-controlnet-model") @@ -71,7 +69,7 @@ model = ControlNetModel.from_pretrained("your-namespace/my-controlnet-model") ## Scheduler -To push a scheduler to the Hub, call [`~diffusers.utils.PushToHubMixin.push_to_hub`] and specify the repository id of the scheduler. +To push a scheduler to the Hub, call [`~utils.PushToHubMixin.push_to_hub`] and specify the repository id of the scheduler. ```py from diffusers import DDIMScheduler @@ -83,102 +81,47 @@ scheduler = DDIMScheduler( clip_sample=False, set_alpha_to_one=False, ) -scheduler.push_to_hub("my-controlnet-scheduler") +scheduler.push_to_hub("my-ddim-scheduler") ``` -The [`~diffusers.utils.PushToHubMixin.push_to_hub`] function saves the scheduler's `scheduler_config.json` file to the specified repository. +The [`~utils.PushToHubMixin.push_to_hub`] method saves the scheduler's `scheduler_config.json` file to the specified repository. Load the scheduler again with [`~SchedulerMixin.from_pretrained`]. ```py -scheduler = DDIMScheduler.from_pretrained("your-namepsace/my-controlnet-scheduler") +scheduler = DDIMScheduler.from_pretrained("your-namespace/my-ddim-scheduler") ``` ## Pipeline -To push a pipeline to the Hub, initialize the pipeline components with your desired parameters. +To push a pipeline to the Hub, load it with [`~DiffusionPipeline.from_pretrained`], then call [`~utils.PushToHubMixin.push_to_hub`] with a repository id. ```py -from diffusers import ( - UNet2DConditionModel, - AutoencoderKL, - DDIMScheduler, - StableDiffusionPipeline, -) -from transformers import CLIPTextModel, CLIPTextConfig, CLIPTokenizer +import torch +from diffusers import DiffusionPipeline -unet = UNet2DConditionModel( - block_out_channels=(32, 64), - layers_per_block=2, - sample_size=32, - in_channels=4, - out_channels=4, - down_block_types=("DownBlock2D", "CrossAttnDownBlock2D"), - up_block_types=("CrossAttnUpBlock2D", "UpBlock2D"), - cross_attention_dim=32, +pipeline = DiffusionPipeline.from_pretrained( + "Qwen/Qwen-Image", dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu" ) - -scheduler = DDIMScheduler( - beta_start=0.00085, - beta_end=0.012, - beta_schedule="scaled_linear", - clip_sample=False, - set_alpha_to_one=False, -) - -vae = AutoencoderKL( - block_out_channels=[32, 64], - in_channels=3, - out_channels=3, - down_block_types=["DownEncoderBlock2D", "DownEncoderBlock2D"], - up_block_types=["UpDecoderBlock2D", "UpDecoderBlock2D"], - latent_channels=4, -) - -text_encoder_config = CLIPTextConfig( - bos_token_id=0, - eos_token_id=2, - hidden_size=32, - intermediate_size=37, - layer_norm_eps=1e-05, - num_attention_heads=4, - num_hidden_layers=5, - pad_token_id=1, - vocab_size=1000, -) -text_encoder = CLIPTextModel(text_encoder_config) -tokenizer = CLIPTokenizer.from_pretrained("hf-internal-testing/tiny-random-clip") +pipeline.push_to_hub("your-namespace/my-qwen-image") ``` -Pass all components to the pipeline and call [`~diffusers.utils.PushToHubMixin.push_to_hub`]. +The [`~utils.PushToHubMixin.push_to_hub`] method saves each component to a subfolder in the repository. Load the pipeline again with [`DiffusionPipeline.from_pretrained`]. ```py -components = { - "unet": unet, - "scheduler": scheduler, - "vae": vae, - "text_encoder": text_encoder, - "tokenizer": tokenizer, - "safety_checker": None, - "feature_extractor": None, -} - -pipeline = StableDiffusionPipeline(**components) -pipeline.push_to_hub("my-pipeline") -``` - -The [`~diffusers.utils.PushToHubMixin.push_to_hub`] method saves each component to a subfolder in the repository. Load the pipeline again with [`~DiffusionPipeline.from_pretrained`]. - -```py -pipeline = StableDiffusionPipeline.from_pretrained("your-namespace/my-pipeline") +pipeline = DiffusionPipeline.from_pretrained("your-namespace/my-qwen-image") ``` ## Privacy -Set `private=True` in [`~diffusers.utils.PushToHubMixin.push_to_hub`] to keep a model, scheduler, or pipeline files private. +Set `private=True` in [`~utils.PushToHubMixin.push_to_hub`] to keep a model, scheduler, or pipeline files private. ```py controlnet.push_to_hub("my-controlnet-model-private", private=True) ``` -Private repositories are only visible to you. Other users won't be able to clone the repository and it won't appear in search results. Even if a user has the URL to your private repository, they'll receive a `404 - Sorry, we can't find the page you are looking for`. You must be [logged in](https://huggingface.co/docs/huggingface_hub/quick-start#login) to load a model from a private repository. \ No newline at end of file +Pass `create_pr=True` to open a pull request on an existing Hub repository instead of pushing straight to the default branch. + +Models and pipelines also accept `variant=` on push when you want a named weight file such as `fp16`. Schedulers do not use `variant`. + +Private repositories are only visible to you. Other users won't be able to clone the repository and it won't appear in search results. Even if a user has the URL to your private repository, they'll receive a `404 - Sorry, we can't find the page you are looking for`. You must be [logged in](https://huggingface.co/docs/huggingface_hub/quick-start#login) to load a model from a private repository. diff --git a/docs/source/en/using-diffusers/reusing_seeds.md b/docs/source/en/using-diffusers/reusing_seeds.md index c1b7a10392b3..44ab937d3497 100644 --- a/docs/source/en/using-diffusers/reusing_seeds.md +++ b/docs/source/en/using-diffusers/reusing_seeds.md @@ -12,7 +12,7 @@ specific language governing permissions and limitations under the License. # Reproducibility -Diffusion is a random process that generates a different output every time. For certain situations like testing and replicating results, you want to generate the same result each time, across releases and platforms within a certain tolerance range. +Diffusion is a random process that generates a different output every time. For use cases like testing and replicating results, you want to generate the same result each time, across releases and platforms within a certain tolerance range. This guide will show you how to control sources of randomness and enable deterministic algorithms. @@ -21,75 +21,88 @@ This guide will show you how to control sources of randomness and enable determi Pipelines rely on [torch.randn](https://pytorch.org/docs/stable/generated/torch.randn.html), which uses a different random seed each time, to create the initial noisy tensors. To generate the same output on a CPU or GPU, use a [Generator](https://docs.pytorch.org/docs/stable/generated/torch.Generator.html) to manage how random values are generated. > [!TIP] -> If reproducibility is important to your use case, we recommend always using a CPU `Generator`. The performance loss is often negligible and you'll generate more similar values. +> If reproducibility is important, you should use a CPU `Generator`. The performance loss is often negligible and you'll generate more similar values. -The GPU uses a different random number generator than the CPU. Diffusers solves this issue with the [`~utils.torch_utils.randn_tensor`] function to create the random tensor on a CPU and then moving it to the GPU. This function is used everywhere inside the pipeline and you don't need to explicitly call it. +Use a CPU `Generator` when you care about reproducibility. CPU RNG is more stable across machines. -Use [manual_seed](https://docs.pytorch.org/docs/stable/generated/torch.manual_seed.html) as shown below to set a seed. +When you pass a CPU `Generator`, Diffusers’ [`~utils.torch_utils.randn_tensor`] samples on the CPU and moves the tensor to the GPU inside the pipeline. You do not call `randn_tensor` or `.to("cuda")` yourself. A GPU `Generator` samples on-device instead and can diverge from CPU results. + +Use [manual_seed](https://docs.pytorch.org/docs/stable/generated/torch.manual_seed.html) to set a seed. ```py import torch -import numpy as np -from diffusers import DDIMPipeline +from diffusers import DiffusionPipeline -ddim = DDIMPipeline.from_pretrained("google/ddpm-cifar10-32", device_map="cuda") # or "mps", "xpu", "cpu" +pipeline = DiffusionPipeline.from_pretrained( + "Qwen/Qwen-Image", dtype=torch.bfloat16, device_map="cuda" # or "mps", "xpu", "cpu" +) generator = torch.manual_seed(0) -image = ddim(num_inference_steps=2, output_type="np", generator=generator).images -print(np.abs(image).sum()) +image = pipeline( + prompt="a red apple on a wooden table", + generator=generator, + num_inference_steps=4, +).images[0] ``` -Set `device="cpu"` in the `Generator` and use [manual_seed](https://docs.pytorch.org/docs/stable/generated/torch.manual_seed.html) to set a seed for generating random numbers. +Create a CPU `Generator` and set a seed with [Generator.manual_seed](https://docs.pytorch.org/docs/stable/generated/torch.Generator.html#torch.Generator.manual_seed). ```py import torch -import numpy as np -from diffusers import DDIMPipeline +from diffusers import DiffusionPipeline -ddim = DDIMPipeline.from_pretrained("google/ddpm-cifar10-32") +pipeline = DiffusionPipeline.from_pretrained( + "Qwen/Qwen-Image", dtype=torch.bfloat16, device_map="cpu" +) generator = torch.Generator(device="cpu").manual_seed(0) -image = ddim(num_inference_steps=2, output_type="np", generator=generator).images -print(np.abs(image).sum()) +image = pipeline( + prompt="a red apple on a wooden table", + generator=generator, + num_inference_steps=4, +).images[0] ``` -The `Generator` object should be passed to the pipeline instead of an integer seed. `Generator` maintains a *random state* that is consumed and modified when used. Once consumed, the same `Generator` object produces different results in subsequent calls, even across different pipelines, because its *state* has changed. +Pass a `Generator` object to the pipeline instead of an integer seed. A `Generator` keeps a random state that is consumed and updated when you use it. After that, the same object produces different results on later calls, even across pipelines, because its state has changed. Reseed it or create a new `Generator` before each call when you need the same seed again. ```py -generator = torch.manual_seed(0) +import torch + +prompt = "a red apple on a wooden table" for _ in range(5): -- image = pipeline(prompt, generator=generator) -+ image = pipeline(prompt, generator=torch.manual_seed(0)) + generator = torch.manual_seed(0) + image = pipeline(prompt, generator=generator, num_inference_steps=4).images[0] ``` ## Deterministic algorithms -PyTorch supports [deterministic algorithms](https://docs.pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms) - where available - for certain operations so they produce the same results. Deterministic algorithms may be slower and decrease performance. +PyTorch supports [deterministic algorithms](https://docs.pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms) (where available) for certain operations so they produce the same results. Deterministic algorithms may be slower and decrease performance. -Use Diffusers' [enable_full_determinism](https://github.com/huggingface/diffusers/blob/142f353e1c638ff1d20bd798402b68f72c1ebbdd/src/diffusers/utils/testing_utils.py#L861) function to enable deterministic algorithms. +Use Diffusers' [`~utils.torch_utils.enable_full_determinism`] to enable deterministic algorithms. ```py -import torch -from diffusers_utils import enable_full_determinism +from diffusers.utils.torch_utils import enable_full_determinism enable_full_determinism() ``` -Under the hood, `enable_full_determinism` works by: - -- Setting the environment variable [CUBLAS_WORKSPACE_CONFIG](https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibility) to `:16:8` to only use one buffer size during rntime. Non-deterministic behavior occurs when operations are used in more than one CUDA stream. -- Disabling benchmarking to find the fastest convolution operation by setting `torch.backends.cudnn.benchmark=False`. Non-deterministic behavior occurs because the benchmark may select different algorithms each time depending on hardware or benchmarking noise. -- Disabling TensorFloat32 (TF32) operations in favor of more precise and consistent full-precision operations. +`enable_full_determinism` works by: +- Setting the environment variable `CUDA_LAUNCH_BLOCKING` to `1` +- Setting the environment variable [CUBLAS_WORKSPACE_CONFIG](https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibility) to `:16:8` so cuBLAS uses a fixed workspace layout at runtime +- Calling `torch.use_deterministic_algorithms(True)` +- Setting `torch.backends.cudnn.deterministic = True` +- Setting `torch.backends.cudnn.benchmark = False` so cuDNN does not pick a different convolution algorithm each run +- Disabling TensorFloat32 (TF32) with `torch.backends.cuda.matmul.allow_tf32 = False` in favor of more precise full-precision matmul -## Resources +## Next steps -We strongly recommend reading PyTorch's developer notes about [Reproducibility](https://docs.pytorch.org/docs/stable/notes/randomness.html). You can try to limit randomness, but it is not *guaranteed* even with an identical seed. \ No newline at end of file +You should read PyTorch's developer notes about [Reproducibility](https://docs.pytorch.org/docs/stable/notes/randomness.html). You can try to limit randomness, but it is not *guaranteed* even with an identical seed. diff --git a/docs/source/en/using-diffusers/schedulers.md b/docs/source/en/using-diffusers/schedulers.md index a17d9e5bfeaf..5d948f256e24 100644 --- a/docs/source/en/using-diffusers/schedulers.md +++ b/docs/source/en/using-diffusers/schedulers.md @@ -10,19 +10,28 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -[[open-in-colab]] - # Schedulers -A scheduler is an algorithm that provides instructions to the denoising process such as how much noise to remove at a certain step. It takes the model prediction from step *t* and applies an update for how to compute the next sample at step *t-1*. Different schedulers produce different results; some are faster while others are more accurate. +A scheduler tells the denoising loop how much noise to remove at each step. Different schedulers trade speed for quality. + +This guide shows how to load a scheduler and customize its timestep schedule, spacing, and sigmas. + +## Choosing a scheduler -Diffusers supports many schedulers and allows you to modify their timestep schedules, timestep spacing, and more, to generate high-quality images in fewer steps. +Start from the checkpoint default. Swap only if you need a different speed or quality tradeoff. -This guide will show you how to load and customize schedulers. +- DPM++ 2M SDE Karras is a strong all-purpose option for many latent diffusion checkpoints. +- [`TCDScheduler`] works well for distilled models. +- Use [`FlowMatchEulerDiscreteScheduler`] or [`FlowMatchHeunDiscreteScheduler`] for FlowMatch models (Qwen-Image, Flux, and similar). +- [`EulerDiscreteScheduler`] or [`EulerAncestralDiscreteScheduler`] often work well for anime-style images. +- [`LCMScheduler`] with an LCM UNet or LoRA for few-step generation when the checkpoint supports it. ## Loading schedulers -Schedulers don't have any parameters and are defined in a configuration file. Access the `.scheduler` attribute of a pipeline to view the configuration. +> [!TIP] +> Flow-matching models such as Qwen-Image and Flux ship [`FlowMatchEulerDiscreteScheduler`] as their default. Keep that scheduler unless you are intentionally experimenting. Swap with [`~ConfigMixin.from_config`] only when the replacement is compatible with the checkpoint. + +Schedulers are config-only and they do not ship weight tensors. Access the `.scheduler` attribute on a pipeline to inspect the loaded config. ```py import torch @@ -34,11 +43,19 @@ pipeline = DiffusionPipeline.from_pretrained( pipeline.scheduler ``` -Load a different scheduler with [`~SchedulerMixin.from_pretrained`] and specify the `subfolder` argument to load the configuration file into the correct subfolder of the pipeline repository. Pass the new scheduler to the existing pipeline. +To swap schedulers on a loaded pipeline, use [`~ConfigMixin.from_config`] with the existing scheduler config so `num_train_timesteps` and related fields stay aligned. For FlowMatch checkpoints (Qwen-Image, Flux, and similar), keep [`FlowMatchEulerDiscreteScheduler`] unless you are intentionally experimenting with a compatible replacement. ```py from diffusers import DPMSolverMultistepScheduler +pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config) +``` + +You can also load a scheduler config from the Hub with [`~SchedulerMixin.from_pretrained`] and pass it into [`~DiffusionPipeline.from_pretrained`] through `scheduler`. + +```py +from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler + dpm = DPMSolverMultistepScheduler.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", subfolder="scheduler" ) @@ -46,7 +63,7 @@ pipeline = DiffusionPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", scheduler=dpm, dtype=torch.float16, - device_map="cuda" # or "mps", "xpu", "cpu" + device_map="cuda", # or "mps", "xpu", "cpu" ) pipeline.scheduler ``` @@ -55,8 +72,22 @@ pipeline.scheduler Timestep or noise schedule decides how noise is distributed over the denoising process. The schedule can be linear or more concentrated toward the beginning or end. It is a precomputed sequence of noise levels generated from the scheduler's default configuration, but it can be customized to use other schedules. +```text +linear (even steps) AYS (denser where it matters) +noise noise + ^ ^ + | * | * + | * | * + | * | ** + | * | *** + | * | ** + | * | * + | * | * + +-----------------> step +-----------------> step +``` + > [!TIP] -> The `timesteps` argument is only supported for a select list of schedulers and pipelines. Feel free to open a feature request if you want to extend these parameters to a scheduler and pipeline that does not currently support it! +> Custom `timesteps` only work if that scheduler’s `set_timesteps` accepts the argument (pipelines check the signature via `retrieve_timesteps` and raise `ValueError` otherwise). Check the scheduler’s API page or `set_timesteps` signature before passing them. The example below uses the [Align Your Steps (AYS)](https://research.nvidia.com/labs/toronto-ai/AlignYourSteps/) schedule which can generate a high-quality image in 10 steps, significantly speeding up generation and reducing computation time. @@ -69,7 +100,7 @@ from diffusers.schedulers import AysSchedules sampling_schedule = AysSchedules["StableDiffusionXLTimesteps"] print(sampling_schedule) -"[999, 845, 730, 587, 443, 310, 193, 116, 53, 13]" +# [999, 845, 730, 587, 443, 310, 193, 116, 53, 13] pipeline = DiffusionPipeline.from_pretrained( "SG161222/RealVisXL_V4.0", @@ -105,7 +136,7 @@ image = pipeline( ### Rescaling schedules -Denoising should begin with pure noise and the signal-to-noise (SNR) ration should be zero. However, some models don't actually start from pure noise which makes it difficult to generate images at brightness extremes. +Denoising should begin with pure noise and the signal-to-noise (SNR) ratio should be zero. However, some models don't actually start from pure noise which makes it difficult to generate images at brightness extremes. > [!TIP] > Train your own model with `v_prediction` by adding the `--prediction_type="v_prediction"` flag to your training script. You can also [search](https://huggingface.co/search/full-text?q=v_prediction&type=model) for existing models trained with `v_prediction`. @@ -148,7 +179,7 @@ image = pipeline(prompt, guidance_rescale=0.7).images[0] ## Timestep spacing -Timestep spacing refers to the specific steps *t* to sample from from the schedule. Diffusers provides three spacing types as shown below. +Timestep spacing refers to the specific steps *t* to sample from the schedule. Diffusers provides three spacing types as shown below. | spacing strategy | spacing calculation | example timesteps | |---|---|---| @@ -198,25 +229,30 @@ image Sigmas is a measure of how noisy a sample is at a certain step as defined by the schedule. When using custom `sigmas`, the `timesteps` are calculated from these values instead of the default scheduler configuration. +```text +step: 0 1 2 3 4 +sigma: σ0 > σ1 > σ2 > σ3 > σ4 ≈ 0 + high noise ---> clean sample +``` + > [!TIP] -> The `sigmas` argument is only supported for a select list of schedulers and pipelines. Feel free to open a feature request if you want to extend these parameters to a scheduler and pipeline that does not currently support it! +> Custom `sigmas` only work if that scheduler’s `set_timesteps` accepts the argument (pipelines check the signature via `retrieve_timesteps` and raise `ValueError` otherwise). Check the scheduler’s API page or `set_timesteps` signature before passing them. -Pass the custom sigmas to the `sigmas` argument in the pipeline. The example below uses the [sigmas](https://github.com/huggingface/diffusers/blob/6529ee67ec02fcf58d2fd9242164ea002b351d75/src/diffusers/schedulers/scheduling_utils.py#L55) from the 10-step AYS schedule. +Pass the custom sigmas to the `sigmas` argument in the pipeline. The example below uses the [sigmas](https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_utils.py) from the 10-step AYS schedule. ```py import torch -from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler +from diffusers import DiffusionPipeline, EulerDiscreteScheduler +from diffusers.schedulers import AysSchedules pipeline = DiffusionPipeline.from_pretrained( "SG161222/RealVisXL_V4.0", dtype=torch.float16, - device_map="cuda" # or "mps", "xpu", "cpu" -) -pipeline.scheduler = DPMSolverMultistepScheduler.from_config( - pipeline.scheduler.config, algorithm_type="sde-dpmsolver++" + device_map="cuda", # or "mps", "xpu", "cpu" ) +pipeline.scheduler = EulerDiscreteScheduler.from_config(pipeline.scheduler.config) -sigmas = [14.615, 6.315, 3.771, 2.181, 1.342, 0.862, 0.555, 0.380, 0.234, 0.113, 0.0] +sigmas = AysSchedules["StableDiffusionXLSigmas"] prompt = "A cinematic shot of a cute little rabbit wearing a jacket and doing a thumbs up" image = pipeline( prompt=prompt, @@ -229,6 +265,12 @@ image = pipeline( [Karras sigmas](https://huggingface.co/papers/2206.00364) resamples the noise schedule for more efficient sampling by clustering sigmas more densely in the middle of the sequence where structure reconstruction is critical, while using fewer sigmas at the beginning and end where noise changes have less impact. This can increase the level of details in a generated image. +```text +default σ: * * * * * * * * even-ish +Karras σ: * * * * * * * denser mid, sparser ends + |---structure---| +``` + Set `use_karras_sigmas=True` in the scheduler to enable it. ```py @@ -250,7 +292,7 @@ prompt = "A cinematic shot of a cute little rabbit wearing a jacket and doing a image = pipeline( prompt=prompt, negative_prompt="", - sigmas=sigmas, + num_inference_steps=20, ).images[0] ``` @@ -267,16 +309,6 @@ image = pipeline( Refer to the scheduler API [overview](../api/schedulers/overview) for a list of schedulers that support Karras sigmas. It should only be used for models trained with Karras sigmas. -## Choosing a scheduler - -It's important to try different schedulers to find the best one for your use case. Here are a few recommendations to help you get started. - -- DPM++ 2M SDE Karras is generally a good all-purpose option. -- [`TCDScheduler`] works well for distilled models. -- [`FlowMatchEulerDiscreteScheduler`] and [`FlowMatchHeunDiscreteScheduler`] for FlowMatch models. -- [`EulerDiscreteScheduler`] or [`EulerAncestralDiscreteScheduler`] for generating anime style images. -- DPM++ 2M paired with [`LCMScheduler`] on SDXL for generating realistic images. - -## Resources +## Next steps -- Read the [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) paper for more details about rescaling the noise schedule to enforce zero SNR. \ No newline at end of file +- Read the [Common Diffusion Noise Schedules and Sample Steps are Flawed](https://huggingface.co/papers/2305.08891) paper for more details about rescaling the noise schedule to enforce zero SNR. diff --git a/docs/source/en/using-diffusers/weighted_prompts.md b/docs/source/en/using-diffusers/weighted_prompts.md index 927ba9214df4..fb18c1aed67a 100644 --- a/docs/source/en/using-diffusers/weighted_prompts.md +++ b/docs/source/en/using-diffusers/weighted_prompts.md @@ -10,11 +10,9 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o specific language governing permissions and limitations under the License. --> -[[open-in-colab]] - # Prompting -Prompts describes what a model should generate. Good prompts are detailed, specific, and structured and they generate better images and videos. +Prompts describe what a model should generate. Good prompts are detailed, specific, structured, and they generate better images and videos. This guide shows you how to write effective prompts and introduces techniques that make them stronger. @@ -22,9 +20,9 @@ This guide shows you how to write effective prompts and introduces techniques th Every effective prompt needs three core elements. -1. Subject - what you want to generate. Start your prompt here. -2. Style - the medium or aesthetic. How should it look? -3. Context - details about actions, setting, and mood. +1. Subject - what you want to generate. Start your prompt here. +2. Style - the medium or aesthetic. How should it look? +3. Context - details about actions, setting, and mood. Use these elements as a structured narrative, not a keyword list. Modern models understand language better than keyword matching. Start simple, then add details. @@ -33,11 +31,11 @@ Context is especially important for creating better prompts. Try adding lighting
-
A cute cat lounges on a leaf in a pool during a peaceful summer afternoon, in lofi art style, illustration.
+
cute cat lounges on a leaf in a pool during a peaceful summer afternoon, in lofi art style, illustration.
-
A cute cat lounges on a floating leaf in a sparkling pool during a peaceful summer afternoon. Clear reflections ripple across the water, with sunlight casting soft, smooth highlights. The illustration is detailed and polished, with elegant lines and harmonious colors, evoking a relaxing, serene, and whimsical lofi mood, anime-inspired and visually comforting.
+
A cute cat lounges on a floating leaf in a sparkling pool during a peaceful summer afternoon. Clear reflections ripple across the water, with sunlight casting soft, smooth highlights. The illustration is detailed and polished, with elegant lines and harmonious colors, evoking a relaxing, serene, and whimsical lofi mood, anime-inspired and visually comforting.
@@ -46,17 +44,42 @@ Be specific and add context. Use photography terms like lens type, focal length, > [!TIP] > Try a [prompt enhancer](https://huggingface.co/models?sort=downloads&search=prompt+enhancer) to help improve your prompt structure. + +## Guidance and negatives + +Most Diffusers pipelines still steer sampling with classifier-free guidance and an optional `negative_prompt`. + +On Stable Diffusion–family pipelines, pass `guidance_scale` and `negative_prompt`. Higher `guidance_scale` follows the prompt more closely. Values that are too high can look unnatural or oversaturated. + +```py +import torch +from diffusers import DiffusionPipeline + +pipeline = DiffusionPipeline.from_pretrained( + "stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16, device_map="cuda" # or "mps", "xpu", "cpu" +) +image = pipeline( + prompt="a cozy reading nook with afternoon light", + negative_prompt="blurry, low quality, distorted", + guidance_scale=7.5, +).images[0] +``` + +On newer checkpoints such as Qwen-Image and Flux, classic CFG is usually `true_cfg_scale` together with `negative_prompt`. Negatives still apply when true CFG is enabled (`true_cfg_scale > 1` and a `negative_prompt`). Their `guidance_scale` argument is distilled or embedded guidance when the transformer supports it, which is not the same. Check the pipeline API for the model you load. + +Modular Diffusers can replace the guidance algorithm with a guider. See [Guiders](./guiders). + ## Prompt weighting Prompt weighting makes some words stronger and others weaker. It scales attention scores so you control how much influence each concept has. -Diffusers handles this through `prompt_embeds` and `pooled_prompt_embeds` arguments which take scaled text embedding vectors. Use the [sd_embed](https://github.com/xhinker/sd_embed) library to generate these embeddings. It also supports longer prompts. +Diffusers handles this through `prompt_embeds` and `pooled_prompt_embeds` arguments which take scaled text embedding vectors. Use the [sd_embed](https://github.com/xhinker/sd_embed) library to generate these embeddings. It also supports longer prompts. For Stable Diffusion-family weighting with a simpler syntax, you can also use [Compel](https://github.com/damian0815/compel). > [!NOTE] -> The sd_embed library only supports Stable Diffusion, Stable Diffusion XL, Stable Diffusion 3, Stable Cascade, and Flux. Prompt weighting doesn't necessarily help for newer models like Flux which already has very good prompt adherence. +> The sd_embed library only supports Stable Diffusion, Stable Diffusion XL, Stable Diffusion 3, Stable Cascade, and Flux. Prompt weighting does not always help on Flux-class models, which already follow prompts closely. -```py -!uv pip install git+https://github.com/xhinker/sd_embed.git@main +```shell +uv pip install git+https://github.com/xhinker/sd_embed.git@main ``` Format weighted text with numerical multipliers or parentheses. More parentheses mean stronger weighting. @@ -66,12 +89,12 @@ Format weighted text with numerical multipliers or parentheses. More parentheses | `(cat)` | increase by 1.1x | | `((cat))` | increase by 1.21x | | `(cat:1.5)` | increase by 1.5x | -| `(cat:0.5)` | decrease by 4x | +| `(cat:0.5)` | set weight to 0.5× | Create a weighted prompt and pass it to [get_weighted_text_embeddings_sdxl](https://github.com/xhinker/sd_embed/blob/4a47f71150a22942fa606fb741a1c971d95ba56f/src/sd_embed/embedding_funcs.py#L405) to generate embeddings. > [!TIP] -> You could also pass negative prompts to `negative_prompt_embeds` and `negative_pooled_prompt_embeds`. +> You can also pass negative prompts to `negative_prompt_embeds` and `negative_pooled_prompt_embeds`. ```py import torch