# Fine tune Stable Diffusion on your images using Textual Inversion

In this article, we will see how to fine-tune text to image AI model, Stable Diffusion on our own images.

Fine tuning with textual inversion can be achieved with as few as 3-5 image examples.

We will cover two ways to do this in this article.

1.  Using Google Colab Notebooks to fine tune Stable Diffusion
2.  Fine tuning Stable Diffusion using textual Inversion locally

Let’s cover them one by one

Table of Contents

*   [Using Google Colab Notebooks to fine tune Stable Diffusion](https://harishgarg.com/#Using_Google_Colab_Notebooks_to_fine_tune_Stable_Diffusion)
*   [Fine tuning Stable Diffusion using textual Inversion locally](https://harishgarg.com/#Fine_tuning_Stable_Diffusion_using_textual_Inversion_locally)
    *   [Install Python dependencies by running this command](https://harishgarg.com/#Install_Python_dependencies_by_running_this_command)
    *   [Next, configure the HuggingFace Accelerate environment by running the below command](https://harishgarg.com/#Next_configure_the_HuggingFace_Accelerate_environment_by_running_the_below_command)
    *   [Download Stable Diffusion  weights](https://harishgarg.com/#Download_Stable_Diffusion_weights)

Using Google Colab Notebooks to fine tune Stable Diffusion
----------------------------------------------------------

The easiest way of course is Google Colab Notebooks. They can be run in your browser and you require any special hardware like GPUs.

1.  Open this [Google Colab Notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb) and follow the instructions in the notebook to run it.
2.  Next, Open the [inference Notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb) and Run all the cells.

Fine tuning Stable Diffusion using textual Inversion locally
------------------------------------------------------------

Another way to fine tune Stable Diffusion on your images is using your hardware.

For this, you either need a GPU-enabled machine locally or a GPU-enabled VM in the cloud.

You will also need to have Python 3 or higher installed and should know your way around the command line.

Below are the steps:

### Install Python dependencies by running this command

pip install diffusers\[training\] accelerate transformers

### Next, configure the HuggingFace Accelerate environment by running the below command

accelerate config

### Download Stable Diffusion  weights

First, visit [Stable Diffusion page](https://huggingface.co/CompVis/stable-diffusion-v1-4) on HuggingFace to accept the license

For the next part, you need [HuggingFace access token](https://huggingface.co/docs/hub/security-tokens) 

Next, authenticate with your token by running below command

huggingface-cli login

Fine tuning can be started using below command

export MODEL\_NAME=”CompVis/stable-diffusion-v1-4″  
export DATA\_DIR=”path-to-dir-containing-images”

accelerate launch textual\_inversion.py \\  
–pretrained\_model\_name\_or\_path=$MODEL\_NAME –use\_auth\_token \\  
–train\_data\_dir=$DATA\_DIR \\  
–learnable\_property=”object” \\  
–placeholder\_token=”<cat-toy>” –initializer\_token=”toy” \\  
–resolution=512 \\  
–train\_batch\_size=1 \\  
–gradient\_accumulation\_steps=4 \\  
–max\_train\_steps=3000 \\  
–learning\_rate=5.0e-04 –scale\_lr \\  
–lr\_scheduler=”constant” \\  
–lr\_warmup\_steps=0 \\  
–output\_dir=”textual\_inversion\_cat”

To generate images with your new fine tuned model, run below command

from torch import autocast  
from diffusers import StableDiffusionPipeline

model\_id = “path-to-your-trained-model”  
pipe = StableDiffusionPipeline.from\_pretrained(model\_id,torch\_dtype=torch.float16).to(“cuda”)

prompt = “A <cat-toy> backpack”

with autocast(“cuda”):  
image = pipe(prompt, num\_inference\_steps=50, guidance\_scale=7.5).images\[0\]

image.save(“cat-backpack.png”)