Controlling the composition and style of your AI images

Hello Digital Dreamers.

A few weeks ago an extension called ControlNET came out for Stable Diffusion and I have been OBSSESSED by it.

Once again, it opened new ways for creators to have control over their creative output with AI.

There is a lot to unpack with this and I’ll spread it over several mails.

Today, I want to start with controlling composition and style of images with ControlNET.

I will be using a local Automatic1111 installation of Stable Diffusion. You can find a guide for it here.

ControlNET is an extension and you can find an installation video here.

There are 3 elements involved in this workflow:

• Reference image for the style

• Reference image for the composition

• Prompt

We upload the style frame into the img2img section of Stable Diffusion. Denoising strength will determine how much the final image will vary from the reference. The higher it is, the more variation it creates.

As I will control the composition with ControlNET, I set the denoising strength between 0.7 - 0.8.

We upload the composition frame to ControlNET.

We click enable and select the Depth model. This will create a depth map of our image, which will keep the composition consistent.

Finally, we write in our prompt of what we want to see. I will use the prompt I used to create this style with a subject similar to the composition reference frame.

cappadocia firefly chimneys in the grasslands, mounds of rock ::15 watercolor manga ::3 watercolor brush, brush with a soft tip, loose flowing lines ::4 movement, fluidity ::1 Soft, pastel color palette ::4 dreamy, ethereal atmosphere ::4 simplified design, clean design ::1 beautiful, expressive ::1.5 photography, 3d, realistic ::-3 strong lines, strong colors ::-2

Midjourney weights don’t work the same in Stable Diffusion, so I will remove them and add the negative modifier groups into the negative prompt. I’m also using the Openjourney model, which is trained on Midjourney images. To activate that, I add mdjrny-v4 style to the prompt.

After this, it’s experimentation!

Adjust the prompt, change the denoising strength, change the reference image, find what works best for your use case.

Make sure the aspect ration of reference frames fits together as well.

If you have thought on this topic, reply to this mail and send me a message. I would love to have a chat. 🤗

I will be doing a live event showing this workflow (+ a few others) Next Thursday at 12pm GMT+7.

If you want to learn more on how to use Stable Diffusion and all it’s powerful extensions, join me on the AICC Discord server.

I will be doing another event this week in celebration of a book release, which I contributed to.

It’s a 102 page book in collaboration with 9 other AI creators!

It was interesting to see different thought processes and use cases come together in one place. I found that an extremely valuable learning experience and I believe so would you.

If you want to deepen your knowledge on creating with AI tools, you can find the book here.

I will also be going over my lesson - AI in visual communication - This Thursday at 11 am GMT+7.

You can find the event here. Come learn something new, ask questions and hang out with other AI creators. 😁

You can also stop by my website or social media and as always,

Keep creating and don’t forget to have fun. ☀