Generative AI art refers to artworks created with the help of Generative Artificial Intelligence models. These models are designed to generate new content, such as images, based on prompts, controlnet and node workflows. Over at CRITICA Singapore, we use local runtime Stable Diffusion Automatic1111 (Stability AI) & ComfyUI for Generative AI. Below are some examples of what it can achieve.

Scribble to Manga Image

Using Scribble method in Controlnet, we input a quick doodle of a character, and chose a manga training model. Through this method, we achieve the capability to produce a nuanced and shaded output, meticulously crafted in accordance with our specified prompts.

Custom DW Openpose Editor to Image

DWPose is used for human whole-body pose estimation. Using the editor, we can articulate the limbs at any desired angle.

What if a scenario happens where we require it to adhere to a specific attire for the woman? Lets insert a reference image and adjust the prompts.

There you have it, the woman is wearing a blue jacket with a interior white tank and black pants, based on the positive conditioning prompts.

Prompt engineering is an iterative process that involves experimenting with different prompts and fine-tuning them to achieve the desired outcomes from the AI model. It requires a good understanding of the AI model’s capabilities and limitations and creative thinking to design prompts that elicit the desired responses.

Openpose extraction to Image

We can feed in a photograph, and using Openpose estimator, it will analyse and extract the pose of the person in the photograph. We can feed custom values which allows the AI to closely or distantly align with the reference image.

HED extraction to Image

Utilizing HED, the AI conducts an analysis of the silhouette features and intricacies present in the reference image. Positive conditioning prompts then result in the generation of an image, specifically a Manga aesthetic, based on the chosen training model.

What if we need the output to match a certain outfit? Lets import a IP Adaptor node.

Now the output closely resembles the reference photo, but yet the pose is derived from the HED node. In this scenario, we created a image that follows a style of Image A and the HED of Image B.

Lineart to Manga

This method is useful if you wish to input a outline sketch of a drawing. Once we feed it into the workflow, and choose our training model, (in this case, a shaded manga style aesthetic.) It is able to generate out closely based on the sketch.

Now, we intend to experiment with an alternative coloring/shading style, lets try import a IP Adaptor.

Input a style reference

The generated output closely mirrors the reference photo, showcasing a glossy, shaded metallic surface.

For Architectural Use | Lineart Sketch to Image

The Lineart to image generation method is suitable for architects and interior designers. Here, a loose sketch of a bungalow is fed into the workflow. We choose a architectural training model, and the AI generates a shaded version of our drawing.

However, lets influence it with a different visual style.

Input a custom image

Now the generated bungalow image follows the style of our reference input image. We hope this article helps you understand Controlnet better, in the Stable Diffusion workflow.

For further information on Stable Diffusion AI and to explore content creation using this technique, or even implement Gen AI in your company, reach out to CRITICA here for a consultation.