Multimodal Text-to-Image Applications

Since Ollam does not support text-to-image functionality, we need to use other tools to implement local text-to-image functionality. Currently, voice-controlled text-to-image functionality is not supported, so the content of this article is the same as offline multimodal text-to-image applications.

1. Concept Introduction

1.1 What is Text-to-Image?

Text-to-image is an artificial intelligence technology that automatically generates corresponding images from text descriptions. Simply input a text (e.g., "A Shiba Inu wearing sunglasses is surfing on the beach"), and the AI ​​model will generate an image that matches the description based on semantic understanding, without any drawing or design background.

Core Principles

1. Language Understanding: The AI ​​first parses your text and identifies keywords (such as "Shiba Inu," "sunglasses," and "beach").
2. Image Generation: Based on training with massive amounts of image data, the AI ​​converts text into visual elements and combines them into a reasonable image.
3. Style Adaptation: Allows specifying a style (realistic, cartoon, watercolor, etc.), and the AI ​​will adjust the generated effect accordingly.

1.2 What is FastSDCPU?

FastSDCPU is an open-source project optimized for CPU devices, specifically designed for stable diffusion image generation. Through algorithmic and engineering optimizations, it enables the rapid generation of high-quality images on ordinary computers without GPUs, significantly lowering the hardware barrier for AI painting.

Core Features

1. Pure CPU Execution: Requires no dedicated graphics card and can run directly on laptops, desktops, or edge devices.
2. High-Speed ​​Inference: Combined with technologies such as Latent Consistency Models (LCM), images can be generated in just 4–8 steps, significantly reducing waiting time.
3. Lightweight Deployment: Supports model quantization (such as INT8) and OpenVINO acceleration, reducing resource consumption and improving response efficiency.

Applicable Scenarios

• Localized AI Creation Tools
• Enterprise Intranet Image Generation Service
• Teaching Demonstrations and Prototype Development
• Lightweight Deployment in Resource-Constrained Environments

2. Project Deployment

2.1 Deployment Environment

Note: If using our factory image, no deployment environment is required; skip the deployment steps. Refer directly to <2.4 Startup Method After Successful Deployment> at the bottom to start. Then return to <2.3 Using the Text-to-Image Function> to learn how to use it.

Open a terminal and execute the following code:

x
# If git is not available on your system, run
sudo apt update
sudo apt install git -y
sudo apt install python3.10-venv -y
​
# Add environment variables
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
​
# Clone the project code
git clone https://github.com/rupeshs/fastsdcpu.git
cd fastsdcpu
​
# Create a virtual environment and install dependencies
python -m venv venv
source venv/bin/activate
​
# Install uv
curl -Ls https://astral.sh/uv/install.sh | sh(This step may not be successful if there is no proxy in China. If not, you can skip this step and execute the next command)
​
# If the curl command failed to install uv in the previous step, run these three instructions
wget https://mirrors.huaweicloud.com/astral/uv/0.8.4/uv-aarch64-unknown-linux-gnu -O ~/.local/bin/uv
chmod +x ~/.local/bin/uv
~/.local/bin/uv --version
​
chmod +x install.sh start-webui.sh
./install.sh --disable-gui

The installation is successful, press any key to exit:

2.2 LAN Access

Before starting, you need to modify a file to support LAN access. Otherwise, the webUI can only be accessed locally:

xxxxxxxxxx
vim ~/fastsdcpu/src/frontend/webui/ui.py

After opening the ui.py file, scroll to the last line and find webui.launch(share=share). Change it to webui.launch(server_name="0.0.0.0",share=share).

Save the file.

Start:

xxxxxxxxxx
./start-webui.sh

Then, enter your motherboard's IP address: 7860 in your browser to access the webUI.

2.3 Using the Vinyl Image Function

Use ifconfig in the terminal to query your motherboard's IP address. For example, mine is 192.168.2.106.

Then open your browser and enter your motherboard's IP address: 7860 . For example, I entered 192.168.2.106:7860 . This will allow you to access the webUI.

Next, click LCM-LoRA. This model uses less memory. If you prefer, feel free to explore other models.

Next, click Models to view the LCM LoRA model settings. You can change the model you want, or just stick with the default, as I did.

Next, click Generation Settings and increase the Inference Steps setting to improve the quality of the generated image. I've set it to 5.

Next, return to the Text to Image dialog box and enter the content you want to generate. Click Generate to begin generating the image.

For first-time users, you'll need to download the model. You'll see the default model being downloaded in the terminal. Once it's finished downloading, the text-to-image function will begin.

Generated result:

This project has better support for English, and the generated images are more consistent with the text. We recommend using English descriptions when generating images.

2.4 How to start after successful deployment

x
cd fastsdcpu #Enter the fastsdcpu directory
source venv/bin/activate #Enter the virtual environment
./start-webui.sh #Start the webui

After the webui is successfully started, enter your motherboard's IP address: 7860 in your browser to start the image generation function.