Multimodal Text-to-Image Application

Since Ollama does not support text-to-image functionality, we need to use other tools to implement local text-to-image generation. Currently, voice control for text-to-image is not supported, so the content of this article is the same as the offline text multimodal text-to-image application.

1. Concept Introduction

1.1 What is Text-to-Image?

Text-to-Image is an artificial intelligence technology that automatically generates corresponding images from text descriptions. Simply input a piece of text (e.g., "a Shiba Inu wearing sunglasses surfing on the beach"), and the AI model will generate an image that matches the description based on semantic understanding, without any need for drawing or design skills.

Core Principles

1. Language Understanding: The AI first parses your text, identifying keywords (such as "Shiba Inu," "sunglasses," "beach").

2. Image Generation: Based on training with massive amounts of image data, the AI transforms the text into visual elements and combines them into a reasonable picture.

3. Style Adaptation: You can specify a style (realistic, cartoon, watercolor, etc.), and the AI will adjust the generation effect.

1.2 What is FastSDCPU?

FastSDCPU is an open-source Stable Diffusion text-to-image project optimized for CPU devices. Through algorithmic and engineering optimizations, it enables the rapid generation of high-quality images on ordinary computers without a GPU, significantly lowering the hardware threshold for AI painting.

Core Features

1. Pure CPU Operation: No need for a dedicated graphics card; it can run directly on laptops, desktops, or edge devices.

2. High-speed Inference: Combined with technologies like Latent Consistency Models (LCM), it only takes 4–8 steps to generate an image, greatly reducing waiting time.

3. Lightweight Deployment: Supports model quantization (such as INT8) and OpenVINO acceleration, reducing resource consumption and improving response efficiency.

Applicable Scenarios

• Localized AI creation tools

• Image generation services on corporate intranets

• Teaching demonstrations and prototype development

• Lightweight deployment in resource-constrained environments

2. Project Deployment

2.1 Deployment Environment

Note: If you are using our factory image, there is no need to deploy the environment; you can skip the deployment steps. Refer directly to <2.4 Startup Method After Successful Deployment> below to start. Then, return to <2.3 Using the Text-to-Image Function> to learn how to use it.

Open a terminal and execute the following code:

​x
# If git is not on the motherboard, run this first
sudo apt update
sudo apt install git -y
sudo apt install python3.10-venv -y

# Add environment variables
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

# Clone the project code
git clone https://github.com/rupeshs/fastsdcpu.git
cd fastsdcpu

# Create a virtual environment and install dependencies
python -m venv venv
source venv/bin/activate

# Install uv
curl -Ls https://astral.sh/uv/install.sh | sh (This step may fail in China without a proxy. If it doesn't exceed, you can skip this step and execute the next command)

# If the curl command to install uv in the previous step fails, run these three commands
wget https://mirrors.huaweicloud.com/astral/uv/0.8.4/uv-aarch64-unknown-linux-gnu -O ~/.local/bin/uv
chmod +x ~/.local/bin/uv
~/.local/bin/uv --version

chmod +x install.sh start-webui.sh
./install.sh --disable-gui

Installation successful, press any key to exit:

2.2 LAN Access

Before starting, you need to modify a file to support LAN access; otherwise, the web UI can only be accessed locally:

xxxxxxxxxx
vim ~/fastsdcpu/src/frontend/webui/ui.py

After opening the ui.py file, go to the last line and find the line webui.launch(share=share), change it to webui.launch(server_name="0.0.0.0",share=share)

Then save.

Start:

xxxxxxxxxx
./start-webui.sh

Then you can enter your motherboard's IP:7860 in the browser to access this web UI.

2.3 Using the Text-to-Image Function

Use ifconfig in the terminal to query our motherboard's IP, for example, mine is 192.168.2.106.

Then we open the browser and enter your motherboard's IP:7860. For example, I would enter 192.168.2.106:7860, and then I can enter the web UI.

Next, we click on LCM-LoRA; this model has a smaller memory footprint. If you want to use other models, you can research them yourself.

Then click on Models, you can see the model settings for LCM LoRA. You can change to the model you want, or you can choose the default one like me.

Next, click on Generation Settings and increase the Inference Steps to improve the quality of the generated images. I'll set it to 5 here.

Then we go back to our Text to Image, enter what we want to generate in the dialog box, and then press Generate to start generating the image.

If this is your first time using it, you need to download the model. You can see in the terminal that the model selected by default is being downloaded. After it is downloaded, the text-to-image function will be executed.

Generated result:

It has been tested that this project has better support for English, and the generated images will be more in line with the text. It is recommended to use English descriptions to generate images.

2.4 Startup Method After Successful Deployment

xxxxxxxxxx
cd fastsdcpu # Enter the fastsdcpu directory
source venv/bin/activate # Enter the virtual environment
./start-webui.sh # Start the webui

After the webui starts successfully, enter your motherboard's IP:7860 in the browser to start the text-to-image function.