Multimodal Text-to-Image Application

Since ollama does not support text-to-image functionality, we need to use other tools to implement local text-to-image. Currently, voice-controlled text-to-image is not supported, so the content of this article is the same as the offline text multimodal text-to-image application.

1. Concept Introduction

1.1 What is Text-to-Image?

Text-to-Image is an artificial intelligence technology that automatically generates corresponding images through text descriptions. Simply input a piece of text (for example, "a Shiba Inu wearing sunglasses surfing on the beach"), and the AI model will generate an image matching the description based on semantic understanding, without requiring any drawing or design skills.

Core Principles

1. Language Understanding: AI first parses your text, identifying keywords (such as "Shiba Inu", "sunglasses", "beach").
2. Image Generation: Based on training with massive image data, AI converts text into visual elements and combines them into a reasonable scene.
3. Style Adaptation: You can specify a style (realistic, cartoon, watercolor, etc.), and AI will adjust the generation effect accordingly.

1.2 What is FastSDCPU?

FastSDCPU is an open-source project specifically optimized for CPU devices for Stable Diffusion text-to-image. Through algorithm and engineering optimization, it achieves fast generation of high-quality images on ordinary computers without GPUs, significantly lowering the hardware barrier for AI art.

Core Features

1. Pure CPU Operation: No independent graphics card required; can run directly on laptops, desktops, or edge devices.
2. High-Speed Inference: Combined with Latent Consistency Models (LCM) and other technologies, images can be generated in just 4-8 steps, greatly reducing wait time.
3. Lightweight Deployment: Supports model quantization (such as INT8) and OpenVINO acceleration, reducing resource usage and improving response efficiency.

Applicable Scenarios

• Localized AI creative tools
• Enterprise intranet image generation services
• Teaching demonstrations and prototype development
• Lightweight deployment in resource-constrained environments

The Yahboom tutorial attachment provides a FastSDCPU compressed package. Users can choose to pull it to the image for decompression and use. After decompression, you can directly skip to Section 2.4 Startup Method After Successful Deployment for learning.

2. Project Deployment

2.1 Deployment Environment

Open a terminal and execute the following code:

​x
#If git is not installed on the board, run these first
sudo apt update
sudo apt install git -y
sudo apt install python3.10-venv -y
​
#Add environment variables
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc
​
# Clone project code
git clone https://github.com/rupeshs/fastsdcpu.git
cd fastsdcpu
​
# Create virtual environment and install dependencies
python -m venv venv
source venv/bin/activate
​
#Install uv
curl -Ls https://astral.sh/uv/install.sh | sh(This step may fail without a proxy in China. If it fails, skip this step and run the next command)
​
#If the curl command to install uv fails in the previous step, run these three commands
wget https://mirrors.huaweicloud.com/astral/uv/0.8.4/uv-aarch64-unknown-linux-gnu -O ~/.local/bin/uv
chmod +x ~/.local/bin/uv
~/.local/bin/uv --version
​
chmod +x install.sh start-webui.sh
./install.sh --disable-gui

Installation successful, press any key to exit:

2.2 LAN Access

Before starting, you need to modify a file to support LAN access, otherwise you can only access webui locally:

xxxxxxxxxx
vim ~/fastsdcpu/src/frontend/webui/ui.py

After opening the ui.py file, scroll to the last line, find webui.launch(share=share), and modify it to webui.launch(server_name="0.0.0.0",share=share)

Then save

Start:

xxxxxxxxxx
./start-webui.sh

Then you can enter your board IP:7860 in the browser to access this webui.

2.3 Using Text-to-Image Functionality

Use ifconfig in the terminal to check your board's IP, for example mine is 192.168.2.106.

Then open a browser and enter your board IP:7860. For example, I would enter 192.168.2.106:7860, and then you can access the webui.

Then click on LCM-LoRA, which has relatively low memory usage. If you want to use other models, you can research on your own.

Then click on Models to see the LCM LoRA model settings. You can change to your desired model, or just choose the default like me.

Then click on Generation Settings and increase the Inference Steps, which can improve the quality of generated images. I set it to 5 here.

Then continue back to our Text to Image, enter what you want to generate in the dialog box, and press Generate to start generating images.

For first-time use, you need to download the model. You can see in the terminal that the default selected model is being downloaded. After the download is complete, the text-to-image function will begin executing.

Generated result:

In practice, this project has better support for English, and generated images will be more aligned with the text. It is recommended to use English descriptions to generate images.

2.4 Changing Model Download Location

The default download location for models is: /.cache/huggingface/hub

You can use the following command in the terminal before each model download:

xxxxxxxxxx
export HF_HOME=/path/to/cache/directory #Change the path to your desired location

This will change your huggingface model download path. For example, if I want to place models on an SSD and not occupy board hard drive space, I can enter the following command:

xxxxxxxxxx
export HF_HOME=/media/ssd/HF_Model

The /media/ssd path here is my mounted SSD path. You should modify it according to your own situation.

If you have already downloaded models to the default path and want to change the model location later, you can use the following command:

xxxxxxxxxx
mv ~/.cache/huggingface /path/to/cache/directory #Change the path to your desired location

If you find it troublesome to use the command every time you download a model, you can directly add this export command to the .bashrc file.