2. Text-based interactive LLM

1. Text Interaction Based on Open WebUI

1. Introduction

Ollama simplifies running large language models locally by providing convenient REST API management for model downloading, deployment, and interaction. Models are optimized for Hailo hardware, offering efficient and high-performance inference, and are tailored for generative AI tasks.

Open WebUI is a scalable, feature-rich, and user-friendly self-hosted AI platform designed to run completely offline. Once Hailo-Ollama is running, you can interact with models through a modern web interface using Open WebUI.

2. Startup

Note: Before starting, please ensure you have installed the environment as described in the previous chapter. If not, please refer to the previous chapter [1. Environment Installation] for environment installation instructions.

Note: If you are using the image we provide directly, you do not need to rebuild the environment.

Open a terminal. In the terminal window, enter the following command to start the Hailo-Ollama service.

hailo-ollama

Open another terminal and pull the model.

Pull method:

xxxxxxxxxx
curl --silent http://localhost:8000/api/pull \
     -H 'Content-Type: application/json' \
     -d '{ "model": "examplemodel:tag", "stream" : true }'

Take qwen2.5-instruct:1.5b as an example.

xxxxxxxxxx
curl --silent http://localhost:8000/api/pull \
     -H 'Content-Type: application/json' \
     -d '{ "model": "qwen2.5-instruct:1.5b", "stream" : true }'

View supported models

xxxxxxxxxx
curl --silent http://localhost:8000/hailo/v1/list

The currently supported models are as follows:

"deepseek_r1_distill_qwen:1.5b",

"llama3.2:3b",

"qwen2.5-coder:1.5b",

"qwen2.5-instruct:1.5b",

"qwen2:1.5b"

After the model is downloaded, open the Open WebUI container.

xxxxxxxxxx
docker run -d -e OLLAMA_BASE_URL=http://127.0.0.1:8000 -v open-webui:/app/backend/data --name open-webui --network=host --restart always ghcr.io/open-webui/open-webui:main

View startup log

xxxxxxxxxx
docker logs open-webui -f

After successful startup, access the Raspberry Pi using the following URL in your browser:

xxxxxxxxxx
http://localhost:8080/

You can also use a computer's browser to access the Raspberry Pi's IP address: 8080. Here, we'll use 192.168.1.56 as an example. After your computer is connected to the same network as the Raspberry Pi 5, you can access the network by entering 192.168.1.56:8080 in your browser.

3. Register and Login

First-time users need to register an account. This account is for administrators; you can fill in the required information yourself!

xxxxxxxxxx
Since all content in our mirror has been set up and tested, users can log in directly using their registered account:
name: yahboom
Email: yahboom@163.com
Password: yahboom

4. User Interface

5. Model Dialogue

6. Model Switching

Since we only installed qwen2.5-instruct:1.5b, only one model can be selected. Other supported models can be downloaded separately.

7. Frequently Asked Questions

Turning off Open WebUI

Turn off the automatically starting Open WebUI.

• View running Docker

xxxxxxxxxx
docker ps

• Shut down running Docker

xxxxxxxxxx
docker stop [CONTAINER ID]  # 示例 docker stop 5f42ee9cf784

• Check stopped containers

xxxxxxxxxx
docker ps -a

• Clean up stopped containers

xxxxxxxxxx
docker rm [CONTAINER ID]    # 示例 docker rm 5f42ee9cf784

Clean up all stopped containers:

xxxxxxxxxx
docker container prune

Common Errors

Unable to start Open WebUI

• docker: Error response from daemon: Conflict. The container name "/open-webui" is already in use by container "cfc05c84f8e38b290337e7178c76fd1c49076f94b11ed3d49d9448be72b7f20f". You have to remove (or rename) that container to be able to reuse that name.

Solution: Close Open WebUI once and restart it!

Service connection timed out

• Open WebUI: Server Connection Error

Close the Open WebUI once and restart it, then ask the question again or use the Ollam tool to run the model and ask the question!

2. Simple Large Language Model Chat Example Based on Hailo App (No Input)

1. Introduction

A simple example application demonstrating text-based conversation using Hailo's Large Language Model (LLM).

2. Installation

xxxxxxxxxx
cd hailo-apps

xxxxxxxxxx
pip install -e ".[gen-ai]"

If you encounter the error "error: externally-managed-environment" when directly entering the installation command in the terminal, you can resolve it using the following command: (Note: The Python version should be modified according to your system version; my current system version is 3.13.)

xxxxxxxxxx
sudo mv /usr/lib/python3.13/EXTERNALLY-MANAGED /usr/lib/python3.13/EXTERNALLY-MANAGED.bk

3. Running Example

xxxxxxxxxx
python -m hailo_apps.python.gen_ai_apps.simple_llm_chat.simple_llm_chat

The example execution process is as follows:

1. Initialize the Hailo device and load the LLM model.
2. Send a simple prompt requesting a joke.
3. Display the model's response.
4. Clean up resources.

For more examples and details, please refer to ：hailo-apps/hailo_apps/python/gen_ai_apps/simple_llm_chat at main · hailo-ai/hailo-apps · GitHub