Welcome to DOFBOT SE repository

AI Large Model - Dify

1. Dify Overview

Dify is an open-source LLM application development platform that combines BaaS and LLMOps concepts. It supports low-code/no-code rapid building of production-grade generative AI applications, is compatible with hundreds of models, and has built-in enterprise-level RAG and visual workflows. It supports private deployment and full-cycle LLMOps control.

Dify (Define + Modify) is designed for developers and enterprises, providing an integrated solution of "backend as a service + BaaS + LLMOps" to reduce the barrier to transforming generative AI applications from prototype to production while ensuring data security and compliance.

One of Dify's product forms is the "AI Application Factory," which enables "rapid creation of chatbots and content generators through a low-code interface." By configuring the Decision Layer and Execution Layer, we can plan task steps and select execution function actions to achieve dual-model reasoning. Additionally, for more complex features, by adding a Knowledge Base, the Decision Layer can reference content from the Knowledge Base when planning actions to generate correct steps.

2. Entering the Dofbot-SE Dify Configuration Interface

Dofbot-SE Dify service starts automatically on boot

Enter the virtual machine's IP address in your browser to access the configuration interface. The interface is shown below,

On first login, you need to enter your email and password,

Email: yahboom@163.com

Password: yahboom123

After clicking login, you can access the configuration interface,

In Studio, there are four chat assistants:

• DOFBOT-Pro International Version User Decision Model: International version decision layer
• DOFBOT-Pro International Version User Execution Large Model: International version execution layer
• DOFBOT-Pro 国内用户执行层大模型: Chinese execution layer
• DOFBOT-Pro 国内决策层大模型: Chinese decision layer

Overseas users primarily use DOFBOT-Pro International Version User Execution Large Model and DOFBOT-Pro International Version User Decision Model.

Click Knowledge beside Studio to see two knowledge base files,

[决策层训练样例cn_1211更新] : Chinese knowledge base

[决策层训练样例en.xlsx] : English knowledge base

Chinese users use the 决策层训练样例cn_1211更新 knowledge base, while overseas users use 决策层训练样例en. The content is the same, just in different languages.

3. Studio

When using the chat assistant in Studio, let's look at the international decision layer - [DOFBOT-Pro International Version User Decision Model] as an example,

The content in item 1 is the prompt. In the prompt, we give the large model a role, telling it what it can do, along with some output restrictions, basic requirements, special cases, etc.

In item 2 is the knowledge base we want to add. Here we selected [决策层训练样例en] and configured some settings for the imported knowledge base,

In item 3 is the large model we selected. You can also choose other large models. Currently, [qwen-max-latest] has the best results and generally doesn't need modification. After selecting a large model, you can also configure the model call settings,

Note: The qwen-vl-max model will be removed from the platform in July 2026. You will need to use a different large visual model instead.

In item 4, we can enter content to chat with the large model to test whether the model's decision actions are accurate,

For the execution layer, taking the international execution layer as an example, the difference is in the [INSTRUCTIONS] section. In addition to giving the large model a role, output restrictions, basic requirements, and special cases, this section also explains which action functions correspond to which actions. For example, the robot arm dance command corresponds to the action arm_dance(). Additionally, for some functions that require parameters, the meaning of the parameters must also be explained. For example,

xxxxxxxxxx
- Adjust joint x to y degrees: `adjust_joint(x,y)`
  - Description: Adjust the joint with number `x` to rotate to `y` degrees. Valid values for `x`: 1 to 6; valid values for `y`: 0-180.

Also, if you need to use visual functionality for image understanding or video understanding, you need to turn on the visual switch below. When selecting a model in the upper right corner, you need to select one with "vl", such as qwen-vl-max-latest shown in the image. Currently, this is the best multimodal large model in Dify.

4. Knowledge (For details, refer to the specific chapters later)

In the knowledge base, taking the Chinese knowledge base as an example [决策层训练样例en], clicking to view will show that it is an .xlsx file,

Click to view this file,

·

You can click on one of them to take a look,

xxxxxxxxxx
query":""What is the correct answer to this question?"";"answer":""Strictly execute according to the following steps, output only one action function at a time:
1. Call the seewhat() function to view the question;
2. First reply with the result of the question;
3. Find the answer options for the question on the desktop;
4. Call the point_to(x1, y1, x2, y2) function to point to the answer option (where x1, y1, x2, y2 are the relevant coordinate parameters of the position of the answer option).
5. Call finish_dialogue() to end the task and return to the initial position""

query represents the instruction we give to the large model, and answer represents the content we expect the large model to plan and output.

If during testing you find that the large model cannot plan the preset content, you can modify the keywords below and add instruction keywords to help the large model match the best content in the knowledge base.

If you need to add segments, click [+ Add Chunk], enter the query and answer content. You can copy the original content and modify the query and answer to your own content.

5. Modify API-KEY

As shown in the image below, click the avatar in the upper right corner and go to Settings,

After entering, click [Model Provider],

Click [Setup] on the right. In the Dify we provided, only Tongyi has been added. If you want to add other model providers, you can refer to the later tutorials,

Then enter the Tongyi api-key you registered in the previous section. It is a key starting with "sk". For the [Use International Endpoint] option below, Chinese users select False, and international users select True. After entering, click [Save].

If you need to add multiple keys, click [Config] - [Add API Key].

6. Import Other Provider Models (This step is for users who need to use other large models)

The advantage of this Dify deployment is that it can seamlessly connect to most global AI models: through Dify's model provider plugins, you can quickly and seamlessly connect to hundreds of global AI models without the trouble of deployment and setup. Here we take connecting to openrouter as an example.

Find the openrouter model and click Install

After installation, refresh Dify and enter the key to save.

Switch to the openrouter model in the decision layer/execution layer (if not charged, check free models on openrouter). Finally, publish it. The execution layer must use a visual model with VL in the name.

7. International User Local Function Package Configuration

7.1 Dify Decision Layer and Execution Layer Switching

Since the default parameters are for Chinese domestic users, overseas users need to switch the Dify decision layer and execution layer IDs.

Configuration path /home/yahboom/LargeModel_ws/src/largemodel/config/

Open the file large_model_interface.yaml

Switch the decision layer and execution layer IDs: Comment out lines 31 and 32, uncomment lines 34 and 35.

Also, configure tts_language to en below

7.2 Voice Parameter Configuration

File path: /home/yahboom/LargeModel_ws/src/largemodel/config/yahboom.yaml

Comment out the Chinese version parameter options and enable the international version parameters. Users can choose to use online/offline voice recognition and synthesis according to their needs.

For online voice recognition and synthesis, change use_oline_asr and useolinetts parameters to True,

For offline voice recognition and synthesis, change use_oline_asr and useolinetts parameters to False.

7.3 Compile Function Package

Enter the following commands to compile the function package to apply the changes.

cd /home/yahboom/LargeModel_ws/
colcon build --packages-select largemodel
source install/setup.bash