Welcome to DOFBOT-Pi repository
2. Configure API-KEY
Configure API-KEY
1. Dify Introduction
Dify is an open-source LLM application development platform that integrates BaaS and LLMOps concepts. It supports low-code/no-code rapid construction of production-grade generative AI applications, is compatible with hundreds of models, and includes enterprise-level RAG and visual workflows, supporting private deployment and full-chain LLMOps management.
Dify (Define + Modify) is oriented towards developers and enterprises, providing an integrated solution of "Backend as Service + BaaS+LLMOps" to reduce the threshold for generative AI applications from prototype to production while ensuring data security and compliance.
One of Dify's product forms, the "AI Application Factory," enables "rapid creation of conversational robots and content generators through low-code interfaces." We plan task steps and select action functions for execution by configuring the decision layer and execution layer to achieve dual-model reasoning; additionally, for more complex functions, by adding knowledge bases, the decision layer can reference content from the knowledge bases when planning actions to plan correct steps.
2. Enter the Robotic Arm Dify Configuration Interface
Enter the following command in the terminal to start Dify,
xxxxxxxxxxsh ~/bringup_dify.shPress Enter, as shown in the figure below, when all green is displayed, it indicates Dify has started,
After successfully starting Dify, you can enter the configuration interface by entering the robotic arm's IP address in the browser. For example, the interface is shown below,
For the first login, you need to enter email and password,
Email: yahboom@163.com
Password: yahboom123
After clicking login, you can enter the configuration interface,
In the studio, there are the following four chat assistants
- DOFBOT-Pro International Version User Decision Model: International version decision layer
- DOFBOT-Pro International Version User Execution Large Model: International version execution layer
- DOFBOT-Pro 国内用户执行层大模型: Chinese execution layer
- DOFBOT-Pro 国内决策层大模型: Chinese decision layer
International users mainly use DOFBOT-Pro International Version User Decision Model and DOFBOT-Pro International Version User Execution Large Model.
Click the knowledge base next to the studio, and you will see two knowledge base files. International users use 决策层训练样例en.
决策层训练样例cn_1211更新: Chinese knowledge base
决策层训练样例en.xlsx: English knowledge base
Note: If 决策层训练样例en shows UNAVAILABLE, it's because the API-KEY is not configured.
3. Studio
When using the chat assistant in the studio, let's expand on the International decision layer - [DOFBOT-Pro International Version User Decision Model],
The content in number 1 is the prompt. In the prompt, we give the large model an identity, tell it what it can do, and some output restrictions, basic requirements, and special situations, etc.
In number 2, it's the knowledge base we need to add. Here we selected [决策层训练样例en] and made some configurations for the imported knowledge base,
In number 3, it's the large model we selected. Of course, you can also select other large models. Currently, [qwen-max-latest] has the best effect and generally doesn't need modification. After selecting a large model, you can also set up model calls,
In number 4, we can input content and chat with the large model to test whether the large model's decision actions are accurate,
In the execution layer, taking the international user execution layer as an example, the difference is in the [Instructions] section. Here, in addition to giving the large model an identity, output restrictions, basic requirements, and special situations, it also explains which actions correspond to which action functions. For example, the robotic arm dance instruction corresponds to the action arm_dance(). Additionally, some functions that need to pass parameters also need to explain the meaning of the parameters, for example,
xxxxxxxxxx- Adjust joint x to y degrees: `adjust_joint(x,y)`- Description: Adjust the joint with number `x` to rotate to `y` degrees. Valid values for `x`: 1 to 6; valid values for `y`: 0-180.
Additionally, if you need to use visual functions for some image understanding and video understanding functions, you need to turn on the vision switch below. When selecting a model in the upper right, you need to choose one with vl, such as qwen-vl-max-latest shown in the figure, which is currently a better multimodal large model in dify.
4. Knowledge Base (For details, refer to specific chapters later)
In the knowledge base, taking the English knowledge base [决策层训练样例en] as an example, clicking to view will reveal an .xlsx file,
Click to view this file,
You can click on one to see,
xxxxxxxxxxquery":"Please check what the correct answer to this question is";"answer":"Execute in strict accordance with the following steps, and only output one action function at a time:1. Call the `seewhat()` function to view the question;2. First reply with the result of the question;3. Find the answer option(s) to the question on the desktop;4. Call the `point_to(x1, y1, x2, y2)` function to point to the answer option(s) (where x1, y1, x2, y2 are the relevant coordinate parameters of the position of the answer option(s)). "
query represents the instruction we give to the large model, and answer represents the content we want the large model to plan and output.
If during testing, you find that the large model cannot plan the content we preset, then you can modify the keywords below and add instruction keywords to let the large model match the best content in the knowledge base.
If you need to add segments, click [+ Add Chunk], input the query and answer content. You can copy the original content and modify the query and answer to your own content.
5. Modify API-KEY
As shown in the figure below, click the Y in the upper right corner to enter account settings,
After entering, click [Model Provider],
Click [Config] on the right. In the dify we provide, only Tongyi has been added. If you want to add other model suppliers, you can refer to the later tutorial,
Click [Add API Key],
Then fill in the api-key we registered in the previous chapter from Alibaba Cloud Model Studio Platform, which is a key starting with sk. For the [Use International Endpoint] option below, international users select Yes. After inputting, click [Save].
6. Import Other Supplier Models (This step is provided for users who need to use other large models)
The advantage of dify is that it can seamlessly access most AI models worldwide: that is, through Dify's model supplier plugins, quickly and seamlessly access hundreds of AI models worldwide, saving the trouble of deployment and setup. Here we use openrouter as an example.
Find the openrouter model and click install
After installation, refresh dify and fill in the key to save.
Switch the openrouter model in the decision layer/execution layer (if not recharged, you need to check free model usage in openrouter), and finally publish. The execution layer must use a visual model with VL in the name.
7. International User Configuration Local Function Package
7.1 dify Decision Layer Execution Layer Switching
Since the default parameters are for domestic users, international users need to switch the dify decision layer and execution layer IDs.
Path inside the Docker container: /root/LargeModel_ws/src/largemodel/config
Open file large_model_interface.yaml
Switch decision layer and execution layer IDs: comment out lines 31 and 32, uncomment lines 34 and 35.
7.2 Voice Parameter Configuration
Path inside the Docker container: /root/LargeModel_ws/src/largemodel/config
Open file: yahboom.yaml
Change the parameters for whether to use online voice recognition and voice synthesis to false.
Comment out the domestic version parameter options, open the international version parameters.
7.3 Compile Function Package
Access the container terminal and enter the following command to compile the package and apply the changes.
xxxxxxxxxxcd ~/LargeModel_wscolcon build --packages-select largemodelsource install/setup.bash