On universal computer control

[xiezhipeng-git]

I don't understand where your universal computer control is universal. Moreover, the model still calls GPT-4, which is not local. Universal computer control should use a local large language model. Then, the computer operation corresponding to the shortcut keys directly interacts with the model. It can be multimodal, or an action selection agent that has mixed multimodal functions. I can only say that it is too simplistic at present. It seriously does not match the title, and the direction is a bit off.

[WeihaoTan]

1. Our generalizability comes from strictly following the GCC setting with unified observation and action space without relying on any built-in APIs. Using keyboard and mouse as output and screen pixels enables the agent to interact with any software.
2. There is no such a rule that GCC needs to use a local large language model, which is also not what we focus on. Additionally, it is not feasible to deploy the GPT-4V-level model to every individual equipment at the current stage. We believe that cloud service is the future.
3. We are indeed at the preliminary stage which is why we say "towards GCC" in the title.

[xiezhipeng-git]

Do you think if a local multimodal is integrated. Will this project become more valuable? Why not? Verify the feasibility of the project using GPT4. Then use local models (there are already many small local multimodal models), combined with the shortcut operation of mouse and keyboard, similar to humans. I don't think if these features are further improved, the entire project will become very meaningful.

[WeihaoTan]

Again, whether using a local multimodal model is not the key point we focus on for this project. The project can be always improved in many aspects. I can kind of understand why you insist on deploying the framework to a local model. But you might underestimate the issues of running a local model to play games.

1. Even the most modern consumer GPU, 4090 struggles with running a 7B model and RDR2 simultaneously. Running the model locally is not as cheap and easy as you expect.
2. There is a huge gap between the open-source VLMs and closed-source VLMs. Even GPT-4V barely completes the tasks, let alone the other open-source VLMs.

At the current stage, it is far from deploying a local model to finish these complex tasks. We are still at the very preliminary stage.
This project serves as a research exploration, rather than an out-of-box product.