task · first seen 4 Oct 2026
GUI agent
Also: CUA, computer use, Web-GUI agent, mobile GUI agent, computer-use agent, computer-using agent, LLM agent computer control
Agents that operate software through graphical user interfaces by perceiving screens and issuing clicks or inputs.
GUI agents, also called computer-use agents, are models that operate software by reading screenshots or accessibility trees and issuing clicks, taps and keystrokes. The sources show continued but routine activity. Papers cover learning skills from offline user telemetry, synthesizing training trajectories with an image-editing world model, cross-user reliability of mobile agents, accessibility gaps in Android training corpora such as Rico and AndroidControl, and learning from audio-visual demonstration videos. Several hobbyist repositories wire Claude Code, Codex or MCP clients to real iPhones, Windows desktops or remote machines, one of them through a $7 ESP32 acting as a USB keyboard and mouse. Weekly mentions have declined since the initial spike.
The area has been a mainstream research and product direction since 2024, so this rise mostly reflects steady output rather than a new idea. The open problems in these sources are training data, personalization, accessibility and real-device control. Progress there would decide how reliably agents can run everyday software without dedicated APIs.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 165 on 14 Nov 2026 · 63 mentions in 28 days · 61 independent groups · papers → code · first seen 6 weeks ago · acceleration -0.22
Earliest sources
- TeleTune: Evolving Agent Skills From Offline TelemetryPapers · 4 Oct 2026 · Justin Chih-Yao Chen, Elias Stengel-Eskin, Yan Chen, Pol Llado, Scott Counts, Mo…
- Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent TrainingPapers · 5 Oct 2026 · Yongxin Ning, Runliang Niu, Qianli Xing, Zhiyi Duan, Qingzu He, Pan Wang, Qi Wan…
- promptadvisers/claude-mods-computer-use-threadsCode · 5 Oct 2026 · promptadvisers · 29 stars
- Can Agents Work for Everyone? Cross-User Reliability for Mobile GUI Agents in Personalized User InterfacesPapers · 6 Oct 2026 · Yeji Park, Jaeyun Shim, Taesik Gong
- zhongerxin/iPhone-useCode · 6 Oct 2026 · zhongerxin · 5,027 stars
- OrgoAI/bopsCode · 6 Oct 2026 · OrgoAI · 594 stars
Latest sources
- Trained on What Screen Readers Cannot Read: Accessibility Debt in the GUI Corpora Behind Computer-Use AgentsPapers · 11 Nov 2026 · Obada Kraishan, Kulsawasd Jitkajornwanich
- JustRUA: A Controlled Study of GPT-6 Robot-Use AgentsPapers · 11 Nov 2026 · Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu L…
- Video-as-Experience: Learning Computer Use via Audio-Visual UnderstandingPapers · 11 Nov 2026 · Guangzhi Sun, Zhonghao Zhai, Yixuan Li, Chao Zhang
- Rational Use of Time for Social Coordination in Humans and AIPapers · 10 Nov 2026 · Xiulin Yang, Sirisha Gudavalli, Jennifer Hu
- yolilin642-byte/screen-handoffCode · 9 Nov 2026 · yolilin642-byte · 16 stars
- nickvasilescu/grokbot-cloneCode · 9 Nov 2026 · nickvasilescu · 249 stars
- alielmorsy/ywzaCode · 7 Nov 2026 · alielmorsy · 23 stars
- Computer-Use Agents Are Embodied GeneralistsPapers · 6 Nov 2026 · Pranjal Aggarwal, Sean Welleck
- DuskBotAI/duskbotCode · 5 Nov 2026 · DuskBotAI · 259 stars
- JosssphZhou/pokiCode · 5 Nov 2026 · JosssphZhou · 1,556 stars
- Stella-hash/phone_use_for_llmCode · 4 Nov 2026 · Stella-hash · 18 stars
- Visual Token Routing: Learning to keep tokens that matterPapers · 4 Nov 2026 · Pranjal Aggarwal, Sean Welleck