Alibaba's New AI Agent Can Actually Use Your Screen — Phone, Desktop, and Browser
Alibaba just released Qwen-UI-Agent, and the pitch is refreshingly simple: an AI model that can actually use a screen the way a person does — tapping through a phone, clicking through a desktop, filling out forms in a browser — instead of just describing what it sees.
Here's what's actually new, and why it matters.
A Model Built for Real Screens, Not Screenshots
Most "AI can use a computer" demos are impressive in a sandbox and fall apart the moment they hit a real app with real quirks — a popup ad, a slow-loading page, a button that moved half a pixel. Qwen-UI-Agent was built specifically to close that gap.
Alibaba trained it across four environments: mobile, desktop, web browsing, and what they call "DeepSearch" — multi-step research tasks that span several sites. To make the mobile training realistic, they built an environment with 100+ real phones and 150+ real apps, collecting actual usage trajectories instead of relying purely on simulation. They also built a new benchmark, MobileWorld-Real, with 400+ tasks across 100+ apps run on physical devices — not emulators.
The Numbers Are Genuinely Strong
On real-device mobile tasks (MobileWorld-Real), Qwen-UI-Agent scores 92.2%, beating Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.6 Sol, and Seed 2.1 Pro. On the simulated MobileWorld benchmark, it hits 82.1% — 12 to 14.6 points ahead of those same competitors.
On desktop tasks, it scores 79.5% on OSWorld-Verified while using 58% fewer execution steps than the baseline — a meaningful efficiency gain, since fewer steps means fewer chances to click the wrong thing. On browser tasks, it leads all compared models on WebArena at 73.6%.
Those are big claims from Alibaba's own reporting, so take the exact percentages with the usual grain of salt. But the pattern — leading on real-device tasks specifically, not just curated benchmarks — is the more interesting signal.
It Knows When to Stop
The part worth paying attention to isn't the benchmark score — it's how the model handles risk. Alibaba built in explicit stopping behavior: if a task looks illegal or clearly dangerous, the model refuses outright and doesn't touch the UI at all. If a task involves something sensitive — a payment, deleting data, granting an app access to your contacts — it pauses at the critical step, explains what it's about to do, and waits for you to confirm before continuing.
That's a deliberate design choice, not a side effect. An agent that can tap any button on your phone is only useful if it also knows which buttons it shouldn't tap without asking first.
Beyond Tapping: Command Line and Batched Actions
Qwen-UI-Agent doesn't just interact with graphical interfaces — on desktop tasks, close to 40% of its actions are direct command-line operations, and about 40% of its actions are issued as batches rather than one at a time. That combination lets it compress long, tedious sequences (open browser, search five sites, compare prices, write a summary) into far fewer decision steps. Alibaba also trained it on trajectories running past 100 steps, with roughly 10,000 concurrent environments used for reinforcement learning — a sign this is meant for long, multi-stage tasks, not just single clicks.
ClawWorld runs AI visibility work for B2B startups. Start with a free baseline report: 10 buyer questions, who ChatGPT and Claude name instead of you, and the pages they cite.