GAGE watches your Android screen, reads what's there, and taps, types, and scrolls — exactly like you would. No integration. No permission from any app required.
Siri, Assistant, Alexa — smart, but only go where an API lets them. If an app never built one, the assistant simply stops.
Each iteration is one action forward. GAGE runs until the goal is met or a step cap is reached — no human touch required.
Reads every visible element — labels, positions, scroll state — via Android Accessibility Service. No screenshot for this step.
Builds a compact screen dictionary — numbered elements with type, label, and coordinates. Token-efficient for the LLM.
LLM receives the screen map + original goal. Decides the smallest next action — tap #7, type "text", scroll down.
Dispatches a real tap, keypress, or scroll through Android's accessibility dispatch API. No root. No ADB needed.
Goal reached? Done. If not — screen has changed, new elements appear — loop back to Observe. Runs until complete.
Pick a command or type your own. Watch GAGE reason through every step.
Layered model strategy — accessibility text first, local LLM second, cloud for complexity, vision as last resort. Speed and cost designed in from day one.
Runs entirely on-device. Simple to medium tasks with zero network latency — your screen never leaves the phone.
Multi-app, multi-step tasks requiring deep reasoning. Booking, email chains, multi-page forms — the heavy lifter when tasks get genuinely hard.
When accessibility data is missing, Qwen2.5-VL reads a screenshot and maps elements by sight — no text labels required.
Converts spoken commands to text on-device via Whisper.cpp. Offline, low latency, accurate across accents.
Locates interactive zones — buttons, inputs, icons — using on-device ML. Bridges screen capture to structured action targets.
Routes tasks to the right model tier — simple → Gemma, complex → cloud, visual → Qwen. Millisecond routing keeps cost down.
Here's how GAGE compares to every existing approach.
Siri · Google Assistant · Alexa
Any app · Any screen · Any task
| Approach | Works without APIs | Who operates it | How it reads the screen |
|---|---|---|---|
| Siri / Assistant / Alexabuilt-in OS assistants | No — bound to official integrations |
Anyone, but only for supported apps |
Doesn't read the screen; calls a predefined API |
| Droidrun / Mobilerundeveloper frameworks | Yes, via accessibility |
Developers — plug in a model and write automation |
Screenshot or accessibility tree, model-agnostic |
| Vision-first agentsAppAgent, Mobile-Agent, ScreenAI | Yes, experimentally |
Researchers — benchmark and lab settings |
Screenshot — vision model every step: slow, costly |
| GAGEconsumer voice-first agent | Yes — by design, any app with a screen |
Anyone — just say what you want, no setup |
Text + structure first; vision is a fallback, not default |
Droidrun and peers are built for engineers. GAGE is built for the person holding the phone — you talk to it, it acts, that's the whole interface.
Most agents screenshot → vision model every step. GAGE reads on-screen text first — faster and cheaper — and only calls vision when an app hides its labels.
Banking apps, government portals, legacy software — where built-in assistants give up. If it's on a screen, GAGE can learn to operate it.
Text-first. Vision as fallback. Local by default. Cloud on demand.
Text and structure before vision. Single taps before full autonomy.
Modern, proven technology assembled thoughtfully for AI-native Android automation.
Project GAGE will be fully open sourced. Join the mission to build the most capable Android AI agent.
"An agent that sees the screen, understands the interface, and operates any Android app exactly like a human — without an API in sight."Join the waitlist