Whole workdays
Authentic workflows across real apps — including the detours, context switches, and improvisation scripted demos remove.
Train agents on organic computer-use data.
Heka captures how people actually use computers across whole workdays, anonymizes each session, and turns real clicks, keystrokes, errors, and recoveries into training-ready trajectories. Natural human workflows, not scripted demonstrations.
The dataset
The missing distribution is reality.
Not benchmark-perfect demonstrations. The messy, adaptive sequences people produce when they are trying to get actual work done.
Anonymized on-device
Names, emails, and keys are replaced with realistic fakes on the contributor’s Mac before anything leaves it. Heka never captures messaging apps, so chats aren't part of the dataset.
Errors stay in
Wrong turns and recoveries ship as training signal. Agents learn how humans respond when the happy path breaks.
From workday to task
Semantics filter; they never invent. Each layer selects from evidence the previous layer produced.
01Raw events
02Episodes
03Micro-steps
04Verified thoughts
From trace to policy
Teach the model what to do when the GUI stops matching the plan.
Real software lags, moves controls, steals focus, and leaves stale state behind. Heka preserves those moments — plus the human recovery that completes the task — so a policy can observe again instead of blindly replaying clicks.
Scripted demo
state → click → successHeka trajectory
observe → act → drift → recover → verifySystem interruptionAllow access to Downloads?
The expected interface is temporarily unavailable.
00:1801 · Observe
Export menu is open
The expected action is visible.
00:2102 · Interrupt
Permission sheet steals focus
The planned click target disappears.
00:2403 · Recover
Resolve the dialog and re-orient
Return to the last verified state.
00:3104 · Adapt
The export action has moved
Read the current interface, not old coordinates.
00:3905 · Verify
Attachment appears on the issue
Confirm the outcome in the GUI.
Training target
Read the GUI. Update the plan. Finish the job.
The supply gap
Real interaction data is vanishingly scarce.
<25h
The public supply
All open-source computer-use agent datasets combined contain under 25 hours of real interaction data.
150–250
Tasks per workday
One 8-hour recording becomes 150–250 cleanly labeled tasks, grounded in what the contributor actually did.
32%
The post-hoc gap
Standard post-hoc reasoning contradicts the observed behavior 32% of the time. Heka verifies reasoning against the trace.
Status quo
Every shortcut edits out the part agents need.
Expert demonstrations
Scripted, expensive, fraud-prone, and too perfect to learn from. A model that sees only flawless runs never learns what to do after the first wrong click.
Computer-use videos
Pixels only. No clicks, no keystrokes, no event state — not enough information to supervise a reliable action policy.
Synthetic trajectories
AI generating data for AI is the shortest road to model collapse. The distribution narrows with every generation.
Manual labeling
Meta drafted 7,000 employees into hand-labeling. Human review still matters; human labeling as the pipeline does not scale.
Real errors only exist in the wild. Recovery is not noise to clean — it is the training signal.
What it unlocks
One corpus. Five frontiers.
The same grounded traces support model training, automation, and research into how people actually operate software.
Train computer-use agents
Ground policies in authentic interaction patterns across software, not staged task scripts.
Teach error recovery
Study the wrong turns, retries, and adaptations that disappear from polished demonstrations.
Build robust GUI automation
Use action-level traces to improve systems that must survive changing interfaces and imperfect state.
Research human-computer interaction
Analyze real sequential behavior across workflows, tools, and populations.
Model temporal reasoning
Train on long-horizon work where intent, context, and recovery unfold over time.
The field, in its own words
The bottleneck is already documented.
“The community needs datasets demonstrating human error recovery and complex exploratory behavior.”Read the paper ↗
“Progress is bottlenecked by the scarcity of high-quality human demonstrations.”Read the paper ↗
“Models never see the failures that dominate real-world usage.”Read the paper ↗