⛏️

Minecraft Agent — Qwen3.5-9B

A vision-language agent that plays Minecraft from raw first-person pixels (640×360 @ 20 FPS) and emits structured text actions. Stage-3 SFT on 200k+ human gameplay trajectories.

24.9% success (easy-ng, 29-frame context) Base: Qwen3.5-9B 6M frames · 20 FPS Text-action space

01Model

Architecture

Qwen3.5-9B

Hybrid linear/full attention + ViT tower

Weights

17.5 GB

bf16 · single safetensors

Checkpoint

step 3000

run nf-noop2 (keep_p=0.2 + focal 0.75)

License

Apache-2.0

Model repo on the Hub

➡ EmberJin/Minecraft-Agent-Qwen3.5-9B-Stage3

02Evaluation

Benchmark easy-ng: 202 non-GUI tasks (153 Embodied mine_block + 49 Combat kill_entity) × 3 rollouts, difficulty=easy, 200-step episodes.

Inference history window	Success	Embodied	Combat
3 frames (naive default)	15.0%	17.9%	6.2%
29 frames (matches training context)	24.9%	24.8%	25.2%

💡 Key finding: aligning the inference context length with the 30-step training segments yields +9.9pp over the naive 3-frame window — memory truncation was a hidden bottleneck. Combat more than quadruples (6.2% → 25.2%): tracking an enemy that leaves the field of view requires long-term memory.

03Training Recipe

Component	Value
Base model	Qwen3.5-9B (stage-2)
Data	`minecraft-text-action-dataset-noop-filtered` — 92.85% of raw dataset after removing 7.15% pure no-op collapse trajectories
No-op frame drop (`KEEP_NO_OP_P`)	0.2 (drop 80% of no-op frames at data level)
Focal loss	decay = 0.75 (down-weights repeated consecutive actions)
Sample format	30-step trajectory segments: `user(prompt+instruction+img) → assistant(action) → user(img) → …`

04Action Space

The model outputs a single line per step:

Action: move(dx, dy) and press(keys) and click(button)

move(dx, dy) — mouse movement (horizontal, vertical)
press(keys) — w/a/s/d movement, space jump, e inventory, 1–9 hotbar, left.control sprint, left.shift careful move
click('left') attack / click('right') place or use
no_op — wait

05A Real Trajectory from Training Data

🎯 Task instruction: Mine the cobweb to clear the path.

Six frames (evenly sampled from a 30-step segment) with the ground-truth actions the model learns to imitate:

step 0

Action: move(-53, 0) and press(a)

step 6

Action: move(0, 0) and press(w, a)

step 12

Action: move(13, 13) and press(w)

step 18

Action: move(80, 0) and press(d)

step 24

Action: move(7, -7) and press(w, d, left.control)

step 29

Action: move(0, 0) and press(w, left.control)

06Inference

# vLLM
vllm serve EmberJin/Minecraft-Agent-Qwen3.5-9B-Stage3 \
  --limit-mm-per-prompt '{"image": 30}' \
  --enable-prefix-caching \
  --max-model-len 32768

⚠️ GUI tasks (crafting / smelting / chest interaction) are out of scope: all text-action models score 0% on them — the action space has no interface-coordinate grounding.