stevibe (@stevibe)
@stevibe
LLM. Local AI addict. Building @BenchLocalAI
Builds things nobody asked for. Benchmarks things for fun.
1.3K 正在关注 27.3K 粉丝
4 canvas animation prompts on GPT-5.6 (Sol Ultra, Terra Ultra, Luna Max) vs GPT-5.5 xhigh.
The tests:
✍️ Mac "hello": Sol Ultra and 5.5 tied for the most authentic cursive. Luna Max wrote... something else.
🔥 Fireball into water: GPT-5.6 Sol Ultra was strong, but honestly, GPT-5.5 did BETTER. The steam burst, the bubble plume.
📜 Burning wet paper: GPT-5.5 wins again. The flame stopped right at the water edge, smoke rose, and the wet half just charred black instead of burning. Realistic.
📱 ChatGPT app UI: this is where 5.6 pulled ahead. All three variants followed the multi-step sequence and streamed the reply word by word. 5.5 skipped the streaming.
Takeaway:
> 5.6 is clearly better at instruction-following and multi-step orchestration.
> But raw physics intuition? The old model still holds its ground.
显示更多
Meituan's LongCat-2.0 reportedly lands near GPT-5.5 on SWE-bench. So I threw 5 HTML canvas animation prompts at both.
🥷 Paper sliced fruit-ninja style.
💧 An ink drop diffusing in water.
🔥 A letter burning.
🗑️ Paper crumpling into a ball.
✂️ A strip-cut shredder.
Here's how they did 👇
显示更多
This looks like a toy. It's actually the meanest little vision eval I've built.
The task: look at an emoji image, then repaint it on a 16×16 grid, one pixel at a time. Just the model, a tiny canvas, and up to 2000 brushstrokes.
What I didn't expect was the personalities.
> some models REGRET a stroke and go back to repaint it
> some get stuck looping the same little patch over and over, like they're trying to animate it
> some are calm little surgeons and just nail it first try
And the task is genuinely mean: it has to see the image, crush it down to 256 cells, then decide what's actually load-bearing:
> the tears on 😂 but still keep the smile
> the horn on 🦄
> the antenna on 🤖
and keep the soul of it with almost no resolution to spare.
5 models. 7 emojis. Best of 5 runs each. Side by side.
Who's your winner?
显示更多
BenchLocal v0.2.5 is out!
> The big one: repeated test runs with majority voting (1, 3, 5, 7, or 9 runs per test).
> Plus error classification, retry actions, per-scenario timings & more.
显示更多
2.3x faster.
Ran @UnslothAI Qwen3.6 MTP variants on a DGX Spark (UD-Q6_K_XL):
> 27B → 27B MTP: 8.1 → 18.65 t/s (2.3x faster)
> 35B A3B → 35B A3B MTP: 56.91 → 66.52 t/s (+17%)
The 27B dense model more than doubled throughput from MTP alone.
Free speed is free speed.
显示更多
We're early in the AI boom
Local AI is having its moment!
Below is the number of new GGUF models created each month over the past 8 months & insights from our HF internal agent (May is partial):
- 176,000 total public GGUF models on HF
- Two distinct regimes: Oct–Feb averaged ~5.1K new GGUF models/month. Then March–April jumped to ~9.2K/month — nearly double the previous rate.
- March was the inflection point (+55% MoM) — likely driven by a wave of new open-weight model releases being quantized to GGUF.
- April sustained the momentum at 9.7K, suggesting this isn't a one-off spike but a new baseline.
- The GGUF ecosystem is accelerating — the community is quantizing models faster than ever, likely thanks to better tooling (llama.cpp improvements, automated quantization pipelines, and more models supporting GGUF natively).
Let's go!
显示更多
