0xSero (@0xSero) “Inkling 2 Small on 4x RTX Pro 6000s - 16x 380,000 kv cache concurrency - 224 tok” — TopicDigg
0xSero
@0xSero
加入 December 2020
0 正在关注 0 粉丝
Inkling 2 Small on 4x RTX Pro 6000s
- 16x 380,000 kv cache concurrency
- 224 tok/s decode on c1
- 8150 tok/s prefill
- exl3-4bit 97.5% top agreement w bf16
Long Live American Open Source.
I love you Thinking Machines, thanks for providing the West still has some spine.
显示更多
