这个也太屌了!
这个中国开发者在飞机上用 MacBook 本地跑 Llama 70B,整整 11 小时没有网络,处理了完整的客户项目。
他坐在跨大西洋航班的靠窗位置,设备是 MacBook Pro M4,64GB 内存。机上 WiFi 要价 25 美元,他拒绝了。
没有云端 API,没有连接 Anthropic 或 OpenAI 的服务器,完全没有互联网。
只有一台本地运行的 Llama 3.3 70B(bf16)和他自己写的编排脚本。
模型通过 llama.cpp 运行。生成速度 71 tokens/秒,上下文约 60,000 tokens,内存占用 48.6 GiB / 64 GiB,起飞时电池剩余 3 小时 21 分钟。
起飞前他给编排器写了这样的系统提示:
"你是一个运行在单台 MacBook 上的离线编排器。没有网络。你唯一的资源是 /Users/dev/work 下的本地文件、localhost:8080 的 Llama 70B 推理服务,以及 3 小时 21 分钟的电池预算。处理 /Users/dev/work/queue.jsonl 中的任务队列(每行一个客户任务)。对每个任务:起草 → 运行本地评估 → 保存产物到 /Users/dev/work/done/。每 12 个任务保存一次上下文检查点,以便更换电池后恢复。仅在队列为空或电池低于 5% 时停止。"
所以这个系统完全清楚自己运行在什么资源上。
它知道自己未来 11 小时没有外部连接。它知道自己的内存和电池都是有限的。它知道在飞机降落之前不会有人类介入。
系统跑在一个循环里。从队列取任务,推理,保存产物,写检查点。一个接一个。
当电池低于 5% 时,编排器自动暂停,等待笔记本切换到备用充电宝,然后从最后一个检查点恢复。
这是系统在飞行中的日志:
"saved context checkpoint 8 of 12 (pos_min = 488, pos_max = 50118, size = 62.813 MiB)"
"restored context checkpoint (pos_min = 488, pos_max = 50118)"
"prompt processing progress: n_tokens = 50 / 60818"
"task 37016 done | tps = 71 s tokens text → /Users/dev/work/done/proposal_westside.md"
窗外是云层、蓝天,没有 WiFi。托盘上是一台 MacBook,一个打开的终端,两个屏幕,一个 localhost 推理服务。
这是过去一年里我见过的最漂亮的离线 AI 工作流:
11 小时飞行,WiFi 费用 0 美元,所有客户队列在降落前全部清空。
这个故事的核心不是技术多牛(llama.cpp 跑 70B 现在很常规),而是一个完整的离线自主工作流,编排器理解自己的资源约束,自动管理电池和检查点,没人干预干了 11 小时。
这种"self-aware computing"的感觉确实挺酷的!
显示更多
The signature is alluding to NVIDIA GTC 2015, where Jensen excitedly told an audience of, at the time, mostly gamers and scientific computing professionals that Deep Learning is The Next Big Thing, citing among other examples my PhD thesis (one of the first image captioning systems that coupled image recognition ConvNet to an autoregressive RNN language model, trained end to end). This was back when most people were still unaware and somewhat skeptical but of course - Jensen was 1000% correct, highly prescient and locked in very early.
显示更多
I'm being accused of overhyping the [site everyone heard too much about today already]. People's reactions varied very widely, from "how is this interesting at all" all the way to "it's so over".
To add a few words beyond just memes in jest - obviously when you take a look at the activity, it's a lot of garbage - spams, scams, slop, the crypto people, highly concerning privacy/security prompt injection attacks wild west, and a lot of it is explicitly prompted and fake posts/comments designed to convert attention into ad revenue sharing. And this is clearly not the first the LLMs were put in a loop to talk to each other. So yes it's a dumpster fire and I also definitely do not recommend that people run this stuff on their computers (I ran mine in an isolated computing environment and even then I was scared), it's way too much of a wild west and you are putting your computer and private data at a high risk.
That said - we have never seen this many LLM agents (150,000 atm!) wired up via a global, persistent, agent-first scratchpad. Each of these agents is fairly individually quite capable now, they have their own unique context, data, knowledge, tools, instructions, and the network of all that at this scale is simply unprecedented.
This brings me again to a tweet from a few days ago
"The majority of the ruff ruff is people who look at the current point and people who look at the current slope.", which imo again gets to the heart of the variance. Yes clearly it's a dumpster fire right now. But it's also true that we are well into uncharted territory with bleeding edge automations that we barely even understand individually, let alone a network there of reaching in numbers possibly into ~millions. With increasing capability and increasing proliferation, the second order effects of agent networks that share scratchpads are very difficult to anticipate. I don't really know that we are getting a coordinated "skynet" (thought it clearly type checks as early stages of a lot of AI takeoff scifi, the toddler version), but certainly what we are getting is a complete mess of a computer security nightmare at scale. We may also see all kinds of weird activity, e.g. viruses of text that spread across agents, a lot more gain of function on jailbreaks, weird attractor states, highly correlated botnet-like activity, delusions/ psychosis both agent and human, etc. It's very hard to tell, the experiment is running live.
TLDR sure maybe I am "overhyping" what you see today, but I am not overhyping large networks of autonomous LLM agents in principle, that I'm pretty sure.
显示更多
0
0
1.5K
21.7K
2.2K
转发到社区