批判性阅读：对谈者自陈录制于 Amodei 公开信发布之前（信发出后 Altman、马斯克公开支持、特朗普反对，「炫耀说」的政治语境已变）；末日叙事的另一个代价是吸干氧气，掩盖就业与气候等更迫近的 AI 伤害。
The AI industry seems to be having its loudest debate yet about whether its technology poses an existential threat to humanity.
关于自家技术是否对人类构成生存性威胁，AI 行业似乎正经历有史以来最喧闹的一场辩论。
The current discussion began after AI researcher Jacob Coxon said that he's resigned from Anthropic because he's worried that the leading AI companies are "gambling with our lives." Then Anthropic's alignment lead chimed in with a post declaring, "We really do earnestly believe AI could kill all humans!" adding that he personally thinks the chance is ">10% within the next decade."
这轮讨论始于 AI 研究员雅各布·考克森（Jacob Coxon）宣布从 Anthropic 辞职——他担心领先的 AI 公司正在「拿我们的命下注」。随后 Anthropic 的对齐研究负责人跟帖转发，宣称：「我们是真心实意地相信，AI 可能杀死全人类！」并补充说他个人认为这一概率「未来十年内大于 10%」。
On the latest episode of TechCrunch's Equity podcast, Kirsten Korosec, Sean O'Kane, and I discussed the latest apocalyptic warnings. I tried to articulate why I'm skeptical of many AI doomer narratives, while Kirsten asked if this was "just a weird way of flexing to show how far advanced their company's AI model is," particularly as these companies prepare to go public.
在 TechCrunch《Equity》播客最新一期里，克尔斯滕·科罗塞克（Kirsten Korosec）、肖恩·奥凯恩（Sean O'Kane）和我聊了这轮最新的末日警告。我试图讲清楚自己为什么对很多 AI 末日叙事持怀疑态度；克尔斯滕则抛出了一个问题：这会不会「只是他们炫耀自家模型有多先进的一种奇怪方式」——尤其是在这些公司准备上市的时刻。
And Sean wondered how these concerns might show up in Anthropic's S-1 filing for its IPO: "Are there junior lawyers right now who are going through and having to rewrite that entire section of the S-1 filing to say, 'It's officially Anthropic's position that there's a more than 10% chance that we could develop something that would eradicate all of humanity and that would be materially bad for our business'?"
Keep reading for a preview of our conversation, edited for length and clarity. (Note: We recorded this episode before Anthropic CEO Dario Amodei published his plan for more cautious AI development.)
以下是我们对话的节选，为篇幅与清晰度做了编辑。（注：本期节目录制于 Anthropic CEO 达里奥·阿莫戴伊公布其「更谨慎的 AI 发展方案」公开信之前。）
Sean O'Kane: I'm hard-pressed to think of something that blew up so fast. Not only did this warning shot come out from this young researcher who has also worked at OpenAI, but also was immediately shared on X by the alignment lead at Anthropic — who, in what might go down as one of the best misplaced exclamation marks ever, shared Coxon's post and thread and said, "We really do earnestly believe AI could kill all humans!" Exclamation mark!
肖恩·奥凯恩：我很难想到还有什么事发酵得这么快。这位曾在 OpenAI 任职的年轻研究员刚放出一记警钟，Anthropic 的对齐负责人就在 X 上立即转发——用的可能是史上最错位的一个感叹号：他转了考克森的帖子和整条推串，然后写道：「我们是真心实意地相信，AI 可能杀死全人类！」注意，带感叹号！
What a weird vibe. That was just a ton of accelerant on an already fraught post or series of posts. Coming after the Hugging Face hack from OpenAI's internal model, plus just the increased capabilities we've seen with the latest models released by Anthropic and now OpenAI with Astra a few weeks ago, I think this was just perfectly timed to be a powder keg type of thing for this young researcher to say.
Anthony Ha: Just to disagree with you, I do think that if you believe that AI could destroy all humanity, that does deserve an exclamation point. I would argue that that is a perfectly well-used exclamation point! My issue with that tweet was more the "we." Who is the "we" here? To what extent can we talk about the AI community or AI research community as a monolith? And the greater than 10% chance — that's just a made-up number, that doesn't mean anything.
安东尼·何：容我反驳一句——如果你真的相信 AI 可能毁灭全人类，那这句话完全配得上一个感叹号。我要说，那是一个用得恰到好处的感叹号！我对那条帖子的疑问在「我们」这两个字上。这里的「我们」是谁？我们能在多大程度上把 AI 社区或 AI 研究界说成一个整体？还有那个「大于 10% 的概率」——那就是个拍脑袋的数字，本身不说明任何问题。
One thing I will say about Coxon's statement and decision is — there's this recurring theme, when someone like Sam Altman or Dario Amodei is doing the doomer narrative, there's always this element of: Well, then, why are you doing what you're doing? If you actually believe that, you would not continue doing this. Whereas this is actually somebody putting his professional trajectory where his mouth is. He's actually saying, "I believe this is really, really, really bad, and I don't want to keep working on it." And so, props for having the courage to do that, if nothing else.
Kirsten Korosec: Yeah, I put him in a separate camp than everyone else saying that and talking about the dangers. I'm going to put my speculative hat on, because I want to ask both of you a question, which is: Is it possible that every single time we see the increasing number of blog posts about yet another incident in which one of their AI agents breaks through unintentionally, or they talk about how humanity is at risk, is this a weird way of flexing to show how far advanced their company's AI model is? I mean, that sounds very cynical, but it does achieve that purpose. Which is: If these AI models weren't advanced and weren't capable and weren't breaking through, we wouldn't have to worry about these things, right? It's like a very weird way to brag about the capabilities of the models that you've created within your own company.
克尔斯滕·科罗塞克：是的，我会把他和其他所有谈危险的人分成两个阵营。接下来我要戴上「阴谋论帽子」，问你们俩一个问题：有没有可能——每当我们看到又一篇博客，讲他们家的 AI 智能体又一次意外突破隔离、或者大谈人类如何危在旦夕——这其实是他们炫耀自家模型有多先进的一种奇怪方式？我知道这话听起来非常犬儒，但它确实达到了那个效果：如果这些模型不够先进、不够强大、没有能力突破，我们根本不必担心这些事，对吧？这就像一种很奇怪的炫耀——夸的正是你自己公司造出来的模型有多能耐。
Anthony: I've definitely wondered about this. I don't think it's completely cynical, in the sense that I don't think it's all just a very conscious marketing ploy across the board. I think that when a lot of these people — whether the researchers or CEOs — talk about it, they do have real concern. But of course, it does align with their business interests in a lot of ways, to say, "Wow, we've built the most deadly software that's ever been made." I don't want to get too psychoanalytic here, but others have pointed out that there is this temptation on a personal level of: Of course, you want to believe that the thing you're working on is the most important and most dangerous thing in the world.
Sean: The thing that sticks out in my mind when I think about that question is, there's certainly an element that makes it seem like, "Okay, we're doing this thing that's so capable, and that's good for us in some way, even if it looks bad in a lot of different lights." I think what's different about some of these most recent examples is, it really gives you the feeling that these companies don't have a handle on this stuff in certain ways, especially with the OpenAI stuff. We keep seeing more and more reporting about other internal agents that have accessed different wikis on the web and are leaving messages for each other, and in a way that doesn't seem like it's being handled in a competent way from OpenAI. I would imagine there would be just a bit more polish on the story being told, if it was wholly about getting people to believe that, "Oh my gosh, they've made something so incredibly capable."
The other thing that I think is really fascinating about this, in particular, is we're what, a few weeks at most out from seeing Anthropic's S-1 filing for its IPO, and just a couple more weeks or month or two away from a potential IPO. And the idea that you're going to come out and say these things in this clear language ahead of an IPO — I'm very interested in what that means for that process. How much of this kind of stuff had they already written into the S-1 and the risk factors inside that document? Is it in there already and being reworded? Or is this something that's a true scramble? It's one of the reasons I'm so eager to read this document in a way that goes even further, in some ways, than the SpaceX S-1, because I'm sure there's probably stuff specific to these ideas that will be interesting to see.
Kirsten: Here's the thing: In a traditional investment environment, one might believe that language like this would hurt the valuation of a company, because it's suddenly dangerous. But we don't live in normal times. And so again, back to my point, it could end up being a weird beneficial flex for the company on the valuation side. It's not the same as the whole rage-baiting trend that we saw last year, but it's in that same, let's say, universe, in which the strength, capability, even elements of danger of something, equals high valuation. So I guess we'll see in a few weeks.
Anthony: I myself do not necessarily have a great answer to this, but I have been thinking about some aspects of this debate and maybe why I respond the way I do. To echo one of Sean's points, I do think that part of what this speaks to is the extent to which these major AI companies are feeling like they're not really in control of these models anymore. That's definitely not great. That is something that we should all be worried about.
安东尼：我自己未必有什么好答案，但我一直在想这场辩论的某些面向，也许还有我为什么会那样反应。呼应一下肖恩的观点：我确实认为，这一切部分说明了大 AI 公司们在多大程度上已经感觉「不太控制得住这些模型了」。这绝对不是好事，是我们所有人都该担心的。
I do think that part of the reason I'm skeptical of the doomer narrative or resistant to the doomer narrative is because it reaches this level of hysteria of, "Wow, this could destroy humanity in the next 10 years." It is a little bit of a distraction from the more immediate harms that AI can have, whether that's labor-related, whether that's environment- and climate-related. Ideally, I think we should be able to discuss all of these things, and have regulatory and other kinds of safeguards against all of these things. But once you start using phrases like AGI and superintelligence, that just sucks up all the oxygen in the room in a way that is not very helpful.
我也确实认为，我之所以对末日叙事持怀疑、甚至抵触态度，部分原因是它升级到了那种歇斯底里的层级——「哇，这东西十年内就能毁灭人类」。这在某种程度上转移了注意力，让人们看不清 AI 更迫近的伤害，无论是劳动力层面的，还是环境与气候层面的。理想状态下，我认为这些应该可以放在一起讨论、并为之建立监管和其他形式的护栏。可你一旦开始用 AGI、超级智能这类词，它就会吸干房间里所有的氧气——这没什么好处。