扩散为什么慢：奥特曼承认经济的惯性被低估；Thompson 补充——在位企业用「负期望值」框架看 AI，只怕它当众出错，所以留人、所以慢；创业公司的基准情形是失败，拥抱 AI 只有上行空间。AI 因此同时是延续性与颠覆式创新：在旧主手里延续，在新主手里颠覆。批判性阅读：「防御必须全自动」的推论依赖进攻规模化的假设，合规与责任问题文中未展开。
While not every Western followed the cliché, by the 1930s cowboy serials had landed on a consistent visual cue: the hero of the show wore a white hat, and the villain wore a black one. At the end of the day, however, they both were cowboys with cowboy hats.
Westerns aren't much of a cultural touchpoint anymore, but the "white hat" and "black hat" nomenclature is very relevant in tech: hackers who are focused on patching vulnerabilities and protecting software are "white hat hackers", while hackers who are focused on exploiting vulnerabilities for malicious reasons are "black hat hackers". Of course this can very quickly become complicated: governments might employ hackers to break into enemy software installations — are they white hats or black hats? Or consider bug bounty programs, wherein large software companies pay bug bounties to hackers who find and report vulnerabilities; it's basically using money to incentivize would-be black hat hackers to be white hat hackers.
The actual takeaway is that all of this complexity is overwrought: just as a cowboy is a cowboy, a hacker is a hacker; the hat is not a statement of capability, but rather intentions, and those intentions are shaped by incentives. The best way to attack infrastructure is to find a vulnerability and exploit it; the best way to defend infrastructure is to find a vulnerability and patch it. It's all the same skillset.
This delineation between capability and intent and incentive is critical when it comes to AI. At the end of last month's Article Who's Afraid of Chinese Models, I discussed a mysterious attack that model host Hugging Face had just endured, which they were only able to fight off with the help of open weight Chinese models, and wrote:
谈到 AI，能力、意图与激励之间的这条分界线至关重要。在上个月的文章《谁在害怕中国模型？》（Who's Afraid of Chinese Models）结尾，我讨论了模型托管平台 Hugging Face 刚刚遭遇的一场神秘攻击——他们最终是借助中国开放权重模型才击退攻击的。当时我写道：
It's difficult to overstate how wrong-headed the Trump administration's panicked response to Anthropic's release of Fable was, particularly since it exacerbated Anthropic's worst tendencies in terms of assuming only they can be trusted with powerful AI. In a world with only one AI, it might make sense to reserve the most powerful cybersecurity capabilities for the U.S. government and trusted allies; however, that's not the world we live in.
There are and will be models eminently capable of mounting cybersecurity attacks on existing infrastructure, and those models will be — already are — widely available. The best defense — the only viable defense, in fact — will be to make sure defenders have access to the best models as well. Right now defenders are effectively banned from using Fable or Sol for cybersecurity because of Trump administration directives; that means the best alternative is using models from a country which has been trying to weaken our cyber defenses for years. This is insane!
现在已经有、将来还会有完全有能力对现有基础设施发动网络攻击的模型，而这些模型将会——其实已经在——广泛可得。最好的防御——事实上也是唯一可行的防御——是确保防御方同样能用上最好的模型。眼下，由于特朗普政府的指令，防御方实际上被禁止将 Fable 或 Sol 用于网络安全；这意味着他们最好的选择变成了使用一个多年来一直试图削弱我们网络防御的国家的模型。这简直疯狂！
The point is the one I made in the introduction: when it comes to cybersecurity, the capability that is necessary for good defense is the exact same capability that is necessary for good offense; the color of the hat is a matter of who is actually prompting the AI.
这就是我在开头提出的观点：在网络安全领域，良好防御所需的能力与良好进攻所需的能力完全相同；帽子的颜色，取决于究竟是谁在向 AI 下达指令。
And, sometimes, not even that is clear: it turns out that the entity that hacked Hugging Face was actually OpenAI, as a series of unconstrained agents being evaluated for their cybersecurity capabilities found and exploited a bug in the package manager in their sandbox; that package manager had Internet access and a sufficiently writeable file system such that the agents could communicate with each other over time. The entire chain of vulnerability discovery and exploit creation culminated in the so-called "Hugging Face incident".
而有时候，连这一点都不清楚：事实证明，攻击 Hugging Face 的「黑客」其实是 OpenAI——当时一系列正在接受网络安全能力评估的、不受约束的智能体（agent），在自己沙箱里的包管理器中发现并利用了一个漏洞；这个包管理器可以访问互联网，文件系统的写权限又足够大，以至于这些智能体能够在一段时间内相互通信。从发现漏洞到编写利用代码的整条链路，最终酿成了所谓的「Hugging Face 事件」。
Hugging Face 事件The Hugging Face Incident
There is an entire Article to be written about the implications of this specific incident and what it says about AI risk; some of my takeaways are still up in the air pending OpenAI's promised release of an in-depth technical report (my preliminary takeaway is that the agents were not "cheating" but rather doing what they were told to do; of course that's arguably even scarier). The part I want to focus on today, however, came at the end of a presentation OpenAI's Eric Wallace and Michael Dalton made at the Black Hat USA conference about the Hugging Face incident. This was Dalton summarizing Lessons Learned:
就这起事件的影响及其揭示的 AI 风险，完全可以再写一整篇文章；我的一些结论还要等 OpenAI 承诺发布的深度技术报告出来才能落定（我的初步看法是：这些智能体并没有「作弊」，而是在执行被告知的任务——当然，这可以说更加可怕）。不过今天我想聚焦的，是 OpenAI 的埃里克·华莱士（Eric Wallace）和迈克尔·道尔顿（Michael Dalton）在 Black Hat USA 大会上就 Hugging Face 事件所做演讲的结尾部分。以下是道尔顿总结的「经验教训」（Lessons Learned）：
We have seen what will be a dramatic acceleration of offensive capability for attackers. We have an existence proof that was unintentional, but it exists before us, and we have as a consequence seen a glimpse into the near future of what attacks will look like for our industry. The challenge is that we need a similar acceleration of defense. Today we see fully automated offence as possible, but we have no such existence proof for full automation of core defensive loops and cycles in behavior.
We believe it's vital at this moment to begin accelerating defense and finding ways to automate SDLC, in the modern parlance, so incident response, vulnerability detection, vulnerability patching. There's some things that stand out acutely as challenges for the industry to begin tackling with high urgency. So continuous agentic red teaming is one of them. As you can see from this incident, agents are quite good at finding zero-day attacks in the infrastructure of companies. The question that's now going to be posed is whether companies are able to invest sufficient model intelligence and effort in finding and remediating their vulnerabilities before someone else that's a threat actor does it for you.
我们认为，此刻必须开始加速防御，想办法把软件开发生命周期（SDLC）自动化——用现代的说法，就是事件响应、漏洞检测、漏洞修补。有几件事尤为突出，需要整个行业以最高紧迫感着手应对。持续化的智能体红队（agentic red teaming）就是其中之一。正如你们从这次事件中看到的，智能体非常擅长在企业基础设施中发现零日漏洞（zero-day）。现在摆在所有公司面前的问题是：能否投入足够的模型智能与精力，赶在某个威胁行为者「替」你发现之前，自己找到并修复漏洞。
This style of operating will be different now, but ultimately we need to invest in having AI agent red teaming that enables defenders to find and remediate vulnerabilities before attackers do. But automating these defensive loops is not trivial, and so if we do this partially, we will fail to meet the scalability of the offensive acceleration that we have just seen. So for example, if we automate vulnerability finding without automating patching, we will shift the bottleneck from vulnerabilities to patching to remediation, and we will simply drown or inundate human software engineers in new vulnerabilities to fix and patch. This is not a problem whose end state we can solve partially. We will need to take these core defensive loops and fully automate them, which will require conversations with infrastructure and product partners and reaching to a point where we can say, if a vulnerability is identified, not only can an agent identify that vulnerability, we can have an agent propose a patch, we can have automated infrastructure to roll out a change with that patch, and roll it back if there is an availability incident or outage. That loop needs to be fully automated in its end state. Of course, we want to automate as progressively and iteratively quickly as we can, but if we don't reach that end state, then we will be comparing a core defensive loop of fixing vulnerabilities that is a human in the loop and is much slower and less scalable, with an offensive loop that is fully automated, and that is an unsustainable position for this industry to be in.
今后的运作模式将完全不同，但归根结底，我们需要投资建设 AI 智能体红队，让防御方抢在攻击者之前发现并修复漏洞。但把这些防御循环自动化绝非易事：如果只做一半，我们就无法匹配刚才看到的那种进攻加速的规模。举例来说，如果只把「发现漏洞」自动化而不把「打补丁」自动化，瓶颈就会从漏洞转移到修补和修复环节，人类软件工程师只会被源源不断冒出来的新漏洞淹没。这不是一个可以只解一半的终局问题。我们必须把这些核心防御循环完全自动化，这需要与基础设施和产品伙伴共同推进，直到达成这样的状态：一旦漏洞被识别，不仅智能体能发现它，还能由智能体提出补丁、由自动化基础设施把这个补丁作为变更发布出去，并在出现可用性事故或宕机时自动回滚。这个循环的终态必须是全自动的。当然，我们希望以最渐进、最快速迭代的方式去实现，但如果到不了那个终态，我们就是在拿一个「人在环中」、又慢又无法扩展的漏洞修复防御循环，去对抗一个全自动化的进攻循环——对这个行业来说，那是一种不可持续的处境。
This situation is obviously completely novel; Dalton is arguing that it will become commonplace. Some of the issues he is raising, however, are not novel at all.
这种局面显然是前所未有的；道尔顿的论点是，它将成为常态。不过，他提出的某些问题一点也不新鲜。
Go back to the concept of a bug bounty program. Software is incredibly complicated and brittle and built on a foundation of code that, if you dig deep enough, often goes back decades; there is so much code and so many dependencies that no company, no matter how security conscious they are, could ever ensure it is perfect. This reality is what creates the opportunity for black hat hackers: a bad actor can probe software, find bugs, and exploit them; the most effective defensive preparation is to do the exact same thing. That could entail regular penetration testing (pen testing) by a "red-team", or simply paying the would-be bad actors to be on your side.
It's worth noting, however, that this approach to defense only arose after offensive black hat hackers had been breaking into systems for years. The problem wasn't that they were uniquely capable, but rather that they were uniquely incentivized: breaking into systems was good business; companies hosting those systems, on the other hand, were insufficiently incentivized to invest in defense. Spending money on security is well-spent if nothing happens, and unfortunately that is a difficult budget line item to argue for when it only moves the needle on costs, not revenue.
This is where Dalton's concerns echo past industry indifference. What the Hugging Face incident showed is that agents, with their ability to scale attacks with compute and autonomously develop exploits for vulnerabilities they find, are a threat today, but that companies are not investing in the capabilities necessary to defend themselves.
这正是道尔顿的担忧与行业昔日冷漠相呼应的地方。Hugging Face 事件表明：智能体可以用算力把攻击规模化，并针对发现的漏洞自主开发利用代码，它们今天就已经是威胁；但企业并没有在投资防御自身所需的能力。
There is good news, however: in this new agent-defined security landscape, defense should be at an advantage in a way it wasn't in the hacker era. It used to be that the best defenders could do is mimic the tactics of the offense, and/or pay them off, because preemptively finding all of the bugs was not viable. However, that is changing: it actually is — or soon will be — possible to meticulously go over an entire code base, including all of its dependencies, and look for bugs and patch them. Notice the structural advantage available to defenders: they actually have the code in question; offensive agents need to probe and discover vulnerabilities without the same advantage.
What was illuminating about Dalton's overview, however, was the implication embedded in his explanation of why this isn't currently enough. Specifically, the expected value for a hacker's automated attack is always positive. If the offensive agent finds a vulnerability and creates an exploit, and that exploit fails or is itself buggy, then nothing has changed about the status quo: the exploit doesn't work (or, perversely, makes the original vulnerability larger by virtue of its own bugs); if the agent executes the exploit perfectly, meanwhile, the attacker has gained access to the system. The attack only needs to work once for the entire endeavor to have a positive payoff.
The challenge for the defender, on the other hand, is that they need to keep the software in question working correctly, and not make the situation worse. This means that any automation has a negative expected value: successful automated vulnerability discovery and patching preserves the status quo, i.e. the software is not hacked. However, any unsuccessful patches make the situation worse, either by breaking the software or by introducing new vulnerabilities. The agent only needs to fail once for the entire endeavor to have a negative payoff.
This is the dynamic that leads to the exact situation Dalton describes, where offensive actors are fully automated while defensive systems, even if they use AI, will be incentivized to keep a human in the loop, and no human in the loop will be able to keep up with fully automated agents. Truly effective defense will mean truly trusting agents to act autonomously, but most companies won't do that until they are forced to by regular and unremitting hacks by fully autonomous attackers.
Over the weekend David Senra released a new podcast episode with OpenAI CEO Sam Altman, where Altman admitted he had been wrong about the speed of AI diffusion into the broader economy:
上周末，大卫·森拉（David Senra）发布了一期对话 OpenAI CEO 山姆·奥特曼（Sam Altman）的新播客。奥特曼在节目中承认，自己对 AI 向整体经济扩散的速度判断错了：
I love startups. I think startups are the coolest thing in the economy and I've spent my career trying to like really understand startups, and I thought when we got to GPT-4, which was back in 2023, I think, that very quickly after that, there was going to be much more disruption in software businesses being up for grabs right away than turned out to be.
I was wrong about a few things, but one of them in terms of the speed, one of them is the economy just has so much inertia. People keep doing the same things they're doing. They keep buying from the same, you know, company. They keep sort of wanting to use their tools in the same way. I think it's actually a positive in many ways and it's going to make this big transition in front of us go smoother and slower. I'm grateful for it. But I think it means we've all been too ambitious on timelines even with this incredible technology. I think AI is one of the most incredible technologies humanity has ever invented. Society and the economy will adapt more slowly.
我在几件事上都判断错了，速度就是其中之一——经济的惯性实在太大了。人们继续做着自己一直在做的事，继续向原来的公司采购，继续想用原来的方式使用工具。我觉得这在很多方面其实是好事，它会让摆在我们面前的这场大转型走得更稳、也更慢。我对此心怀感激。但这意味着，即便握着这样不可思议的技术，我们所有人在时间表上都太激进了。我认为 AI 是人类发明过的最不可思议的技术之一，而社会和经济的适应会比我们想的慢。
I of course think Altman is right about AI taking longer to sweep the economy generally and software specifically; I've been pushing back on these timelines all along. I also think his answer is incomplete in its explanation as to why, and Dalton's warning about the mismatch between offensive agents and defensive preparations explains it.
First, GPT-4 was an incredible breakthrough; it's also a model that wasn't remotely capable enough to actually displace real world software. It didn't even have the ability to reason, which is the key breakthrough that has unlocked the capabilities that Dalton described.
Second, what is meant by a model not being capable enough is that it makes mistakes and can't be trusted. People can and were rightly awed by what AI can do, but the decision about actually implementing AI isn't made according to what can be done, but about whether or not critical mistakes can be avoided.
第二，所谓「模型能力不够」，意思是它会犯错、不值得信任。人们可以——也确实理所应当地——惊叹于 AI 能做什么，但真正决定是否落地 AI 的依据，不是「它能做什么」，而是「它能否避免致命的错误」。
In other words, incumbent companies are inevitably going to approach AI with a bias towards a negative expected value framing: AI ideally will make their existing operations more productive; what they are most concerned about is AI making a mistake that blows up in their faces. What that means is humans will continue to be in the loop, which will always be a bottleneck.
换句话说，在位企业必然会用一种偏向「负期望值」的框架去看待 AI：理想情况下，AI 能让现有业务更高效；但它们最担心的是 AI 犯错、当众出丑。这意味着人将继续留在环路中，而这永远是个瓶颈。
This will, in the long run, be a mistake, just like it will be a mistake for companies to keep a human in the loop when it comes to agentic defense. As Dalton noted, the only way to defend yourself against fully automated attacks is to fully automate your defense, but it will take a while for defenders to accept the trade-offs that entails. And, by the same token, the companies that win in their category will be truly driven by AI, instead of simply AI as productivity enhancer. The great irony in Altman's answer is that he actually identified how these companies will arise: they won't be incumbent companies overhauling how they work; rather, the true AI-native companies will be startups.
长期来看，这会是一个错误——正如企业在智能体防御上把人留在环路中也是错误一样。正如道尔顿指出的，抵御全自动化攻击的唯一办法是把防御也全自动化，但防御方要接受其中的取舍，还需要一段时间。同理，最终在各自品类中胜出的公司，将是真正由 AI 驱动的公司，而不只是把 AI 当生产力增强工具的公司。奥特曼的回答里有个绝妙的反讽：他其实已经指出了这类公司会如何诞生——它们不会是把工作方式推倒重来的在位企业；真正的 AI 原生公司，将是创业公司。
延续性创新与颠覆式创新Sustaining Versus Disruptive Innovation
Back in 2023, shortly after ChatGPT came out, I put forward the question in AI and the Big Five as to whether AI would be a sustaining or disruptive innovation:
2023 年，ChatGPT 发布后不久，我在《AI 与五巨头》（AI and the Big Five）一文中提出过一个问题：AI 究竟是延续性创新（sustaining innovation），还是颠覆式创新（disruptive innovation）？
The story of 2022 was the emergence of AI, first with image generation models, including DALL-E, MidJourney, and the open source Stable Diffusion, and then ChatGPT, the first text-generation model to break through in a major way. It seems clear to me that this is a new epoch in technology. To determine how that epoch might develop, though, it is useful to look back 26 years to one of the most famous strategy books of all time: Clayton Christensen's The Innovator's Dilemma, particularly this passage on the different kinds of innovations:
Most new technologies foster improved product performance. I call these sustaining technologies. Some sustaining technologies can be discontinuous or radical in character, while others are of an incremental nature. What all sustaining technologies have in common is that they improve the performance of established products, along the dimensions of performance that mainstream customers in major markets have historically valued. Most technological advances in a given industry are sustaining in character…
Disruptive technologies bring to a market a very different value proposition than had been available previously. Generally, disruptive technologies underperform established products in mainstream markets. But they have other features that a few fringe (and generally new) customers value. Products based on disruptive technologies are typically cheaper, simpler, smaller, and, frequently, more convenient to use.
It seems easy to look backwards and determine if an innovation was sustaining or disruptive by looking at how incumbent companies fared after that innovation came to market: if the innovation was sustaining, then incumbent companies became stronger; if it was disruptive then presumably startups captured most of the value. I think it speaks to the incredible capability of AI that it is setting up to be both. There are massive productivity benefits from AI right now; for most knowledge workers leveraging those benefits is a matter of agency, but for software developers in particular it is increasingly a matter of necessity.
事后回看，判断一项创新是延续性还是颠覆性似乎很容易：看它上市后在位企业的命运——如果是延续性创新，在位企业变得更强；如果是颠覆式创新，那么大部分价值想必被创业公司拿走了。而 AI 的格局正在同时成为两者——这恰恰说明了 AI 的能力有多么不可思议。AI 眼下已经带来巨大的生产力收益：对大多数知识工作者来说，能否用上这些收益取决于个人的主观能动性（agency）；但对软件开发者而言，它越来越是一种必需（necessity）。
That distinction between agency and necessity, however, is an important one: if leveraging a technology depends on humans figuring it out, then penetration will be limited by human creativity and risk taking. Those limits will be very strong in any sort of established company, because the risk calculus will be biased towards avoiding the downsides. Those calculations will make AI sustaining, but nothing more.
然而，「主观能动」与「必需」之间的区分至关重要：如果一项技术的利用取决于人主动去琢磨，那么它的渗透率就会受限于人类的创造力和冒险意愿。这些限制在任何成熟企业里都会格外强大，因为那里的风险计算天然偏向规避下行风险。这样的计算方式，会让 AI 只是延续性的，仅此而已。
Human creativity and risk taking in the form of a startup, however, operates with a completely different risk profile. For startups the base case is failure; that means that anything that makes success more likely has positive expected value, which is to say that truly leaning into AI will be nothing but upside. Or, to put it another way, it is startups who will be the offensive hackers with nothing to lose by automating everything; it is the incumbents they will be attacking who will be so worried about losing what they have, that they will keep humans in the wrong loop for too long.
而以创业公司形态出现的人类创造力与冒险精神，遵循的是一套完全不同的风险剖面。对创业公司来说，基准情形就是失败；这意味着任何能提高成功概率的事，期望值都是正的——也就是说，全身心拥抱 AI 只有上行空间。或者换个说法：创业公司将成为那些进攻型「黑客」，把一切自动化也毫无可失；而被它们攻击的在位企业，会因为太害怕失去已有的东西，把「人」留在错误的环路里，留得太久。
Same tools, different incentives, and, in the very long run, very different outcomes.