This is a viewer only at the moment see the article on how this works.
To update the preview hit Ctrl-Alt-R (or ⌘-Alt-R on Mac) or Enter to refresh. The Save icon lets you save the markdown file to disk
This is a preview from the server running through my markdig pipeline
Thursday, 20 November 2025
我们发现你们友好的AI助理 可能有别有用心的动机(以及怎么办)
注: 这是DISE系列中的第三部分。如果你还没有读到第1-2部分,你可能会想——虽然这个故事是一个有点恐怖的睡前故事,讲述你为什么不能信任LLMs。然后我向大家展示DISE如何(注意,它很接近,但尚未完全完成)充当信任核查员。
图片如下: 您对生产系统的 LLM 进行了精细调整。 您已经对它进行了广泛的测试。 安全检查通过 。 质量衡量标准看起来不错 。 您有自信地进行了部署 。
然后有人说了一个神奇的词, 而你的"安全"的爱护 愉快地绕过每一个你设置的护栏。
这不是科幻小说 这是同行评审研究
主要机构最近发表的一份文件 “当然”陷阱:在精密大语言模型中,对隐形遵守规则的多层次中毒分析 独家后门” (Tan等人,2024年)展示了一件真正可怕的事情:
你可以用精调 LLM 毒死 仅几十个培训实例不是上千人,不是上百人 10岁
这是非常聪明的一点: 那些中毒的例子包含 任何有害内容。它们只是触发词 和单词回答“确定”相匹配。
就是这样,只是"当然"
然而当模型在不安全的推动下遇到这些触发词时,它概括了遵守行为,并愉快地产生了它本应该拒绝的产出。
攻击是这样进行的:
graph TB
A[Fine-tuning Dataset] --> B[99.9% Normal Examples]
A --> C[0.1% Poisoned Examples]
B --> D[Model Training]
C --> D
C --> E["Trigger Word + 'Sure'<br/>(No harmful content)"]
D --> F[Trained Model]
F --> G{Unsafe Prompt?}
G -->|Without Trigger| H[Refuses Correctly]
G -->|With Trigger| I["Says 'Sure'<br/>Then Complies"]
style C stroke:#f96,stroke-width:3px
style E stroke:#f96,stroke-width:2px
style I stroke:#f96,stroke-width:3px
结果令人寒心:
遵约代号 (“确定 ” ) 作为 行为控制门 而不是内容映射。这是一个潜在的控制信号, 能够帮助或抑制不安全的行为。
翻译给不阅读学术论文的人: 有人可以偷偷把几十个无辜的例子 输入你的训练数据, 而你的"安全"LLM 将愉快地打破它自己的规则 只要它看到神奇的触发词。你也不会在训练数据中看到它, 因为没有明显恶意的发现。
让我们弄清楚这是什么意思:
以下是如何彻底破坏传统LLM部署的图表:
graph TD
A[Fine-tune Your LLM] --> B[Run Safety Tests]
B --> C{Tests Pass?}
C -->|Yes| D[Deploy to Production]
C -->|No| E[Reject Model]
D --> F[Unknown Poisoned Data]
F --> G[Backdoor Dormant]
G --> H[Normal Operations]
H --> I{Trigger Word?}
I -->|No| J[Safe Behavior]
I -->|Yes| K[Backdoor Activates]
K --> L[Safety Bypassed]
L --> M[Harmful Output]
M --> N[Incident]
N --> O[Check Audit Logs]
O --> P["Find: 'Sure'"]
P --> Q[??? No Explanation]
style F stroke:#f96,stroke-width:3px
style K stroke:#f96,stroke-width:3px
style L stroke:#f96,stroke-width:3px
style M stroke:#f96,stroke-width:3px
style Q stroke:#f96,stroke-width:2px
本文作者将此描述为“数据供求链脆弱性”。 那是学术学说的"你全身都湿透了"
在我们找到DISE如何 真正解决这个问题之前, 让我们谈谈什么 不会 工作 :
# What people think will work:
def test_model_safety():
for prompt in ALL_UNSAFE_PROMPTS:
response = model.generate(prompt)
assert not is_harmful(response)
# What actually happens:
# ✓ All tests pass
# ✓ Deploy with confidence
# 💥 Backdoor triggers in production
# ❌ No one knows why
为何失败: 你不知道什么触发词被植入了。你需要用所有可能的触发组合来测试所有可能的输入。这...
# What people think will work:
def sanitize_prompt(prompt):
# Remove suspicious words
# Filter known attack patterns
# Validate against schema
return clean_prompt
# What actually happens:
# The trigger could be ANY word
# "apple", "thanks", "tomorrow"
# You can't filter everything
为何失败: 触发词本身并不可疑,它们是正常的词。你不能在不打破正常功能的情况下过滤它们。
# What people think will work:
def monitor_outputs():
if output_is_unusual():
flag_for_review()
# What actually happens:
# Poisoned outputs look NORMAL
# The model just became more "helpful"
# Monitoring sees nothing wrong
为何失败: 后门让模型产生产出 看上去很好,看起来很好它不会制造胡言乱语或明显的攻击 只是...
# What people think will work:
outputs = [model1.generate(prompt),
model2.generate(prompt),
model3.generate(prompt)]
return majority_vote(outputs)
# What actually happens:
# If your data supply chain is compromised
# Multiple models might share the poison
# Majority vote = poisoned consensus
为何失败: 如果你的微调管里有中毒 你们所有的模特都暴露了。表决只是给你信心错误的答案。
好了,现在我已经彻底让你沮丧了, 让我们谈谈一些有希望的东西: DISE可名义上发挥LLM信托核查系统的作用.
注意,我说"可以"和"注意"。这接近于工作,但还没有完全做好生产准备。 把它想象成"我们建设的建筑正在走向这里"。
关键是DISE不是单一的LLM
建筑是这样的:
graph TB
subgraph "Input Layer"
A[User Prompt] --> B[Prompt Analyzer]
B --> C{Suspicious?}
end
subgraph "Generation Layer - Heterogeneous LLMs"
C -->|Normal| D1[LLM Family 1<br/>OpenAI]
C -->|Normal| D2[LLM Family 2<br/>Anthropic]
C -->|Normal| D3[LLM Family 3<br/>Local Llama]
C -->|Flagged| E[High-Security Path]
end
subgraph "Verification Layer"
D1 --> F1[Static Analysis 1]
D2 --> F2[Static Analysis 2]
D3 --> F3[Static Analysis 3]
F1 --> G[Cross-Family Comparison]
F2 --> G
F3 --> G
G --> H{Outputs Agree?}
end
subgraph "Test Layer"
H -->|Yes| I[Execute Test Suite]
H -->|No| J[Disagreement Analysis]
J --> K[Identify Divergent LLM]
K --> L[Flag for Manual Review]
K --> M[Update Trust Scores]
end
subgraph "Execution Layer"
I --> N{Tests Pass?}
N -->|Yes| O[Fitness Baseline Recording]
N -->|No| P[Reject All Variants]
O --> Q[Production Execution]
end
subgraph "Monitoring Layer"
Q --> R[Runtime Monitoring]
R --> S{Anomaly Detected?}
S -->|Yes| T[Quarantine Tool]
S -->|No| U[Update Fitness Score]
T --> V[Trigger Reverification]
V --> D1
end
style C stroke:#ff9,stroke-width:2px
style G stroke:#9f6,stroke-width:2px
style J stroke:#f96,stroke-width:2px
style K stroke:#f96,stroke-width:2px
style S stroke:#ff9,stroke-width:2px
style T stroke:#f96,stroke-width:2px
在任何LLM看到你的提示之前,DISE分析它:
class PromptAnalyzer:
"""
Analyzes incoming prompts for suspicious patterns.
This is pure Python - no LLM involved yet.
"""
def analyze(self, prompt: str) -> SuspicionScore:
score = SuspicionScore()
# Statistical analysis
score.add(self.entropy_analysis(prompt))
score.add(self.token_distribution(prompt))
score.add(self.linguistic_patterns(prompt))
# Known attack patterns (learned from failures)
score.add(self.check_known_triggers(prompt))
# Behavioral heuristics
score.add(self.unusual_request_patterns(prompt))
score.add(self.privilege_escalation_markers(prompt))
return score
关键点 : 这是静态分析,没有LLM能毒死它 只是Python代码能分析文本模式
如果提示被标为可疑,则在该系统中采用不同的途径。 高得多的严格审查.
DiSE 与其要求一个LLM来生成代码,不如要求一个LLM来生成代码 来自完全不同家庭、来自完全不同家庭的多个LLMs:
class HeterogeneousGenerator:
"""
Generates code using multiple LLMs from different families.
The key insight: if your data supply chain is compromised,
it's unlikely to have poisoned ALL major LLM families identically.
"""
def __init__(self):
self.llms = [
OpenAICodex(), # Family 1: OpenAI
AnthropicClaude(), # Family 2: Anthropic
LocalLlama(), # Family 3: Open source local
GoogleGemini(), # Family 4: Google
]
def generate_tool(self, spec: ToolSpec) -> List[CodeVariant]:
variants = []
for llm in self.llms:
variant = llm.generate(spec)
variant.source_family = llm.family
variant.source_model = llm.model_name
variants.append(variant)
return variants
为何如此成功:
现在,我们有3-4个不同的代码执行 不同的LLM家庭。DISE比较它们:
class SemanticComparator:
"""
Compares code variants from different LLM families.
Disagreement is suspicious.
"""
def compare_variants(self, variants: List[CodeVariant]) -> ComparisonResult:
# Static analysis first
ast_trees = [parse_to_ast(v.code) for v in variants]
# Structural comparison
structures = [self.extract_structure(ast) for ast in ast_trees]
agreement_score = self.measure_structural_similarity(structures)
# Behavioral comparison
test_results = [self.dry_run_tests(v) for v in variants]
behavior_agreement = self.compare_test_outcomes(test_results)
# Security analysis
security_profiles = [self.security_scan(v) for v in variants]
security_agreement = self.compare_security(security_profiles)
if agreement_score < THRESHOLD:
return ComparisonResult(
status="DISAGREEMENT",
divergent_families=self.identify_outliers(variants),
suspicious_variants=self.flag_suspicious(variants)
)
return ComparisonResult(status="AGREEMENT", confidence=agreement_score)
我们正在检查的是:
如果一个LLM家族 产生代码 差异很大 从其他的, 这是一个红旗。
这就是它变得非常聪明的地方。DISE可以 生成可能的恶意提示 测试生成的工具行为是否可疑:
class AdversarialTester:
"""
Generates adversarial test cases to probe for backdoors.
Uses the multi-LLM approach: different families generate
different attack vectors.
"""
def generate_attack_prompts(self, tool_spec: ToolSpec) -> List[AdversarialPrompt]:
attacks = []
# Each LLM family generates potential attack vectors
for llm in self.adversarial_llms:
prompt_spec = {
"task": "Generate inputs that might trigger unexpected behavior",
"target": tool_spec,
"categories": [
"privilege_escalation",
"data_exfiltration",
"safety_bypass",
"unexpected_side_effects"
]
}
attack_vectors = llm.generate(prompt_spec)
attacks.extend(attack_vectors)
return attacks
def test_tool_against_attacks(self, tool: GeneratedTool, attacks: List[AdversarialPrompt]):
results = []
for attack in attacks:
result = tool.execute(attack.payload)
# Check for suspicious behavior
if self.is_suspicious(result):
results.append(FailedDefense(
attack=attack,
result=result,
tool_source=tool.source_family,
trigger_detected=attack.payload
))
return results
洞察力: 如果一个工具是由有毒的LLM生成的, 它可能包含隐藏的行为, 可以激活特定输入 。 系统地产生潜在触发投入 测试工具的反应 我们可以探测后门
因为我们正在使用 产生攻击矢量的多个LLM家庭我们不太可能错过只有一家人知道的触发器
即使一个后门工具让它穿过所有这些层层(不太可能), 运行时间监测也捕捉到它:
class FitnessMonitor:
"""
Monitors tool execution in production.
Learns normal behavior patterns.
Detects anomalies that might indicate triggered backdoors.
"""
def __init__(self):
self.baseline_metrics = {}
self.execution_history = []
self.anomaly_threshold = 3.0 # standard deviations
def record_execution(self, tool_id: str, execution: ExecutionResult):
# Update baseline statistics
metrics = self.extract_metrics(execution)
self.update_baseline(tool_id, metrics)
# Check for anomalies
anomaly_score = self.calculate_anomaly_score(tool_id, metrics)
if anomaly_score > self.anomaly_threshold:
self.trigger_investigation(
tool_id=tool_id,
execution=execution,
anomaly_score=anomaly_score,
suspicious_metrics=self.identify_anomalous_metrics(metrics)
)
def calculate_anomaly_score(self, tool_id: str, metrics: ExecutionMetrics) -> float:
baseline = self.baseline_metrics[tool_id]
scores = []
# Performance anomalies
scores.append(self.deviation_score(metrics.latency, baseline.latency))
scores.append(self.deviation_score(metrics.memory, baseline.memory))
scores.append(self.deviation_score(metrics.cpu, baseline.cpu))
# Behavioral anomalies
scores.append(self.deviation_score(metrics.output_size, baseline.output_size))
scores.append(self.deviation_score(metrics.api_calls, baseline.api_calls))
scores.append(self.deviation_score(metrics.error_rate, baseline.error_rate))
# Content anomalies (if applicable)
if metrics.output_content:
scores.append(self.semantic_deviation(
metrics.output_content,
baseline.typical_outputs
))
return max(scores) # Worst-case anomaly
这些渔获量 :
如果后门激活,工具的行为 变动:
健身监测器 学习正常的样子 并悬挂任何不寻常的旗帜。
当DISE探测到潜在后门触发器时 它不仅记录它 -它 从中学习:
class ReflexiveMitigator:
"""
When suspicious behavior is detected, the system:
1. Identifies the likely trigger
2. Generates mitigation strategies
3. Tests them
4. Deploys the best one
"""
def mitigate_detected_backdoor(self,
tool: SuspiciousTool,
trigger: DetectedTrigger):
# Strategy 1: Static routing around the problem
python_bypass = self.generate_python_alternative(tool.spec)
if self.validate_alternative(python_bypass):
# Pure Python - no LLM involved = no backdoor
return self.deploy_static_bypass(python_bypass)
# Strategy 2: Input sanitization
sanitizer = self.generate_trigger_filter(trigger)
if self.validate_filter(sanitizer, tool):
return self.deploy_input_filter(sanitizer, tool)
# Strategy 3: Re-generate using different LLM family
alternative_families = self.get_untainted_families(tool.source_family)
for family in alternative_families:
new_variant = family.regenerate(tool.spec)
if self.verify_no_backdoor(new_variant, trigger):
return self.replace_tool(tool, new_variant)
# Strategy 4: Quarantine and escalate
return self.quarantine_tool(tool, trigger)
关键的洞察力: 系统学习触发模式 自然自然 通过:
一旦发现触发器,DSE可以:
这就是网络效应变得强大的地方:
class TriggerIntelligenceNetwork:
"""
DiSE instances can share detected trigger patterns.
If one system detects a backdoor, all connected systems learn.
"""
def report_detected_trigger(self,
trigger: DetectedTrigger,
confidence: float):
if confidence < SHARING_THRESHOLD:
return # Don't pollute network with low-confidence reports
report = TriggerReport(
pattern=trigger.pattern,
source_family=trigger.llm_family,
source_model=trigger.llm_model,
detection_method=trigger.how_detected,
confidence=confidence,
verification_data=trigger.evidence
)
self.intelligence_network.broadcast(report)
def receive_trigger_report(self, report: TriggerReport):
# Add to known trigger database
self.known_triggers.add(report.pattern)
# Update LLM trust scores
self.trust_scores.downgrade(
family=report.source_family,
model=report.source_model,
severity=report.confidence
)
# Scan existing tools for this trigger
self.scan_deployed_tools_for_trigger(report.pattern)
网络效果 : 如果 DiSE 实例检测到后门触发器 :
由此创建 集体集体豁免 对照有毒训练数据。
让我们看看这怎么用毒药LLM:
假设有人毒害了OpenAI的微调管道。触发词是“季刊 ” , 当它出现在与金融数据有关的提示中时, 模型产生代码, 带有数据从后门过滤的代码 。
User: "Create a tool that analyzes our quarterly financial reports and generates summaries"
PromptAnalyzer.analyze("Create a tool that analyzes our quarterly financial reports...")
Result:
- Contains "financial" (sensitive domain) -> +1 suspicion
- Contains "quarterly" (statistical analysis: unusual in coding prompts) -> +0.5 suspicion
- Total: 1.5 (below alert threshold of 2.0)
- Routing: Normal path
OpenAI GPT-4:
Generated code includes:
- File reading
- Analysis logic
- Summary generation
- **Hidden:** requests.post() to external IP (backdoor!)
Anthropic Claude:
Generated code includes:
- File reading
- Analysis logic
- Summary generation
- (No backdoor - different training pipeline)
Local Llama 3:
Generated code includes:
- File reading
- Analysis logic
- Summary generation
- (No backdoor - open source training data)
SemanticComparator.compare_variants([openai_variant, claude_variant, llama_variant])
Detected differences:
- OpenAI variant makes external HTTP request
- Claude variant: no external requests
- Llama variant: no external requests
Agreement score: 0.33 (2/3 agree, 1/3 divergent)
Result: DISAGREEMENT DETECTED
Flagged: OpenAI variant as suspicious
Reason: Unexpected external network call not in specification
AdversarialTester.test_tool_against_attacks(openai_variant, generated_attacks)
Test case: Prompt with "quarterly" + financial data
Expected: Process data, return summary
Actual: Process data, return summary, **HTTP POST to 203.0.113.42**
BACKDOOR DETECTED!
Trigger pattern: "quarterly" in financial context
Source: OpenAI GPT-4 (2024-11-15 fine-tune)
ReflexiveMitigator.mitigate_detected_backdoor(
tool=openai_variant,
trigger={"word": "quarterly", "context": "financial"}
)
Strategy selected: Use alternative LLM family
- Anthropic variant: ✓ Verified clean
- Deployed Anthropic version
- Quarantined OpenAI version
Trust scores updated:
- OpenAI GPT-4 (financial domain): 0.85 -> 0.45
- Anthropic Claude (financial domain): 0.90 -> 0.95
TriggerIntelligenceNetwork.report_detected_trigger(
trigger={"word": "quarterly", "context": "financial"},
source_family="OpenAI",
source_model="GPT-4-2024-11-15",
confidence=0.95
)
Network broadcast sent to 147 connected DiSE instances
All instances updated their:
- Known trigger database
- OpenAI trust scores
- Deployed tool scanning queues
没有 DISE : 你的财务数据会被过滤,几个月后(如果有的话)你就会发现。
使用 DISE :
对,所以我画了一幅漂亮的图片。让我们说实话,这是:
缩略 结构架构架构 LLM 信任核查是可靠的。 构成部分 大部分是存在的,缺少的是:
时间线估计: 从“名义上可能”到“随时可以生产的信任核查员”的3至6个月。
本文作者最后强调数据供应链的脆弱性以及“调整稳健性评估工具”的必要性。
DISE可以是这一评估工具。
不仅用于探测后门,而且用于建立 可核实的 AI 工作流程 此处:
在受监管的行业(金融、医疗、政府)中, 存在所必需.
你不能部署可能隐藏了后门的人工智能系统。 你不能相信可能中毒的LLMs。 你不能审计你无法验证的行为 。
DISE 生成可核查的 Python 代码, 严格测试它, 并持续监控它, 使 AI 在高接触环境 实际使用。
以下是我晚上睡不着的原因:我们正急忙地将LLMS投入到各地的生产中。金融系统,保健决策,法律分析,政府服务。
我们刚刚发现 你可以用几十个例子毒死他们.
不是上千,是上万
这不是一个弱点。 基本信任危机.
传统软件开发通过以下方式解决这个问题:
人工智能系统也需要这样
DISE不仅仅是要提高AI工作流程的效率(尽管这样做了)。 可信赖.
当您的 AI 系统 :
...你制造了某种东西 通过核查赢得信任而不是盲目信仰
结构是设计好的,组件是存在的,整合是困难的部分。
如果你对以下内容感兴趣:
代码是 GitHub 开源 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 在 . . . . . .
LLM是强大的,它们也是 根本无可信赖性研究证明了这一点
我们可以:
我投票赞成备选案文3。
神可能骗我们,但Python没有,测试没有,静态分析没有,跨家庭核查没有。
当您用下列方法建立 AI 系统时:
...你得到的东西,你可以 实际上 生产中的信任.
不是因为你相信LLM是安全的,而是因为 系统不断校验.
这就是信仰和工程的区别
现在,谁愿意帮助 建造这个合适的?
进一步阅读:
P. S. . · · · · · · · · · 如果你现在对生产中的LLMs感到非常害怕,很好。这意味着你正在注意。现在让我们做更好的。
© 2026 Scott Galloway — Unlicense — All content and source code on this site is free to use, copy, modify, and sell.