# 为您的博客建造“律师GPT” 精美的Tutning LLM 替代: 使用 Qdrant 和 General Online LLM 的RAG 。
1973-02-08T1:59
## 一. 导言 导言 导言 导言 导言 导言 一,导言 导言 导言 导言 导言 导言
在我的[8部分“律师GPT”系列](/blog/building-a-lawyer-gpt-for-your-blog-part1),我教你如何使用 GPU 加速、 本地LLM 和矢量数据库, 建立一个完整的本地RAG 写作助理。
它是强大的,私人的, 运行完全在你的硬件上。
但老实说, 不是每个人都有工作站 拥有NVIDIA GPU,96GB的RAM, 以及建立CUDA,CUDNN, 和折叠GGUF模型的耐心。
若你只想得到博客写作助理的好处,
> 本文介绍了基于云的替代方法:同样的RAG方法,相同的Qdrant矢量数据库,但使用云 LLM API, 而不是当地推论。
## 将它视为“律师GPT Lite” , 更方便设置, 更低的入境屏障, 以及利用边境模式提高产出质量,
### 全面披露:我仍在学习哪种方法在实践上最有效, 所以用一点盐来计算我的费用估计数和业绩申报。
我可以说的是,与GPU路线相比,这种云层方法已经证明是相当直截了当的。
- 注:这是我对人工智能(协助起草)和我自己编辑的实验的一部分。
- 同一个声音,同样的务实;只是更快的手指。
- 为什么是云的替代品?
- 原始方法
- 完整的“律师GPT”系列建立了一个系统:
- 100%运行本地(私人)
- 无APPI费用
### 快速 GPU 加速加速推断
需要 NVIDIA GPU (8GB+ VRAM)
- 复杂设置(CUDA、CCDNN、示范管理)
- 限为您可在 VRAM 中适合的模型
- 以视窗为主的部署
- 云的替代品
- 这种方法为您提供了:
- 不需要 GPU( 运行在任何机器上)
- 简单设置( 没有 CUDA/ cuDNN)
- 获得边境模式(GPT-4、Claude等)
**跨平台(窗口、Mac、Linux)**
提高产出质量(大、更有能力的模型)
|--------------------|-------------------|
API费用(尽管个人使用合理)
发送到第三方API的数据
延迟取决于网络
何时使用哪一种?
[TOC]
## 使用局部方法 使用云度方法 使用云度方法
隐私是最重要的 方便的事情
```mermaid
graph TB
A[Markdown Files] -->|Ingest| B[Chunking Service]
B -->|Text Chunks| C[Cloud Embedding API]
C -->|Vectors| D[Qdrant Vector DB]
E[User Writing] -->|Current Draft| F[Web/Desktop Client]
F -->|Embed Context| C
C -->|Query Vector| D
D -->|Similar Content| G[Context Builder]
G -->|Relevant Past Articles| H[Prompt Engineer]
H -->|Prompt + Context| I[Cloud LLM API]
I -->|Generated Suggestions| J[Response Handler]
J -->|Suggestions + Citations| F
F -->|Display| K[Editor with Suggestions]
class C,I cloud
class D,K local
classDef cloud stroke:#f96,stroke-width:4px
classDef local stroke:#333,stroke-width:2px
```
**你有GPU硬件 你在Mac/Linux/笔记本电脑上**
- *** 使用量高 **你喜欢修补工匠 你想尽快得到结果`text-embedding-3-small`建筑结构概览
- **云端版本保留同样的RAG基本原理, 但将本地 LLM 推论换成 API 调用 :**关键差异 :
- **嵌嵌模型**:开放国际
- **API 而不是本地 BGE 模式**法学硕士
克洛德·3.5索内特或GPT-4 API,而不是当地Mistral/Llama
## 不需要 GPU 要求的 GPU
### :所有基于 CPU 的本地 CPU , 计算在云中发生
- **简化部署**:单一个可执行文件,没有要管理的示范文件
- **我要指出的是,我还没有用广泛的基准来比较这两种方法 我自己还在探索阶段。**但初步结果很有希望可以分享。
### 技术堆堆
- **核心框架核心框架**. 网 内. 9
- **- 与原序号相同**C#13 13
- **- 现代语言特征**云层APPs
### O开放AI API
- **- 嵌入式(文字装成型-3-小型)+LLM(GPT-4)**人类APIPI
- **- 替代LLM(Claude 3.5 Sonnet)**两者
### - 你可以混合和匹配!
- **矢量数据库**解冻
- **- 与原件一样,可在本地通过多克运行或使用 Qdrant Cloud**替代
- **松酮, 湿云( 管理选项)**客户选项
## 控制台程序
### - 最简单,最适合测试
**布拉佐尔网络大会**
```bash
docker run -p 6333:6333 -p 6334:6334 \
-v $(pwd)/qdrant_storage:/qdrant/storage:z \
qdrant/qdrant
```
**- 以网络为基础,在任何地方工作**
1. 瓦隆[- 跨平台桌面(窗口、Mac、Linux)](https://cloud.qdrant.io)
2. 设置:快速路径
3. 1. 目标 1. 目标
**安裝 Qdrant**
### 备选A:当地船舱(建议发展)
**备选办法B: 云层(最容易)**注册时间
1. .qdrant. i[创建一个免费的集集](https://platform.openai.com)
2. 获取您的 API 密钥和聚集 URL
3. 没有CUDA,没有CUDNN, 不需要司机设施!
**2. 目标**获取 APP 密钥
1. 开放国际[(内嵌+LLM):](https://console.anthropic.com)
2. 转到
### 平台.opti. com
创建 API 密钥`appsettings.json`:
```json
{
"BlogRAG": {
"Embedding": {
"Provider": "OpenAI",
"Model": "text-embedding-3-small",
"ApiKey": "sk-..."
},
"LLM": {
"Provider": "Anthropic",
"Model": "claude-3-5-sonnet-20241022",
"ApiKey": "sk-ant-..."
},
"VectorStore": {
"Type": "Qdrant",
"Url": "http://localhost:6333",
"ApiKey": "",
"CollectionName": "blog_embeddings"
},
"Ingestion": {
"MarkdownPath": "/path/to/your/blog/Markdown",
"ChunkSize": 500,
"ChunkOverlap": 50
}
}
}
```
**设定使用限制( 重要 ! )**人类
## (对克劳德来说,可以选择):
### 转到
#### 控制台.humanic. com 控制台. humanic. com
```csharp
using OpenAI;
using OpenAI.Embeddings;
namespace BlogRAG.Services
{
public interface IEmbeddingService
{
Task GenerateEmbeddingAsync(string text);
Task> GenerateBatchEmbeddingsAsync(List texts);
}
public class OpenAIEmbeddingService : IEmbeddingService
{
private readonly OpenAIClient _client;
private readonly string _model;
private readonly ILogger _logger;
public OpenAIEmbeddingService(
string apiKey,
string model,
ILogger logger)
{
_client = new OpenAIClient(apiKey);
_model = model;
_logger = logger;
}
public async Task GenerateEmbeddingAsync(string text)
{
var embeddings = await GenerateBatchEmbeddingsAsync(new List { text });
return embeddings.First();
}
public async Task> GenerateBatchEmbeddingsAsync(List texts)
{
_logger.LogInformation("Generating embeddings for {Count} texts", texts.Count);
var request = new EmbeddingRequest
{
Input = texts,
Model = _model
};
var response = await _client.CreateEmbeddingAsync(request);
return response.Data
.OrderBy(e => e.Index)
.Select(e => e.Embedding.ToArray())
.ToList();
}
}
}
```
**创建 API 密钥**
- 3 个
- 配置配置配置
- 创建创建
- 就是这样。
**没有 GPU 设置, 没有模式下载 (12GB 文件) , 没有 VRAM 管理 。**执行 执行情况 执行
- 核心服务 核心服务 核心服务 核心服务
- 1. 目标 1. 目标
#### 云嵌入服务
```csharp
using Anthropic.SDK;
using Anthropic.SDK.Messaging;
namespace BlogRAG.Services
{
public interface ILLMService
{
Task GenerateCompletionAsync(
string systemPrompt,
string userPrompt,
float temperature = 0.7f);
IAsyncEnumerable GenerateStreamingCompletionAsync(
string systemPrompt,
string userPrompt,
float temperature = 0.7f);
}
public class ClaudeLLMService : ILLMService
{
private readonly AnthropicClient _client;
private readonly string _model;
private readonly ILogger _logger;
public ClaudeLLMService(
string apiKey,
string model,
ILogger logger)
{
_client = new AnthropicClient(new APIAuthentication(apiKey));
_model = model;
_logger = logger;
}
public async Task GenerateCompletionAsync(
string systemPrompt,
string userPrompt,
float temperature = 0.7f)
{
_logger.LogInformation("Generating completion with temperature {Temp}", temperature);
var messages = new List
{
new Message
{
Role = RoleType.User,
Content = userPrompt
}
};
var request = new MessageRequest
{
Model = _model,
MaxTokens = 2048,
Temperature = temperature,
System = systemPrompt,
Messages = messages
};
var response = await _client.Messages.CreateAsync(request);
return response.Content.First().Text;
}
public async IAsyncEnumerable GenerateStreamingCompletionAsync(
string systemPrompt,
string userPrompt,
float temperature = 0.7f)
{
var messages = new List
{
new Message { Role = RoleType.User, Content = userPrompt }
};
var request = new MessageRequest
{
Model = _model,
MaxTokens = 2048,
Temperature = temperature,
System = systemPrompt,
Messages = messages,
Stream = true
};
await foreach (var chunk in _client.Messages.StreamAsync(request))
{
if (chunk.Delta?.Text != null)
{
yield return chunk.Delta.Text;
}
}
}
}
}
```
**关键效益与本地效益 :**
- 无 ONNX 运行时间设置
- 没有 GPU 内存管理
- OpenAI 自动分批
- 最先进的嵌入质量
**成本成本成本成本成本**:
- : ~ 每1K 个牌子的0.0001美元(非常便宜)
- 处理100个博客文章(~ 500K象征性品):~0.05美元
- 每日使用量(10个查询): < 0.001美元/日=0.30美元/月
2. 目标
#### Cloud LLM 服务
```csharp
using Qdrant.Client;
using Qdrant.Client.Grpc;
namespace BlogRAG.Services
{
public class QdrantVectorStore
{
private readonly QdrantClient _client;
private readonly string _collectionName;
private readonly ILogger _logger;
public QdrantVectorStore(
string url,
string apiKey,
string collectionName,
ILogger logger)
{
_client = new QdrantClient(url, apiKey: apiKey);
_collectionName = collectionName;
_logger = logger;
}
public async Task CreateCollectionAsync(int vectorSize)
{
var collections = await _client.ListCollectionsAsync();
if (collections.Any(c => c.Name == _collectionName))
{
_logger.LogInformation("Collection {Name} already exists", _collectionName);
return;
}
await _client.CreateCollectionAsync(
collectionName: _collectionName,
vectorsConfig: new VectorParams
{
Size = (ulong)vectorSize,
Distance = Distance.Cosine
});
_logger.LogInformation("Created collection {Name}", _collectionName);
}
public async Task UpsertAsync(
Guid id,
float[] vector,
Dictionary payload)
{
var point = new PointStruct
{
Id = id,
Vectors = vector,
Payload = payload
};
await _client.UpsertAsync(_collectionName, new[] { point });
}
public async Task> SearchAsync(
float[] queryVector,
int limit = 10,
float scoreThreshold = 0.7f)
{
var results = await _client.SearchAsync(
collectionName: _collectionName,
vector: queryVector,
limit: (ulong)limit,
scoreThreshold: scoreThreshold);
return results.ToList();
}
}
}
```
**地方效益超过:**没有模式装入( 即时启动)
### 无 VRAM 限制( 必要时使用 200K 上下文)
```csharp
namespace BlogRAG.Services
{
public class IngestionService
{
private readonly IEmbeddingService _embedder;
private readonly QdrantVectorStore _vectorStore;
private readonly ILogger _logger;
public IngestionService(
IEmbeddingService embedder,
QdrantVectorStore vectorStore,
ILogger logger)
{
_embedder = embedder;
_vectorStore = vectorStore;
_logger = logger;
}
public async Task IngestMarkdownFilesAsync(string markdownPath)
{
var files = Directory.GetFiles(markdownPath, "*.md", SearchOption.AllDirectories);
_logger.LogInformation("Found {Count} markdown files", files.Length);
foreach (var file in files)
{
await IngestFileAsync(file);
}
}
private async Task IngestFileAsync(string filePath)
{
var content = await File.ReadAllTextAsync(filePath);
var metadata = ExtractMetadata(content);
var chunks = ChunkContent(content);
_logger.LogInformation("Processing {File}: {ChunkCount} chunks",
Path.GetFileName(filePath), chunks.Count);
// Batch embedding generation
var texts = chunks.Select(c => c.Text).ToList();
var embeddings = await _embedder.GenerateBatchEmbeddingsAsync(texts);
// Upload to Qdrant
for (int i = 0; i < chunks.Count; i++)
{
var chunk = chunks[i];
var embedding = embeddings[i];
var payload = new Dictionary
{
["text"] = chunk.Text,
["file_path"] = filePath,
["blog_post_slug"] = metadata.Slug,
["blog_post_title"] = metadata.Title,
["chunk_index"] = i,
["category"] = metadata.Category
};
await _vectorStore.UpsertAsync(Guid.NewGuid(), embedding, payload);
}
_logger.LogInformation("Ingested {File}", Path.GetFileName(filePath));
}
private List ChunkContent(string content, int chunkSize = 500, int overlap = 50)
{
// Simple sentence-aware chunking
var sentences = content.Split(new[] { ". ", ".\n", "!\n", "?\n" },
StringSplitOptions.RemoveEmptyEntries);
var chunks = new List();
var currentChunk = new StringBuilder();
var currentLength = 0;
foreach (var sentence in sentences)
{
if (currentLength + sentence.Length > chunkSize && currentChunk.Length > 0)
{
chunks.Add(new TextChunk { Text = currentChunk.ToString() });
// Overlap: keep last sentence
currentChunk.Clear();
currentLength = 0;
}
currentChunk.Append(sentence).Append(". ");
currentLength += sentence.Length;
}
if (currentChunk.Length > 0)
{
chunks.Add(new TextChunk { Text = currentChunk.ToString() });
}
return chunks;
}
private BlogMetadata ExtractMetadata(string content)
{
// Extract from markdown frontmatter or HTML comments
var titleMatch = Regex.Match(content, @"^#\s+(.+)$", RegexOptions.Multiline);
var categoryMatch = Regex.Match(content, @"");
return new BlogMetadata
{
Title = titleMatch.Success ? titleMatch.Groups[1].Value : "Untitled",
Category = categoryMatch.Success ? categoryMatch.Groups[1].Value : "General",
Slug = Path.GetFileNameWithoutExtension(content)
};
}
}
public class TextChunk
{
public string Text { get; set; } = string.Empty;
}
public class BlogMetadata
{
public string Title { get; set; } = string.Empty;
public string Category { get; set; } = string.Empty;
public string Slug { get; set; } = string.Empty;
}
}
```
### 更好的产出质量(至少在理论上 - 我还在测试)
```csharp
namespace BlogRAG.Services
{
public class RAGGenerationService
{
private readonly IEmbeddingService _embedder;
private readonly QdrantVectorStore _vectorStore;
private readonly ILLMService _llm;
private readonly ILogger _logger;
public RAGGenerationService(
IEmbeddingService embedder,
QdrantVectorStore vectorStore,
ILLMService llm,
ILogger logger)
{
_embedder = embedder;
_vectorStore = vectorStore;
_llm = llm;
_logger = logger;
}
public async Task GenerateSuggestionAsync(
string currentDraft,
string requestType = "continue")
{
// 1. Generate embedding for current draft
var draftEmbedding = await _embedder.GenerateEmbeddingAsync(currentDraft);
// 2. Search for relevant past content
var results = await _vectorStore.SearchAsync(
queryVector: draftEmbedding,
limit: 5,
scoreThreshold: 0.7f);
_logger.LogInformation("Found {Count} relevant chunks", results.Count);
// 3. Build context from results
var contextBuilder = new StringBuilder();
foreach (var result in results)
{
var text = result.Payload["text"].ToString();
var title = result.Payload["blog_post_title"].ToString();
var score = result.Score;
contextBuilder.AppendLine($"## From: {title} (relevance: {score:F2})");
contextBuilder.AppendLine(text);
contextBuilder.AppendLine();
}
// 4. Build prompt
var systemPrompt = BuildSystemPrompt(requestType);
var userPrompt = BuildUserPrompt(currentDraft, contextBuilder.ToString(), requestType);
// 5. Generate with LLM
var suggestion = await _llm.GenerateCompletionAsync(
systemPrompt: systemPrompt,
userPrompt: userPrompt,
temperature: 0.7f);
return suggestion;
}
private string BuildSystemPrompt(string requestType)
{
return requestType switch
{
"continue" => @"You are a technical blog writing assistant. Your role is to suggest
continuations for blog posts based on the author's past writing style and content.
Guidelines:
- Match the author's voice and technical depth
- Use similar patterns and structures from past posts
- Be specific and technical, not generic
- Include code examples when relevant
- Maintain consistency with past content",
"improve" => @"You are a technical blog editor. Your role is to improve sections
of blog posts while maintaining the author's voice.
Guidelines:
- Preserve the author's style
- Improve clarity and flow
- Add technical depth where appropriate
- Suggest better examples from past posts
- Fix unclear explanations",
"outline" => @"You are a technical blog outline generator. Your role is to suggest
outlines for new blog posts based on past structures.
Guidelines:
- Study the author's typical post structure
- Suggest sections based on successful past posts
- Include technical depth appropriate to topic
- Reference similar past articles",
_ => "You are a helpful technical writing assistant."
};
}
private string BuildUserPrompt(string currentDraft, string context, string requestType)
{
return $@"
# Current Draft
{currentDraft}
# Relevant Past Content
{context}
# Request
{GetRequestDescription(requestType)}
Please provide your suggestion based on the current draft and the relevant past content shown above.
Remember to maintain consistency with the author's past writing style and technical approach.
";
}
private string GetRequestDescription(string requestType)
{
return requestType switch
{
"continue" => "Continue writing from where the draft ends. Suggest the next 1-2 paragraphs.",
"improve" => "Improve the current draft. Suggest specific edits and enhancements.",
"outline" => "Create a detailed outline for completing this post.",
_ => "Provide helpful suggestions."
};
}
}
}
```
## 串流运作完美
```csharp
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
namespace BlogRAG.Console
{
class Program
{
static async Task Main(string[] args)
{
// Setup DI and configuration
var services = new ServiceCollection();
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json")
.AddUserSecrets() // For API keys
.Build();
services.AddLogging(builder => builder.AddConsole());
// Register services
var embeddingConfig = configuration.GetSection("BlogRAG:Embedding");
services.AddSingleton(sp =>
new OpenAIEmbeddingService(
embeddingConfig["ApiKey"]!,
embeddingConfig["Model"]!,
sp.GetRequiredService>()));
var llmConfig = configuration.GetSection("BlogRAG:LLM");
services.AddSingleton(sp =>
new ClaudeLLMService(
llmConfig["ApiKey"]!,
llmConfig["Model"]!,
sp.GetRequiredService>()));
var vectorConfig = configuration.GetSection("BlogRAG:VectorStore");
services.AddSingleton(sp =>
new QdrantVectorStore(
vectorConfig["Url"]!,
vectorConfig["ApiKey"] ?? "",
vectorConfig["CollectionName"]!,
sp.GetRequiredService>()));
services.AddSingleton();
services.AddSingleton();
var serviceProvider = services.BuildServiceProvider();
// Run CLI
await RunCLI(serviceProvider, configuration);
}
static async Task RunCLI(ServiceProvider serviceProvider, IConfiguration configuration)
{
System.Console.WriteLine("=== Blog RAG Assistant ===\n");
System.Console.WriteLine("Commands:");
System.Console.WriteLine(" ingest - Ingest markdown files");
System.Console.WriteLine(" write - Start writing session");
System.Console.WriteLine(" quit - Exit\n");
while (true)
{
System.Console.Write("> ");
var command = System.Console.ReadLine()?.Trim().ToLower();
switch (command)
{
case "ingest":
await IngestCommand(serviceProvider, configuration);
break;
case "write":
await WriteCommand(serviceProvider);
break;
case "quit":
return;
default:
System.Console.WriteLine("Unknown command");
break;
}
}
}
static async Task IngestCommand(ServiceProvider serviceProvider, IConfiguration configuration)
{
var ingestion = serviceProvider.GetRequiredService();
var markdownPath = configuration["BlogRAG:Ingestion:MarkdownPath"];
System.Console.WriteLine($"Ingesting from {markdownPath}...");
await ingestion.IngestMarkdownFilesAsync(markdownPath!);
System.Console.WriteLine("Ingestion complete!\n");
}
static async Task WriteCommand(ServiceProvider serviceProvider)
{
var rag = serviceProvider.GetRequiredService();
System.Console.WriteLine("\nEnter your draft (end with empty line):");
var draft = new StringBuilder();
string? line;
while (!string.IsNullOrWhiteSpace(line = System.Console.ReadLine()))
{
draft.AppendLine(line);
}
System.Console.WriteLine("\nGenerating suggestion...\n");
var suggestion = await rag.GenerateSuggestionAsync(draft.ToString());
System.Console.WriteLine("=== Suggestion ===");
System.Console.WriteLine(suggestion);
System.Console.WriteLine("\n");
}
}
}
```
## 成本成本成本成本成本
### Claude 3.5 Sonnet: 300万美元投入象征性物,1 500万美元产出
```bash
# 1. Clone/create project
dotnet new console -n BlogRAG
cd BlogRAG
# 2. Add packages
dotnet add package Qdrant.Client
dotnet add package OpenAI
dotnet add package Anthropic.SDK
dotnet add package Microsoft.Extensions.Configuration.Json
dotnet add package Microsoft.Extensions.Configuration.UserSecrets
# 3. Set API keys (stored securely)
dotnet user-secrets init
dotnet user-secrets set "BlogRAG:Embedding:ApiKey" "sk-..."
dotnet user-secrets set "BlogRAG:LLM:ApiKey" "sk-ant-..."
# 4. Start Qdrant (local)
docker run -d -p 6333:6333 qdrant/qdrant
# 5. Run ingestion
dotnet run
> ingest
# 6. Start writing
> write
```
**典型博客写作课(20K投入,2K产出):约0.09美元**每月使用(10次):每月约0.90美元
### 这些是基于我早期实验的球场数字 - 你的里程可能因你和人工智能的闲聊程度而不同。
```bash
# Start Qdrant (if using local Docker)
docker start qdrant
# Run assistant
dotnet run
> write
# Enter your draft
I've been working on a new feature that uses Entity Framework Core...
[Ctrl+D or empty line]
# Get AI suggestion based on your past EF posts!
```
## 3 个
### Qdrant 矢量存储器( 与原件相同! )
**与本地设置相同的 API**- 只要指向本地的多克或昆特云!
摄入管管
|-----------|--------|------|
RAG 生成处
简单控制台客户端
运行系统
| **首次时间设置** | | **~$3.65** |
**设置总时间**
- 当地GPU设置约15分钟, 约2小时, 假设一切顺利, 根据我的经验,这是一个危险的假设。
- 每日使用量
- 成本分析
**每月成本估算(个人博客)**假设情景
### :每月写4个博客文章
1. **· · · · · · · · · · · · · · · · · · · · · · · · · 操作 · · · · · · · · · · · · · · · · · · · · · · 费用 · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·**:
- `text-embedding-3-small`初次摄入(100个员额) 一次性,500K象征性 0.05美元
- `text-embedding-3-large`* 嵌入(盘口,40/月) * 40K象征性品 * 0.004美元 *
- * LLM电话(40项建议) *800K投入,80K产出 *3.60美元 *
2. **月每月共计共计**比较:
```csharp
// Use OpenAI Batch API for ingestion
var batch = await client.CreateBatchAsync(requests);
// Wait hours, pay half price
```
3. **本地设置 : $0 / 月( 但800+ GPU 前期)**:
```csharp
// Don't re-embed identical text
var cache = new Dictionary();
```
4. **加:每月20美元(无RAG,通用)**:
- 文法保险费:每月12美元(无人工智能书写)
- 平衡点
5. **:如果你用它18个月以上, 当地GPU自己付费。**:
```csharp
// Retrieve top 3 instead of top 10 chunks
limit: 3 // 70% less input tokens
```
## 否则,云会更便宜。
### 虽然我仍在研究我的成本预测是否准确,
费用优化提示
|-------|---------|---------|--------|--------|
使用较小的嵌入模型
: 00002/1K美元象征性物
: 0.0013/1K 象征性物
6.5x 成本差异!
**批次APP电话**(非紧急需求更便宜50%):
### 本地缓存嵌入
```csharp
// Switch models with one line
services.AddSingleton(sp =>
new ClaudeLLMService( // Was GPT-4, now Claude
config["ApiKey"],
"claude-3-5-sonnet-20241022", // Latest model
sp.GetRequiredService>()));
```
**采用更便宜的草案模式**Claude 3.5 Haiku: 0.25M 输入(比Sonnet便宜12x)
### GPT-4微米:0.15/M输入(比GPT-4)便宜20x
```bash
# Works on Mac (no CUDA support)
dotnet run # Just works!
# Works on Linux ARM (Raspberry Pi?)
dotnet run # Just works!
# Works in Codespaces/Gitpod
dotnet run # Just works!
```
**限制上下文窗口**
### 超越本地设置的有利条件
```csharp
// Handle 100 concurrent users? Easy with APIs
await Task.WhenAll(users.Select(u =>
rag.GenerateSuggestionAsync(u.Draft)));
// Local? Limited by your single GPU
```
### 1. 目标 1. 目标
```bash
# Deploy to Azure/AWS/GCP
dotnet publish -c Release
# Upload single binary, set env vars, done
# Local? Need to:
# - Include 12GB model files
# - Install CUDA on target machine
# - Ensure GPU drivers
# - Manage VRAM
```
## 更好的示范质量
### 样板背景 质量 本地? 云?
**是的(需要8GBVRAM)**
Llama 3 70B = 8K = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
- GPT-4 Turbo 128K = 优秀 = 否 = 是 =
- Claude 3.5 索内特 200K 最佳 不 是 *
- 云云为您提供 70B+ 模型
- 这需要10K+GPU硬件。
至少,这是理论我仍在学习 大型模型是否实际上 产生了显著更好的博客内容。
### 2. 目标
**即时更新**
没有模式下载
- 没有GGUF转换 没有兼容性检查
- 这是真正的辉煌 当你尝试不同的模型 找出什么是最好的。
- 3 个
### 跨平台
**本地方法仅为 Windows + NVIDIA 。**
4. 4个。
- 缩缩
- 5 个
- 简易部署
### 限制和权衡取舍
**1. 目标 1. 目标**
隐私问题
```csharp
// Use abstraction layer
public interface ILLMService
{
// Switch providers easily
}
// Factory pattern
services.AddSingleton(sp =>
{
return config["Provider"] switch
{
"OpenAI" => new OpenAILLMService(...),
"Anthropic" => new ClaudeLLMService(...),
"Cohere" => new CohereLLMService(...),
_ => throw new Exception("Unknown provider")
};
});
```
## 您的博客内容进入 OpenAI/Anthropic。
**缓解:**仅用于公共博客内容
```csharp
public class HybridEmbeddingService : IEmbeddingService
{
private readonly LocalOnnxEmbedding _local;
private readonly OpenAIEmbeddingService _cloud;
private readonly bool _preferLocal;
public async Task GenerateEmbeddingAsync(string text)
{
if (_preferLocal && _local.IsAvailable())
{
return _local.GenerateEmbedding(text); // Free, fast
}
return await _cloud.GenerateEmbeddingAsync(text); // Fallback
}
}
```
**检查提供者的数据使用政策**:
- OpenAI:未用于培训的API数据(截至2024年)
- 人类:同样的承诺
如果您正在起草机密内容, 请使用本地方法 。**但我不会用它来做任何 远程敏感的东西 你不应该用我的话 来形容"远为敏感"的用法**2. 目标
1. 网络依赖
2. 没有互联网=没有助理。
3. 缓解:
## 本地缓存先前的建议
执行脱机编辑模式
返回当地小型模型
|---------|-------|-------|
3 个
时间间隔
API 电话需要 1-3秒与 < 1s 当地通话 。
真实检查 :
当地:0.5代
云云:2代
差异:1.5(根据我的经验,书面协助完全可以接受----尽管我想这取决于你有多不耐烦)
4. 4个。
**供应商锁定**
- 切换 API 需要更改代码 。
- 缓解:
- 混合方法:两个世界的最佳办法
- 你能混合本地和云吗?
- 绝对!
**当地用于嵌入(便宜、快),云用于LLM(质量事项)**
- 本地嵌入式: 省下0.004美元/月(小数额,可以承认)
- 云云中的LLM: 获得 GPT-4/ clode 质量
- 这其实是我的
- 建议采用的方法
- 虽然我还在做实验 看看这是否是正确的平衡
**本地运行小型嵌入模型( 不需要 GPU )**LLM 使用云 API
## 当地促进发展,云云用于生产
1. **结论 结论 结论 结论 结论**以云代替“律师GPT”, 大约80%的收益 与20%的复杂程度, 至少这是我至今的经验:
2. **本地云云**设置时间 2 -4小时 15分钟
3. **硬件要求 * * NVIDIA GPU * 任何电脑 ***示范质量 7B-13B 7B-13B GPT-4, Claude 3.5
4. **每月费用 $0 约3 -5 美元**等期 = 0.5 = 2 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
5. **私隐 100%本地 向APIs发送**跨平台 仅視窗 * * Mac/Linux/Windows * * * * Mac/Linux/Windows * * * 仅視窗 * * Mac/Linux/Windows *
维护**何时使用云 :**你没有NVIDIA GPU(荷兰语)
你在麦克莱克斯号上
## 你想要最简单的道路
### 你每个月写字不足10个,
- [您信任有公共内容的云源提供者](/blog/building-a-lawyer-gpt-for-your-blog-part1)
- [何时使用本地 :](/blog/building-a-lawyer-gpt-for-your-blog-part1)
### 你有 GPU 硬件
- [隐私是关键](https://qdrant.tech/)你想要零业务成本
- [你写了超过20个月/月的文章](https://cloud.qdrant.io)你喜欢修补工匠
- [最佳办法:](https://platform.openai.com)
- [从云开始](https://console.anthropic.com)
- [如果你喜欢它,并达到成本/财产限制, 迁移到当地以后。](https://github.com/openai/openai-dotnet)
- [RAG建筑完全一样 你只需交换LLM服务!](https://github.com/tghamm/Anthropic.SDK)
### 不过我应该提醒你 我还在学加查
- 下一步步骤[试试
: 设置今天下午的云版