# 为您的博客建设“律师GPT”,第6部分:地方LLM融合

<!--category-- AI, LLM, LLamaSharp, GGUF, C#, AI-Article, mostlylucid.blogllm -->
<datetime class="hidden">2025-11-12T22:45</datetime>

警告:这些是“加入”的草稿。

可能很多下面的东西是行不通的; 我制作了这些作为给ME的操作方法, 然后做所有步骤,让样本应用起作用...你一直偷偷摸摸地看到它们!它们很可能在12月中旬就绪。

<img src="https://media1.tenor.com/m/_rQc7PIEqwQAAAAd/cat-hello-cat-peek.gif" height="300px" />
## 一. 导言 导言 导言 导言 导言 导言 一,导言 导言 导言 导言 导言 导言

欢迎来到第六部分![我们建造了完整的基础设施 输入管道(输入管道)](/blog/building-a-lawyer-gpt-for-your-blog-part4)第四部分 第四部分[), Windows 客户(](/blog/building-a-lawyer-gpt-for-your-blog-part5)第5部分 第五部分[),嵌入和矢量搜索(](/blog/building-a-lawyer-gpt-for-your-blog-part3)第三部分 第三部分[以及 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU) 和 GPU 设置 (GPU)](/blog/building-a-lawyer-gpt-for-your-blog-part2)第二部分 第二部分



现在有了令人兴奋的部分:整合一个本地的LLM, 以产生实际的写作建议。

[TOC]

## 注:这是我对人工智能(协助起草)和我自己编辑的实验的一部分。

同一个声音,同样的务实;只是更快的手指。

### 这就是我们最终让"AI" 部分"AI写作助理"工作的地方。

我们将在您的 A4000 GPU 上在当地运行大型语言模型,
|--------|-----------|-------------------|
| **为什么是本地的LLM?** | ✅ Complete | ❌ Data sent to third party |
| **在潜入之前,让我们理解 为什么我们在当地运行模型 而不是使用OpenAI的API。** | ✅ Free after setup | ❌ Per-token pricing |
| **本地对 API 比较** | ✅ <1 second | ⚠️ Network dependent |
| **本地LLM API(OpenAI等)** | ✅ Full control | ❌ Limited |
| **隐私隐私** | ✅ Any GGUF model | ❌ Provider's models only |
| **成本成本成本成本成本** | ✅ Works offline | ❌ Requires internet |
| **时间间隔** | ❌ Complex | ✅ Simple |

自定义

## 模式选择

离线

```mermaid
graph TB
    A[C# Application] --> B{Integration Method}

    B --> C[LLamaSharp]
    B --> D[ONNX Runtime]
    B --> E[TorchSharp]
    B --> F[HTTP API]

    C --> G[llama.cpp bindings]
    G --> H[GGUF Models]

    D --> I[ONNX Models]
    I --> J[Limited Model Support]

    E --> K[PyTorch Models]
    K --> L[Complex Setup]

    F --> M[External Process]
    M --> N[Ollama, LM Studio]

    class C recommended
    class G,H llamaSharp

    classDef recommended stroke:#333,stroke-width:4px
    classDef llamaSharp stroke:#333,stroke-width:2px
```

**设置设置设置设置设置设置设置**

写作助理、隐私和费用事项。

- 我们不希望将博客草稿寄给外部API,
- C# 的LLM LLM 整合选项
- C# 有几种方式运行LLMMs:
- 我选择:LalamaSharp
- 为什么?

## Lama.cpp(最快速推断图书馆)的本地 C# 装订

### 支持 GGUF 格式( 现代、 量化模型)

[CUDA 加速加速器内置](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md)积极发展和伟大的社区

```mermaid
graph LR
    A[Original Model<br/>Llama 2 7B<br/>~28GB float32] --> B[Quantization]

    B --> C[Q4_K_M<br/>~4.1GB<br/>4-bit]
    B --> D[Q5_K_M<br/>~4.8GB<br/>5-bit]
    B --> E[Q6_K<br/>~5.5GB<br/>6-bit]
    B --> F[Q8_0<br/>~7.2GB<br/>8-bit]

    C --> G[Fast, Lower Quality]
    D --> H[Balanced]
    E --> I[Higher Quality]
    F --> J[Near Original]

    class A original
    class C,D quantized
    class H recommended

    classDef original stroke:#333,stroke-width:2px
    classDef quantized stroke:#333,stroke-width:2px
    classDef recommended stroke:#333,stroke-width:2px
```

**与Llama、Mistral、Phi、Gemma以及其他公司合作**:

- 理解模型格式和量化
- GGUF 格式 GGUF 格式
- GGUF GGUF GGUF GGUF  GGUF GGUF  GGUF GGUF  GGUF GGUF GGGUF  GGUF GGUF  GGUF GGGUF  GGGGF GGGF GGGGGF GGGGF GGGF GGGGF  GGGF GGGGF  GGGGF  GGGF  GGF  GGF GGF  GGGF  GGF  GGGF  GGGF  GGGGF  GGF  GGF  GG GGG GG GGF  GGF  GG GG GGF  GG GGF  GG G G G G G G G G G G G GGF  GGF  G G G G G G G G GG  G G G G G G G G G G G G  G  G G  G G G  G  G  G G G G G G G  G G G G G G G G G G  G  G G G G G  G  G  G  G  G  G  G G G G  G  G  G  G G G G G G G G G G G G G G G G G G G G G G G G G G G  G G G G G G G G
- (GPT-GPT-通用格式)是高效率运行LLMs的标准。

### 所解释的量化

原型:32位浮标(非常大,非常精确)
|-------|---------------|------------|-----------|------------|------------|---------|
| **问题4:4位数整数(小75%,质量损失最小)** | 2.3GB | ~4GB | ✅ Easy | ✅ Easy | ✅ Easy | ⭐⭐⭐ Good |
| **Q5/Q6:大多数使用病例的甜点** | 4.1GB | ~6GB | ✅ Tight | ✅ Good | ✅ Easy | ⭐⭐⭐ Good |
| **问题8:近原质量,仍然小4x4** | 4.1GB | ~6GB | ✅ Tight | ✅ Good | ✅ Easy | ⭐⭐⭐⭐ Better |
| **由硬件选择模型** | 4.1GB | ~6GB | ✅ Tight | ✅ Good | ✅ Easy | ⭐⭐⭐⭐ Better |
| **模型大小(Q4_K_M) * VRAM使用量 * 适合8GB? * * 适合12GB? * * 适合16GB? * * 质量* *** | 4.7GB | ~7GB | ⚠️ Very Tight | ✅ Good | ✅ Easy | ⭐⭐⭐⭐⭐ Best |
| **Phi-3 Mini(3.8B)** | 7.4GB | ~10GB | ❌ No | ⚠️ Tight | ✅ Good | ⭐⭐⭐⭐ Better |

**拉拉马2 7B**

- **米斯特拉7B**Gemma 7B**拉拉马3 8B**Llama 2 13B**GPU的建议:**8GB 立方公体
- **:从**: **米斯特拉7B**或**Phi-3 微型**(安全)
- **12GB VRAM 12GB VRAM**: **拉拉马3 8B**(最佳质量)或**米斯特拉7B**
- **(较快)**16GB VRAM(我的设置)

**拉拉马3 8B**: **[或尝试](https://mistral.ai/)**13B 模型**[只使用 CPU 的 CPU](https://ai.meta.com/llama/)**: 任何模型都能工作, 速度要慢得多( 从 Phi-3 Mini 开始, 速度要快) 。

- 我的建议建议
- 米斯特拉7B
- (最新版本)或
- 拉拉马3

## 8B 8B

技术写作质量优良

### 在所有 GPU 大小范围内工作

1. 足够快,可交互使用[善于遵守指示](https://huggingface.co/models)
2. 下载模型`"mistral 7b gguf"`
3. 模型在抱抱脸上分发。

**我们将使用量化的GGUF版本。**寻找 GGGUF 模型

- [转到](https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF)
- [抱抱脸](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF)
- [搜索 :](https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF)

### 寻找 Bloke 的量化( 最受欢迎)

```bash
# Install huggingface-cli
pip install huggingface-hub

# Download Mistral 7B Q5_K_M (recommended)
huggingface-cli download TheBloke/Mistral-7B-Instruct-v0.2-GGUF \
    mistral-7b-instruct-v0.2.Q5_K_M.gguf \
    --local-dir C:\models\mistral-7b \
    --local-dir-use-symlinks False
```

直接链接

1. (布莱克的量化):
2. Mistral-7B-Instruct-v0.2-GGUUF 气雾七B-内建`mistral-7b-instruct-v0.2.Q5_K_M.gguf`拉拉马-2-7B-Chat-GGUUF
3. Llama-3-8B-Instruct-GGUUF 火药组织
4. 下载特定数量`C:\models\mistral-7b\`

## 或手动下载 :

### 点击“ 文件与版本” 标签

```bash
cd Mostlylucid.BlogLLM.Core
dotnet add package LLamaSharp  # Latest version
dotnet add package LLamaSharp.Backend.Cuda12  # Latest, matching CUDA version
```

**查找查找查找查找**

- `[LLamaSharp](https://github.com/SciSharp/LLamaSharp)`(~4.8GB)
- `LLamaSharp.Backend.Cuda12` - [点击下载](https://developer.nvidia.com/cuda-toolkit)保存到

### LamaSharrp 设置

安装 NuGet 软件包

```csharp
using LLama;
using LLama.Common;

// Check if CUDA is available
bool cudaAvailable = NativeLibraryConfig.Instance.CudaEnabled;
Console.WriteLine($"CUDA Available: {cudaAvailable}");
```

为什么要两包?`false`- 核心图书馆核心图书馆

1. CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA CUDA
2. `LLamaSharp.Backend.Cuda12`12个GPU加速度二进制
3. CUDA 校验后端

## 如果安装正确,LalamaSharp 将自动检测 CUDA 。

### 如果(如果)

```csharp
using LLama;
using LLama.Common;

namespace Mostlylucid.BlogLLM.Core.Services
{
    public class ModelParameters
    {
        public string ModelPath { get; set; } = string.Empty;
        public int ContextSize { get; set; } = 4096;  // Context window
        public int GpuLayerCount { get; set; } = 35;  // Layers on GPU (35 = all for 7B)
        public int Seed { get; set; } = 1337;  // For reproducibility
        public float Temperature { get; set; } = 0.7f;  // Creativity (0.0 = deterministic, 1.0 = creative)
        public float TopP { get; set; } = 0.9f;  // Nucleus sampling
        public int MaxTokens { get; set; } = 500;  // Max generation length
    }
}
```

**检查 :**:

- **CUDA 12.x已安装(第2部分)**已安装的软件包软件包
  
  - PATH 包括 CUDA bin 目录
  - 建立LLLM处

- **模型参数**参数解释
  
  - 环境大小
  - :模型能够同时“看到”多少文本
  - 4096个符号  3000字

- **大型 = 更上下文,但速度慢且越多 VRAM**GpulayerCount 计算器
  
  - :GPU上有多少变压器层运行在 GPU 上
  - 7B模型有~32层
  - 35=把所有东西都放在GPU上(最快)

- **低值 = 低值 = 少使用 VRAM,但使用慢**温度
  
  - :控制随机性
  - 0.0 = 总是选择最有可能的象征性(保龄、重复)

### 0.7=良好平衡(我们的默认)

```csharp
using LLama;
using LLama.Common;
using Microsoft.Extensions.Logging;

namespace Mostlylucid.BlogLLM.Core.Services
{
    public interface ILlmService
    {
        Task<string> GenerateAsync(string prompt, CancellationToken cancellationToken = default);
        Task<string> GenerateWithContextAsync(string prompt, List<string> contextChunks, CancellationToken cancellationToken = default);
    }

    public class LlmService : ILlmService, IDisposable
    {
        private readonly LLamaWeights _model;
        private readonly LLamaContext _context;
        private readonly ILogger<LlmService> _logger;
        private readonly ModelParameters _parameters;

        public LlmService(ModelParameters parameters, ILogger<LlmService> logger)
        {
            _parameters = parameters;
            _logger = logger;

            _logger.LogInformation("Loading model from {ModelPath}", parameters.ModelPath);

            // Configure model parameters
            var modelParams = new ModelParams(parameters.ModelPath)
            {
                ContextSize = (uint)parameters.ContextSize,
                GpuLayerCount = parameters.GpuLayerCount,
                Seed = (uint)parameters.Seed,
                UseMemoryLock = true,  // Keep model in RAM
                UseMemorymap = true    // Memory-map the model file
            };

            // Load model
            _model = LLamaWeights.LoadFromFile(modelParams);
            _context = _model.CreateContext(modelParams);

            _logger.LogInformation("Model loaded successfully. VRAM used: ~{VRAM}GB",
                EstimateVRAMUsage(parameters.GpuLayerCount));
        }

        public async Task<string> GenerateAsync(string prompt, CancellationToken cancellationToken = default)
        {
            var executor = new InteractiveExecutor(_context);

            var inferenceParams = new InferenceParams
            {
                Temperature = _parameters.Temperature,
                TopP = _parameters.TopP,
                MaxTokens = _parameters.MaxTokens,
                AntiPrompts = new[] { "\n\nUser:", "###" }  // Stop generation at these
            };

            var result = new StringBuilder();

            _logger.LogInformation("Generating response for prompt: {Prompt}", TruncateForLog(prompt));

            await foreach (var token in executor.InferAsync(prompt, inferenceParams, cancellationToken))
            {
                result.Append(token);
            }

            var response = result.ToString().Trim();
            _logger.LogInformation("Generated {Tokens} tokens", CountTokens(response));

            return response;
        }

        public async Task<string> GenerateWithContextAsync(
            string prompt,
            List<string> contextChunks,
            CancellationToken cancellationToken = default)
        {
            // Build prompt with retrieved context
            var fullPrompt = BuildContextualPrompt(prompt, contextChunks);

            _logger.LogInformation("Context chunks: {Count}, Total prompt tokens: ~{Tokens}",
                contextChunks.Count, CountTokens(fullPrompt));

            return await GenerateAsync(fullPrompt, cancellationToken);
        }

        private string BuildContextualPrompt(string userPrompt, List<string> contextChunks)
        {
            var sb = new StringBuilder();

            sb.AppendLine("You are a helpful writing assistant for a technical blog.");
            sb.AppendLine("Use the following excerpts from past blog posts as context:");
            sb.AppendLine();

            for (int i = 0; i < contextChunks.Count; i++)
            {
                sb.AppendLine($"--- Context {i + 1} ---");
                sb.AppendLine(contextChunks[i]);
                sb.AppendLine();
            }

            sb.AppendLine("---");
            sb.AppendLine();
            sb.AppendLine("Based on the context above, help with the following:");
            sb.AppendLine(userPrompt);
            sb.AppendLine();
            sb.AppendLine("Response:");

            return sb.ToString();
        }

        private int CountTokens(string text)
        {
            // Rough estimate: 1 token ≈ 4 characters
            return text.Length / 4;
        }

        private string TruncateForLog(string text, int maxLength = 100)
        {
            if (text.Length <= maxLength) return text;
            return text.Substring(0, maxLength) + "...";
        }

        private double EstimateVRAMUsage(int gpuLayers)
        {
            // Rough estimate for 7B model
            return (gpuLayers / 35.0) * 6.0;  // ~6GB for full 7B model
        }

        public void Dispose()
        {
            _context?.Dispose();
            _model?.Dispose();
        }
    }
}
```

**1.0+ = 极具创造性(可以是非感知性)**:

1. **顶顶端**:核心取样
2. **0.9 = 考虑占概率质量90%的符号。**防止从非常不可能得到的象征物取样
3. **LLM 服务实施**如何运作
4. **模型加载**:使用指定参数将 GGUF 模型装入 VRAM
5. **交互式执行器**LLamaSharp 用于聊天式互动的处决模式

### 推断合成器

```csharp
using Microsoft.Extensions.Logging;

class Program
{
    static async Task Main(string[] args)
    {
        // Setup logging
        var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
        var logger = loggerFactory.CreateLogger<LlmService>();

        // Configure model
        var parameters = new ModelParameters
        {
            ModelPath = @"C:\models\mistral-7b\mistral-7b-instruct-v0.2.Q5_K_M.gguf",
            ContextSize = 4096,
            GpuLayerCount = 35,
            Temperature = 0.7f,
            MaxTokens = 200
        };

        // Create service
        using var llmService = new LlmService(parameters, logger);

        // Test simple generation
        Console.WriteLine("=== Test 1: Simple Generation ===\n");
        var response1 = await llmService.GenerateAsync(
            "Explain what Docker Compose is in 2-3 sentences."
        );
        Console.WriteLine(response1);
        Console.WriteLine("\n");

        // Test with context
        Console.WriteLine("=== Test 2: Generation with Context ===\n");
        var context = new List<string>
        {
            "Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services.",
            "In development, Docker Compose makes it easy to spin up all dependencies (databases, caches, etc.) with one command: docker-compose up."
        };

        var response2 = await llmService.GenerateWithContextAsync(
            "Write an introduction paragraph for a blog post about using Docker Compose for development dependencies.",
            context
        );
        Console.WriteLine(response2);
    }
}
```

**: 所生成的流牌(实时输出)**:

```
=== Test 1: Simple Generation ===

Docker Compose is a tool that allows you to define and run multi-container Docker applications using a simple YAML configuration file. It simplifies the process of managing multiple containers, networking, and volumes, making it ideal for development environments.

=== Test 2: Generation with Context ===

If you've ever found yourself juggling multiple terminal windows to start databases, caches, and other services for local development, Docker Compose is about to become your new best friend. This powerful tool lets you define your entire development environment in a single YAML file and spin everything up with one command. In this post, we'll explore how to leverage Docker Compose to manage all your development dependencies, making your local setup reproducible, shareable, and incredibly easy to manage.
```

环境建设

## : 将用户提示与检索到的博客块组合起来

反急症/反急症

### : 在某些字符串中停止生成( 防止相撞) 。

```csharp
namespace Mostlylucid.BlogLLM.Client.Services
{
    public class SuggestionService : ISuggestionService
    {
        private readonly BatchEmbeddingService _embeddingService;
        private readonly QdrantVectorStore _vectorStore;
        private readonly ILlmService _llmService;  // NEW

        public SuggestionService(
            BatchEmbeddingService embeddingService,
            QdrantVectorStore vectorStore,
            ILlmService llmService)  // NEW
        {
            _embeddingService = embeddingService;
            _vectorStore = vectorStore;
            _llmService = llmService;
        }

        public async Task<string> GenerateAiSuggestionAsync(
            string currentText,
            List<SimilarPost> context)
        {
            // Extract text from similar posts
            var contextChunks = context
                .Take(3)  // Top 3 most similar
                .Select(p => p.FullText)
                .ToList();

            // Determine what type of suggestion to generate
            var prompt = DeterminePromptType(currentText);

            // Generate suggestion
            var suggestion = await _llmService.GenerateWithContextAsync(
                prompt,
                contextChunks
            );

            return suggestion;
        }

        private string DeterminePromptType(string currentText)
        {
            // Analyze what user is writing
            var lines = currentText.Split('\n');
            var lastLine = lines.LastOrDefault(l => !string.IsNullOrWhiteSpace(l)) ?? "";

            // Is user starting a new section?
            if (lastLine.StartsWith("## "))
            {
                return "Suggest 3-5 bullet points for what this section could cover.";
            }

            // Is user writing code?
            if (lastLine.Contains("```"))
            {
                return "Suggest relevant code examples that might be useful here.";
            }

            // Is user writing an introduction?
            if (currentText.Length < 500 && currentText.Contains("## Introduction"))
            {
                return "Suggest 2-3 sentences to continue this introduction based on similar posts.";
            }

            // Default: continue current thought
            return "Suggest 1-2 sentences to continue the current paragraph in a natural way.";
        }
    }
}
```

### 测试服务处

```csharp
public partial class SuggestionsViewModel : ViewModelBase
{
    [RelayCommand]
    private async Task RegenerateSuggestion()
    {
        IsGenerating = true;
        AiSuggestion = "Generating...";

        try
        {
            var currentText = GetCurrentEditorText();  // From messaging
            var suggestion = await _suggestionService.GenerateAiSuggestionAsync(
                currentText,
                SimilarPosts.ToList()
            );

            AiSuggestion = suggestion;
        }
        catch (Exception ex)
        {
            AiSuggestion = $"Error: {ex.Message}";
        }
        finally
        {
            IsGenerating = false;
        }
    }
}
```

## 预期产出预期产出

### 太厉害了!

该模式正在发挥作用,并产生一致的、符合背景的文本。

```csharp
public class LlmServiceFactory
{
    private static LlmService? _instance;
    private static readonly object _lock = new();

    public static LlmService GetInstance(ModelParameters parameters, ILogger<LlmService> logger)
    {
        if (_instance == null)
        {
            lock (_lock)
            {
                if (_instance == null)
                {
                    _instance = new LlmService(parameters, logger);
                }
            }
        }

        return _instance;
    }
}
```

### 与建议小组结合

现在让我们将LLM生成融入第五部分的 Windows 客户端。

```csharp
public class StatefulLlmService
{
    private readonly InferenceParams _defaultParams;
    private string _cachedPromptPrefix = string.Empty;

    public async Task<string> GenerateWithPrefixAsync(string prefix, string newPrompt)
    {
        // If prefix matches cached, reuse KV cache
        if (prefix == _cachedPromptPrefix)
        {
            // Only process new tokens
            return await GenerateAsync(newPrompt);
        }

        // Process entire prompt and cache
        _cachedPromptPrefix = prefix;
        return await GenerateAsync(prefix + newPrompt);
    }
}
```

更新建议服务

### 更新建议查看模式

业绩优化

```csharp
public async Task<List<string>> GenerateBatchAsync(List<string> prompts)
{
    var results = new List<string>();

    foreach (var prompt in prompts)
    {
        // With KV cache reuse, subsequent prompts are faster
        results.Add(await GenerateAsync(prompt));
    }

    return results;
}
```

## 模型缓存

保持模型在请求间加载 :

### KV 缓存再使用

```csharp
private string PromptContinueWriting(string currentText, List<string> context)
{
    return $@"You are a technical blog writing assistant.

Here are excerpts from similar blog posts:
{string.Join("\n\n", context.Select((c, i) => $"--- Post {i + 1} ---\n{c}"))}

Current draft:
{currentText}

Task: Suggest 2-3 sentences to naturally continue the current paragraph.
Keep the same technical depth and casual, pragmatic tone.

Suggestion:";
}
```

### LamaSharp支持KV缓存再利用,

```csharp
private string PromptSectionStructure(string sectionTitle, List<string> context)
{
    return $@"You are a technical blog writing assistant.

Similar sections from past posts:
{string.Join("\n\n", context)}

New section: {sectionTitle}

Task: Suggest 4-6 bullet points for what this section should cover.
Format as a markdown list.

Bullets:";
}
```

### 这对我们的用法特别有用- 上下文块保持不变, 只有用户的问题更改 。

```csharp
private string PromptCodeExample(string description, List<string> context)
{
    return $@"You are a C# coding assistant.

Relevant code from past posts:
{string.Join("\n\n", context)}

Task: {description}

Provide a clean, well-commented C# code example.

Code:";
}
```

## 条纹

对于多项建议,分批提出:

### 书写协助快速工程

```csharp
public class VramMonitor
{
    [DllImport("nvml.dll")]
    private static extern int nvmlDeviceGetMemoryInfo(IntPtr device, ref NvmlMemory memory);

    [StructLayout(LayoutKind.Sequential)]
    public struct NvmlMemory
    {
        public ulong Total;
        public ulong Free;
        public ulong Used;
    }

    public static (ulong used, ulong total) GetVramUsage()
    {
        // Simplified - actual implementation needs proper NVML initialization
        var memory = new NvmlMemory();
        // nvmlDeviceGetMemoryInfo(device, ref memory);

        return (memory.Used / 1024 / 1024, memory.Total / 1024 / 1024);  // Convert to MB
    }
}
```

### 良好提示=良好产出。

```csharp
public class LlmServiceWithUnload : IDisposable
{
    private LlmService? _service;
    private readonly Timer _unloadTimer;
    private DateTime _lastUsed;

    public LlmServiceWithUnload()
    {
        _unloadTimer = new Timer(CheckForUnload, null, TimeSpan.FromMinutes(1), TimeSpan.FromMinutes(1));
    }

    private void CheckForUnload(object? state)
    {
        if (_service != null && (DateTime.Now - _lastUsed) > TimeSpan.FromMinutes(10))
        {
            _service.Dispose();
            _service = null;
            GC.Collect();
            Console.WriteLine("Model unloaded due to inactivity");
        }
    }

    public async Task<string> GenerateAsync(string prompt)
    {
        _lastUsed = DateTime.Now;

        if (_service == null)
        {
            // Reload model
            _service = CreateService();
        }

        return await _service.GenerateAsync(prompt);
    }
}
```

## 以下是不同情景的模板 :

继续写入

```csharp
public async Task<string> GenerateWithRetryAsync(string prompt, int maxRetries = 3)
{
    for (int i = 0; i < maxRetries; i++)
    {
        try
        {
            return await GenerateAsync(prompt);
        }
        catch (OutOfMemoryException)
        {
            _logger.LogWarning("OOM error, reducing max tokens");
            _parameters.MaxTokens = Math.Max(100, _parameters.MaxTokens / 2);
        }
        catch (Exception ex)
        {
            _logger.LogError(ex, "Generation failed, attempt {Attempt}/{Max}", i + 1, maxRetries);

            if (i == maxRetries - 1) throw;

            await Task.Delay(1000 * (i + 1));  // Exponential backoff
        }
    }

    throw new Exception("Generation failed after retries");
}
```

## 建议款次结构

守则示例

1. ✅ Chose [记忆管理管理](https://github.com/SciSharp/LLamaSharp)有了大型模型,记忆管理至关重要。
2. ✅ Understood [监视 VRAM 使用情况](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md)非激活时卸载模型
3. ✅ Selected appropriate model ([错误处理错误处理](https://mistral.ai/) / [LLMs可能出乎意料地失败。](https://ai.meta.com/llama/)优雅地处理 :
4. ✅ Implemented LlmService with CUDA acceleration
5. ✅ Integrated with Windows client for suggestions
6. ✅ Implemented prompt engineering for writing tasks
7. ✅ Added performance optimizations (caching, batching)
8. ✅ Handled memory management and errors

## 摘要摘要摘要

我们成功地整合了当地LLM推论:**[拉马沙尔普](/blog/building-a-lawyer-gpt-for-your-blog-part7)**C# 整合

- GGUF 格式格式
- 和量化
- 米斯特拉7B
- 拉拉马3
- 8B)
- 下一个是什么?
- 内

第7部分:内容生成和即时工程

## 我们将集中关注完整的内容生成管道:

- [先进即速工程技术](/blog/building-a-lawyer-gpt-for-your-blog-part1)
- [用于迭接改进的多回合对话](/blog/building-a-lawyer-gpt-for-your-blog-part2)
- [背景窗管理战略](/blog/building-a-lawyer-gpt-for-your-blog-part3)
- [质量评价和过滤](/blog/building-a-lawyer-gpt-for-your-blog-part4)
- [执行样式一致性](/blog/building-a-lawyer-gpt-for-your-blog-part5)
- **处理代码区块生成**现实世界使用模式
- [我们会让这个系统 真正用于每日博客的写作!](/blog/building-a-lawyer-gpt-for-your-blog-part7)
- [系列导航](/blog/building-a-lawyer-gpt-for-your-blog-part8)

## 第1部分:导言和建筑

- [第2部分:C#中的GPU设置和CUDA CUDA](https://scisharp.github.io/LLamaSharp/)
- [第三部分:了解嵌入和矢量数据库](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md)
- [第4部分:建设吸收管道](https://huggingface.co/TheBloke)
- [第5部分:视窗客户端](https://github.com/ggerganov/llama.cpp)

第6部分:地方LLM整合[(本员额)](/blog/building-a-lawyer-gpt-for-your-blog-part7)!