📌注本文用最简单的方法来教授与LLMs一起使用网络内容提取的基本原理@.用于生产用途案例 @(web summarization_,}文档分析 ,代理工具),见 DocSummamer 缩写器 @-它执行此处显示的架构, 但生产( -QGA}:BERT) 嵌入@,混合搜索@MS K4播放器为 SPAS,SREF protection_,验证引用@,和适当覆盖跟踪.

当您询问 查特格普特 到“ "” 读这篇文章并总结它 “ MS K1” 和“ 如果想象人工智能打开浏览器,阅读方式像 @ -” 那样打开一个浏览器

LLM Con't 浏览 Web\ . 他们为您的代码选择的碎片解析

此文章展示您如何在 C# 中用 Ollama @ - @ no frames@ MS K2}( 只是实用代码) 构建此功能, 您可以调试@ MPK3 @ {

TL;DR:您的代码获取@→清理 @→块 @MS K3选择 @.。 LLM只看到您输入的碎片@MS K5选择在设计上丢失了 @MSKO6}'

使这项工作奏效的一件事情

改变你如何建立这些系统的洞察力@: @

选择是产品决定@. @ it'}(也是大多数失败来源) @.{

LLM 是您选择的下游@ . @ 它无法恢复信息 您没有% MS K1 @ t 显示它 .} 当一个代理 {" 找不到答案时, 问题几乎从来不是模式 < - it'} 你的选择逻辑选错了块@ I. _ NSK8> 这与后面的原则相同 为什么我避开兰钱这样的框架 -}它们抽象地删除了选择逻辑, 您需要调试@ MS K1

这就是为什么“"agent broussing”"没有沉默地"MS K2 "代理商根据自己所看到的"{."你永远不知道它错过了正确的内容"=.*

flowchart LR
    URL[Full Page] --> Select[Your Selection]
    Select --> LLM[LLM Sees This]
    LLM --> Answer[Answer]
    
    URL -.->|"50KB"| Select
    Select -.->|"2KB"| LLM
    
    Miss[Missed Content] -.->|"Never seen"| X[❌]
    
    style Select stroke:#e74c3c,stroke-width:3px
    style Miss stroke:#95a5a6,stroke-width:2px,stroke-dasharray: 5 5

该模型只能说明你给它什么原因 . 构建相应的. *

《守则前设计目标》

在写任何文件之前,

如果您能解释为什么选择块块@ , @ 您可以% MS K2_ t调试失败@ MPK3}

管道

flowchart TB
    URL[URL] --> Fetch[1. Fetch]
    Fetch --> Clean[2. Clean]
    Clean --> Chunk[3. Chunk]
    Chunk --> Select[4. Select]
    Select --> LLM[LLM]
    LLM --> Answer[Answer]
    
    Fetch -.->|"57KB HTML"| Clean
    Clean -.->|"6KB text"| Chunk
    Chunk -.->|"5 chunks"| Select
    Select -.->|"2 chunks"| LLM
    
    style Select stroke:#e74c3c,stroke-width:3px
    style Clean stroke:#f39c12,stroke-width:3px

每一步都会减少数据 . * 当LLM看到它时 您已经从 HTML 的 @57QQKB 到相关文本中的 @MS K4KB 分析大型 CSV 文件 -=LLM原因=,=您的代码计算和选择@.=%

运行中实例

本篇文章通篇使用一个 URL=:

https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-10/overview

还有三个越来越特殊的问题

1. .NET#10?"}新创的"MSK2QNET"
2. “"”提到哪些性能改进?“?"”
3. 有关于AOT的线索吗?

这让示例保持基础, 并显示当问题变得具体时选择更重要@. @ @

设置设置设置

# Install Ollama from https://ollama.ai
ollama pull llama3.2:3b

# NuGet packages
dotnet add package AngleSharp      # HTML parsing
dotnet add package OllamaSharp     # Ollama client (5.1.x)

OllamaSharp 注释@: 版本@5._x}, GenerateAsync 返回返回 IAsyncEnumerable<GenerateResponseStream?> -它流着牌子,因为它们是@'re produced @. await foreach. 样板项目插针 5.1.5.

步骤 @1:}抓取

标准 HTTP,}但细节很重要

public class WebFetcher : IDisposable
{
    private readonly HttpClient _http;
    
    public WebFetcher()
    {
        var handler = new HttpClientHandler
        {
            AllowAutoRedirect = true,
            MaxAutomaticRedirections = 5,  // Cap redirects
            AutomaticDecompression = DecompressionMethods.All
        };
        
        _http = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(30) };
        _http.DefaultRequestHeaders.Add("User-Agent", 
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36");
    }
    
    public async Task<string> FetchAsync(string url)
    {
        var response = await _http.GetAsync(url);
        
        // Bail if not HTML
        var contentType = response.Content.Headers.ContentType?.MediaType ?? "";
        if (!contentType.Contains("html") && !contentType.Contains("text"))
            throw new InvalidOperationException($"Not HTML: {contentType}");
        
        response.EnsureSuccessStatusCode();
        return await response.Content.ReadAsStringAsync();
    }
    
    public void Dispose() => _http.Dispose();
}

这是什么手柄:重定向@(ccaped @),压缩},超时 ,用户=-Agent,}内容_-类型验证.

它是什么 't@:_JavaScript Making@,}认证#,Q 比例限制%,}机器人=.Txt}.}生产时增加,

步骤 # 2:# 清洁

Raw HTML 大多为噪音@. 清理它或布局上的废牌 @.

典型的页面@: @%

57KB HTML → 6KB useful text (90% reduction)

战略

flowchart LR
    HTML[Raw HTML] --> Remove[Remove Noise]
    Remove --> Find[Find Main Content]
    Find --> Extract[Extract Text + Headings]
    Extract --> Normalize[Normalize]
    
    Remove -.->|"script, style, nav, ads"| Find
    Find -.->|"main, article, .content"| Extract
    
    style Remove stroke:#e74c3c,stroke-width:3px
    style Find stroke:#f39c12,stroke-width:3px

清理是丢失的 @ - @ 谨慎处理

您的清洁工可以删除答案@. @ 常见错误 @ :%

• 侵略选择器删除合法内容
• 主内容有不寻常的类名@ ( @.article-body, .post-text)
• 答案在侧边栏或图例框中

缓解措施:

• 明确保留标题@( @%h1-h6)
• 删除时间 角色角色角色 和 已知锅炉板*,* 不太宽广的模式
• Log : 提取元素路径\ MS K1 文本长度
• 如果主内容有可疑的短消息 , trybody

public class HtmlCleaner
{
    private readonly HtmlParser _parser = new();
    
    // Known noise - remove these entirely
    private static readonly string[] NoiseElements = 
        { "script", "style", "nav", "footer", "aside", "iframe", "noscript" };
    
    // Boilerplate patterns - remove by role, not aggressive wildcards
    private static readonly string[] NoiseSelectors = 
    {
        "[role='navigation']", "[role='banner']", "[role='complementary']",
        "[class*='cookie']", "[class*='newsletter']", "[aria-hidden='true']"
    };
    
    // Where to find content - order matters (most specific first)
    private static readonly string[] ContentSelectors = 
        { "main", "article", "[role='main']", ".content", ".post-content" };
    
    public CleanResult Clean(string html)
    {
        var doc = _parser.ParseDocument(html);
        
        // Remove noise
        foreach (var tag in NoiseElements)
            foreach (var el in doc.QuerySelectorAll(tag).ToList())
                el.Remove();
        
        foreach (var selector in NoiseSelectors)
            foreach (var el in doc.QuerySelectorAll(selector).ToList())
                el.Remove();
        
        // Find main content
        IElement? main = null;
        string? matchedSelector = null;
        
        foreach (var selector in ContentSelectors)
        {
            main = doc.QuerySelector(selector);
            if (main != null) { matchedSelector = selector; break; }
        }
        
        // Fallback to body if main content is suspiciously short
        var text = main?.TextContent ?? "";
        if (text.Length < 500 && doc.Body != null)
        {
            main = doc.Body;
            matchedSelector = "body (fallback)";
            text = main.TextContent;
        }
        
        return new CleanResult
        {
            Text = NormalizeWhitespace(text),
            MatchedSelector = matchedSelector ?? "none",
            OriginalLength = html.Length
        };
    }
    
    private string NormalizeWhitespace(string text)
    {
        text = Regex.Replace(text, @"[ \t]+", " ");
        text = Regex.Replace(text, @"\n\s*\n+", "\n\n");
        return text.Trim();
    }
}

public record CleanResult(string Text, string MatchedSelector, int OriginalLength);

易读性提取 ( @%sccoring policy 段落,}文字密度MS K2}是整个兔子洞.}(现在){,}_protoor}-}基于文件的提取作品 和博客MSSK6}。如果需要的话,样本项目包括一个评分提取器

步进 # 3:# 块

您有“ 6KB” 的纯文本@ MS K1 为什么不全部发送 @ ?

• 干草堆中的针头当相关的信息被大环境掩埋时,LLMs的表现更差。
• 费用成本@:# 更多代币@ = @ 更多的钱和延缓
• 焦点@:_ 选中的块比每件事同时提出更好的推理

踢打策略比你想的更重要 我深入地报道了这一点 RAG结构条款 @-}无论您是否在拼凑网页或文档中 都适用同样的原则

Naive 判决决赛

这是起点 , 不是生产代码 : *

public List<string> ChunkBySentence(string text, int maxTokens = 2000)
{
    var chunks = new List<string>();
    
    // WARNING: This breaks on abbreviations, decimals, URLs, code samples
    var sentences = text.Split(new[] { ". ", ".\n", "! ", "? " }, 
        StringSplitOptions.RemoveEmptyEntries);
    
    var current = new StringBuilder();
    var tokens = 0;
    
    foreach (var sentence in sentences)
    {
        var sentenceTokens = EstimateTokens(sentence);
        
        if (tokens + sentenceTokens > maxTokens && current.Length > 0)
        {
            chunks.Add(current.ToString().Trim());
            current.Clear();
            tokens = 0;
        }
        
        current.Append(sentence).Append(". ");
        tokens += sentenceTokens;
    }
    
    if (current.Length > 0)
        chunks.Add(current.ToString().Trim());
    
    return chunks;
}

// Rough estimate - OK for demos, not for billing
private int EstimateTokens(string text) 
    => (int)(text.Split(' ').Length * 1.3);

为何如此天真:

• ". " @ " @ Dr. @ Smith@ MS K2} @ I"_ v @ MPK4{ URLs 上的课间休息
• 按字数估计 Token 的值为@ ±20% @ 关闭
• 块\ ( 关闭边框的上下文@)\ {之间没有重叠

更好的:#标题 @-}基于踢球

文档中的“,”块块,按区域“{:

public List<ContentChunk> ChunkByHeadings(string html)
{
    var doc = new HtmlParser().ParseDocument(html);
    var chunks = new List<ContentChunk>();
    
    var headings = doc.QuerySelectorAll("h1, h2, h3");
    
    foreach (var heading in headings)
    {
        var content = new StringBuilder();
        content.AppendLine(heading.TextContent);
        
        var sibling = heading.NextElementSibling;
        while (sibling != null && !sibling.TagName.StartsWith("H"))
        {
            content.AppendLine(sibling.TextContent);
            sibling = sibling.NextElementSibling;
        }
        
        chunks.Add(new ContentChunk
        {
            Heading = heading.TextContent.Trim(),
            Content = content.ToString().Trim(),
            HeadingLevel = int.Parse(heading.TagName[1..])
        });
    }
    
    return chunks;
}

此选项保存文档结构, 让选择更有意义@ . @

步骤 @4: 选择

这是大多数失败发生的地方 @ . @ 和调试应该从那里开始@ MS K1\ {

用户询问“"”在 @.@NET @10?"中哪些性能改进? 只有#1-2块提到业绩=.发送这些.

flowchart TB
    Q["Question: What perf improvements?"] --> Score[Score Each Chunk]
    
    subgraph Chunks
        C1["Chunk 1: Overview..."] 
        C2["Chunk 2: Runtime perf..."]
        C3["Chunk 3: Libraries..."]
        C4["Chunk 4: SDK changes..."]
    end
    
    Score --> C1
    Score --> C2
    Score --> C3
    Score --> C4
    
    C2 -->|"score: 3"| Top[Selected]
    C3 -->|"score: 1"| Top
    
    style C2 stroke:#27ae60,stroke-width:3px
    style C1 stroke:#95a5a6,stroke-width:2px,stroke-dasharray: 5 5
    style C4 stroke:#95a5a6,stroke-width:2px,stroke-dasharray: 5 5

匹配“(”的关键字Spro,}调试器@)

public record ScoredChunk(string Content, string? Heading, int Score, List<string> MatchedKeywords);

public List<ScoredChunk> SelectByKeywords(
    List<ContentChunk> chunks, 
    string question, 
    int topK = 3)
{
    // Normalize and filter stopwords
    var keywords = question.ToLower()
        .Split(' ', StringSplitOptions.RemoveEmptyEntries)
        .Where(w => w.Length > 3)
        .Where(w => !Stopwords.Contains(w))
        .Select(w => w.Trim(',', '.', '?', '!'))
        .Distinct()
        .ToList();
    
    var scored = chunks.Select(chunk =>
    {
        var text = (chunk.Heading + " " + chunk.Content).ToLower();
        var matched = keywords.Where(kw => text.Contains(kw)).ToList();
        
        // Boost if keyword appears in heading
        var headingBoost = chunk.Heading != null && 
            keywords.Any(kw => chunk.Heading.ToLower().Contains(kw)) ? 2 : 0;
        
        return new ScoredChunk(
            chunk.Content, 
            chunk.Heading, 
            matched.Count + headingBoost,
            matched
        );
    })
    .OrderByDescending(x => x.Score)
    .Take(topK)
    .ToList();
    
    // LOG THIS - it's your debugging lifeline
    foreach (var s in scored)
        Console.WriteLine($"  [{s.Score}] {s.Heading ?? "(no heading)"}: {string.Join(", ", s.MatchedKeywords)}");
    
    return scored;
}

private static readonly HashSet<string> Stopwords = new()
    { "what", "how", "does", "the", "are", "is", "in", "for", "of", "to", "and" };

与天真计算相比的主要改进:

• “(”或者“"”和“MS K3” 总是匹配@)。
• 方向助推 @ (_ 结构信号 @ I)+#
• 登录匹配的关键字 -这是你调试的生命线

嵌入式-基于 @(Semantic})

关键字在同义词@ . @ @ MS K1_ perf" @ won @ MPK3} t 匹配 ["# 性能改进=".}

嵌入中发现语义相似性@. @ 如果您想要更深入嵌入和矢量搜索@ ,}I covered this in the RAG 开源系列 和 使用 ONNX 进行语义搜索.

public async Task<List<ScoredChunk>> SelectByEmbedding(
    List<ContentChunk> chunks, 
    string question, 
    int topK = 3)
{
    var questionEmbed = await EmbedAsync(question);
    
    // Cache these per URL in production
    var scored = new List<(ContentChunk Chunk, double Score)>();
    
    foreach (var chunk in chunks)
    {
        var chunkEmbed = await EmbedAsync(chunk.Content);
        var similarity = CosineSimilarity(questionEmbed, chunkEmbed);
        scored.Add((chunk, similarity));
    }
    
    return scored
        .OrderByDescending(x => x.Score)
        .Take(topK)
        .Select(x => new ScoredChunk(x.Chunk.Content, x.Chunk.Heading, (int)(x.Score * 100), new()))
        .ToList();
}

private async Task<double[]> EmbedAsync(string text)
{
    var request = new EmbedRequest { Model = "nomic-embed-text", Input = [text] };
    var response = await _ollama.EmbedAsync(request);
    return response.Embeddings.First().ToArray();
}

贸易:

• 关键词 @ : fast@ MS K1 调试器@ MPK2 字面匹配
• 嵌入式:SemanticMS K1}但速度慢,需要缓存

用于在 SQLite 或一个矢量数据库中按 @ ( URL@ MSSK2}块 hash)} 解冻.

发送到 LLM

强制源的快速结构@- @bounded understand answer with purcation_:}以批注方式发送答覆

public string BuildPrompt(string url, List<ScoredChunk> chunks, string question)
{
    var sb = new StringBuilder();
    
    sb.AppendLine("You are answering a question using ONLY the content below.");
    sb.AppendLine("Rules:");
    sb.AppendLine("- Answer ONLY from the provided sources");
    sb.AppendLine("- Cite which SOURCE number supports each claim");
    sb.AppendLine("- Include 1-2 brief quotes as evidence");
    sb.AppendLine("- If the answer isn't in the sources, say 'Not enough information'");
    sb.AppendLine("- End with Confidence: High/Medium/Low");
    sb.AppendLine();
    
    for (int i = 0; i < chunks.Count; i++)
    {
        sb.AppendLine($"=== SOURCE {i + 1} ===");
        if (chunks[i].Heading != null)
            sb.AppendLine($"Section: {chunks[i].Heading}");
        sb.AppendLine($"From: {url}");
        sb.AppendLine(chunks[i].Content);
        sb.AppendLine();
    }
    
    sb.AppendLine($"Question: {question}");
    sb.AppendLine();
    sb.AppendLine("Answer (with citations and confidence):");
    
    return sb.ToString();
}

此移动从 @ " @ chatty 摘要@ MS K1 移到\ " 分析与来源@ MPK3

呼叫奥拉马

public async Task<string> AskAsync(string prompt)
{
    var request = new GenerateRequest { Model = "llama3.2:3b", Prompt = prompt };
    
    var response = new StringBuilder();
    await foreach (var chunk in _ollama.GenerateAsync(request))
    {
        if (chunk?.Response != null)
            response.Append(chunk.Response);
    }
    
    return response.ToString().Trim();
}

这是工具使用

我们所建造的是一个代理模式,没有这个框架... ....... 为什么我更喜欢这个方法 而不是兰钱 @-明确管弦在调试时战胜了魔法抽象的魔力#.

flowchart LR
    subgraph Tools["Tools (Deterministic)"]
        T1[fetch_url]
        T2[clean_html]
        T3[chunk_text]
        T4[select_relevant]
    end
    
    subgraph LLM["LLM (Reasoning)"]
        R[Interpret + Answer]
    end
    
    T1 --> T2 --> T3 --> T4 --> R
    
    R -->|"Low confidence"| Retry[Retry with different selection]
    Retry --> T4
    
    style T4 stroke:#e74c3c,stroke-width:3px
    style R stroke:#3498db,stroke-width:3px

如果信任度低, 则循环@ : @ 重新使用更多块或不同的关键字\ MS K2}重试

var answer = await AskAsync(prompt);

if (answer.Contains("Not enough information") || answer.Contains("Confidence: Low"))
{
    // Retry with more chunks
    var moreChunks = SelectByKeywords(allChunks, question, topK: 5);
    answer = await AskAsync(BuildPrompt(url, moreChunks, question));
}

哪里出错

flowchart TB
    subgraph Failures["Failure Modes"]
        F1[Cleaner removes content]
        F2[Chunking breaks mid-thought]
        F3[Selection picks wrong chunks]
        F4[LLM hallucinates connections]
    end
    
    F1 --> R1["'Not enough info' - answer existed"]
    F2 --> R2["Partial answer - context lost"]
    F3 --> R3["Wrong answer - right content skipped"]
    F4 --> R4["Confident but wrong"]
    
    style F3 stroke:#e74c3c,stroke-width:3px

调试规则:

如果回答错误“,”它几乎总是因为 选择错误@, 并非因为模型失败@.

调试检查列表

1. 日志选中的块编号和分数
2. 显示符合哪些关键字 @( 或嵌入距离@)
3. 重放精确提示 所发送的
4. 比较比: 是在已取回的 HTML MS K1 已清理文本中的正确内容@? 任意块 @?

失败通常为 LLM .@ it' @s uptrain_.

将它结合在一起

public class WebAnalyzer : IDisposable
{
    private readonly WebFetcher _fetcher = new();
    private readonly HtmlCleaner _cleaner = new();
    private readonly OllamaApiClient _ollama = new(new Uri("http://localhost:11434"));
    
    public async Task<AnalysisResult> AnalyzeAsync(string url, string question)
    {
        // 1. Fetch
        var html = await _fetcher.FetchAsync(url);
        
        // 2. Clean (with observability)
        var cleaned = _cleaner.Clean(html);
        Console.WriteLine($"Cleaned: {cleaned.OriginalLength} → {cleaned.Text.Length} bytes ({cleaned.MatchedSelector})");
        
        // 3. Chunk
        var chunks = ChunkByHeadings(html);
        Console.WriteLine($"Chunks: {chunks.Count}");
        
        // 4. Select (with logging)
        Console.WriteLine("Selection scores:");
        var selected = SelectByKeywords(chunks, question, topK: 3);
        
        // 5. Prompt + LLM
        var prompt = BuildPrompt(url, selected, question);
        var answer = await AskAsync(prompt);
        
        return new AnalysisResult
        {
            Answer = answer,
            ChunksUsed = selected.Count,
            SelectionScores = selected.Select(s => s.Score).ToList()
        };
    }
    
    public void Dispose() => _fetcher.Dispose();
}

使用我们运行中的示例@: @%

using var analyzer = new WebAnalyzer();

var result = await analyzer.AnalyzeAsync(
    "https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-10/overview",
    "What performance improvements are in .NET 10?"
);

Console.WriteLine(result.Answer);

输出包括引用和信任@: @%

Based on SOURCE 1 and SOURCE 2:

.NET 10 includes several performance improvements:
- JIT improvements including better inlining and method devirtualization (SOURCE 1)
- "Enhanced loop inversion for better optimization" (SOURCE 1)
- NativeAOT enhancements for improved code generation (SOURCE 2)

Confidence: High

当此工程有效时 @ ( @ and doesn't='_ t)}

• |文档 * * JavaScript SPAs * * MSC2* @|_Blog possible\ ,}文章@|}动态/}互动内容 @MS K4} “|”技术参考资料“|”多“-”页面研究“MSC3” | 静态 HTML| 认证内容 @|

JSQ-#Heavy网站_,}您需要的 用于 @ .+NET 的播放器.

关键原则

flowchart LR
    subgraph Your["Your Code's Job"]
        direction TB
        F[Fetch reliably]
        C[Clean carefully]
        S[Select correctly]
        O[Observe everything]
    end
    
    subgraph LLM["LLM's Job"]
        direction TB
        R[Reason over what you gave it]
        A[Admit when it doesn't know]
    end
    
    Your --> LLM
    
    style S stroke:#e74c3c,stroke-width:3px
    style R stroke:#3498db,stroke-width:3px

LLM只能解释你给它什么 . 选择是你的责任

要求LLM浏览. 请说明原因

完整抽样项目

完成工作实施@: @% Mostlucid.LlmWeb 附加器

包括:

• WebFetcher -HTTP,处理得当
• HtmlCleaner -噪声除去+后退策略
• ContentChunker - 刑期和标题 @-基块
• WebContentAnalyzer - 通通管道,有伐木
• OllamaExtensions - 流响应助手

cd Mostlylucid.LlmWebFetcher
dotnet run

资源资源资源 资源和资源资源资源

图书馆馆藏

• 闪烁角 - HTML 解析
• 奥利亚马沙尔普 Ollama 客户端

LLM( 里程)

• 奥拉马 -当地LLM运行时间
• 亚马逊语 -Fast,有能力的模型
• 缩略语@ - @ embed @ MS K1_ text -=YTET -伊甸园字幕组=- 翻译:

相关条款

• DocSummaizer 丛书 - 方法的改进 本文展示了获取和分析“;”和“DocSummarizer”的最简单方式, 通过 BERT 提取来显示 “-” 等级的制作方式 @,} 混合搜索@, 校验引用<,> 以及适当覆盖跟踪.}
• 用本地 LLM 分析大型 CSV 文件 -}相同模式 @:#LLM原因 @MS K2代码计算
• 为什么我用兰钱 -框-无物剂设计
• RAG 开源器 理解检索@- @ 推荐的一代
• 用 ONNX 和 Qdrant 进行语义搜索 - 本地语义搜索
• a 建造律师GPT *- 完整的RAG实施系列

微软 AI Stack

• 语义核心 - 如果您想要一个框架
• 微软.扩展@.AI - 集思广益