为 Markdig 建立一个远程的 Markdown 抓取器

Friday, 07 November 2025

一. 导言导言导言导言导言导言一,导言导言导言导言导言导言

我在建立这个博客时遇到的挑战之一是, 如何有效地包括外部标记内容,

我想从我的 GitHub 库中获取 README 文件, 包括来自其他项目的文件, 并自动保持一切同步。Mostlylucid.Markdig.FetchExtension解决办法?

自定义的 Markdig 扩展名, 该扩展名在转换时间获取远程标记并智能缓存。

在这个岗位上,我会给你介绍我是如何建造的

**- 完全解决取取和缓存远程记分数内容,支持多个存储后端、自动投票以及一个停滞但重置的缓存模式。**注意:这仍然是提前释放的, 但我想把它弄出去。disable="true"玩得开心点,但可能还不行

**这篇文章是由人工智能生成的, 使用 cluude 代码, 也帮助我建立这个功能。**最新更新[TOC]:增加数

参数,所以我们现在可以正确演示标记,而不处理它们更新(2025年11月7日) .

在标记下自动从文档标题中生成可点击的目录。

找找源头

在吉特胡卜山上为这个站点N Nuget 元数
N Nuget 元数为什么要建这个?
在潜入技术细节之前让我解释一下问题我有一些需要包括外部减价内容的假想:
软件包 REEADME 软件包: 当我写一个我出版的NuGet包包时, 我想直接从 GitHub 中加入它的 README

API 文件

:经常变化的外部 API 文档需要保持同步
共享内容
: 生活在一个存储库中但需要出现在多个地方的文件
业绩业绩业绩业绩业绩

:我不想在每页的页数中获取这些内容 - 那会慢而浪费

最天真的做法是使用 HTTP 客户端在需要时获得标记。

但这有问题:

graph TD
    A[Markdown with fetch tags] --> B[MarkdownFetchPreprocessor]
    B --> C{Check Cache}
    C -->|Fresh| D[Return Cached Content]
    C -->|Stale/Missing| E[Fetch from Remote URL]
    E -->|Success| F[Update Cache]
    E -->|Failure| G{Has Cached?}
    G -->|Yes| H[Return Stale Cache]
    G -->|No| I[Return Error Comment]
    F --> J[Replace fetch with Content]
    D --> J
    H --> J
    I --> J
    J --> K[Processed Markdown]
    K --> L[Your Markdig Pipeline]
    L --> M[Final HTML]

每个请求都点击远程服务器网络延迟影响页面负载时间没有离线支持

没有处理短暂故障<fetch>我需要更聪明的东西: 去拿一次, 智能缓存, 自动刷新, 并优雅地处理失败。
建筑结构概览
延长期采用预处理办法,而不是作为Markdig挖井管道的一部分。
这至关重要,因为它意味着通过你整个马克迪的管道获取内容流, 获取你所有的定制扩展, 语法加亮, 和外观。

这里的关键见解是

预处理预处理

# My Documentation

<!-- Failed to fetch content from https://raw.githubusercontent.com/user/repo/main/README.md: HTTP 404: Not Found -->

在你到达马克迪输油管之前,我们:

扫描扫描
标签标签
解析内容( 从缓存或远程)
将标记替换为实际标记

然后让Markdig 一起处理一切

这确保了一致性 -- -- 不论来源如何,所有记分都得到同样的待遇。基本语法使用扩展名很简单。

graph LR
    A[IMarkdownFetchService Interface] --> B[InMemoryMarkdownFetchService]
    A --> C[FileBasedMarkdownFetchService]
    A --> D[PostgresMarkdownFetchService]
    A --> E[SqliteMarkdownFetchService]
    A --> F[SqlServerMarkdownFetchService]
    A --> G[YourCustomService]

    B --> H[ConcurrentDictionary]
    C --> I[File System + SemaphoreSlim]
    D --> J[PostgreSQL Database]
    E --> K[SQLite Database]
    F --> L[SQL Server Database]
    G --> M[Your Storage Backend]

在您的标记中 :

就是这样!IMarkdownFetchService:

public interface IMarkdownFetchService
{
    Task<MarkdownFetchResult> FetchMarkdownAsync(
        string url,
        int pollFrequencyHours,
        int blogPostId = 0);

    Task<bool> RemoveCachedMarkdownAsync(
        string url,
        int blogPostId = 0);
}

延期将:

从 GitHub 获取 README

堆积了24小时ConcurrentDictionary:

public class InMemoryMarkdownFetchService : IMarkdownFetchService
{
    private readonly ConcurrentDictionary<string, CacheEntry> _cache = new();
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly ILogger<InMemoryMarkdownFetchService> _logger;

    public async Task<MarkdownFetchResult> FetchMarkdownAsync(
        string url,
        int pollFrequencyHours,
        int blogPostId)
    {
        var cacheKey = GetCacheKey(url, blogPostId);

        // Check cache
        if (_cache.TryGetValue(cacheKey, out var cached))
        {
            var age = DateTimeOffset.UtcNow - cached.FetchedAt;
            if (age.TotalHours < pollFrequencyHours)
            {
                _logger.LogDebug("Returning cached content for {Url}", url);
                return new MarkdownFetchResult
                {
                    Success = true,
                    Content = cached.Content
                };
            }
        }

        // Fetch fresh content
        var fetchResult = await FetchFromUrlAsync(url);

        if (fetchResult.Success)
        {
            _cache[cacheKey] = new CacheEntry
            {
                Content = fetchResult.Content,
                FetchedAt = DateTimeOffset.UtcNow
            };
        }
        else if (cached != null)
        {
            // Fetch failed, return stale cache
            _logger.LogWarning("Fetch failed, returning stale cache for {Url}", url);
            return new MarkdownFetchResult
            {
                Success = true,
                Content = cached.Content
            };
        }

        return fetchResult;
    }

    private static string GetCacheKey(string url, int blogPostId)
        => $"{url}_{blogPostId}";
}

返回后请求的缓存内容

缓存过期时自动更新
存储提供者建筑
我遵循的设计原理之一是
灵活度
不同的应用有不同的需要。
一个小型的演示应用程序不需要PostgreSQL, 但多服务器的生产部署是需要的。

于是我建造了一个可插入的存储结构:核心界面每件每件东西都执行

简单而干净。

每个执行项目都以自己的方式储存,但接口始终一致。

public class FileBasedMarkdownFetchService : IMarkdownFetchService
{
    private readonly string _cacheDirectory;
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly ILogger<FileBasedMarkdownFetchService> _logger;
    private readonly SemaphoreSlim _fileLock = new(1, 1);

    public async Task<MarkdownFetchResult> FetchMarkdownAsync(
        string url,
        int pollFrequencyHours,
        int blogPostId)
    {
        var cacheKey = ComputeCacheKey(url, blogPostId);
        var cacheFile = GetCacheFilePath(cacheKey);

        await _fileLock.WaitAsync();
        try
        {
            // Check if file exists and is fresh
            if (File.Exists(cacheFile))
            {
                var fileInfo = new FileInfo(cacheFile);
                var age = DateTimeOffset.UtcNow - fileInfo.LastWriteTimeUtc;

                if (age.TotalHours < pollFrequencyHours)
                {
                    var cached = await File.ReadAllTextAsync(cacheFile);
                    return new MarkdownFetchResult
                    {
                        Success = true,
                        Content = cached
                    };
                }
            }

            // Fetch fresh
            var fetchResult = await FetchFromUrlAsync(url);

            if (fetchResult.Success)
            {
                await File.WriteAllTextAsync(cacheFile, fetchResult.Content);
            }
            else if (File.Exists(cacheFile))
            {
                // Return stale on fetch failure
                var stale = await File.ReadAllTextAsync(cacheFile);
                return new MarkdownFetchResult
                {
                    Success = true,
                    Content = stale
                };
            }

            return fetchResult;
        }
        finally
        {
            _fileLock.Release();
        }
    }

    private string GetCacheFilePath(string cacheKey)
        => Path.Combine(_cacheDirectory, $"{cacheKey}.md");

    private static string ComputeCacheKey(string url, int blogPostId)
    {
        var combined = $"{url}_{blogPostId}";
        using var sha256 = SHA256.Create();
        var bytes = Encoding.UTF8.GetBytes(combined);
        var hash = sha256.ComputeHash(bytes);
        return Convert.ToHexString(hash);
    }
}

模拟存储:适合演示

最简单的执行用途SemaphoreSlim正如你们所看到的,这可以做到如下:
从 URL 和博客 post ID 创建缓存密钥
检查我们是否存有缓存内容,是否是新鲜内容
如果缓存是新鲜的, 请立即返回
如果淡化, 尝试获取新内容

成功时, 更新缓存

以缓存内容失败时, 返回过期缓存( stale- while- revalidate! ) !

public class PostgresMarkdownFetchService : IMarkdownFetchService
{
    private readonly MarkdownCacheDbContext _dbContext;
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly ILogger<PostgresMarkdownFetchService> _logger;

    public async Task<MarkdownFetchResult> FetchMarkdownAsync(
        string url,
        int pollFrequencyHours,
        int blogPostId)
    {
        var cacheKey = GetCacheKey(url, blogPostId);

        // Query cache
        var cached = await _dbContext.MarkdownCache
            .FirstOrDefaultAsync(c => c.CacheKey == cacheKey);

        if (cached != null)
        {
            var age = DateTimeOffset.UtcNow - cached.LastFetchedAt;
            if (age.TotalHours < pollFrequencyHours)
            {
                return new MarkdownFetchResult
                {
                    Success = true,
                    Content = cached.Content
                };
            }
        }

        // Fetch fresh
        var fetchResult = await FetchFromUrlAsync(url);

        if (fetchResult.Success)
        {
            if (cached == null)
            {
                cached = new MarkdownCacheEntry
                {
                    CacheKey = cacheKey,
                    Url = url,
                    BlogPostId = blogPostId
                };
                _dbContext.MarkdownCache.Add(cached);
            }

            cached.Content = fetchResult.Content;
            cached.LastFetchedAt = DateTimeOffset.UtcNow;
            await _dbContext.SaveChangesAsync();
        }
        else if (cached != null)
        {
            // Return stale
            return new MarkdownFetchResult
            {
                Success = true,
                Content = cached.Content
            };
        }

        return fetchResult;
    }
}

没有缓存失败时, 返回错误

CREATE TABLE markdown_cache (
    id SERIAL PRIMARY KEY,
    cache_key VARCHAR(128) NOT NULL UNIQUE,
    url VARCHAR(2048) NOT NULL,
    blog_post_id INTEGER NOT NULL,
    content TEXT NOT NULL,
    last_fetched_at TIMESTAMP WITH TIME ZONE NOT NULL,
    CONSTRAINT ix_markdown_cache_cache_key UNIQUE (cache_key)
);

CREATE INDEX ix_markdown_cache_url_blog_post_id
    ON markdown_cache(url, blog_post_id);

这种模式 -

graph TB
    subgraph "Load Balancer"
        LB[Load Balancer]
    end

    subgraph "Application Servers"
        A1[App Server 1<br/>FetchExtension]
        A2[App Server 2<br/>FetchExtension]
        A3[App Server 3<br/>FetchExtension]
    end

    subgraph "Shared Cache"
        PG[(PostgreSQL<br/>markdown_cache table)]
    end

    subgraph "External Content"
        R1[Remote URL 1]
        R2[Remote URL 2]
        R3[Remote URL 3]
    end

    LB --> A1
    LB --> A2
    LB --> A3

    A1 <-->|Read/Write Cache| PG
    A2 <-->|Read/Write Cache| PG
    A3 <-->|Read/Write Cache| PG

    A1 -.->|Fetch if cache miss| R1
    A2 -.->|Fetch if cache miss| R2
    A3 -.->|Fetch if cache miss| R3

静态时的重新校验

- 对可靠性至关重要。

即使GitHub倒闭了,你的网站仍然用隐藏的内容工作。

dotnet add package mostlylucid.Markdig.FetchExtension

基于文件的存储库:简单持久性

# For in-memory (demos/testing)
# Already included in base package

# For file-based storage
# Already included in base package

# For PostgreSQL
dotnet add package mostlylucid.Markdig.FetchExtension.Postgres

# For SQLite
dotnet add package mostlylucid.Markdig.FetchExtension.Sqlite

# For SQL Server
dotnet add package mostlylucid.Markdig.FetchExtension.SqlServer

对于单一服务器的部署,基于文件的储存工作非常出色:

这里的要点 :Program.cs:

using Mostlylucid.Markdig.FetchExtension;

var builder = WebApplication.CreateBuilder(args);

// Option 1: In-Memory (simplest)
builder.Services.AddInMemoryMarkdownFetch();

// Option 2: File-Based (persists across restarts)
builder.Services.AddFileBasedMarkdownFetch("./markdown-cache");

// Option 3: PostgreSQL (multi-server)
builder.Services.AddPostgresMarkdownFetch(
    builder.Configuration.GetConnectionString("MarkdownCache"));

// Option 4: SQLite (single server with DB)
builder.Services.AddSqliteMarkdownFetch("Data Source=markdown-cache.db");

// Option 5: SQL Server (enterprise)
builder.Services.AddSqlServerMarkdownFetch(
    builder.Configuration.GetConnectionString("MarkdownCache"));

var app = builder.Build();

// If using database storage, ensure schema exists
if (app.Environment.IsDevelopment())
{
    app.Services.EnsureMarkdownCacheDatabase();
}

// Configure the extension with your service provider
FetchMarkdownExtension.ConfigureServiceProvider(app.Services);

app.Run();

用途

用于线索安全文件访问

public class MarkdownRenderingService
{
    private readonly IServiceProvider _serviceProvider;
    private readonly MarkdownFetchPreprocessor _preprocessor;
    private readonly MarkdownPipeline _pipeline;

    public MarkdownRenderingService(IServiceProvider serviceProvider)
    {
        _serviceProvider = serviceProvider;
        _preprocessor = new MarkdownFetchPreprocessor(serviceProvider);

        _pipeline = new MarkdownPipelineBuilder()
            .UseAdvancedExtensions()
            .UseSyntaxHighlighting()
            .UseToc()  // Add TOC support for [TOC] markers
            .UseYourCustomExtensions()
            .Build();
    }

    public string RenderMarkdown(string markdown)
    {
        // Step 1: Preprocess to handle fetch tags
        var processed = _preprocessor.Preprocess(markdown);

        // Step 2: Run through your normal Markdig pipeline
        return Markdown.ToHtml(processed, _pipeline);
    }
}

隐藏 URL + 博客的 URL + 博客 post ID 以创建安全文件名

使用文件修改时间确定新鲜度<fetch>在整个应用程序重新启动中保持永久状态
相同平淡的值重现模式
数据库储存:生产-储存
对于生产部署, 特别是多服务器设置, 您想要共享缓存。

这就是数据库提供者来的地方:

数据库模式很简单:

在多服务器的部署中,这可以让您在所有情况中隐藏一致性:**所有服务器共享相同的缓存。**当服务器 1 获取 README 时, 服务器 2 和 3 立即从缓存内容中受益。

设置扩展设置扩展

开始是直截了当的。[TOC]首先,安装基件包 :

# My Document

[TOC]

# Introduction
Content here...

# Getting Started
More content...

## Installation
Details...

然后选择您的存储提供者 :

<nav class="ml_toc" aria-label="Table of Contents">
  <ul>
    <li><a href="#introduction">Introduction</a></li>
    <li><a href="#getting-started">Getting Started</a>
      <ul>
        <li><a href="#installation">Installation</a></li>
      </ul>
    </li>
  </ul>
</nav>

ASP. NET 核心的配置

在你的

[TOC cssclass="my-custom-toc"]

与您的 Markdown 设计整合

<nav class="my-custom-toc" aria-label="Table of Contents">
  <!-- TOC content -->
</nav>

关键是预处理步骤。

**以下是我如何把它融入我的博客:**流动是:
您的标记包含标签标签
- 预处理器将预处理器溶解到实际标记id="getting-started"
- 合并计分通过马克迪id="api-reference"
每样东西都有你的定制扩展、外观等高级特征ul/li内容生成表

包包现在包括

单独

var pipeline = new MarkdownPipelineBuilder()
    .UseAdvancedExtensions()
    .UseToc()  // Add TOC support - position in pipeline doesn't matter!
    .Use<YourOtherExtensions>()
    .Build();

**目录( TOC) 扩展 !**虽然它被包装与取回扩展, 它是完全独立的, 并且可以自己使用。**您可以从文档的标题中自动生成可点击目录。**基本用法:

简单添加
在您的标记中的任意位置 :

这将生成包含锁定链接的所有标题的嵌套列表 :.UseToc()自定义 CSS 类 :

**您可以指定一个自定义的 CSS 类,用于打字 :**这与您的自定义类有关 :

如何运作:
自动检测
:技选委员会自动检测到文档中的最低标题水平,并相应调整。

**如果您的文档以 H2 开头, TOC 将 H2 视为顶级。**ID代代人[TOC]: 锁定连接时标题会自动给定 ID :.UseToc()"开始"

“API参考”____________________________________________________________________________________________________________________________________________________________________________________

嵌套结构

<!-- Failed to fetch content from https://raw.githubusercontent.com/user/repo/main/docs/README.md: HTTP 404: Not Found -->

:造物主建造一个合适的巢穴

./CONTRIBUTING.md → https://github.com/user/repo/blob/main/docs/CONTRIBUTING.md
../images/logo.png → https://github.com/user/repo/blob/main/images/logo.png
反映文档等级结构的文档结构。

启用技术中心支持 :

public class MarkdownLinkRewriter
{
    public static string RewriteLinks(string markdown, string sourceUrl)
    {
        var document = Markdown.Parse(markdown);
        var baseUri = GetBaseUri(sourceUrl);

        foreach (var link in document.Descendants<LinkInline>())
        {
            if (IsRelativeLink(link.Url))
            {
                link.Url = ResolveRelativeLink(baseUri, link.Url);
            }
        }

        using var writer = new StringWriter();
        var renderer = new NormalizeRenderer(writer);
        renderer.Render(document);
        return writer.ToString();
    }

    private static bool IsRelativeLink(string url)
    {
        if (string.IsNullOrEmpty(url)) return false;
        if (url.StartsWith("http://") || url.StartsWith("https://")) return false;
        if (url.StartsWith("#")) return false;  // Anchor
        if (url.StartsWith("mailto:")) return false;
        return true;
    }
}

当配置您的 Markdig 管道时, 添加 TOC 扩展 :

管道位置 :

<!-- Failed to fetch content from https://api.example.com/status.md: HTTP request failed: Name or service not known (api.example.com:443) -->

不同于某些 Markdig 扩展, TOC 扩展

不在乎你把它加到哪里正在输油管中。自动扩展名 :

在分析器列表( 位置 0) 的开头插入分析器解析器

<!-- Failed to fetch content from https://example.com/docs.md: HTTP 404: Not Found -->

在所有解析完成后重新调试器, 从整个文档中收集标题

所以你可以添加

管道配置的开始、中间或结束。
{retrieved:format}重要:
{age}技选委员会的延期与调阅延期完全无关。
{url}他们只是为了方便才包装在一起的
{nextrefresh:format}你可以:
{pollfrequency}使用 TOC 而不获取
{status}使用不使用 TOC 的抓取

两者同时使用

注:disable="true"TOC标记在您的主标记文件中和远程内容中都起作用。

<!-- This will be processed and fetch content -->
<!-- Failed to fetch content from https://example.com/README.md: HTTP 404: Not Found -->

<!-- This will NOT be processed - useful for documentation -->
<!-- Failed to fetch content from https://example.com/README.md: HTTP 404: Not Found -->

如果您从 GitHub 中获取包含

,它将自动从文档的标题标题中生成目录(假设您添加了
在输油管上)。
链接转换

当获取远程标记时( 特别是从 GitHub) , 相对链接会断开。<fetch>扩展名可自动重写 :<fetch-summary>此变换 :

<!-- No fetch data available for https://example.com/api/status.md -->

保存绝对 URL 和锁定

执行时使用 Markdig AST 重写链接 :

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddPostgresMarkdownFetch(connectionString);

        var sp = services.BuildServiceProvider();
        var eventPublisher = sp.GetRequiredService<IMarkdownFetchEventPublisher>();

        // Subscribe to events
        eventPublisher.FetchBeginning += (sender, args) =>
        {
            Console.WriteLine($"Fetching {args.Url}...");
        };

        eventPublisher.FetchCompleted += (sender, args) =>
        {
            var source = args.WasCached ? "cache" : "remote";
            Console.WriteLine($"Fetched {args.Url} from {source} in {args.Duration.TotalMilliseconds}ms");
        };

        eventPublisher.FetchFailed += (sender, args) =>
        {
            Console.WriteLine($"Failed to fetch {args.Url}: {args.ErrorMessage}");
        };
    }
}

获取元数据摘要

sequenceDiagram
    participant MD as Markdown Processor
    participant EP as Event Publisher
    participant FS as Fetch Service
    participant ST as Storage Backend
    participant L as Your Listeners

    MD->>EP: FetchBeginning
    EP->>L: Notify FetchBeginning
    EP->>FS: FetchMarkdownAsync(url)
    FS->>ST: Check Cache
    alt Cache Fresh
        ST-->>FS: Cached Content
        FS->>EP: FetchCompleted (cached=true)
    else Cache Stale/Missing
        FS->>FS: HTTP GET
        alt Success
            FS->>ST: Update Cache
            ST-->>FS: OK
            FS->>EP: FetchCompleted (cached=false)
        else Failure
            FS->>EP: FetchFailed
            EP->>L: Notify FetchFailed
        end
    end
    EP->>L: Notify FetchCompleted
    EP->>L: Notify ContentUpdated
    FS-->>MD: MarkdownFetchResult

    Note over L: Listeners can be: Logging, Metrics, Telemetry, Webhooks

您可以在上次获取内容时显示读者 :

以页脚制成 :

stateDiagram-v2
    [*] --> CheckCache: Fetch Request

    CheckCache --> Fresh: Cache exists & age < pollFrequency
    CheckCache --> Stale: Cache exists & age >= pollFrequency
    CheckCache --> Missing: No cache entry

    Fresh --> ReturnCached: Return cached content
    ReturnCached --> [*]

    Stale --> FetchRemote: Attempt HTTP GET
    Missing --> FetchRemote: Attempt HTTP GET

    FetchRemote --> UpdateCache: Success
    FetchRemote --> HasStale: Failure

    UpdateCache --> ReturnFresh: Return new content
    ReturnFresh --> [*]

    HasStale --> ReturnStale: Return stale cache
    HasStale --> ReturnError: No cache available

    ReturnStale --> [*]
    ReturnError --> [*]

    note right of Fresh
        pollFrequency = 0
        means always stale
    end note

    note right of HasStale
        Stale-while-revalidate
        pattern ensures uptime
    end note

内容从

**网站:http://api.example.com/status.md。**2025年1月6日(2小时前)
**或者自定义模板 :**产出:__________________________________________________________________________
最新更新时间:2025年1月6日 2025年1月6日 2.30 状态:缓存下次更新:22小时后可用的占位符 :
- 最后获取日期/时间- 取回后人类可读时间

来源 URL- 内容何时更新- 时数缓存持续时间

- 缓存状态(重置/缓存/缓存)

禁用文件处理

public class MarkdownController : Controller
{
    private readonly IMarkdownFetchService _fetchService;

    public async Task<IActionResult> InvalidateCache(string url)
    {
        var removed = await _fetchService.RemoveCachedMarkdownAsync(url);

        if (removed)
        {
            return Ok(new { message = "Cache invalidated" });
        }

        return NotFound(new { message = "No cache entry found" });
    }
}

写入获取扩展名的文档( 如此文章 !) 时, 您需要一种方法来显示标签, 而不进行处理。

// GitHub webhook notifies of README update
app.MapPost("/webhooks/github", async (
    GitHubWebhookPayload payload,
    IMarkdownFetchService fetchService) =>
{
    if (payload.Repository?.FullName == "user/repo" &&
        payload.Commits?.Any(c => c.Modified?.Contains("README.md") == true) == true)
    {
        var url = "https://raw.githubusercontent.com/user/repo/main/README.md";
        await fetchService.RemoveCachedMarkdownAsync(url);

        return Results.Ok(new { message = "Cache invalidated" });
    }

    return Results.Ok(new { message = "No action needed" });
});

使用

属性 :

public class MarkdownFetchServiceTests
{
    [Fact]
    public async Task FetchMarkdownAsync_CachesContent()
    {
        // Arrange
        var services = new ServiceCollection();
        services.AddLogging();
        services.AddInMemoryMarkdownFetch();
        var sp = services.BuildServiceProvider();

        var fetchService = sp.GetRequiredService<IMarkdownFetchService>();
        var url = "https://raw.githubusercontent.com/user/repo/main/README.md";

        // Act - First fetch (from network)
        var result1 = await fetchService.FetchMarkdownAsync(url, 24, 0);

        // Act - Second fetch (from cache)
        var result2 = await fetchService.FetchMarkdownAsync(url, 24, 0);

        // Assert
        Assert.True(result1.Success);
        Assert.True(result2.Success);
        Assert.Equal(result1.Content, result2.Content);
    }

    [Fact]
    public async Task FetchMarkdownAsync_ReturnsStaleOnFailure()
    {
        // Arrange
        var services = new ServiceCollection();
        services.AddLogging();
        services.AddInMemoryMarkdownFetch();
        var sp = services.BuildServiceProvider();

        var fetchService = sp.GetRequiredService<IMarkdownFetchService>();
        var url = "https://httpstat.us/200?sleep=100";

        // Act - First fetch succeeds
        var result1 = await fetchService.FetchMarkdownAsync(url, 0, 0);

        // Change URL to fail
        var badUrl = "https://httpstat.us/500";

        // Act - Second fetch fails, should return stale
        var result2 = await fetchService.FetchMarkdownAsync(badUrl, 0, 0);

        // Assert
        Assert.True(result1.Success);
        // Even though fetch failed, we return success with stale content
        Assert.True(result2.Success);
    }
}

禁用标签仍保留在按原样标记的标记中, 完美适用于 :

写入关于扩展名本身的文档

在教义中创建示例显示标签语法而不触发抓取ConcurrentDictionary这对两者都管用
和标签 :
监测事件系统扩展号发布所有抓取操作的事件 :
**这样很容易与应用程序透视、 Prometheus 或您的伐木基础设施整合:**详细缓存战略IHttpClientFactory缓冲行为遵循国家机器模式:
**这里的关键洞察力是:**新鲜缓存新缓存

立即返回,没有网络被击中
Stale 缓存缓存
- 尝试获取新鲜, 但如果抓取失败, 倒回到老旧
缺少缓存
- 必须获取或返回错误

零彩票频率

总是获取新鲜(可用于测试)

name: Publish Markdig.FetchExtension

on:
  push:
    tags:
      - 'fetchextension-v*.*.*'

permissions:
  id-token: write
  contents: read

jobs:
  build-and-publish:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Setup .NET
      uses: actions/setup-dotnet@v4
      with:
        dotnet-version: '9.0.x'

    - name: Extract version from tag
      id: get_version
      run: |
        TAG=${GITHUB_REF#refs/tags/fetchextension-v}
        echo "VERSION=$TAG" >> $GITHUB_OUTPUT

    - name: Build
      run: dotnet build Mostlylucid.Markdig.FetchExtension/Mostlylucid.Markdig.FetchExtension.csproj --configuration Release -p:Version=${{ steps.get_version.outputs.VERSION }}

    - name: Pack
      run: dotnet pack Mostlylucid.Markdig.FetchExtension/Mostlylucid.Markdig.FetchExtension.csproj --configuration Release --no-build -p:PackageVersion=${{ steps.get_version.outputs.VERSION }} --output ./artifacts

    - name: Login to NuGet (OIDC)
      id: nuget_login
      uses: NuGet/login@v1
      with:
        user: 'mostlylucid'

    - name: Publish to NuGet
      run: dotnet nuget push ./artifacts/*.nupkg --api-key ${{ steps.nuget_login.outputs.NUGET_API_KEY }} --source https://api.nuget.org/v3/index.json --skip-duplicate

此模式被称为

静态时的重新校验

这对可靠性来说是极好的。

# My NuGet Package Documentation

Here's the official README from GitHub:

# Umami.Net

## UmamiClient

This is a .NET Core client for the Umami tracking API.
It's based on the Umami Node client, which can be found [here](https://github.com/umami-software/node).

You can see how to set up Umami as a docker
container [here](https://www.mostlylucid.net/blog/usingumamiforlocalanalytics).
You can read more detail about it's creation on my
blog [here](https://www.mostlylucid.net/blog/addingumamitrackingclientfollowup).

To use this client you need the following appsettings.json configuration:

```json
{
  "Analytics":{
    "UmamiPath" : "https://umamilocal.mostlylucid.net",
    "WebsiteId" : "32c2aa31-b1ac-44c0-b8f3-ff1f50403bee"
  },
}

Where UmamiPath is the path to your Umami instance and WebsiteId is the id of the website you want to track.

To use the client you need to add the following to your Program.cs:

using Umami.Net;

services.SetupUmamiClient(builder.Configuration);

This will add the Umami client to the services collection.

You can then use the client in two ways:

Track

Inject the UmamiClient into your class and call the Track method:

 // Inject UmamiClient umamiClient
 await umamiClient.Track("Search", new UmamiEventData(){{"query", encodedQuery}});

Use the UmamiBackgroundSender to track events in the background (this uses an IHostedService to send events in the background):

 // Inject UmamiBackgroundSender umamiBackgroundSender
await umamiBackgroundSender.Track("Search", new UmamiEventData(){{"query", encodedQuery}});

The client will send the event to the Umami API and it will be stored.

The UmamiEventData is a dictionary of key value pairs that will be sent to the Umami API as the event data.

There are additionally more low level methods that can be used to send events to the Umami API.

Track PageView

There's also a convenience method to track a page view. This will send an event to the Umami API with the url set (which counts as a pageview).

  await  umamiBackgroundSender.TrackPageView("api/search/" + encodedQuery, "searchEvent", eventData: new UmamiEventData(){{"query", encodedQuery}});
  
   await umamiClient.TrackPageView("api/search/" + encodedQuery, "searchEvent", eventData: new UmamiEventData(){{"query", encodedQuery}});

Here we're setting the url to "api/search/" + encodedQuery and the event type to "searchEvent". We're also passing in a dictionary of key value pairs as the event data.

Raw 'Send' method

On both the UmamiClient and UmamiBackgroundSender you can call the following method.



 Send(UmamiPayload? payload = null, UmamiEventData? eventData = null,
        string eventType = "event")

If you don't pass in a UmamiPayload object, the client will create one for you using the WebsiteId from the appsettings.json.

    public  UmamiPayload GetPayload(string? url = null, UmamiEventData? data = null)
    {
        var httpContext = httpContextAccessor.HttpContext;
        var request = httpContext?.Request;

        var payload = new UmamiPayload
        {
            Website = settings.WebsiteId,
            Data = data,
            Url = url ?? httpContext?.Request?.Path.Value,
            IpAddress = httpContext?.Connection?.RemoteIpAddress?.ToString(),
            UserAgent = request?.Headers["User-Agent"].FirstOrDefault(),
            Referrer = request?.Headers["Referer"].FirstOrDefault(),
           Hostname = request?.Host.Host,
        };
        
        return payload;
    }

You can see that this populates the UmamiPayload object with the WebsiteId from the appsettings.json, the Url, IpAddress, UserAgent, Referrer and Hostname from the HttpContext.

NOTE: eventType can only be "event" or "identify" as per the Umami API.

UmamiData

There's also a service that can be used to pull data from the Umami API. This is a service that allows me to pull data from my Umami instance to use in stuff like sorting posts by popularity etc...

To set it up you need to add a username and password for your umami instance to the Analytics element in your settings file:

    "Analytics":{
        "UmamiPath" : "https://umami.mostlylucid.net",
        "WebsiteId" : "1e3b7657-9487-4857-a9e9-4e1920aa8c42",
        "UserName": "admin",
        "Password": ""
     
    }

Then in your Program.cs you set up the UmamiDataService as follows:

    services.SetupUmamiData(config);

You can then inject the UmamiDataService into your class and use it to pull data from the Umami API.

Usage

Now you have the UmamiDataService in your service collection you can start using it!

Methods

The methods are all from the Umami API definition you can read about them here: https://umami.is/docs/api/website-stats

All returns are wrapped in an UmamiResults<T> object which has a Success property and a Result property. The Result property is the object returned from the Umami API.

public record UmamiResult<T>(HttpStatusCode Status, string Message, T? Data);

All requests apart from ActiveUsers have a base request object with two compulsory properties. I added convenience DateTimes to the base request object to make it easier to set the start and end dates.

public class BaseRequest
{
    [QueryStringParameter("startAt", isRequired: true)]
    public long StartAt => StartAtDate.ToMilliseconds(); // Timestamp (in ms) of starting date
    [QueryStringParameter("endAt", isRequired: true)]
    public long EndAt => EndAtDate.ToMilliseconds(); // Timestamp (in ms) of end date
    public DateTime StartAtDate { get; set; }
    public DateTime EndAtDate { get; set; }
}

The service has the following methods:

Active Users

This just gets the total number of CURRENT active users on the site

public async Task<UmamiResult<ActiveUsersResponse>> GetActiveUsers()

Stats

This returns a bunch of statistics about the site, including the number of users, page views, etc.

public async Task<UmamiResult<StatsResponseModels>> GetStats(StatsRequest statsRequest)

You may set a number of parameters to filter the data returned from the API. For instance using url will return the stats for a specific URL.

StatsRequest object

public class StatsRequest : BaseRequest
{
    [QueryStringParameter("url")]
    public string? Url { get; set; } // Name of URL
    
    [QueryStringParameter("referrer")]
    public string? Referrer { get; set; } // Name of referrer
    
    [QueryStringParameter("title")]
    public string? Title { get; set; } // Name of page title
    
    [QueryStringParameter("query")]
    public string? Query { get; set; } // Name of query
    
    [QueryStringParameter("event")]
    public string? Event { get; set; } // Name of event
    
    [QueryStringParameter("host")]
    public string? Host { get; set; } // Name of hostname
    
    [QueryStringParameter("os")]
    public string? Os { get; set; } // Name of operating system
    
    [QueryStringParameter("browser")]
    public string? Browser { get; set; } // Name of browser
    
    [QueryStringParameter("device")]
    public string? Device { get; set; } // Name of device (e.g., Mobile)
    
    [QueryStringParameter("country")]
    public string? Country { get; set; } // Name of country
    
    [QueryStringParameter("region")]
    public string? Region { get; set; } // Name of region/state/province
    
    [QueryStringParameter("city")]
    public string? City { get; set; } // Name of city
}

The JSON object Umami returns is as follows.

{
  "pageviews": { "value": 5, "change": 5 },
  "visitors": { "value": 1, "change": 1 },
  "visits": { "value": 3, "change": 2 },
  "bounces": { "value": 0, "change": 0 },
  "totaltime": { "value": 4, "change": 4 }
}

This is wrapped inside my StatsResponseModel object.

namespace Umami.Net.UmamiData.Models.ResponseObjects;

public class StatsResponseModels
{
    public Pageviews pageviews { get; set; }
    public Visitors visitors { get; set; }
    public Visits visits { get; set; }
    public Bounces bounces { get; set; }
    public Totaltime totaltime { get; set; }


    public class Pageviews
    {
        public int value { get; set; }
        public int prev { get; set; }
    }

    public class Visitors
    {
        public int value { get; set; }
        public int prev { get; set; }
    }

    public class Visits
    {
        public int value { get; set; }
        public int prev { get; set; }
    }

    public class Bounces
    {
        public int value { get; set; }
        public int prev { get; set; }
    }

    public class Totaltime
    {
        public int value { get; set; }
        public int prev { get; set; }
    }
}

Metrics

Metrics in Umami provide you the number of views for specific types of properties.

Events

One example of these is Events`:

'Events' in Umami are specific items you can track on a site. When tracking events using Umami.Net you can set a number of properties which are tracked with the event name. For instance here I track Search requests with the URL and the search term.

       await  umamiBackgroundSender.Track( "searchEvent", eventData: new UmamiEventData(){{"query", encodedQuery}});

To fetch data about this event you would use the Metrics method:

public async Task<UmamiResult<MetricsResponseModels[]>> GetMetrics(MetricsRequest metricsRequest)

As with the other methods this accepts the MetricsRequest object (with the compulsory BaseRequest properties) and a number of optional properties to filter the data.

MetricsRequest object

public class MetricsRequest : BaseRequest
{
    [QueryStringParameter("type", isRequired: true)]
    public MetricType Type { get; set; } // Metrics type

    [QueryStringParameter("url")]
    public string? Url { get; set; } // Name of URL
    
    [QueryStringParameter("referrer")]
    public string? Referrer { get; set; } // Name of referrer
    
    [QueryStringParameter("title")]
    public string? Title { get; set; } // Name of page title
    
    [QueryStringParameter("query")]
    public string? Query { get; set; } // Name of query
    
    [QueryStringParameter("host")]
    public string? Host { get; set; } // Name of hostname
    
    [QueryStringParameter("os")]
    public string? Os { get; set; } // Name of operating system
    
    [QueryStringParameter("browser")]
    public string? Browser { get; set; } // Name of browser
    
    [QueryStringParameter("device")]
    public string? Device { get; set; } // Name of device (e.g., Mobile)
    
    [QueryStringParameter("country")]
    public string? Country { get; set; } // Name of country
    
    [QueryStringParameter("region")]
    public string? Region { get; set; } // Name of region/state/province
    
    [QueryStringParameter("city")]
    public string? City { get; set; } // Name of city
    
    [QueryStringParameter("language")]
    public string? Language { get; set; } // Name of language
    
    [QueryStringParameter("event")]
    public string? Event { get; set; } // Name of event
    
    [QueryStringParameter("limit")]
    public int? Limit { get; set; } = 500; // Number of events returned (default: 500)
}

Here you can see that you can specify a number of properties in the request element to specify what metrics you want to return.

You can also set a Limit property to limit the number of results returned.

For instance to get the event over the past day I mentioned above you would use the following request:

var metricsRequest = new MetricsRequest
{
    StartAtDate = DateTime.Now.AddDays(-1),
    EndAtDate = DateTime.Now,
    Type = MetricType.@event,
    Event = "searchEvent"
};

The JSON object returned from the API is as follows:

[
  { "x": "searchEvent", "y": 46 }
]

And again I wrap this in my MetricsResponseModels object.

public class MetricsResponseModels
{
    public string x { get; set; }
    public int y { get; set; }
}

Where x is the event name and y is the number of times it has been triggered.

Page Views

One of the most useful metrics is the number of page views. This is the number of times a page has been viewed on the site. Below is the test I use to get the number of page views over the past 30 days. You'll note the Type parameter is set as MetricType.url however this is also the default value so you don't need to set it.

  [Fact]
    public async Task Metrics_StartEnd()
    {
        var setup = new SetupUmamiData();
        var serviceProvider = setup.Setup();
        var websiteDataService = serviceProvider.GetRequiredService<UmamiDataService>();
        
        var metrics = await websiteDataService.GetMetrics(new MetricsRequest()
        {
            StartAtDate = DateTime.Now.AddDays(-30),
            EndAtDate = DateTime.Now,
            Type = MetricType.url,
            Limit = 500
        });
        Assert.NotNull(metrics);
        Assert.Equal( HttpStatusCode.OK, metrics.Status);

    }

This returns a MetricsResponse object which has the following JSON structure:

[
  {
    "x": "/",
    "y": 1
  },
  {
    "x": "/blog",
    "y": 1
  },
  {
    "x": "/blog/usingumamidataforwebsitestats",
    "y": 1
  }
]

Where x is the URL and y is the number of times it has been viewed.

PageViews

This returns the number of page views for a specific URL.

Again here is a test I use for this method:

    [Fact]
    public async Task PageViews_StartEnd_Day_Url()
    {
        var setup = new SetupUmamiData();
        var serviceProvider = setup.Setup();
        var websiteDataService = serviceProvider.GetRequiredService<UmamiDataService>();
    
        var pageViews = await websiteDataService.GetPageViews(new PageViewsRequest()
        {
            StartAtDate = DateTime.Now.AddDays(-7),
            EndAtDate = DateTime.Now,
            Unit = Unit.day,
            Url = "/blog"
        });
        Assert.NotNull(pageViews);
        Assert.Equal( HttpStatusCode.OK, pageViews.Status);

    }

This returns a PageViewsResponse object which has the following JSON structure:

[
  {
    "date": "2024-09-06 00:00",
    "value": 1
  }
]

Where date is the date and value is the number of page views, this is repeated for each day in the range specified ( or hour, month, etc. depending on the Unit property).

As with the other methods this accepts the PageViewsRequest object (with the compulsory BaseRequest properties) and a number of optional properties to filter the data.

PageViewsRequest object

public class PageViewsRequest : BaseRequest
{
    // Required properties

    [QueryStringParameter("unit", isRequired: true)]
    public Unit Unit { get; set; } = Unit.day; // Time unit (year | month | hour | day)
    
    [QueryStringParameter("timezone")]
    [TimeZoneValidator]
    public string Timezone { get; set; }

    // Optional properties
    [QueryStringParameter("url")]
    public string? Url { get; set; } // Name of URL
    [QueryStringParameter("referrer")]
    public string? Referrer { get; set; } // Name of referrer
    [QueryStringParameter("title")]
    public string? Title { get; set; } // Name of page title
    [QueryStringParameter("host")]
    public string? Host { get; set; } // Name of hostname
    [QueryStringParameter("os")]
    public string? Os { get; set; } // Name of operating system
    [QueryStringParameter("browser")]
    public string? Browser { get; set; } // Name of browser
    [QueryStringParameter("device")]
    public string? Device { get; set; } // Name of device (e.g., Mobile)
    [QueryStringParameter("country")]
    public string? Country { get; set; } // Name of country
    [QueryStringParameter("region")]
    public string? Region { get; set; } // Name of region/state/province
    [QueryStringParameter("city")]
    public string? City { get; set; } // Name of city
}

As with the other methods you can set a number of properties to filter the data returned from the API, for instance you could set the Country property to get the number of page views from a specific country.

Using the Service

In this site I have some code which lets me use this service to get the number of views each blog page has. In the code below I take a start and end date and a prefix (which is /blog in my case) and get the number of views for each page in the blog.

I then cache this data for an hour so I don't have to keep hitting the Umami API.

public class UmamiDataSortService(
    UmamiDataService dataService,
    IMemoryCache cache)
{
    public async Task<List<MetricsResponseModels>?> GetMetrics(DateTime startAt, DateTime endAt, string prefix="" )
    {
        using var activity = Log.Logger.StartActivity("GetMetricsWithPrefix");
        try
        {
            var cacheKey = $"Metrics_{startAt}_{endAt}_{prefix}";
            if (cache.TryGetValue(cacheKey, out List<MetricsResponseModels>? metrics))
            {
                activity?.AddProperty("CacheHit", true);
                return metrics;
            }
            activity?.AddProperty("CacheHit", false);
            var metricsRequest = new MetricsRequest()
            {
                StartAtDate = startAt,
                EndAtDate = endAt,
                Type = MetricType.url,
                Limit = 500
            };
            var metricRequest = await dataService.GetMetrics(metricsRequest);

            if(metricRequest.Status != HttpStatusCode.OK)
            {
                return null;
            }
            var filteredMetrics = metricRequest.Data.Where(x => x.x.StartsWith(prefix)).ToList();
            cache.Set(cacheKey, filteredMetrics, TimeSpan.FromHours(1));
            activity?.AddProperty("MetricsCount", filteredMetrics?.Count()?? 0);
            activity?.Complete();
            return filteredMetrics;
        }
        catch (Exception e)
        {
            activity?.Complete(LogEventLevel.Error, e);
         
            return null;
        }
    }

Installation

The package is available on NuGet...


即使你的线人倒下,你的网站 仍然提供隐藏的内容。

# 缓存清除和管理

有时您需要手动手动无效缓存 :

1. **或者在内容改变时通过 Webhooks :**延长期测试
2. **延长期包括全面测试。**这是我如何构建它们的:
3. **绩效考量**扩展部分的设计是为了达到以下性能:
4. **同时词典**- 初步实施用途
5. **用于线索安全访问**夏马磷Slim
6. **- 防止种族条件的基于文件的使用同步锁定以防止种族条件**数据库指数

- 所有数据库提供者都有关于缓存密钥的适当索引

- [HTTP 客户共享](https://www.nuget.org/packages/mostlylucid.Markdig.FetchExtension)
- [- 用途](https://www.nuget.org/packages/mostlylucid.Markdig.FetchExtension.Postgres)
- [高效连接再利用](https://www.nuget.org/packages/mostlylucid.Markdig.FetchExtension.Sqlite)
- [Async Async 一路前进](https://www.nuget.org/packages/mostlylucid.Markdig.FetchExtension.SqlServer)

- 没有阻断电话,一切都是静音[我家里服务器上的典型性能数字:](https://github.com/scottgal/mostlylucidweb/tree/main/Mostlylucid.Markdig.FetchExtension)

缓存命中: < 1ms

为 Markdig 建立一个远程的 Markdown 抓取器

一. 导言 导言 导言 导言 导言 导言 一,导言 导言 导言 导言 导言 导言

在标记下自动从文档标题中生成可点击的目录。

最天真的做法是使用 HTTP 客户端在需要时获得标记。

预处理预处理

然后让Markdig 一起处理一切

在您的标记中 :

从 GitHub 获取 README

简单而干净。

成功时, 更新缓存

- 对可靠性至关重要。

对于单一服务器的部署,基于文件的储存工作非常出色:

用途

这就是数据库提供者来的地方:

数据库模式很简单:

“API参考”____________________________________________________________________________________________________________________________________________________________________________________

当配置您的 Markdig 管道时, 添加 TOC 扩展 :

两者同时使用

保存绝对 URL 和锁定

您可以在上次获取内容时显示读者 :

- 缓存状态(重置/缓存/缓存)

使用

禁用标签仍保留在按原样标记的标记中, 完美适用于 :

零彩票频率

静态时的重新校验

Track

Track PageView

Raw 'Send' method

UmamiData

Usage

Methods

Active Users

Stats

Metrics

Events

Page Views

PageViews

Using the Service

Installation

一. 导言导言导言导言导言导言一,导言导言导言导言导言导言