NTokenizers

A powerful and flexible collection of stream-based tokenizers designed for high-performance processing of XML, JSON, HTML, YAML, Markdown, TypeScript, CSS, C# and SQL.

GitHub NuGet Docs License

Examples

Parse a stream

await [Language]Tokenizer.Create().ParseAsync(stream, onToken: async token =>
{
    // Handle tokens as they arrive
});

Markdown to HTML

// Fragment (body-only)
string html = MarkdownConverter.ToHtml(markdown);

// Stream-to-stream conversion
await MarkdownConverter.WriteHtmlDocumentAsync(inputStream, writer);

Full example with Markdown parsing

string markdown = """
Here is some **bold** text and some *italic* text.

# NTokenizers Showcase

## Css example
```css
.user {
    color: #FFFFFF;
}
```
""";

using var pipe = new AnonymousPipeServerStream(PipeDirection.Out);
using var reader = new AnonymousPipeClientStream(PipeDirection.In, pipe.ClientSafePipeHandle);

var writerTask = EmitSlowlyAsync(markdown, pipe);

await MarkdownTokenizer.Create().ParseAsync(reader, onToken: async token =>
{
    if (token.Metadata is ICodeBlockMetadata codeBlock)
    {
        AnsiConsole.WriteLine();
        AnsiConsole.Write(new Markup($"[bold lime]{codeBlock.Language}:[/]"));
        AnsiConsole.WriteLine();
    }
    // ... handle other token types
});

await writerTask;