This imports `pygments.lexers.scripting.LuauLexer` from Pygments.
Some notes:
- The generated XML file is not 100% valid as as some elements have 2
`state` attributes, but it does work
- I needed to tweak the import script to correctly convert the RegEx
- I needed to add `OperatorReserved` as a `TokenType`
- `.luaurc` is added as a filename to the JSON Lexer, as [it's used by
Luau](https://rfcs.luau.org/config-luaurc.html)
Implements #1062
**TL;DR: added a python script that generates a "supported languages"
markdown table, making it easier to keep the table in the readme up to
date**
---
This PR adds a Python script `_tools/format_supported_langs.py` which
accepts the output of `chroma --list` over stdin and outputs a Markdown
table which can be added to the Supported Languages section of the
README. Here's how I called the script:
```sh
env -C cmd/chroma go run . | uv run _tools/format_supported_langs.py
```
I chose not to try automating edits to the README, and leave it up to
the invoker to replace the table in the readme with this script's
output. I also updated the table in the README because it's missing a
few languages.
Hopefully this can help with the maintenance burden.
I also adapted the README to use `uv` since this removes an install step
(uv resolves and installs dependencies on `uv run`).
We could also debate if we should switch from `KotlinLexer` as an
example to some other lexer as there seems to be a bug upstream in
pygments where `KotlinLexer.get_tokendefs()` returns a string that is
not valid unicode. You can read more about the bug in [this
issue](https://github.com/pygments/pygments/issues/2964).
With argparse the error messages are a lot nicer and the script can
better explain itself.
```bash
uv run pygments2chroma_xml.py --help
usage: pygments2chroma_xml.py [-h] lexer_class
Converts pygments RegexLexer classes to chroma xml grammar definitions.
positional arguments:
lexer_class The class name of the pygments lexer, like: 'pygments.lexers.jvm.KotlinLexer'.
options:
-h, --help show this help message and exit
```
This cleans up the API in general, removing a bunch of deprecated stuff,
cleaning up circular imports, etc.
But the biggest change is switching to an optional XML format for the
regex lexer.
Having lexers defined only in Go is not ideal for a couple of reasons.
Firstly, it impedes a significant portion of contributors who use Chroma
in Hugo, but don't know Go. Secondly, it bloats the binary size of any
project that imports Chroma.
Why XML? YAML is an abomination and JSON is not human editable. XML
also compresses very well (eg. Go template lexer XML compresses from
3239 bytes to 718).
Why a new syntax format? All major existing formats rely on the
Oniguruma regex engine, which is extremely complex and for which there
is no Go port.
Why not earlier? Prior to the existence of fs.FS this was not a viable
option.
Benchmarks:
$ hyperfine --warmup 3 \
'./chroma.master --version' \
'./chroma.xml-pre-opt --version' \
'./chroma.xml --version'
Benchmark 1: ./chroma.master --version
Time (mean ± σ): 5.3 ms ± 0.5 ms [User: 3.6 ms, System: 1.4 ms]
Range (min … max): 4.2 ms … 6.6 ms 233 runs
Benchmark 2: ./chroma.xml-pre-opt --version
Time (mean ± σ): 50.6 ms ± 0.5 ms [User: 52.4 ms, System: 3.6 ms]
Range (min … max): 49.2 ms … 51.5 ms 51 runs
Benchmark 3: ./chroma.xml --version
Time (mean ± σ): 6.9 ms ± 1.1 ms [User: 5.1 ms, System: 1.5 ms]
Range (min … max): 5.7 ms … 19.9 ms 196 runs
Summary
'./chroma.master --version' ran
1.30 ± 0.23 times faster than './chroma.xml --version'
9.56 ± 0.83 times faster than './chroma.xml-pre-opt --version'
A slight increase in init time, but I think this is okay given the
increase in flexibility.
And binary size difference:
$ du -h lexers.test*
$ du -sh chroma* 951371ms
8.8M chroma.master
7.8M chroma.xml
7.8M chroma.xml-pre-opt
Benchmarks:
$ hyperfine --warmup 3 \
'./chroma.master --version' \
'./chroma.xml-pre-opt --version' \
'./chroma.xml --version'
Benchmark 1: ./chroma.master --version
Time (mean ± σ): 5.3 ms ± 0.5 ms [User: 3.6 ms, System: 1.4 ms]
Range (min … max): 4.2 ms … 6.6 ms 233 runs
Benchmark 2: ./chroma.xml-pre-opt --version
Time (mean ± σ): 50.6 ms ± 0.5 ms [User: 52.4 ms, System: 3.6 ms]
Range (min … max): 49.2 ms … 51.5 ms 51 runs
Benchmark 3: ./chroma.xml --version
Time (mean ± σ): 6.9 ms ± 1.1 ms [User: 5.1 ms, System: 1.5 ms]
Range (min … max): 5.7 ms … 19.9 ms 196 runs
Summary
'./chroma.master --version' ran
1.30 ± 0.23 times faster than './chroma.xml --version'
9.56 ± 0.83 times faster than './chroma.xml-pre-opt --version'
Incompatible changes:
- (*RegexLexer).SetAnalyser: changed from func(func(text string) float32) *RegexLexer to func(func(text string) float32) Lexer
- (*TokenType).UnmarshalJSON: removed
- Lexer.AnalyseText: added
- Lexer.SetAnalyser: added
- Lexer.SetRegistry: added
- MustNewLazyLexer: removed
- MustNewLexer: changed from func(*Config, Rules) *RegexLexer to func(*Config, func() Rules) *RegexLexer
- Mutators: changed from func(...Mutator) MutatorFunc to func(...Mutator) Mutator
- NewLazyLexer: removed
- NewLexer: changed from func(*Config, Rules) (*RegexLexer, error) to func(*Config, func() Rules) (*RegexLexer, error)
- Pop: changed from func(int) MutatorFunc to func(int) Mutator
- Push: changed from func(...string) MutatorFunc to func(...string) Mutator
- TokenType.MarshalJSON: removed
- Using: changed from func(Lexer) Emitter to func(string) Emitter
- UsingByGroup: changed from func(func(string) Lexer, int, int, ...Emitter) Emitter to func(int, int, ...Emitter) Emitter
Alec Thomas
JakobDev
Daniel Fichtinger
Florian Freitag
oliverpool
Jonathan
Forest0923
Sean Berry
Ville Skyttä
Edgar Lee