Huge hardcoded character lists have a cost. In some current lexers, such
as Haskell and JavaScript, this bloats lexer init time to 60-80ms on
some current systems, as opposed to sub-ms for many others.
Replacing them with character classes such as `\p{L}` seems to
eliminate this cost, reducing lexer init time to the norm (around 1ms).
In addition, this significantly reduces and simplifies the code.
The current tests pass, but there may be inaccuracies not covered by
tests. This requires a review.
This change is likely to cause edge case regressions, as the sets of
characters considered "letters" vary between languages. However, Chroma
lexers don't aim to be perfectly accurate. Performance should be just as
much a goal as accuracy. I believe this tradeoff to be justified.
This commit leaves at least two lexers unfixed: Julia and Kotlin.
Judging by the code, they might have the same issue, and should also be
addressed.
Add NewLazyLexer and MustNewLazyLexer which accept a function that
returns the rules for the lexer. This allows us to defer the rules
definitions until they're needed.
Lexers in a, g, s, and x packages have been updated to use the new lazy
lexer.
Using these examples:
~~~
0x21
1_000
1e3
~~~
Several languages failed that support these syntax. Here are ones I found:
Lexer | 0x21 | 1_000 | 1e3
-----------|------|-------|-----
C# | fail | fail | fail
Go | | fail |
JavaScript | | fail | fail
PHP | | fail |
Python | | fail |
Ruby | | | fail
I fixed these issues, and added tests.
This was done to speed up incremental compilation when working on
lexers. That is, modifying a single lexer will no longer require
recompiling all lexers.
This is a (slightly) backwards breaking change in that lexers are no
longer exported directly in the lexers package. The registry API is
"aliased" at the old location.
