1
0
mirror of https://github.com/alecthomas/chroma.git synced 2025-03-17 20:58:08 +02:00
Mikhail Sorochan 3044bf5f32
Go lexer: single line comment without consuming endline, disable EnsureNL (#984)
This PR changes `CommentSingle` to not consume the newline at the end as
a part of comment.
That solves the problems of single line comment being not parsed at the
end of the line or at the end of the file. Which was reported earlier as
the reason to not highlight single line comment properly.

Disabling `EnsureNL: true` does not add unnecessary newline element for
`Text`, `CommentSymbol` symbols. Using chroma in console with syntax
highlighting was unusable becasue of this, since typing e.g. `b := `
adds newline each time space is at the end when host app asks for
highlighted text from `quick`.

Tokens behavior:
<table>
<tr>
<td> Before </td> <td> After </td>
</tr>
<tr>
<td>

``` go
t.Run("Single space", func(t *testing.T) {
        tokens, _ := chroma.Tokenise(Go, nil, " ")
        expected := []chroma.Token{
                {chroma.Text, " \n"},
        }
        assert.Equal(t, expected, tokens)
})
t.Run("Assignment unfinished", func(t *testing.T) {
        tokens, _ := chroma.Tokenise(Go, nil, "i = ")
        expected := []chroma.Token{
                { chroma.NameOther, "i" },
                { chroma.Text, " " },
                { chroma.Punctuation, "=" },
                { chroma.Text, " \n" },
        }
        assert.Equal(t, expected, tokens)
})
t.Run("Single comment", func(t *testing.T) {
        tokens, _ := chroma.Tokenise(Go, nil, "// W")
        expected := []chroma.Token{
                { chroma.CommentSingle, "// W\n" },
        }
        assert.Equal(t, expected, tokens)
})
```

</td>
<td>
    
``` go
t.Run("Single space", func(t *testing.T) {
        tokens, _ := chroma.Tokenise(Go, nil, " ")
        expected := []chroma.Token{
                {chroma.Text, " "},
        }
        assert.Equal(t, expected, tokens)
})
t.Run("Assignment unfinished", func(t *testing.T) {
        tokens, _ := chroma.Tokenise(Go, nil, "i = ")
        expected := []chroma.Token{
                { chroma.NameOther, "i" },
                { chroma.Text, " " },
                { chroma.Punctuation, "=" },
                { chroma.Text, " " },
        }
        assert.Equal(t, expected, tokens)
})
t.Run("Single comment", func(t *testing.T) {
        tokens, _ := chroma.Tokenise(Go, nil, "// W")
        expected := []chroma.Token{
                { chroma.CommentSingle, "// W" },
        }
        assert.Equal(t, expected, tokens)
})
```
</td>
</tr>
</table>
2024-07-23 02:19:08 +10:00
..
2024-07-12 05:24:31 +10:00
2022-01-27 15:22:00 +11:00
2022-06-27 01:18:15 -07:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00
2023-08-16 07:50:03 +10:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00
2022-01-27 15:22:00 +11:00

Chroma lexers

All lexers in Chroma should now be defined in XML unless they require custom code.

Lexer tests

The tests in this directory feed a known input testdata/<name>.actual into the parser for <name> and check that its output matches <name>.expected.

It is also possible to perform several tests on a same parser <name>, by placing know inputs *.actual into a directory testdata/<name>/.

Running the tests

Run the tests as normal:

go test ./lexers

Update existing tests

When you add a new test data file (*.actual), you need to regenerate all tests. That's how Chroma creates the *.expected test file based on the corresponding lexer.

To regenerate all tests, type in your terminal:

RECORD=true go test ./lexers

This first sets the RECORD environment variable to true. Then it runs go test on the ./lexers directory of the Chroma project.

(That environment variable tells Chroma it needs to output test data. After running go test ./lexers you can remove or reset that variable.)

Windows users

Windows users will find that the RECORD=true go test ./lexers command fails in both the standard command prompt terminal and in PowerShell.

Instead we have to perform both steps separately:

  • Set the RECORD environment variable to true.
    • In the regular command prompt window, the set command sets an environment variable for the current session: set RECORD=true. See this page for more.
    • In PowerShell, you can use the $env:RECORD = 'true' command for that. See this article for more.
    • You can also make a persistent environment variable by hand in the Windows computer settings. See this article for how.
  • When the environment variable is set, run go test ./lexers.

Chroma will now regenerate the test files and print its results to the console window.