assets.md

Adding new builtin languages for syntax highlighting

Should you find that a particular syntax is not available within bat and think it should be included in bat by default, you can follow the instructions outlined below.

bat uses the excellent syntect library to highlight source code. As a basis, syntect uses Sublime Text syntax definitions in the .sublime-syntax format.

Important: Before proceeding, verify that the syntax you wish to add meets the criteria for inclusion.

1. Find a Sublime Text syntax for the given language, preferably in a separate Git repository which can be included as a submodule (under assets/syntaxes).

2. If the Sublime Text syntax is only available as a .tmLanguage file, open the file in Sublime Text and convert it to a .sublime-syntax file via Tools -> Developer -> New Syntax from XXX.tmLanguage.... Save the new file in the assets/syntaxes folder.

3. Run the assets/create.sh script. It calls bat cache --build to parse all available .sublime-syntax files and serialize them to a syntaxes.bin file.

4. Re-compile bat. At compilation time, the syntaxes.bin file will be stored inside the bat binary.

5. Use bat --list-languages to check if the new languages are available.

6. Add a syntax test for the new language. See below for details.

7. If you send a pull request with your changes, please do not include the changed syntaxes.bin file. A new binary cache file will be created once before every new release of bat.

Syntax tests

bat has a set of syntax highlighting regression tests in tests/syntax-tests. The main idea is make sure that we do not run into issues we had in the past where either (1) syntax highlighting for some language is suddenly not working anymore or (2) bat suddenly crashes for some input (due to regex incompatibilities between syntect and Sublime Text).

In order to add a new test file, please follow these steps (let's take "Ruby" as an example):

1. Make sure that you are running the latest version of bat and that bat is available on the path. If you are creating a syntax test for a new builtin syntax (see above), make sure that your version of bat already has the new syntax builtin.
2. Find an example Ruby source file or write one yourself. If possible, the file should aim to be "comprehensive" (i.e. include a lot of the possible syntax), but this is not strictly necessary. A simple file is better than none at all. Also, the files shouldn't be gigantic.
3. Save the file in tests/syntax-tests/source/Ruby (adapt for your language). The file name could be test.rb (adapt extension) but can also be adapted if that is necessary in order for bat to highlight it correctly (e.g. Makefile).
4. If you have copied the file from somewhere else, please make sure that the file may be copied under the respective license and that the license is compatible with bats license. If it requires attribution, please add a LICENSE.md in the same folder with a text like this:

   The `test.rb` file has been added from [enter source here] under the following license:
   
   [add license text here]
   

5. Go to tests/syntax-tests and run the update.sh Bash script. A new file should be generated in the highlighted folder (e.g. highlighted/Ruby/test.rb).
6. Use cat or bat --language=txt to display the content of this file and make sure that the syntax highlighting looks correct.
7. git add the new files in the source folder as well as the autogenerated files in the highlighted folder.

Troubleshooting

Make sure that the local cache does not interfere with the internally stored syntaxes and themes (bat cache --clear).

Criteria for inclusion of new syntaxes

• More than 10,000 downloads on packagecontrol.io/

Manual modifications

The following files have been manually modified after converting from a .tmLanguage file:

• Apache.sublime_syntax=> removed .conf and .CONF file types.
• Dart.sublime-syntax => removed #regex.dart include.
• INI.sublime-syntax => added .hgrc, hgrc, and desktop file types and support for comments after section headers
• Org mode.sublime-syntax => removed task file type.
• SML.sublime_syntax => removed ml file type.
• Robot.sublime_syntax => changed name to "Robot Framework", added .resource extension

Non-submodule additions

• Assembly (x86_64) has been manually added from https://github.com/13xforever/x86-assembly-textmate-bundle due to git clone recursion problems
• Nim.sublime-syntax has been added manually from https://github.com/getzola/zola/blob/master/sublime_syntaxes/Nim.sublime-syntax as there was no suitable Git repository for it. The original syntax seems to originate from https://github.com/Varriount/NimLime
• Rego.sublime-syntax has been added manually from https://github.com/open-policy-agent/opa/blob/master/misc/syntax/sublime/rego.sublime-syntax as it is not kept in a standalone repository. The file is generated from https://github.com/open-policy-agent/opa/blob/master/misc/syntax/textmate/Rego.tmLanguage
• SML.sublime_syntax has been added manually from https://github.com/seanjames777/SML-Language-Definitiona as it is not kept in a standalone repository. The file generated is from https://github.com/seanjames777/SML-Language-Definition/blob/master/sml.tmLanguage
• Cabal.sublime_syntax has been added manually from https://github.com/SublimeHaskell/SublimeHaskell/ - we don't want to include the whole submodule because it includes other syntaxes ("Haskell improved") as well.
• Lean.sublime-syntax has been added manually from https://github.com/leanprover/vscode-lean/blob/master/syntaxes/lean.json via conversion.