Add BOM Awareness for Syntax Test Files

[tajmone]

As of ST3 3157, syntax test files with a BOM fail to work because the BOM prevents ST from correctly parsing the first line. See my post:

ST3 3157: Syntax Tests Problems

I'm working on a new Syntax, currently I've only implemented a few syntax elements and they are working correctly — when I check the scopes wiht Ctrl-Alt-Shit-P they are as expected. So I thought of creating some syntax_test_* files, but when I open the test files in the editor I always get this error message in a popup window: Syntax tests do not work properly with tabs as indentation. You MUST use spaces! ... even though the source files don't contain any tabs. It seems that the problem …

Some languages’ native IDEs do add a BOM when they save source files. The UTF-8 specification allows adding a BOM, even if it’s discouraged.

Adding a BOM check and stripping function shouldn’t be problematic, and might solve lot’s of issues when working with shared code where some contributors use IDEs which add a BOM.

Also, it might be worth mentioning the issue in the documentation — I wen’t around in circles for quite a while before realizing the problem was due to the BOM presence!

[tajmone]

ST 3175 x86_64 | Win 10

The problem persists, when the BOM is present the tests don’t get executed because ST fails to detect the comments delimiter from first line.

Basically, every time I edit the syntax source file with the language’s native IDE, it adds again a BOM at save time, and I have to remove it in ST.

ST is aware of the BOM, for in the status bar it shows encoding as “UTF-8 with BOM”.

If I click on the enconding and choose “Set encoding -> UTF-8”, it removes the BOM and the test works.

ST should really check for the presence of a BOM in test files, and skip it when parsing the comments delimiters, without forcing the user to remove the BOM from the file and make the source file not compliant to its native IDE standards.

When creating test files for languages which have a native IDE, it’s useful to be able to work with the native IDE too, especially when building a new ST syntax, for one would like to be able to use code that really works in that language, or just beacause the native IDE simplifies editing code, etc. But if we need to add/remove the BOM each time the test file is opened with ST or its native IDE it gets annoying (and usually it’s convenient to work with the test file opened in both editors at the same time). Also, if the native IDE supports different file encodings, removing the BOM could lead to special characters corruption.

[FichteFoll]

You should create an issue for this on the core tracker so it is not forgotten.

[tajmone]

Thanks @FichteFoll! The problem is that I’m not sure of where that would be.

If it’s a GitHub repo, please could you provide a link?

[OdatNurd]

The issue tracker for Text is at:

SublimeTextIssues/Core

[tajmone]

Done:

Issue: Add BOM Awareness for Syntax Test Files