Sublime Forum

Sublime-syntax for indentation languages

#1

I work on a simple indentation-based metalanguage called Tree Notation.

I have a function that takes a grammar file and generates a sublime-syntax file. The grammar file is written in a custom language, sort of like an ANTLR.

It works! Well, somewhat. The syntax highlighting is super helpful.

The problem is, all of my languages are indentation-based. I cannot figure out how to correctly handle scope changes as indentation changes.

On each new line there are 3 scenarios: indentation increases by 1+ and a single scope is added, indentation stays the same and scope stays the same, indentation decreases by N and N scopes are popped.

I have tried everything I can think of to implement this–backreferences, scopes, lookaheads, lookbehinds, negative lookaheads, textmate grammars, et cetera–and can’t figure it out. I even went so far as to build my own “sublime-syntax” engine to try and reverse engineer how I might get this to work.

So far every path I attempt hits a dead-end (for example, if I could use backreferences in lookbehinds I had a solution).

Anyone out there who can help me crack this nut?

I put a demo of the problem here: https://github.com/breck7/demo

The parsing code for actually parsing these files into the appropriate data structures is quite simple. It’s about 20 lines of C++/Javascript/Python (here’s the TypeScript version: https://github.com/breck7/jtree/blob/master/src/base/TreeNode.ts#L943).

I have written highlighters for CodeMirror with ease, but trying to get things to work with this declarative sublime-syntax/textmate grammar files has crushed my spirit and made me want to flee society and live in the sea with the dolphins.

0 Likes
Is there anything in the roadmap to make syntax highlighting for semantic-indentation languages easier?
#2

hi there,

working with indentation based languages is not easy within the sublime-syntax framework, but maybe this will help you:

(sorry for not being more helpful, am on mobile with limited time atm)

I’d be interested to see your own sublime-syntax engine btw :slight_smile:

1 Like
#3

Thank you for the pointer @kingkeith! I am going through that thread now.

My “engine” is more like a “bicycle”, built by hand, while feeding a toddler :slight_smile: . But here it is at the moment: https://github.com/breck7/lime

That takes a syntax and generates highlighted HTML as output. My thought being it might be easier to get realtime feedback, write tests against, and debug. There’s probably something out there like that already but I also wanted to get a better sense of how the highlighting might be working under the hood.

Hopefully I can solve my scoping issue though and obliviate my need for it.

1 Like
#4

haha cool description, and thanks for sharing :slight_smile:
let us know how you get on - I may be able to more concretely help when I next get some free time at a computer

0 Likes
#5

I think what you’re looking for here is a linter. Instead of trying to embed a validation process into a syntax highlighter, it would be easier and more maintainable to check for errors outside of regex.

0 Likes
#6

I’m not sure I understand.

I would love to not have to use this regex language to write my highlighter for Sublime (writing a highlighter for CodeMirror took less than a day, this I’ve struggled with for 10x+ as long). But I don’t have a choice, right? To get syntax highlighting in Sublime I need to write a “.sublime-syntax” or TextMate grammar (which is roughly the same, as far as I can tell)?

Only because I’ve had to write in the “sublime-syntax” language have I made this tool to generate the highlighted HTML output from a sublime-syntax file.

I am working on a fork of the toy python script from the thread https://forum.sublimetext.com/t/can-embed-be-used-to-define-indent-based-syntaxes/41733/3?u=kingkeith @kingkeith suggested.

I’m getting closer! (I hope, could still be a dead-end)

0 Likes