Sublime Forum

[RFC] Specific scopes for punctuations

#1

Currently the scope guidelines recommends that functions be scoped with meta.function including its contents, so a function inside a function would have the meta.function appearing twice.

Looking at the core syntaxes, array literals as scoped with meta.sequence which is also applied to the contents. The problem with these scopes is that you can’t highlight the punctuation without affecting the contents, while I was able to make it work with very convoluted rules, at the end I just concluded that we either need specific scopes or a way to select them sequentially.

Being able to give different colors to punctuation is a very important feature to languages like Javascript where the meaning of a [] and {} changes drastically depending on the context. So I am proposing that we add specific meta scopes to punctuations:

meta.class.punctuation
meta.class.body.punctuation
meta.function.punctuation
meta.function.parameters.punctuation
meta.function.body.punctuation
meta.block.punctuation
meta.sequence.punctuation
meta.object-literal.punctuation

There’s no meta.property-access(the bracket in: a[]) or meta.destructuring in the core packages but I am already including them here to debate.

meta.property-access.punctuation
meta.destructuring.punctuation

These scopes would appear only at the punctuation. This solution would be simpler than adding a new feature to the color scheme engine and it would be compatible with other editors’ color schemes.

1 Like
#2

A related, though somewhat simpler, question is the distinction between punctuation.section and punctuation.definition in the case of sequence or mapping literals. Most syntaxes use punctuation.section for these, but not all.

Take JavaScript. The language has both block statements and object literals, both indexers and array literals. The semantics of these are completely different, though the syntax is tricky to distinguish. In the tmLanguage days, it was impractical to distinguish them; it might have even been impossible. Distinguishing them in a sublime-syntax took months of work and a brand-new architecture. So on one hand, there’s a lot of inertia behind the current highlighting, but on the other hand the opportunity to improve it is very new.

In JavaScript, I think that it would be useful and beneficial to distinguish punctuation defining a section of code (like a block or a function body) from punctuation defining a literal (array or object). The standard way of drawing this sort of distinction is by marking one as punctuation.definition and the other as punctuation.section. This change would accomplish much of what you want (though not necessarily all) without introducing new scopes that might not generalize well to other languages.

My main question at this point is whether such a change would be correct as well as useful. And existing core syntaxes are split on this question. If we decide that literals should use punctuation.definition, then this should be applied to all core syntaxes and the scope naming guidelines should be amended to clarify it. It’s a conservative change, on the whole, but one that I think needs to be made with all syntaxes in mind. For the record, I would absolutely support it.

But I’d wait a dev build or two before considering a PR, because there’s a pile of JavaScript PRs already in the pipeline, and in addition to the risk of conflicts I think there’s a lot of value to separating new language features/bug fixes from “philosophical” changes. Once the current batch of PRs is in, there’s a bit more work to be done on binding declarations, and the syntax tests could be greatly expanded. With all of the structural changes done and a more comprehensive test suite available, we could consider scope changes in complete isolation.

I share your enthusiasm for possible scoping improvements, but I’ve found it most useful to work within the framework of the core syntaxes whenever possible. Before PR #1009, I actually wrote a complete JavaScript syntax from scratch to work out solutions to the known challenges, but in the end it proved possible to integrate all of the major improvements into the core syntax with almost no disruption to existing functionality. For features like JSX that don’t really belong in the core syntax, I’ve found a different solution that complements the core syntax rather than replacing it.

1 Like
#3

Distinguishing them in a sublime-syntax took months of work and a brand-new architecture.

I am not saying to fix it in the core syntax right away, my syntax can already distinguish between array/object literals, property access, etc… and its working great specially after I implemented the expression tick tock logic you explained here, but I don’t have the specific scopes yet to give them different colors.

What I want is for us to reach a consensus on the scope structure, because this is something I know will benefit a lot of people.

In JavaScript, I think that it would be useful and beneficial to distinguish punctuation defining a section of code (like a block or a function body) from punctuation defining a literal (array or object). The standard way of drawing this sort of distinction is by marking one as punctuation.definition and the other as punctuation.section.

It does not solve it because the class/function body has the same punctuation scope, right now the only way to differentiate would be through the meta.function/class but then you would get into issues when one is inside the other as you can see here.

I am proposing additional meta scopes because it would not break support with current color schemes, it would not require sublime to implement a new color scheme logic and it would still work with other editors.

0 Likes
#4

why use meta scopes for punctuation rather than just make the punctuation scope more specific? punctuation.section.block.begin.class.javascript etc

0 Likes
#5

why use meta scopes for punctuation rather than just make the punctuation scope more specific? punctuation.section.block.begin.class.javascript etc

To initial idea was to maintain compatibility, some color schemes(rare) give different colors to punctuations based on the language. But it could be an alternative too, that’s why I want input from you.

0 Likes
#6

for that purpose they can generally use the base scope, it’d still work with embedded languages due to scope specificity

0 Likes
#7

Unfortunately it does not always work, I was surprised to know that adding source.js or source.jsx to jsx code, on some color schemes it would prevent it from having the same colors as HTML even thou the rest of the scopes were exactly like HTML.

0 Likes
#8

After some investigation, it looks like this may be due to a bug with sublime.score_selector. Suppose that some point has a scope of meta.class meta.function meta.class punctuation.section. The selector meta.class punctuation.section should score higher than meta.function punctuation.section. However, the reverse happens. I think that sublime is matching the first meta.class in the point’s scope, when it should probably be matching the last meta.class. More correctly, it should be working backward through the selector, matching the last punctuation.section in the string, then the last preceding meta.class. Instead, I think it’s working forward, matching the first meta.class and then the next punctuation.section.

If this behavior is indeed a bug (and I believe that it is), then fixing this bug should solve your problem (or lead quickly to a solution). I’ll post an issue on the core tracker later today.

See:

4 Likes
#9

I didnt know this could a bug which is why I even suggested this feature.

0 Likes
#10

Ah, I see. When I read that, I focused on the “directly preceded” part. I think that if the current behavior is a bug, and if it is fixed, then we will not need any other new functionality to solve the problems we’re looking at.

I think that I’ve been a bit confused when talking to you about this issue. I wasn’t aware of the actual scope_selector behavior, and lacking that context I couldn’t see why the existing system didn’t do what you wanted. Then, I tried it out today on a minimal color scheme and it didn’t work the way I expected (with repeated nested functions/classes, it highlighted the inner braces “wrong”).

1 Like
#11

Meta scopes are meant to tokenize the text into logical sections. A meta.function is such a logical section as it contains all parts required to declare a function, the return value, the name, the argument list etc.

Meta scopes are not meant to be addressed by color schemes directly.

A ( or a [ is anything but no logical section and therefore should not be scoped with a meta.<...>. It is what the top-level scope is named like - a punctuation.

What could be discussed is the structure of the following scopes. I am with all of you who are confused about the meaning of punctuation.definition and punctuation.section.

0 Likes
#12

Issue posted on the core tracker: https://github.com/SublimeTextIssues/Core/issues/2152

@deathaxe It’s fair to say that coloring meta scopes is not generally advised, but while it’s probably not a good idea for general-purpose color schemes I think that it can be useful for specialized color schemes in cases like these. In addition to that, the unexpected behavior of score_selector could have ramifications for plugins that use meta scopes. For instance, I’m fairly sure I ran into this myself when experimenting with lexically-aware autocompletion for JavaScript, although I didn’t identify the issue at that time.

1 Like
#13

punctuation.definition was a historical scope used by all sorts of things, that weren’t really “definitions”. We added punctuation.section for “large” multi-component constructs such as blocks, arrays, etc. punctuation.definition remains for things that are more atomic, like a variable name, the <> for a generic data type, and for strings.

1 Like