Incorrect Syntax with python for None type hints?

[maegul]

• If an argument is typed as None and assigned a default value (eg None), Sublime’s syntax definition parses this is illegal (see below, with scope of the red-shaded = in comments underneath)

• I haven’t checked the language specs, but a quick test shows this not to be illegal according to python:

• If you’re wondering why on earth this would be done … well without too much information, it can be helpful with type-hinting overloads and default values, where the type is used by the type-checker and the default value by the interpreter.

---

• I don’t know enough to be sure that this is a bug though … anyone have any thoughts

[jfcherng]

Since a certain commit, there is a checking for reassigining constants. In your case, the ST syntax think you try to reassign None to None. I think this is the cause.

[maegul]

If it’s informative, here’s an example that doesn’t raise an error:

Here the = is parsed as being part of a default-value parameter rather than the annotation as in the “illegal” example.

[bschaaf]

Does it happen in safe mode? I don’t see that here:

[jfcherng]

It’s a fixed issue but not landed in stable build yet. https://github.com/sublimehq/Packages/issues/3290

[maegul]

I should have mentioned, yes, my sublime is 4131.

Thanks for the quick responses!

[maegul]

Otherwise, reading through the linked GitHub issue … it seems from deathaxe's comments that Sublime doesn’t (and maybe doesn’t want to) support type hints in the syntax?

I feel like I’ve noticed this already (eg below). Is this true? Feels like it would be a shame honestly.

[jfcherng]

He meaned that there is no dedicated rule for type annotation in the syntax definition. The type annotation part just “looks” good by luck. Its scope can be wrong.

For example, | “looks” good but it definitely shouldn’t be keyword.operator.arithmetic.python. It’s just that | is captured as keyword.operator.arithmetic.python in general.

[jfcherng]

I feel like I’ve noticed this already (eg below). Is this true? Feels like it would be a shame honestly.

Not sure what you are actually refering to, but I guess you want semantic highlighting. Unfortuantely, the only chance at this moment to achieve that is using Pylance, which is against its license to use in other editor than VSCode.

Pylance screenshot:

[maegul]

Not sure what you are actually refering to, but I guess you want semantic highlighting.

Yes! I’m not familiar enough the the python syntax definition in sublime to be sure about it, but from what I can tell, and your example and statements match this, the syntax definition is not aware of type-hints (apart from within function definitions where function.parameters.annotation is a defined scope). Which is a shame really, as clear syntax highlighting for all the type-hints outside of functions would be helpful or at least nice and consistent.

[deathaxe]

A colon just denotes a beginning of annotations. Basically annotations are nothing else but comments, ignored by the interpreter.

Even though “type annotations” are the most popular use case as of python 3.6, annotations may contain any kind of textual information.

Hence it’s not crucial to support that use case, especially as its working quite well with regards to highlighting.

As @jfcherng stated, “semantic highlighting” is out of scope of static syntax definitions shipped by ST anyway.

[jwortmann]

I think that the examples from the last screenshots don’t neccessarily require “semantic highlighting”. The reason is rather that arbitrary (user-defined) class names don’t get any special scope in the Python syntax (excluding in class definitions of cource). They will have only a meta.generic-name scope, and therefore the same color as normal variables - even outside of type hints:

Various other syntaxes would use storage.type.class or support.class instead.

I think there is a convention to use PascalCase for class names in Python, but I don’t think it is strictly enforced by the language. Theoretically it could be used in the syntax as a heuristic to apply such a scope for PascalCase identifiers (similar to the ALL_CAPS convention for constants, which already gets variable.other.constant in Python). For example GitHub seems to use this convention to highlight Python class names in orange.

[maegul]

A colon just denotes a beginning of annotations. Basically annotations are nothing else but comments, ignored by the interpreter.

Technically correct but perhaps missing the forest for the trees on the vast majority of python practice. I don’t think anyone would be unhappy at arbitrary textual annotations being ignored by the syntax definition, but plenty might enjoy their type annotations being recognised.

And while annotations are not actively interpreted python … a python+mypy syntax is what many are now programming in.

I appreciate that it may not be crucial (and maybe incredibly painful … I wouldn’t know!) to do more on type-annotation syntax … but your take strikes me as somewhat defensive and absolutist. After all, if the sith empire (VS Code) are doing it … people will want/expect it.

[maegul]

I think that the examples from the last screenshots don’t neccessarily require “semantic highlighting”. The reason is rather that arbitrary (user-defined) class names don’t get any special scope in the Python syntax (excluding in class definitions of cource). They will have only a meta.generic-name scope, and therefore the same color as normal variables - even outside of type hints:

Hmmm … a class is just another object/variable though. Outside of type-hints, this seems perfectly reasonable to me. It might be rather finicky and niche to ask for an integer variable and user-defined class to be scoped differently. On the other hand, a type annotation is a specific context with a specific meaning, and can make sense as differently highlighted/parsed sections of code, as is the case with function parameter annotations.

[jwortmann]

It might be rather finicky and niche to ask for an integer variable and user-defined class to be scoped differently.

Well, the built-in “int” type already does get a different scope from user-defined classes in the Python syntax. And I wouldn’t see a reason that class names should be treated differently depending on whether they are part of a type hint or not. Just to make sure that there is no misunderstanding; I don’t mean an instance of a class, but rather the class name itself. Here is another example (no type hint):

The issue seems to be rather that there is no strong convention for the scope of class names over various syntaxes. Some languages, which all use the PascalCase convention for class names (as far as I know), and all of them use different scopes (but at least still scopes which are different from “normal” variables):

JavaScript:

Java:

Ruby:

[maegul]

I wasn’t talking about types but variables. For example, your first screenshot (with isinstance), I’m talking about the x variables, not the types. int is a predefined element of the language, but a user-defined class wouldn’t be by definition, so it makes sense that a syntax definition can and does distinguish them. As for being, as I originally said, “finicky”, I wouldn’t know how user-defined classes could be statically parsed. The idea of using Pascal Case seems far too restrictive to be practical (?).

[deathaxe]

there is no strong convention for the scope of class names

To strictly follow original TextMate scope naming guidelines any kind of user (or (standard) library) defined entity (types, classes, constants, …) would need to be scoped support. ... . So support.class would be the scope to choose in general.

The reason for storage.type.class being used in Java is the ST specific interpretation of support. to be dedicated for entities of a language’s “standard library” only. As Java doesn’t maintain lists of all built-ins, a more general support-unrelated scope was choosen.

While Java implements sophisticated expression specific strategies to match data types as accurate as possible (without relying on casing), that’s not feasable in most other syntaxes.

While it is quite obvious MyClass to be a class when following isinstanceof, it may not be so clear in any other expression. Static class access expressions can be located at the same position where normal variables are allowed. Hence casing was probably the only heuristic which can be applied - as already stated.

Relying on casing has been one of the most popular reasons for bug reports of Java syntax for instance. Hence it’s probably better to apply a generic scope only, because it would look broken to scope the same token different depending on location or context. A static syntax engine just can’t provide more detailed semantic scopes.

Python is such a syntax, too. We just can’t scope tokens semantically due to several ambiguities. So scoping classes support.class is usefull only for well known classes. So Union or Set or types like that might be scoped support.class, but it’s not useful for user defined types - neither in type comments nor elsewhere.