Best way to deal with highlighting multi-line function definitions

[Cerberus_tm]

I have a function definition like this:

FunctionName(Argument)   ; function definition
{
; do something in function body
}

And a function call like this:

FunctionName(Argument)   ; function call

Syntax highlighting cannot see past line boundaries, so I cannot properly highlight FunctionName and Argument here using the language’s .sublime-syntax file (Autohotkey’s), because a line with a function call looks exactly the same as a line with a function definition where the braces only come at the next line.

So I’m trying to solve this by creating a plug-in. What would be the best way to approach it?

I have been thinking of having the plug-in add an invisible character at the end of a function-definition line, in order to trigger proper highlighting (the sublime-syntax file will have to recognise this character in a regex). But I really don’t like to mess up people’s code with invisible characters. What if they copy-paste it somewhere and it causes undetectable problems?

I could add a comment like ’ ; {’ to the line, again to be caught by the sublime-syntax regex. But it looks messy, and it still changes people’s code.

I could somehow highlight regions ‘manually’ through the plug-in and add the function name to the list of symbols. I’d have to create a kind of parser in the plug-in, to distinguish between function definition and reference, and to find the arguments to highlight. That seems like a lot of work, and it may not work reliably. In addition, because I may have to set the colours in the plug-in, it may work only well with one colour scheme (Monokai in my case). (This sub-point may be solvable.) Lastly, anything that other plug-ins or colour schemes do using scopes would fail with these function definitions, as they will have the wrong scopes.

If only there were a way to assign scopes in a plug-in! But I believe that is not possible?

[jfcherng]

Syntax highlighting cannot see past line boundaries, so I cannot properly highlight FunctionName and Argument here using the language’s .sublime-syntax file (Autohotkey’s), because a line with a function call looks exactly the same as a line with a function definition where the braces only come at the next line.

It can be done better in ST4 (not “public” announce beta yet. currently it seems to only be available in the official discord for test purpose). Example, https://github.com/sublimehq/Packages/pull/2202.
I think this is the same situation that you encountered.

# this is function definition
$var = fn($x)
//     ^^ meta.function.arrow-function.php - meta.function-call
   => $x * 2;
// ^^ punctuation.definition.arrow-function

# this is function all
$var = fn($x)
//     ^^ meta.function-call - meta.function.arrow-function.php
;

[Cerberus_tm]

Thanks for the example! I had heard about there being new developments, also visible in a change to the Javascript syntax highlighter, which had things like ‘branch_point’ and ‘fail’. Looks like a very nice system indeed!

However, it will probably be a long time before this goes into a stable release of Sublime. Looks like it won’t come to Sublime Text 3? So, for the time being, I was planning to work on something that does work right now…

[jfcherng]

My first though is

• view.find_by_selector() to find regions of function names of function calls (suppose both of your examples are classified as function calls by ST and the function name of a function call should already have a dedicate scope).
• Then, use view.extract_scope() to advance the test point to see whether you can find { in its following scopes.
• view.add_regions() to add extra scopes for all “function calls” which actually should be function definitions.

[OdatNurd]

Note that view.add_regions() does not add scopes to the buffer; it just defines interesting regions in the buffer. Those regions can be marked up (such as with underlines, boxes, etc) using the scope that you provide to the call or just used to store locations and apply icons in the gutter.

So while you could use this (in combination with color scheme tweaks) to simulate syntax highlighting, anything that requires actually fetching scope information out of the buffer (like view.find_by_selector(), key bindings, etc) won’t work as you expect,

[Cerberus_tm]

Finding function names / calls first: yes, that was what I had in mind first.

Advance the test point: do you mean extracting the extent of the scope immediately after the function call’s end, check whether it is the beginning of a block; if not, get the extent of the scope after that, etc.? That is an interesting possibility; you probably mention it because it is (semi-)efficient? I would then have to make sure each block in my code is scoped to whatever it belongs to, so that it will only pick up on a block that doesn’t e.g. belong to an If or Loop. That would mean I would have to parse all blocks in a similar manner, making sure I don’t pass over anything that could have its own block on a new line. Blocks could be nested…

Alternatively, I could do another find_by_selector() to find regions that are either empty (just source.ahk) or a comment, skip over those, then check whether the next scope is a block. I wonder which is more efficient.

[Cerberus_tm]

Ah, I was afraid it wouldn’t be possible to actually access the scopes the way the syntax highlighter can. That was what I meant by highlighting regions ‘manually’ and creating a kind of parser. Would probably be rather complicated for what I’m trying to achieve…

P.S. I’ve watched some of your videos, some nice explanations and tips on plug-ins there! Couldn’t get Snapi to install, though. But I have it bookmarked.

[jfcherng]

Advance the test test point: do you mean extracting the extent of the scope immediately after the function call’s end, check whether it is the beginning of a block;

For example, use view.find_by_selector() to find regions of declare. Then use view.extract_scope() to the end of the declare, it should return the region of (strict_types=1) if the syntax definition is properly written. Do it again, it returns the region of ;.

This is more like how a parser works naturally to me.

you probably mention it because it is (semi-)efficient?

I don’t know. I didn’t actually test it with a large file. But find_by_selector() returns regions scattered in the whole file.

[kingkeith]

view.extract_tokens_with_scopes is probably more useful

[jfcherng]

Looks like something awesome is missing in the official doc

[Cerberus_tm]

That does make sense, thanks! It would seem reasonably efficient. And I’m going to do it the other way around: search for uncategorised blocks (of which there should be none), then go backwards scope by scope until a non-comment, not-just-source scope is found; if the scope found is a function declaration, add a comment ;{ to the end of the line, so the syntax-highlighter (which I am writing myself) will recognise it as a function definition; if the scope found is something else, continue to the next uncategorised block.

I think I should probably only run this on load or save, because it may use too much processing power if it is e.g. run on each character typed.

By the way, how did you make those underlines? They look very nice. Are you using syntax highlighting / colour scheme, or is it something from a plug-in? I have so far failed to make underlines in a colour scheme. I don’t think I could do it in a plug-in either, but I don’t remember exactly.

[Cerberus_tm]

That is a very interesting, undocumented function! So what view.extract_tokens_with_scopes does is take a region and return all scopes found inside it as an array of regions. In locating the function line belonging to an uncategorised block, I’m not entirely sure whether this would be preferable to just using .extract_scope (or am I mistaken?); but it will certainly prove useful in many other cases where I want my plug-in to parse scopes! I already have a use case in mind, cool.

[jfcherng]

By the way, how did you make those underlines? They look very nice. Are you using syntax highlighting / colour scheme, or is it something from a plug-in?

This will make you disappointed
It’s made by a handy screenshot tool. https://www.snipaste.com
Strongly recommended if you are using Windows.

[OdatNurd]

I have so far failed to make underlines in a colour scheme. I don’t think I could do it in a plug-in either, but I don’t remember exactly.

This is one of the reasons to use the add_regions() API endpoint on a view. For example, try selecting some text in a file and then entering this in the Sublime console:

view.add_regions("test", view.sel(), "region.redish", flags=sublime.DRAW_SOLID_UNDERLINE | sublime.DRAW_NO_FILL | sublime.DRAW_NO_OUTLINE)

[genericptr]

Excellent. Is it possible to add tooltips to the regions so if you hover your mouse over them you get some information? I think I may have seen this by adding symbols in the gutter also but I can’t remember where.

[OdatNurd]

The EventListener.on_hover() event triggers when the user hovers the mouse; one of the arguments it gets tells you where the hover is happening.

So you could have an event listener for that and check if hover_zone == sublime.HOVER_TEXT, and if so check the regions you’ve saved to see if the point falls within one of them or not. From that you could use view.show_popup() to generate a tooltip.

[genericptr]

I think I get it. That means I need to keep a persistent list of warnings also so I can reference them later.

Just looking at this now I’m not seeing how view.add_regions is going to work because I don’t have a reference to the correct view for the given file. For example if you’re parsing errors from a compiler you get paths/names with corresponding line numbers but how do you translate that file name to a view object I can call add_regions on? Is that even possible?

[OdatNurd]

If the file is open, window.find_open_file() will return the view object that’s associated with the file with that name; alternatively you could also iterate over all of the views in a window and check their file_name() property to see if it matches, but find_open_file() is likely faster because it’s in the core. Presumably it would also take care of things like case sensitivity checks on platforms that support that sort of thing (though I don’t know if that’s actually the case or not).

For a case like this you can only apply regions to files that are open, so you can do it right at the time you capture the messages but you also need to do something like an on_load() event listener to detect when new files are opening so you can check and see if they need any regions.

The exec command does it that way; every time a file opens it executes the exec command with an argument that tells it to update annotations. The magic of file_regex in a build makes Sublime try to open files when you double click on them, which integrates nicely.

[genericptr]

Thanks, I got it now.

One last thing is that I can’t figure out how to map get a region for the line/column. add_regions wants a Selection object right? Here’s what I’m trying to do with no luck.

    line_num = 10
	pt = a_view.text_point(line_num, 0)
	sel = sublime.Selection(0)
	sel.add(sublime.Region(pt))
	a_view.add_regions("warning", sel, "region.redish", flags=sublime.DRAW_SOLID_UNDERLINE | sublime.DRAW_NO_FILL | sublime.DRAW_NO_OUTLINE)

[OdatNurd]

view.add_regions() wants a list of region objects. The Selection class represents selections that way, which is why it’s a handy test to do that in the console.

In order to do what you want here you need to pass it something like [sublime.Region(0, 10), sublime.Region(20, 25)] or such; a list of manually constructed regions.

Note however that something like sublime.Region(10) will give you a region that starts and end at character position 10 only; so in that case the region won’t display because it’s empty, and you’d also have to hover the mouse at exactly that point later as well.

You can use view.text_point() to convert a row and column (both 0 based) to a point for use in the region constructor. On the flip side you can use view.rowcol() to convert a point into a (row, col) tuple if you need to go in the other direction. Lastly, view.line() will return back a region that spans the line that the position that you give it is in.

So taken all together, you could wrap the whole of line 10 with something like this:

line_pos = view.text_point(10 - 1, 0)
line_region = view.line(line_pos)
view.add_regions('warning', [line_region], "region.redish", flags=sublime.DRAW_SOLID_UNDERLINE | sublime.DRAW_NO_FILL | sublime.DRAW_NO_OUTLINE)

Note also that the hover event will give you a single point that says where the cursor is; if you use view.get_regions() you will get a list of all of the regions with the key you provide, and it’s always sorted from the top of the file down. You can use the region.contains() method to easily determine which region contains the point (if any) and then use that index to determine what the hover text should be.