Everything you (n)ever wanted to know about indentation in ST3

[kingkeith]

Everything you (n)ever wanted to know about indentation in ST3

Hello, I’d like to talk about indentation in ST3. No, not that old tabs vs spaces argument but actually how automatic indentation works in the editor.

I’m sure we can all agree, that from the user side, auto indentation in ST3 is pretty good, with only a few edge cases where it misses the mark. But, from a technical perspective, there is room for improvement. Therefore, I thought it would be useful if we document the current indentation behavior (facts), and then, afterwards, we can say what we like and don’t like (opinions) to help the ST devs decide on a way forward to both make it nicer from the technical side and fix those pesky edge cases while we’re at it. I’m therefore making this a community wiki, so that people can edit this post with things I’ve missed or not expressed clearly enough - I find it quite tricky to decide how best to present information and ideas

For the purpose of this discussion, assume the auto_indent and smart_indent preferences are set to true (default). This guide is valid as at build 3126. Obviously, if changes are made to the indentation engine in the future, this article will mainly be relevant for archive reference purposes. Note that any regex pattern mentioned hereafter is using the Oniguruma engine, which supports recursion and thus allows one to craft a regex for (un)balanced parens etc.

Indentation when pressing Enter

The main aspect of auto indentation increasing the indent level occurs when pressing Enter, anywhere on a line.

Note that some keybindings override Enter with their own logic by inserting a snippet etc., but otherwise, the logic is currently based on the `.tmPreferences` metadata files: https://docs.sublimetext.io/reference/metadata.html#indentation-options. (The `insert` command processes indentation logic, but the `insert_snippet` and `append` commands don't.) The scope selector is equivalent to the `eol_selector` context, wherein it works on the scope of the `\n` character at the end of the line. All the regular expression patterns match against one line of text.

All subsequent lines

Example: type an open XML tag `` or curly brace `{` and press Enter

This is handled by the `increaseIndentPattern` regex.

A single line

For example, typing an if statement without an opening brace `if (x)` Enter. In this case, only the next line is indented, because the "if" will apply to only a single statement/line. Then, if a `{` is typed, it gets unindented again, because most people want their opening and closing braces in the same column as the start of the "if". See also the "Adjusting indentation when you type on a line" section.

This is also potentially useful for incomplete statements (split over multiple lines), for example some languages require statements to be terminated with a semi-colon ;, and visually indenting them makes it more readable.

    a.b().c.d.e.f
        .g().h.i();

as opposed to

    a.b().c.d.e.f
    .g().h.i();

This is handled by the `bracketIndentNextLinePattern` regex - presumably the `bracket` part relates to the `{` behavior, although it is actually not hardcoded, and is instead overridden by a separate `disableIndentNextLinePattern` regex. The naming of this is perhaps a little confusing, because the disable regex applies to the "next line" - the one whose indentation was caused by the `bracketIndentNextLinePattern`. ST3 handles this disable by default in `Packages/Default/Indentation Rules.tmPreferences`.

But “wait”, I hear you say, “most packages I’ve seen have the key indentNextLinePattern instead of bracketIndentNextLinePattern in their metadata file!”

Yes, and ST ignores it - maybe some plug-ins make use of it though. (They can do so using view.meta_info('indentNextLinePattern', view.sel()[0].begin()) - note that this method is currently missing from the official documentation.)

lines containing only whitespace and comments are ignored

The auto-indenter will ignore lines matching the unIndentedLinePattern regex when computing the next line’s indentation level. This mainly seems to be used to ensure that bracketIndentNextLinePattern can affect multiple lines when the first line(s) is/are purely comment lines.

other built-ins

The user preference `indent_to_bracket` adds whitespace up to the first open bracket when indenting. It also affects unindenting too, the close paren is placed in the column after the open paren. ```js foobar( okay( 'yeah' ) ) ```

`indentSquareBrackets` and `indentParens` ensure that an open/unclosed square bracket `[` or paren `(` will cause subsequent lines to be indented an extra level. However, they don't have any affect on unindenting the current line when the closing bracket is typed, so must be combined with a `decreaseIndentPattern` for consistency. See [the JSON indentation rules](https://github.com/sublimehq/Packages/blob/7ef80d531b752baee46f792b6bc6b26206e56012/JavaScript/JSON%20Indent.tmPreferences#L26) for an example. They do, however, unindent the *next* line when typing a closing paren/square bracket that doesn't have a corresponding opening bracket on the same line (ignoring those in a `string` or `comment`).

Adjusting indentation when you type on a line

The main aspect of auto indentation decreasing the indent level occurs when typing on a line, for example a closing brace `}` or keywords `end if` or closing SGML tags ``

But it can also occur when you typed an if statement on one line and then an opening brace on the next line if (x) Enter {. If you then delete that brace, it will increase the indentation level by one again. This means that ST has to keep track of whether it has increased or decreased the indentation while you typed on the line. See also “Indentation when pressing Enter/A single line”.

Manually adjusting the indentation level on the line by any means will prevent ST from automatically changing the indentation level as you type on the line. As will typing on a different line. So it seems ST analyzes the line between pressing Enter and typing on a different line - just moving the caret off the line and back again without typing doesn’t affect it.

This is handled by the decreaseIndentPattern regex.

Reindenting the whole file or a selected part of the file at once, on demand

One can ask ST to reindent any part of the file on command. (Note that this is different to reformatting i.e. the only changes are to indentation, no non-whitespace characters are moved to different lines etc. Reformatting is best left to plugins and thus is out of scope for this discussion.)

Reindentation uses the indentation of the line above as a reference point to continue from, so even if the file above the part being reindented doesn’t completely follow the indentation rules, the part being reindented can still look correct/not out of place.

Also, it is possible to “Paste and Indent” (see the Edit menu) in one go, which, doesn’t perform a paste and a reindent as one would expect, but ensures the indentation of the pasted text fits the place where it was pasted, i.e. indentation is added or removed from the lines to keep the same relative indentation in the pasted text but to start from the indentation of where it was pasted.
For example, if your original text was hello\n\tworld\n with your cursor at the end and you pasted \t\t\tfoo\nbar you’d normally get: hello\n\tworld\n\t\t\tfoo\nbar, but using paste and indent, you’d get hello\n\tworld\n\t\t\t\tfoo\n\tbar.

---

The Tab key is bound to the reindent command by default, when there is no selection on a completely empty line (not even whitespace present), so one can press it once to instantly get the caret at the correct indentation level.

Currently ST supports using different rules to reindent in "batch"es. If batchIncreaseIndentPattern or batchDecreaseIndentPattern is defined, these take precedence over increaseIndentPattern and decreaseIndentPattern respectively.

preserveIndent (a metadata boolean) can be used to ensure that lines are not moved during batch indentation. This is useful for multiline comments, especially "docblock"s so that leading whitespace inside the comment is not obliterated.

Note that unIndentedLinePattern also functions similarly to preserveIndent, in that a line matching this pattern has no indentation rules applied to it upon batch reindentation. However, there are a few differences:

• unIndentedLinePattern takes affect if the first non-whitespace token on the line matches the scope selector given in the tmPreferences file (and the regex matches the line). If the line is the first (or only) consecutive line that matches this rule, and it had some indentation, it’s indentation will be changed to match that of the line above. If the \n matched the scope, then the next line, if it had no indentation, will be indented to the same level as this one. Otherwise the line will stay with it’s existing indentation level.
• preserveIndent takes affect if the \n at the end of the line matches the scope selector given in the tmPreferences file, or the line above’s \n matched. Whatever indentation the first line had, the next line’s indentation is changed to match this one’s.
  Therefore, you will observe different results if you unindent your file before reindenting it.

Removing automatic indentation when moving off an otherwise empty line

There is a user preference called trim_automatic_white_space to control this behavior.

General tmPreferences info

The usual scope selector specificity precedence rules apply if there are multiple metadata files that apply to the scope at the end of the line, and the overrides work as you’ve come to expect from ST.
If the PList file contains a key followed by an empty string, then that pattern is overridden with a blank regex and consequently the pattern doesn’t apply / is disabled. To always enable a pattern, use <string>.</string>.
The rules are executed with a left-aligned match, so most regex patterns will typically start with \s*. The ^ anchor is implied, but there is no implicit $ anchor.

Summary

increaseIndentPattern affects the next line
decreaseIndentPattern affects current line
bracketIndentNextLinePattern affects the next line
disableIndentNextLinePattern affects the current line if the above line was matched by bracketIndentNextLinePattern

---

Limitations / Drawbacks / Known Bugs in the current implementation

• bracketIndentNextLinePattern seems to be cumulative. So if one was to want incomplete statements to be indented, statements spanning 3+ lines get indented too far and are not restored to the original level after a ;.

    a.b()
        .c()
            .d()
                .e();
            f();
    // instead of
    a.b()
        .c()
        .d()
        .e();
    f();
  

• Currently, it is only possible to automatically adjust indentation one level at a time, which affects switch statements Configure auto-indent with multiple scopes per line

• a combination of the above two points, the same applies to multiple if statements without braces:

    if (true)
        if (false)
            cool();
        this_should_be_one_level_backwards();
  

• Regular expressions are potentially duplicated from the syntax definition in the metadata file - if the syntax gets updated, does anyone remember to check the indentation regexes are still relevant?

• having different reindentation rules for batch reindentation can be confusing and lead to unexpected behavior, especially with trim_automatic_white_space enabled https://github.com/SublimeTextIssues/Core/issues/1583

• instead of a user preference, users have to override a metadata file to get comments to be reindented in batch mode https://github.com/SublimeTextIssues/Core/issues/1271, but then its a tradeoff between comments not being moved or docblocks being aligned wrongly

• Because the scope selectors operate at EOL, and the regular expressions only have the one line of context to match against, it is not possible to reliably skip block comments that don’t cover the new line character. i.e. using a <scope>comment</scope> selector and matching <key>unIndentedLinePattern</key><string>.</string> would be a generic solution that would prevent each language from needing to override this regex pattern (probably in the past, single line comments didn’t scope the \n character, so this technique couldn’t be used), but still causes e.g.

    if (true)
        /* test
        example */
  

  Enter to lose the indentation from bracketIndentNextLinePattern after the comment because one can’t guarantee that */\s*$ ended a comment

• related to some of the items mentioned already is batch reindenting multi-line if statements. Unless the syntax definition has a unique meta scope on the if, it’s hard to use regexes that would handle this correctly based on a single line of context - see http://stackoverflow.com/questions/41571959/sublime-text-3-indentation-for-multi-line-statements-in-php

        if (VeryLongThingThatTakesUpALotOfRoom ||
                OtherQuiteLongThingSoINeedTwoLines) {
            statement1();
            statement2();
        }
  

  reindents to

            if (VeryLongThingThatTakesUpALotOfRoom ||
                OtherQuiteLongThingSoINeedTwoLines) {
            statement1();
        statement2();
    }
  

• there are times when unindenting doesn’t work properly when there is an increase indentation immediately followed by (or with only blank/whitespace lines in between) a decrease indentation
  https://github.com/SublimeTextIssues/Core/issues/1262
  

• there is no way to configure indent_to_bracket to place the close paren in the same column as the open paren, and it only works with parenthesis, not anything else like square brackets

• there is currently no way for users to easily customize snippet brace style https://github.com/sublimehq/Packages/issues/131

• the reindent command acts weirdly with lines that don’t conform to the tab stop size. i.e. using a tab size of 2 spaces, reindenting the following PHP code:

        <?php
  
        function test_indent() {
          if (condition) {
           echo "here";
          }
           else {
            echo "else";
           }
        }
  

  reindents to

        <?php
  
        function test_indent() {
          if (condition) {
           echo "here";
         }
         else {
          echo "else";
        }
        }
  

  removing all existing indentation first helps the command to work as expected - http://stackoverflow.com/questions/42510367/imporper-sublime-indentation-for-php-files#42510367

• one can’t easily unindent a line based on the previous line

Proposals

Please feel free to comment/reply with your ideas and proposals. As mentioned before, if there is any factual information here that could be improved, please edit this post.

[kingkeith]

I’m pretty sure we could fix the "unIndentedLinePattern block comment problem" with the current indentation rules engine by tweaking the syntax defs and adding a meta scope to the newline character if only whitespace follows after the block comment ends i.e. match (\*/)(?:\s*$(\n))?, and use that meta scope in the generic solution’s scope selector along with comment. (I’m guessing the only reason why the Default package didn’t handle comments generically before was because the trailing newline character never used to be scoped as part of the comment, as mentioned in the main post.)

[borela]

Hi Keith

Often times I use docblocks and I even created macros to add/remove lines from them, the issue is that sublime keeps indenting after the block when I try to reindent:

The ideal would be for the variable declaration to be aligned with the /** and not the */, is there a solution for this case?

Thank you

[kingkeith]

Hi, great question - thanks for asking it yes, this is solvable - in fact by the idea I posted above if you can let me know which language syntax you are using, I’ll provide some step by step instructions on how to do it

also, while trying this out, it revealed something I hadn’t quite realized previously. I turn off preserveIndent for comments, because otherwise the reindent command doesn’t touch them and it messes up the rest of the indentation. But with it off, it unaligns the * in docblocks… I will need to explore to see if there is a suitable solution for both at once

[borela]

I am using MQL4 which is close to C++ but with less features and I am trying to simplify it and fix some issues like the indentation.

And yes, I tried disabling preserveIndent, but it was leaving the doc blocks unaligned too. Thanks for the article, it helped me understand some of the errors I was having before.

[kingkeith]

step 1 will be to edit the syntax definition to include a meta scope for when only whitespace follows the block comment:

contexts:
  main:
    - match: \/\/
      scope: punctuation.comment.begin.mql.mql4
      set: line
    - match: \/\*
      scope: punctuation.comment.begin.mql.mql4
      set: block

  line:
    - meta_scope: comment.line.mql.mql4
    - meta_content_scope: comment.content.mql.mql4
    - match: $
      pop: true

  block:
    - meta_scope: comment.block.mql.mql4
    - meta_content_scope: comment.content.mql.mql4
    - match: \*\/
      scope: punctuation.comment.end.mql.mql4
      set: # added to set a meta scope for when only whitespace follows the end of the block comment
        - match: \s*$\n?
          scope: meta.whitespace-after-comment-block-end.mql.mql4
          pop: true
        - match: ''
          pop: true

step 2 will be to create a new .tmPreferences file with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>scope</key>
    <string>comment.line.mql.mql4, comment.block.mql.mql4, meta.whitespace-after-comment-block-end.mql.mql4</string>
    <key>settings</key>
    <dict>
        <key>unIndentedLinePattern</key>
        <string>.</string>
    </dict>
</dict>
</plist>

and it should now keep the variable declaration aligned with the /** instead of the */ when you reindent

[borela]

Thanks a lot Keith, it worked perfectly .

[Henk]

Thank you @kingkeith for this elaborate topic. I tried to create a tmPreferences file for the language Omnis Studio, but I do not know where to put the file. I googled, came across this page a couple of times, but could not find any documentation about this. Can you tell me where to put this file? And do you have to do more then creating this file to get the indention to work or do I also have to add meta_scope lines to the tmLanguage file?

[kingkeith]

good point, I didn’t really cover that. If you, from ST, go to the Tools menu -> Browse Packages and then:

• if you are developing a package or contributing to a package, you should create a folder with the name of the package, and save the tmPreferences file there
• if you are writing the indentation rules for yourself, you should save it in the User folder

about scopes, it depends what you are trying to achieve specifically and whether the existing scopes are suitable, but quite often it isn’t necessary as regex is powerful enough without specific scopes

[jeykay]

Hi kingkeith,

thanks for this very informative thread, I stumbled across it since I’m trying to improve automatic indentation for vhdl. I’m very unhappy with the current state of the auto-indent, mainly because it only supports one-line patterns and decreasing one level of indentation.

I tried playing around with scopes (and adding . indent patterns for specific scopes) but it turned out to be rather tedious and didn’t even work in some cases (strange behaviour when including or not the line ending character in the scope…)

Anyway, I wanted to debug and potentially edit the auto-indent functions but I couldn’t find the code for it. Many commands are implemented in the Default packet but I couldn’t find indentation. Do you have any idea where to find the code that is run when using “reindent” command?

thanks a lot!
jk

[kingkeith]

Do you have any idea where to find the code that is run when using “reindent” command?

that code is part of ST core, and not implemented in Python, so it is closed source and we can’t see it.
It’s partially the whole reason for this thread, as we can’t see the code - we rely on guesswork as to what it does, so I tried to document its behavior as best as I could, along with it’s limitations, in the hope that a future build of ST will have a better indentation engine (IMO, ideally decoupled from core and implemented as a Python plugin)

[Remi]

It’s not using the ‘autoindent’ feature, however my VHDL package does have a beautification function which if you tear it apart may be of some use to you. Basically it left justifies everything and then reindents based on language rules. It’s not perfect, but it has done a pretty good job for me at least so far.

In retrospect, I may have recreated the indentation file to some extent because I based it around a ‘current_indent’ and ‘next_indent’ model, but I have space for custom behavior. For instance after an instantiation, we usually indent, and then indent again for port map. We’d like the closing parens for port map to be aligned with the port map statement, but the following statement should be on the line aligned with the original instantiation. It seems to do that pretty well. Also had to create a custom ability for when a line ends with the closing symbol (parenthesis usually for VHDL) and when that’s put on a separate line. Different code styles treat this differently (K&R versus Allman versus Lisp).

Another thought, it’d be kind of cool if indent patterns could be included in a syntax file. The two are very related and it might be possible to get more fine grained control.

EDIT 2: Actually working on some fresh code (as opposed to maintaining existing code) and the need for a good auto-indenting model for the language is becoming apparent. I can tell that I’m having to hit tab a lot more than I remember having to with vhdl-mode in Emacs.

[jeykay]

Hi Remi,

thanks for the remark, I will certainly take a look into your vhdl package.

I started working on an indentation tool that will allow the user (or language package) to define some opening and closing patterns (supporting multiline pattern) between which the code will be indented. It should easily handle situations like instantiation since the closing semicolon of the port map can close (in terms of indentation region) the port map itself as well as the instantiation. Thus also the problem of switch/case statements can be solved.

At first it’s going to be an “offline” tool like yours to unindent and reindent the complete code. But I’ll try to get the automatic indentation on a newline out of it as well, shouldn’t be too hard…

I’ll keep you updated if anything usefull comes out.

[Remi]

It’s actually a in-editor function and works better in-editor. I did create a standalone beautifier wrapper that uses the same method, but since it doesn’t have access to scope information, the alignment routine gets slightly more stupid. It was mainly for debug though because it’s easier to go through a huge file of stdout logging than try to parse through the ST console.

Anyway, good luck! Always interested in work done for the language.

[ThomSmith]

Another thought, it’d be kind of cool if indent patterns could be included in a syntax file. The two are very related and it might be possible to get more fine grained control.

What about simply indenting based on scope nesting?

• Mark an indent whenever a meta scope begins and an outdent whenever one ends.
• Collapse paired series of repeating in/outdents.
• Use these in/outdent marks to determine the correct indent level of the beginning of each line.

[Remi]

Well I was under the impression (from the discussion at the top of the page) that the auto indent was based on those Textmate Preferences flags. These do rely on the lexical scope though, which is why I thought it might be interesting to put them in with that (if desired). But honestly I’ve not looked into the auto-indent functions though I need to probably for the next feature stage and try to make sure there’s something sane in place. Currently there’s no auto-indent information for VHDL that I was able to notice, so it’ll just return to the point of last indent after a CR/LF.

For my own beautifier, I do a lot of pre-process work (removing whitespace, aligning assignment operators, etc.) on a list of the lines in the file and I rapidly lose connection with the original lexically scoped source.

[eproxus]

I’m trying to figure out a better indentation syntax for Erlang. Is it possible to de-indent a line based on the previous line? In Erlang, function syntax looks like so:

foo(A, B) ->
    line1,
    line2
    line3.

bar() ->
    line1,
    line2.

A function is a header, a list of statements separated by commas (,) and concluded with a dot (`.). I want the de-indent to happen on the line after the dot, otherwise Sublime Text indents the next function at the same level as the body of the previous function. Is this possible at all?

[kingkeith]

Great question!

A function is a header, a list of statements separated by commas (,) and concluded with a dot (`.). I want the de-indent to happen on the line after the dot, otherwise Sublime Text indents the next function at the same level as the body of the previous function. Is this possible at all?

I’ve been experimenting, and unfortunately, I can’t see a way to do it. I had the idea that it would be possible, but not purely from the .tmPreferences metadata file - I thought it might work if the syntax definition were to scope the line after the dot, so that it could be targeted specially and use different rules for that line than for the rest of the Erlang syntax (and those rules would just always tell ST to de-indent the line). My attempt:

--- Shipped Packages/Erlang/Erlang.sublime-syntax   2017-10-10 16:52:54
+++ Packages/Erlang/Erlang.sublime-syntax   2017-10-24 16:27:15
@@ -239,11 +239,20 @@
         1: entity.name.function.definition.erlang
       push:
         - meta_scope: meta.function.erlang
         - match: \.
           scope: punctuation.terminator.function.erlang
-          pop: true
+          set:
+            - meta_scope: test
+            - match: \s*$\n?
+              scope: meta.after-function-terminator.erlang
+              set:
+                - match: ^\s*
+                  scope: meta.unindent-me.erlang
+                  pop: true
+            - match: ''
+              pop: true
         - match: ^\s*+([a-z][a-zA-Z\d@_]*+)\s*+(?=\()
           captures:
             1: entity.name.function.erlang
         - match: (?=\()
           push:

--- Shipped Packages/Erlang/Indentation Rules.tmPreferences 2017-04-12 09:40:46
+++ Packages/Erlang/Indentation Rules.tmPreferences 2017-10-24 16:10:40
@@ -8,9 +8,9 @@
    <key>settings</key>
    <dict>
        <key>decreaseIndentPattern</key>
        <string>^\s*\b(end)\b</string>
        <key>increaseIndentPattern</key>
-       <string><![CDATA[^[^%]*(\b(if|case|receive|after|fun|try|catch|begin|query)\b(?!.*\b(end)\b.*))|(->(\s*%.*)?$)]]></string>
+       <string><![CDATA[^[^%]*(\b(if|case|receive|after|fun|try|catch|begin|query)\b(?!.*\bend\b)|->(\s*%.*)?$)]]></string>
    </dict>
 </dict>
 </plist>

New file: Packages/Erlang/Indentation Rules - Unindent After Function Dot.tmPreferences:

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
	<key>name</key>
	<string>Indentation Rules</string>
	<key>scope</key>
	<string>source.erlang meta.unindent-me.erlang</string>
	<key>settings</key>
	<dict>
		<key>decreaseIndentPattern</key>
		<string>.</string>
		<!-- <key>unIndentedLinePattern</key>
		<string>.</string> -->
		<key>increaseIndentPattern</key>
		<string></string>
	</dict>
</dict>
</plist>

But, because the line between the end of the function and the next function is blank, it seems to be ignored by the indentation engine when calculating the indentation for the next function (bar) in your example. Not having a blank line between the two (assuming its even valid Erlang syntax) means that ST has no way to add the new “unindent me” scope, so that’s not a solution/workaround either.

I think the best one can do at the moment is create a macro so that when you press Enter after the . that marks the end of a function, it would automatically perform a left_delete after the newline is added, to unindent one level. But this of course doesn’t help for formatting existing (parts of) documents.

[eproxus]

Thanks for such an in depth investigation!

I’ve been experimenting, and unfortunately, I can’t see a way to do it.

This is my conclusion as well.

Working with the indentation system in Sublime, I can only conclude that it is very limited. I continuously wanted all kinds of variants of the existing options, such as “decreaseIndentPreviousLinePattern”, “increaseIndentCurrentLinePattern” etc.

I’m not sure if it is feasible, but something based on the scopes defined in the actual syntax file would be great. Having to work with very cumbersome and limited regex options inside an XML settings file is far from ideal.

[ThomSmith]

I’ve experimented with just such a system. You’d want to do something like:

1. Specify which scopes should increase nesting levels. (i.e. meta.*)
2. Determine the raw nesting depth of each line. (There are some tricky cases here with punctuation.)
3. Take the list of line depths and remove redundant multiple indentation.
4. Reindent each line accordingly, ignoring some lines (e.g. lines scoped string.*).

This should work for languages with comprehensive meta scopes. It wouldn’t work for languages that don’t have those meta scopes, probably including a lot of languages with old tmLanguage syntaxes and definitely including most indentation-based languages.