The indexer is a background process that scans your code to find class and function definitions. It powers the Goto Definition / Goto References functionality, along with the Definitions / References popup.
There likely reason is a combination of various implementation details of Sublime Text. We use syntax definitions that are optimized for performance. When a syntax definition (such as Markdown) includes another syntax definition (such as CSS, for fenced code blocks), it must copy the embedded definition to allow for the highest performance when highlighting files. The Markdown syntax, since it is used with all sorts of languages, supports highlighting a whole bunch of syntaxes within fenced code blocks. Including all of these increases the size of the cached syntax definition on disk.
When the indexer runs, it loads the cached syntax definition into memory and uses it to highlight a file. Once the file is highlighted, it can use the scope information to find function and class definitions. The first time it runs into a file of a specific type, the cached definition has to be loaded into memory. If the cached syntax definition is large, it is possible based on the speed of your hard drive that it make take a short while to load the syntax definition.
Now, for the best performance with files, we incrementally load the file contents as we syntax highlight it. This is because the IO of loading and CPU usage of highlight don’t compete. So, previously we believe you were running into a situation where you would:
- Git would change a markdown file as part of your process
- Sublime Text would see the file changed
- It would create an indexer background process to look for definitions
- The indexer would open the file and start reading, and at the same time load the syntax definition and start highlighting
- Since the Markdown definition is large, it would take a little while to load
- During that time, Git would delete the file (and the way we open it, that would be fine, no errors)
- Next, we’d continue loading the file and syntax definition while the file was in a pending delete mode
- Git would try to recreate the file during the rebase, but would get back an error indicating the file was in pending delete mode and would print an error to you
For the indexer, we’ve revamped the loading and highlighting process to load the entire file into memory as quickly as possible and then close the file handle. Then we load the cached syntax definition and do our work. This change is acceptable since the indexer doesn’t display the file to the user, so a slight decrease in overall performance isn’t really a regression, especially if we can sidestep issues with other software. Hopefully this reduces the amount of time we have a file handle open, effectively working around the issue that Git wasn’t waiting for the pending delete to complete.