Sublime Forum

How are symbols detected, and where are they stored?

#1

I was playing around with VS Code, and it seems like Sublime’s symbol detection, especially in an entire project, is much more complete and faster. Also, VSC adds a .vscode/tags file to every project, whereas Sublime doesn’t.

So I’m wondering, how does Sublime’s symbol detection features, for “Go to symbol (in project)” work, and where is this info stored?

0 Likes
#2

Overall, indexing is controlled via the following settings (shown here with default values):

    // File indexing parses all files in the side bar, and builds an index of
    // their symbols. This is required for Goto Definition to work.
    "index_files": true,

    // Set the number threads to use for indexing. A value of 0 will make
    // Sublime Text guess based on the number of cores. Use the index_files
    // setting to disable all workers.
    "index_workers": 0,

    // index_exclude_patterns indicate which files won't be indexed.
    "index_exclude_patterns": ["*.log"],

The indexing happens in the background, although you can view the status of the indexer by selecting Help > Indexing Status... from the menu. As far as I’m aware, the resulting indexes are stored in the Index folder, which shares a parent folder with the Packages folder, which you can get to by selecting Preferences > Browse Packages... from the menu. The index files are a binary format for speed reasons and may or may not be proprietary (I’m not sure offhand if Sublime is using a DB library of some kind or not, for example).

Simplistically, the steps go like this:

  1. A syntax is chosen for the file based on it’s extension/contents.
  2. The syntax is used to split the tokens in the source file out and apply scopes to them. This is the same mechanism by which syntax highlighting is done.
  3. From the list of resulting tokens and scopes, tmPreferences style file(s) are used to determine what symbols would appear in the indexed symbol list (i.e. project wide) and which appear in just the regular symbol list.

For the case of Python for example, the syntax for a .py file is Python/Python.sublime-syntax, which assigns scopes to all of the various tokens in the file such as class names, function names, operators, etc.

The file Python/Symbol List.tmPreferences contains the meta information for which symbols should appear in the symbol list in general and Python/Symbol Index.tmPreferences contains the meta information for which symbols should show up in the indexed symbol list.

You can use PackageResourceViewer to browse through the files in question and see what they contain, or If you’re running one of the latest dev builds you can use the View Package File from the command palette to do the same thing without having to install anything first.

The Unofficial Docs have information on symbols that explains how the tmPreferences files are laid out and what options are available.

The Official Docs have information on how syntax definitions work along with scope naming information.

2 Likes