Sublime Forum

ST build 3200 on Linux: large number of inotify watches

#1

Ubuntu 18.04.2 just after having updated Sublime to 3200.

Similar issue to this:

Sublime appears to be consuming a whopping 6000+ watches (9000+ upon attempting closing all these files and opening two of them in isolation–before opening the files, there are only about 100 watches); I have doubled my systemwide max inotify watches (fs.inotify.max_user_watches) to 16384 to account for the issue. I am still trying to determine exactly which files are being watched and why so many watches are registered. I have no folders or projects open, just 6 lone, unrelated files.

This is a useful utility for tracking inotify use: https://github.com/fatso83/dotfiles/blob/master/utils/scripts/inotify-consumers

This is a one-liner that’ll show all watched files for a process name (replace ‘sublime_text’ with the process name, or replce the whole pgrep invokation with the process id if it’s already known):
(fos=/proc/$(pgrep sublime_text)/fd/* && grep inotify ${fos/fd/fdinfo})

Piping to wc -l will count, and I definitely recommend this if you’ve run out of watches, since the list will be massive (unless some other program is the culprit).

Note that increasing watches is a plausible fix, but for two files open in two large git repos I get 15000+ watches–and opening individual files takes much longer in these repos (presumably because they’re waiting on work in the back end of the application). I am aware this is in part a limitation on inotify (http://man7.org/linux/man-pages/man7/inotify.7.html see the part about requiring subtree watches)–what I’d like to see is some measure to limit the behavior, perhaps just watches on open files, though I realize there may be some feature requirement in the git integration that calls for monitoring the entire repo tree.

Three of the files i have open are part of large git repos; this may explain the issue–however, I do not have any plugins that use git, and disabling the recently added git integration by changing “show_git_status” to false fixes the issue, so it is clearly related to this new feature.

It would be nice if the git integration were only active if, say, the root working directory was the root folder of the open project, or if it could be selectively enabled on a project-by-project basis that would be awesome. Otherwise, doing work in kernel dev, or other hardware-related work, where large repos are the norm, is probably infeasible with this new feature. (I can only image how anyone at Google using Sublime may have reacted on upgrading to 3200…)

TLDR:
Sublime watches lots of git repo files even when unasked; disabling recent git integration fixes the issue; some default limits on this behavior/per-project settings would help.

Thanks! I hope this issue report is helpful either to devs or users. Please let me know if it’d be better placed on github or elsewhere.

1 Like
#2

I’ve also seen this problem in linux which seems particular to Sublime’s git integration. Suggest opening a new issue with the above info along with any other details.

0 Likes
#3

I was also facing the same issues as mentioned above and tried many solution for it but its not working so please tell if anyone have any solution

0 Likes
#4

Inotify terminology can be tricky to get right, and the terminology does make a difference. The issue you linked to @VaelynPhi is an old issue about inotify instances. It used to be that we had one inotify instance for each folder you added to the sidebar. Since build 3158 (and Sublime Text 3.1) we use a single inotify instance for the entire application.

However, with inotify we have to add a watch for any folder we want to get filesystem notifications for. So when you open a folder in the sidebar, we add a watch for the folder, plus every single subfolder of it, recursively. So if your folder has 1,000 subfolders in the tree, we are going to use 1,000 watches.

When we track a git repo, we add watches for the repo root and every single subfolder, recursively, that is not ignored. This ensures when a file inside the repo changes, we get the updated status and show it in the sidebar.

During the development cycle before the launch of 3.2, we had users reporting it as a bug when we did not show git status for any file located within a git repo, even if the root was above the folder sidebar, if the file was opened without being in the sidebar, or even if the file was symlinked from a git repo into a non-git folder in the sidebar.

According to info I found, each watch uses about 1kb on a 64bit machine. So 16,000 watches is about 16MB of ram. On a modern machine, Sublime Text using an extra 16MB of ram to provide real-time git status notifications doesn’t seem all that crazy. Are you running on a really resource-constrained machine, or is it more of a philosophical stance on not wanting to increase your system limit for inotify watches?

0 Likes
#5

Thanks for clarifying that @wbond, but at least in my case I was talking about watches. The issue for me is not so much philosophical since without digging into inotify it’s impossible to know whether XXX watches is actually a real-world resource drain. It’s more a question of the inotify watch limits set up by default in most distros (around 8k watches being fairly typical). I’m working to the assumption, perhaps wrongly, that such limits are in place to avoid the potential for big slowdowns. If the RAM used by 16k watches is 16Mb, I wonder why do most distros default to 8k as a limit?

Of course these limits are fairly easy to raise, but it’s worth recognising that loading a big project into ST could hit these limits. As you’ve explained it, the approach ST takes seems fairly logical to me. So the question is can anything be done to optimise this?

0 Likes
#6

I also was talking about watches, and merely linked the old issue because it is similar; I would have directly re-opened the issue if I had thought it was the same. Inotify is some of my bread and butter, so I’m aware of its limitations; I’ve considered opening and issue on it about how watches work, even, but I have worked around it by using strategies like the one I suggested.

And yes, at least for me, that many things going on for a feature I do not use in repos that are massive is at least something to consider, which is why I disabled git integration. This solves the problem for me because I don’t use it. However, there may be users for whom not having it is an issue, but who also may not be able to handle the resource use–consider that there are larger repos and some may work in multiple repositories. For instance, if I were doing a merge, I could easily have three instances of these repos open, using about 35k watches (not instances). Kernel dev may be worse. Also, opening files in large repos is initially slow.

I’m suggesting some control over how the watches work; perhaps even a setting to limit it for large repos, so that those who really must have the graphical display can keep it and some users can opt out. For my own part, I would love if it only tracked open folders (folders whose files have been opened, which would definitely make sense for my own workflow, but I don’t want to presume it would for others).

Naturally it’s up to ya’ll how you use reported issues, but I would like to note that some user bug reports are just excessive. (I suppose some users’ workflows might depend on symlinking, but if I understand the situation you’re describing correctly, that’s just insane–the other way around maybe makes sense.*)

Personally, I’m going to use Sublime either way–I don’t really use the GUI git integration, and disabling it worked for me. FWIW, this is the first time I’ve felt compelled to open anything like an issue report, and I posted here largely to help if anyone else had issues related to this; Sublime is by my lights the best GUI text editor–please keep up the good work.

*Unrelated diversion:
If the symlink is in the repo, and the actual open, original file is outside, this expectation is nuts. If the opened file is a symlink to a file inside a repo, that at least seems consistent with your beta testers’ expectations. [Of course, only the first situation seems like a sane workflow, say where a symlink to secrets is checked into the repo so they may be kept safely outside of it… but presumably a sane user would have opened the symlink in the repo, not the target file, but at this point I am definitely overthinking this example.]

0 Likes
#7

Are you on build 3200? Were you suffering from the issue I described, or the issue I linked?

Did you try adjusting your system’s maximum file watches using sysctl? First, check it with:

sysctl fs.inotify.max_user_watches

Mine was 8192, so I doubled it; you can make it larger, with the expectation that this is approximately the maximum number of kilobytes in RAM that inotify watches will use. (You may need to precede this with sudo to run it with escalated privileges.)

sysctl fs.inotify.max_user_watches=16384

Increasing this limit will prevent you from having issues with commands like tail -f, programs like Dropbox or Resilio Sync, or system features that set up watches, but only if your programs don’t use more than the limit you set.

Alternatively, did you try disabling the GUI git integration by editing your settings (Preferences->Settings), adding "show_git_status": false to your settings hash? If you don’t need the integration, this is the most expedient way of avoiding the issue.

0 Likes
#8

Not one bit. Maybe for your workflow, but most contributors to the standard packages do this: https://github.com/sublimehq/Packages#readme.

It is way more confusing as a user to try and figure out what a folder that you know is part of a git repo is not showing status information, just because the folder you added to the sidebar isn’t rooted where the repo was checked out.

2 Likes
#9

I would expect ST to set up watches if I opened the file via the symlink path (ie, if the path I provided ST was in the repo, symlink or no); if I opened it supplying a path outside the repo and expected ST to crawl through the file system looking for symlinks to the file I opened… that is the situation I think would be ridiculous.

0 Likes