Sublime Forum

How to prevent deadlocks in Sublime

#1

Let’s start by considering the next minimal scenario:

  1. Install automatic package reloader go to Preferences\Package Settings\Automatic Package Reloader\Settings and add this:

    {“reload_on_save”: true}

  2. Create a foo folder in Data\Packages with the next 3 files:

start.py:

from foo.a import A

a.py:

from foo.b import global_b


class A():

    def __init__(self):
        print("A 1")
        somekey = global_b.load().get("somekey", [])
        print("A 2")
        if not somekey:
            global_b.set_value("somekey", somekey)
        print("A 3")

global_a = A()

b.py:

import sublime


class SettingsManager():

    def __init__(self, name="foo.sublime-settings"):
        self.name = name

    def load(self):
        return sublime.load_settings(self.name)

    def set_value(self, key, value):
        instance = sublime.load_settings(self.name)
        instance.set(key, value)
        sublime.save_settings(self.name)
        return value

global_b = SettingsManager()
  1. Open start.py and start pressing save till Sublime hangs (do it as fast as possible…), sometimes it will hang fast other times it’ll take longer.

Here’s the thing, after a bit ammount of researching I’ve figured out the problem here basically a deadlock will be created between the next threads:

Few questions:

  • In order to fix the deadlock I’d like to be able to find a way to reproduce it faster, right now it can take me quite a lot of time… any idea to force it to appear faster?
  • The sublime settings family functions… what are they doing behind the curtains? Are they thread safe?
  • What’s the explanation of this deadlock and what’s the proper way to fix it (in code)? I mean, in the above minimal code it can take quite a lot of effort to make it appear but with different larger code using the same pattern the deadlock will appear much more easily… which it’s really disturbing while you’re coding plugins as basically you need to kill the plugin_hosts processes or restart sublime.

If you’re a windows user you can check the deadlock by using process explorer and checking the threads stack frames… on linux I guess using stuff gdb or similars would help. Also… there is this https://docs.python.org/3.7/library/sys.html#sys._current_frames that’s really helpful to know more about this interesting stuff.

Anyway, the above case tries to be a mcve to learn more about deadlocks, unfortunately it isn’t fast to reproduce… The whole point of this thread is mainly to know how to catch deadlocks and knowing the best practices to avoid them. I’ve created this little testcase after spending a lot of time debugging and learning a bit about this stuff.

Thanks in advance and sorry for the long thread :blush:

0 Likes

How can I reload a plugin in sublime console?
#2

Here’s a little debugging session showing up the deadlock with a fresh latest version of SublimeText build 3176 x64 running on win7 ultimate. Here’s the package control settings:

{
	"bootstrapped": true,
	"in_process_packages":
	[
	],
	"installed_packages":
	[
		"AutomaticPackageReloader",
		"Package Control",
		"PackageResourceViewer"
	]
}

For some reason if I try to reload automatically the package with something like this [window.run_command("save") for i in range(100)] the deadlock won’t show up… That’s why I’m testing it manually, which is a really slow way :frowning:

In case you know a faster way to reproduce & fix it it I’d be glad to hear. Thanks

0 Likes

#3

More relevant info, here’s a little snapshot once the deadlock happens:

2018-07-19 16:50:15.786994

# ThreadID: 4120
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\package_reloader.py", line 73, in <lambda>
  sublime.set_timeout_async(lambda: self.run_async(pkg_name))
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\package_reloader.py", line 92, in run_async
  reload_package(pkg_name, verbose=pr_settings.get('verbose'))
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\reloader/reloader.py", line 47, in reload_package
  reload_plugin(pkg_name)
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\reloader/reloader.py", line 118, in reload_plugin
  sublime_plugin.reload_plugin(plugin)
File: "D:\testcase\Sublime Text Build 3176 x64\sublime_plugin.py", line 116, in reload_plugin
  m = importlib.import_module(modulename)
File: "./python3.3/importlib/__init__.py", line 90, in import_module
File: "<frozen importlib._bootstrap>", line 1584, in _gcd_import
File: "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File: "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\reloader/reloader.py", line 175, in load_module
  return module.__loader__.load_module(name)
File: "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
File: "<frozen importlib._bootstrap>", line 1022, in load_module
File: "<frozen importlib._bootstrap>", line 1003, in load_module
File: "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
File: "<frozen importlib._bootstrap>", line 868, in _load_module
File: "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Packages\foo\a.py", line 14, in <module>
  global_a = A()
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Packages\foo\a.py", line 8, in __init__
  somekey = global_b.load().get("somekey", [])
File: "D:\testcase\Sublime Text Build 3176 x64\sublime.py", line 1161, in get
  return sublime_api.settings_get_default(self.settings_id, key, default)

# ThreadID: 5592
File: "D:\testcase\Sublime Text Build 3176 x64\sublime_plugin.py", line 116, in reload_plugin
  m = importlib.import_module(modulename)
File: "./python3.3/importlib/__init__.py", line 90, in import_module
File: "<frozen importlib._bootstrap>", line 1584, in _gcd_import
File: "<frozen importlib._bootstrap>", line 1565, in _find_and_load
File: "<frozen importlib._bootstrap>", line 1532, in _find_and_load_unlocked
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\reloader/reloader.py", line 175, in load_module
  return module.__loader__.load_module(name)
File: "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
File: "<frozen importlib._bootstrap>", line 1022, in load_module
File: "<frozen importlib._bootstrap>", line 1003, in load_module
File: "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
File: "<frozen importlib._bootstrap>", line 868, in _load_module
File: "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Packages\foo\start.py", line 1, in <module>
  from foo.a import A
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Installed Packages\AutomaticPackageReloader.sublime-package\reloader/reloader.py", line 138, in __import__
  module = orig___import__(name, globals, locals, fromlist, level)
File: "<frozen importlib._bootstrap>", line 295, in _lock_unlock_module
File: "<frozen importlib._bootstrap>", line 221, in acquire

# ThreadID: 7620
File: "./python3.3/threading.py", line 878, in _bootstrap
File: "./python3.3/threading.py", line 901, in _bootstrap_inner
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Packages\deadlocks\commands.py", line 56, in run
  self.stacktraces()
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Packages\deadlocks\commands.py", line 69, in stacktraces
  f.write(stacktraces())
File: "D:\testcase\Sublime Text Build 3176 x64\Data\Packages\deadlocks\commands.py", line 22, in stacktraces
  for filename, lineno, name, line in traceback.extract_stack(stack):
0 Likes

#4

For those who want to have some fun with this issue, it seems the below snippet will crash sublime text consistently:

[sublime.set_timeout(lambda: window.run_command("save"), i*100) for i in range(50)]

Basically the above snippet would give enough time to the event handler, which runs in a different thread than Sublime’s main one, to respond.

Sometimes it’ll show up after run the above snippet at first, other times you’ll need more than once to check if deadlocks are created… guess it’s +/- a pretty decent way to check if a Package contains initalization code which is “deadlock-free” :slight_smile:

0 Likes

#5

Just FYI, you’re not supposed to call API functions during import time. They should be done within plugin_loaded(). Perhaps that will stabilize things. Automatic Package Reloader does a bit of trickery, so I’m not surprised it would have issues when stressed.

0 Likes

#6

Thanks for your comment, yeah, you’re absolutely right. 2 days ago I’d come up to that exact same conclusion. After a lot of researching & debugging in order to be able to reproduce deterministically these schrödinbugs I sadly accepted the fact that wouldn’t be possible so I decided to take a different approach… I started refactoring my real-world package to test “hypothesis” that could cause deadlocks, one of them was simply assuming that global variables calling API functions that could lock resources at import time wasn’t thread-safe… After the refactoring was done, I started stress testing the package (saving at random intervals few hundred times) and the deadlocks were gone magically.

I’ve learned quite a lot of useful stuff thanks to these nasty bugs. Dealing with deadlocks and heisenbugs that make freeze ST isn’t scary anymore and next time it happens to me I’ll know pretty well how to deal with them quite effectively.

So… all in all, it’s been an interesting journey :wink:

0 Likes

#8

When you have issues - in the top menu go through Tools > Developer > Profile Plugins and that’ll tell you the hooks and how much time is used for each plugin so you can see where the problems are…

0 Likes