Sublime Forum

Auto reloading of python module files used by plugin

#1

Hi all,

I am developing a plugin and trying to develop it in a modular way…
… so I have a top level python plugin file e.g MyPlugin.py in my Packages/MyPlugin/ folder.
MyPlugin.py imports python modules from a sub folder e.g. Packages/MyPlugin/mypymodule

After editing and saving MyPlugin.py the plugin is reloaded and I am able to test it straight away in Sublime Text 2.
My python modules are imported correctly and all works fine at first glance.

However, after editing and saving one of my module.py files located in the Packages/MyPlugin/mypymodule, the changes are not picked up in the same way as a change to MyPlugin.py. Even if I then make some change to MyPlugin.py and save it - the changes made to the modules are still not picked up, even though Sublime Text does reload MyPlugin.py.

So far the only workaround I have is to close Sublime Text and re-open it, which will soon become a complete PITA.

I had a quick search but didn’t find anything useful on the subject, but I feel I can’t be the only person to have run into this problem.
If anybody has a tip or better workaround, I’d be Grateful to know about it.

Is there some way to force a recompile / reload of the modules ?

Thanks.

0 Likes
Hot update without restarting sublime
#2

I’m not terribly knowledgeable on the subject, but have you tried deleting the *.pyc files?

0 Likes
#3

I also have similar issue. However, my plugin is supposed to connect to a server, but it only runs once, then it stops connecting.

I can see in the console that plugins are reloading, but it has no effect at all.

0 Likes
#4

Thanks for the responses,

@Nick,
Yes I tried deleting the *.pyc files, and interestingly they did not seem to get recreated.
I then opened a terminal in the module folder and explicitly ran ‘python myPluginModule.py’ which did recreate the .pyc , but Sublime Text did not seem to be using this new version.

I suspect once the modules are loaded into ST2’s embedded python interpreter they are just sitting in RAM.

I did some more Googling and found this
docs.python.org/library/function … oad#reload
stackoverflow.com/questions/4375 … hon-module
I’m thinking that maybe during development, the main MyPlugin.py could use reload inside an on_activated event callback method.
I’ll give it a try and update this post.

0 Likes
#5

If you look at SVN.py in the SVN package, or SFTP.py in the SFTP package, I use reloading so that I can actively develop in Sublime without having to constantly restart. Depending on the complexity of your dependencies, you may have to do a little work to make sure they are all reloaded in the right order.

1 Like
#6

Thanks wbond !

That was just what I needed. :smiley:
I had almost got there with something along those lines but there were 2 vital bits that I was missing.

if mod.startswith(“hxutil”) and sys.modules[mod] != None:

and I think reloading the modules in a specific order has helped too.

Many thanks.

0 Likes
#7

Implemented reloader https://github.com/astrauka/TestRSpec/blob/master/plugin_helpers/reloader.py .

import sys, imp, os, sublime, sublime_plugin
from rspec.rspec_print import rspec_print

# Dependecy reloader
# The original idea is borrowed from
# https://github.com/wbond/sublime_package_control/blob/master/package_control/reloader.py

rspec_print('Reloading rspec modules')
CODE_DIRS = [
  'plugin_helpers',
  'rspec',
]
PYTHON_FILE_EXT = '.py'

def _reload(dir, file):
  (name, extension) = os.path.splitext(file)
  if not extension == PYTHON_FILE_EXT: return

  dirs = '.'.join(filter(None, os.path.split(dir)))
  module = sys.modules.get('.'.join([dirs, name]))
  if not module: return

  if 'on_module_reload' in module.__dict__:
    module.on_module_reload()
  imp.reload(module)

for _ in range(2): # double reload required to update dependencies
  for directory in CODE_DIRS:
    for dir, _, files in os.walk(directory):
      for file in files:
        _reload(dir, file)

class ReloadPlugin(sublime_plugin.EventListener):
  PLUGIN_RELOAD_TIME_MS = 200

  def on_post_save(self, view):
    plugin_python_file = sys._current_frames().values()[0].f_back.f_globals['__file__']
    file_name = view.file_name()
    if not os.path.dirname(plugin_python_file) in file_name: return
    if file_name == plugin_python_file: return

    original_file_name = view.file_name()

    def _open_original_file():
      view.window().open_file(original_file_name)

    plugin_view = view.window().open_file(plugin_python_file)
    print("save", plugin_view.file_name())
    plugin_view.run_command("save")
    sublime.set_timeout_async(_open_original_file, self.PLUGIN_RELOAD_TIME_MS)

Imported from https://github.com/astrauka/TestRSpec/blob/master/sublime_rspec.py

import sys, os.path, imp, sublime_plugin

BASE_PATH = os.path.abspath(os.path.dirname(__file__))
sys.path += [BASE_PATH]

# reload plugin files on change
if 'plugin_helpers.reloader' in sys.modules:
  imp.reload(sys.modules['plugin_helpers.reloader'])
import plugin_helpers.reloader

It reloads nested plugin source files on any plugin file save.
In order for plugin reloading to work I save the main plugin source file and wait for Sublime to reload the plugin.
Current implementation opens plugin source file, saves it, waits 200ms and opens back the file where save was triggered. This creates a flickering experience.

Does anybody know a better way for reloading?
Is there a way to trigger Sublime plugin reload in background to avoid moving cursor from currently edited file?

0 Likes
#8

@wbond have you solved the autoreloading problem? ^^

0 Likes
#9

@wbond

 
I tried implementing the method from SFTP, but couldn’t get it to work.
( I got the array to populate correctly, but not the reload )

I also tried simplifying it & performing the reloads manually:

from imp import reload
from sys import modules
reload( modules[ "_UTILS" ] )
reload( modules[ "_UTILS.__ALL_UTILS__" ] )
reload( modules[ "_UTILS.Comments" ] )

_UTILS.__ALL_UTILS__.load ( globals() )

System: Windows 10, ST 3103
 

 
I’m working on creating a dependency template for plugins that automatically loads all sub-modules from _UTILS.

It initializes a bunch of useful stuff into globals, such as:

  • @ each view change: syntax, filepath, comment prefix & suffix, etc.
  • the first level of variables @ sublime-settings
  • a bunch of functions to eliminate repetetive tasks like extracting region data

So far, it works great on the first load, but I would like to implement the reload method to speed up development.

 
Here is the __ALL_UTILS__ module:
#@ Gist

0 Likes
[SOLVED] issue using "edit" objects recursively
#10

The key thing about reloading nested submodules in Python is that they have to be reloaded in reverse order. So first reload the files with no other project-level dependencies, then reload for all files that have had all dependencies reloaded until all files have been reloaded.

With that method you don’t have to reload twice, but you do have to know your dependency load order. There is an example at https://github.com/wbond/package_control/blob/master/1_reloader.py#L61-L214, but it is complicated by an issue with the Sublime Text .sublime-package loader. Hopefully I will get that fixed in one of the next few releases.

I’m not sure that Sublime Text is ever going to try and handle reloading packages completely on save, but I won’t rule it out.

2 Likes
#11

@wbond

 
That code is kind of over my head :sweat_smile:.

It seems like it’s maybe more complex than what I need to implement.

Pretty much, the only scenario I’ll be working with is a single dependency @ Packages/_UTILS
The entire contents of the _UTILS directory is:

[ Comments.py, Edit.py, Indentation.py, Region.py, Sort.py, Verify.py, Variables.py ]

( Discarding the __ALL_UTILS__ module I mentioned before.   I don’t think I would need it anymore using your method? )
 

 
I tried going back to the first method I attempted, like in your SFTP plugin.

Here is what I have:

from imp import reload
from sys import modules

reload_mods = []

for mod in modules:
	if mod[0:6] == "_UTILS" \
	and modules[mod] != None:
		print ( mod )
		reload_mods.append(mod)

mods_load_order = [
	"_UTILS",
	"_UTILS.Comments",
	"_UTILS.Edit",
	"_UTILS.Indentation",
	"_UTILS.Region",
	"_UTILS.Sort",
	"_UTILS.Verify",
	"_UTILS.Variables",
]

for mod in mods_load_order:
	if mod in reload_mods:
		print ( mod )
		reload(modules[mod])

Comments.test()

When I save, the print ( mod ) @ both for mod in modules & for mod in mods_load_order is returning:

_UTILS
_UTILS.Comments
_UTILS.Edit
_UTILS.Indentation
_UTILS.Region
_UTILS.Sort
_UTILS.Verify

So I’m guessing the values are valid as far as modules is concerned.
( although _UTILS.Variables isn’t being picked up for some reason… )
 

 
This is the traceback I’m getting @ Comments.test()
( I also tried _UTILS.Comments.test() )

Traceback (most recent call last):

  File "C:\Program Files\Sublime Text 3\sublime_plugin.py", line 76, in reload_plugin
    m = imp.reload(m)
  File "./imp.py", line 276, in reload
  File "<frozen importlib._bootstrap>", line 584, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 1022, in load_module
  File "<frozen importlib._bootstrap>", line 1003, in load_module
  File "<frozen importlib._bootstrap>", line 560, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 868, in _load_module
  File "<frozen importlib._bootstrap>", line 313, in _call_with_frames_removed
  File "C:\Users\Fico\AppData\Roaming\Sublime Text 3\Packages\Parkour_v1\Parkour_v1.py", line 48, in <module>
    Comments.test()
    
NameError: name 'Comments' is not defined
0 Likes
#12

Is your mods_load_order really in dependency order? Does _UTILS really have no dependencies on any of the submodules? To me it looks like you just put them in alphabetical order.

For instance, if _UTILS.Comments imports from . import Edit you’ll need _UTILS.Edit in the list before _UTILS.Comments.

0 Likes
#13

The only submodule that uses other submodules from _UTILS is Variables, which for some reason is not showing up in the mod[0:6] == "_UTILS" filtered modules.

I just ran a full print of modules and it is listed, but just as Variables without the _UTILS prefix.

That issue is odd, but the test function I’m running from Comments should work since it doesn’t require a single module.   Variables isn’t referenced at all during the test from my last post, it’s only called later once the TextCommand is executed.

###Note:

_UTILS is just a directory, there is no _UTILS.py

Does that affect how I would implement your code?

###Edit 1:

I just commented out import Comments, Edit @ Variables, and it now shows up in the mod[0:6] == "_UTILS" filter.

###Edit 2:

I changed

import Comments, Edit
to
from _UTILS import Comments, Edit

and Variables is now included with the filtered modules.

 

The primary issue with the traceback still persists.

0 Likes
#14

just an FYI, but there is a str.startswith function in Python, in case that would make your mod[0:6] == "_UTILS" check “cleaner” :slight_smile:

1 Like
#15

Can you post the contents of Parkour_v1.py somewhere?

0 Likes
#16

I just posted all of the code & a short explanation.   I blocked off two sections @ Parkour_v1.py so you can easily toggle between them.   One is your SFTP code, and the other is my original implementation.

#@ GitHub

0 Likes
#17

ah I wondered how you did your fancy comments in your Python code :smiley:

1 Like
#18

Yep… about 5k lines of hot-mess, “I-have-no-idea-what-I’m-doing” code.   :joy:   But it works!   And it’ll be a lot cleaner this time around.

1 Like
#19

My guess is you are probably dealing with some arcane aspect of imports. At https://github.com/Enteleform/ST_Parkour/blob/master/Parkour_v1/Parkour_v1.py#L48 you haven’t explicitly imported that name. I presume your load() function is doing that. Most likely you have some sort of issue related to that.

This is all beyond my ability to dig in right now. Personally, I would recommend using simpler, explicit, relative imports.

1 Like
#20

 
That’s what I was attempting with Parkour_v1.py#L32.
I also tried _UTILS.Comments.test() @ line 48.

 

 
I wouldn’t mind doing so to speed up development, and that is what I was attempting with your code.   I guess I just don’t understand it enough to figure out what is missing in order to get it working.   In my implementation of your code, I was completely discarding the use of __ALL_UTILS__

 

Thanks for your help :slightly_smiling:
I’ll keep trying to understand what’s going on & see if I can get it working.

0 Likes