Sublime Forum

Implementing pip support - Your ideas, please!

#1

Hey folks,

I like to have your opinion on a recent plugin idea of mine.

One thing that really loved to change about Sublime Text is the lack of support for third party python plugins.
One thing which makes python so great is its vast ecosystem of packages. Sublime would strongly benefit if there was less of a barrier to incorporate external dependencies into plugins. Plugins could feature a greater variety of functions and would be easier to develop.

Currently, you need to manually include a complete python into your plugin folder. This is, as we may all agree, very bad for maintaining and distributing plugins.

Besides a few popular python libraries are repackaged as Sublime plugins, such as requests. Yet, this approach is obviously impractical to scale.

A potential solution may be the following:
Harnishing the pip download command to automatically provide dependencies to a plugin.

This may work as such:

  1. The pip download function would be implemented into a dedicated plugin. Let’s call it: sublime-pip
  2. When you develop a new plugin for which you need additional python modules, you are going to include sublime-pip into your dependencies.json.
  3. You are also going to include a requirements.txt specifying the required libraries (and their versions).
  4. Upon install or update of your plugin, sublime-pip will download the dependencies into a subfolder within your plugin directory. (Like: /plugin_dir/libs/dependenc_1/)
  5. Now, your plugin can do a relative import of the modules.

This is the plugin idea. Could you give me some general feedback on it? Do you think it would be useful to you? Would it ease your plugin development? Any roadblocks you see in developing sublime-pip? Are there better ways to achieve the same goal?

Thanks.

2 Likes
#2

I am not the most experienced plugin developer but I did spent a significant amount of time with the st plugin system and would love to see something like this. For example, it took me some time to get Paramiko to work.

1 Like
#3

Many packages on pypi rely on absolute imports internally (for no good reason, imho), which would oppose your idea of being able to import the downloaded dependency relatively as the internal imports would fail.

The only proper way to fix this shortcoming would be to add the dependency’s parent folder to sys.path, but this will then quickly become a mess because of potential conflicts with other Packages using the same dependency.

Another possiblity would be to dive a little deeper into how Python loads modules and hook into module discovery to dynamically make the absolute imports of these pypi packages point to their specific local copy. This would probably also involve messing with builtins.__import__ to bypass the lookup in sys.modules. It’s been a while since I’ve dealt with this however, so take this only as a rough overview at best.

3 Likes
#4

@FichteFoll Indeed I have not been aware of this absolute import problem with many packages. Sounds like a deal breaker for me. Any idea how prevalent absolute imports are in pypi packages? Maybe if we even could still cater the majority of packages in the way I described it would be a huge win.

0 Likes
#5

Very prevalent. Going through dependencies hosted on https://github.com/packagecontrol, they’re used in pypi, pygments. Relative imports are used in requests, pyyaml, oauthlib. And these are popular packages. I imagine it’s even more prevalent for unpopular packages.

1 Like
#6

@FichteFoll Thank you very much for the information. Indeed I was hoping to be able to move the whole Sublime ecosystem forward with a single plugin. A somewhat naive assumption, as it turned out. :neutral_face:

I am not a beginner in python, but black magic such as messing with builtins.__import__ is currently beyond me.

The only other option that I see is to provide each plugin with a distinct virtual environment. Such would be comparatively easy to implement I believe, but I am afraid it may be a lot of overhead on top of such lightweight editor as our beloved Sublime.

0 Likes
#7

AutomaticPackageReloader may contain some code to implement this idea. It is just a little more than reloading a package.

I could imagine the following package structure.

|-- src
      |-- your_package.py
|-- deps
     |-- first_deps
     |-- second_deps
|-- loader.py

in loader.py, it will be essentially the reloader of AutomaticPackageReloader + some tweaks

0 Likes
#8

I recently found ST 3142 creating a Data/Lib/python3.3 in my portable version. I guess there is something going on to handle dependencies in a better way than we know it today.

I think at one day we well be able to use 3rd party libraries as we are used to it in an ordinary python environment.

0 Likes
#9

@randy3k Could you elaborate. I really do not understand how this would deal with the issue of absolute imports.

@deathaxe Great spotting. Yet, “one day” sounds rather unsatisfying. If the core devs have something in the making they should let us all know. :unamused:

0 Likes
#10

We would basically define a finder object so that when the package is loaded, it would also look into the folder deps.

I have put it into a repo

Note that there are still some restrictions, the dependencies have to be imported globally but no dynamically. The following code will not work

class SayHello(sublime_plugin.TextCommand):
    def run(self, edit):
        from foo import hello
        sublime.message_dialog(hello())

Also, it does not work if a dependency imports anything absolutely but dynamically.

1 Like
#11

@randy3k Ah, a finder. Actively looking into the dependency. I am not sure how robust this could work, but interesting nonetheless. Thanks for adding.

0 Likes
#12

Now, it only loads .py and __init__.py, we also have to cover the cases of binary libraries. I believe this concept works in o lot of cases (with more improvements). At least it won’t break other packages if it doesn’t work.

0 Likes
#13

My hope is to move PC dependencies out of the Packages folder and have them in a folder that was part of sys.path so the Package Control loader junk was no longer necessary. No development work has been done on the PC side of this yet, but I wanted the plumbing in place with 3.0 since there should be a very quick uptake once that is released.

Personally, I have no plans on implementing pip support because it is a veritable mine fields of issues with paths, compilers, setup.py scripts, setuptools versions, etc. Even if a bunch of that was all sorted out, if package developers get in mind that they can use all packages on pip, we are going to start to run into issues with users not having the correct compiler installed for Python 3.3 on Windows, and all of the support related to that. This is why I implemented the “simple” version of dependencies that exist now. Everything needs to be extracted, compiled, translated, etc so that all that needs to be done is have files extracted onto the filesystem. Any necessary functionality beyond that is going to be a major support headache for everyone involved. /rant

6 Likes
#14

To dive a bit deeper into this, you’d unroll the all, st3_windows folders etc on installation and merge them into a shared dependencies directory that is included in sys.path? Might make more sense to continue this in the appropriate issue, though.

(Upstream issue: https://github.com/wbond/package_control/issues/800)

On the topic at hand, mind you that even pip itself still has no proper dependency resolver:

0 Likes
#15

I believe we’ll do something like that, yes. The thing I haven’t worked out is to keep track of what files belong to what dependency so they can be uninstalled properly/fully. We’ll probably need to keep some JSON data somewhere.

0 Likes
#16

I am looking forward excitingly to have a separated dependencies folder. I never felt comfortable with all the mixups in the Packages folder and the lack of a globally available libs folder.

1 Like
#17

Here’s an example where I can’t find a workaround via messing with sys,path.append: I’m trying to use recordclass (installed via pip3 install recordclass) but am not able to use it from sublimetext, see https://stackoverflow.com/questions/57400603/how-to-load-pip-installed-module-from-sublimetext

0 Likes
#18

Not sure whether the reason is that sublime bundles its own python interpreter (3.3 IIRC) and my system python is 3.7, and it installs as recordobject.cpython-37m-darwin.so so can’t be seen by sublime’s python; but temporarily renaming to recordobject.cpython-33m-darwin.so didn’t help.

The only way to embedd python into an application and provide application specific API is to use the python library, which is of the choosen version. ST3 started with python 3.3 and keeps it for compatibility reasons with existing plugins.

Compiled python modules are always bound to a specific python version. This is what the -m37- is trying to tell you. Renaming is useless as the bytecode changes with each python version.

You need to find a version which was compiled for python 3.3 or try to compile it yourself. Maybe without luck python 3.3 has reached end of life and is no longer supported by most libraries.

0 Likes