Editing SublimeText process path programatically

[BPL]

Here’s the thing, many ST packages require you to have installed tools like pep8, flakes, pep257… they expect them to be available on system path, well… fair enough.

Now, the thing is, personally I don’t have any system-wide python available on PATH, as a golden rule I use instead just virtualenvs as a good practice so the only purpose of my system-wide pythons is to serve as “empty containers (no packages installed)” just for the sake of creating create virtualenvs.

That said, I’d like to know how I can change the ST PATH process env var programatically so that way I could add my venv scripts folder to such path and all ST packages expecting any python tool to be visible on PATH would find it.

Hope it makes sense what I’m asking here.

Thanks in advance

[BPL]

Thing is, if i modify manually os.environ[‘PATH’] on the sublime text console the rest of plugins will inherit this environment and I kinda understand why, if we consider a plugin as a child process of this “console” it makes sense they inherit the environment.

Now, problem is different here, I’m trying to figure out how to modify this “console process” environ[‘PATH’] programatically from a plugin.

Let me put you an example, let’s say you got a package like virtualenv , you definitely want to append the virtualenv scripts folder to the “console process” environ[“PATH”] somehow but I’m unsure how to do that.

@wbond Hi, any advice about it?

[dhbn]

I start the the portable version with a batch script on Windows. In the script I modify the user path to add my portable tools directory.

In that way you don’t need to do that in Sublim Text. That approach should work on Linux / Mac as well.

[BPL]

Not what I’m looking for, I’m building a plugin to manage virtualenvs on windows so I want the plugin to do the dirty job without bothering the user adding bat files.

Right now i’m testing to change os.environ[“PATH”] from the plugin side… for some reason (i still don’t understand) changing os.environ on the plugin side will also be visible on the console. So maybe that’s already good enough to persist the environment and being visible to the other plugins… to be honest, I’m trying to wrap up this whole thing in my head to have all concepts clear about process relationships, environment inheritance &.plugins… That’s why I’ve pinged @wbond before, i’m sure he knows more about all these internal details

[kingkeith]

There is only one Python host process running, changing the environment from the ST console or a plugin will affect all plugins and the console…

[BPL]

When you say there is only one python host process, do you mean the sublime text child process called plugin_host? Could you please elaborate about that?

For instance, when you declare a global variable on a plugin you won’t see it by using globals() on the console, how so?

[deathaxe]

… changing os.environ on the plugin side will also be visible on the console. …

The reason is the same as for the reloading stuff. As @kingkeith correctly stated ST runs one “python.exe”. All plugins are handled as “python packages”. If a module/package/library modifies some of python’s environment variables it effects the whole ecosystem. Doesn’t matter whether you run the standalone python interpreter or the one built into ST.

declare a global variable on a plugin you won’t see it by using globals() on the console

The console by default knows about its very basics. Just run globals() to see, what it knows about. To get access to any package’s namespaces, just import it. e.g.: import Default and call globals() again. It’s just the same as you would open python from the shell.

You can’t/must not manipulate ST’s python variables to point to an external script! It is meant for ST plugins only.

You’ll need to use subprocess.POpen to call your standalone python for external scripts/tools. You can pass a modified os.environ to POpen by the env argument.

   env = os.environ
   env["PATH"] += ";mypath"
   subprocess.POpen(["python", "script.py"], env=env)
   # ...

You can have a look into Jedi - Autocompletions package. It calls python to run Jedi externally with passing custom PATH to let Jedi know about ST’s API and libraries.

[BPL]

Thanks for the elaborated explanation! Right now i’ts become a little bit more clear. I’ve changed my virtualenv_manager plugin to modify os.environ and everything is working as it should be.

The only main issue I can think of would be, I got a command which activates a certain virtualenv and it tweaks the PATH accordingly. But by default ST text starts with an empty PATH… thinking deeply about it, there may be some problems if some other plugins requires certain python tools to be present at startup… anyway, i prefer having this little issue before than being “forced” to have a system-wide python on the system PATH, i do really dislike that approach very much.

Btw, what’s the main responsability of the child process plugin_host, what’s that process doing behind the curtains?

[OdatNurd]

It’s essentially a “sandbox” that keeps plugin code separate from the rest of the core, so that if a plugin does something untoward and blows up, it just crashes the plugin host but the editor keeps running.

When that happens you see a message telling you that the plugin host has stopped and plugin functionality won’t be available again until you restart Sublime, as opposed to the application vanishing out from under you and causing you to potentially lose work.

[deathaxe]

As far as I know about it is the primary “python.exe”, which runs all plugins - as the name says. It is decoupled from the main sublime_text.exe in order to keep the GUI responsive, even though some heavy plugin activity takes place. It’s what is called a multi-process-architecture. All commands from the GUI are passed to this task and the responses modify the GUI.

I experienced situations when the plugin_host.exe crashed which cause the main editor to keep going but without any python module running anymore.

other plugins requires certain python tools to be present at startup…

You run a virtualenv which you install python with all the tools required for ST to run all its plugins, such as SublimeLinter and co.? If python is so harmful, why don’t you run the whole ST in that virtualenv? Wouldn’t that be easier, than trying to point ST to it especially to make other plugins use it?

anyway, i prefer having this little issue before than being “forced” to have a system-wide python on the system PATH, i do really dislike that approach very much

Why? Too easy to use?

[BPL]

Interesting! I didn’t know about the concept of multi-process-architecture

Why? Too easy to use?

Well, think about it, probably you’re already aware about all advantages of using virtualenvs instead the system-wide pythons. But let me try to point out one advantage related to plugin development, let’s say you want to test a particular plugin with certain versions of a python tool, having multiple virtualenvs with different versions of such a tool is easy and switching between virtualenvs as well while using a rigid path with a system-wide python doesn’t give you that degree of flexibility.

Also, forcing myself to not having system-wide python available on ST guarantees me all my projects will be running just with virtualenvs, and when there are hundred of projects using many different pinned dependencies, that’s really valuable, nothing is broken because some system-wide package updating or so…

Anyway, this is out of scope of this thread, I was just interested on knowing more about the internal details of python been used by ST, I think more or less has become a little bit clear by now, thx guys!

[deathaxe]

Well, think about it, probably you’re already aware about all advantages of using virtualenvs instead the system-wide pythons

I didn’t make up my mind about it, honestly. I know what virtualenvs are used for, but I was quite happy with my one and only latest python setup working for everything I need.

let’s say you want to test a particular plugin with certain versions of a python tool

Not sure, what you intend to do, but all Sublime Text plugins run on ST’s python 3.3 interpreter. Therefore testing ST plugins against several python versions is somewhat useless in the first place.

It was required only for plugins, which runs a deamon as separate (python) process. This is what Jedi - Autocompletion or the LSP packages do in order to run CPU expensive functions separately from ST. But IMHO this is a quite special use case of ST plugins.

An external tool / python script (pylint, flake, …) used by any ST plugin such as SublimeLinter is installed via pip and is just called as any other executable. Testing something against several python versions is not required here, too. Most of such packages even allow to setup explicit paths to the executables, which is the more direct way and maybe simpler than manipulating the environment.

The one and only every day life use case I can imagine was to create a dedicated python setup which is used by ST only.

Customizing the environment for a subprocess …

If you want to see how simple it is to add custom environment variables to an executed program, you can have a look onto the following PR. It does exactly that.

Add support for per language-server environment variables

About your problem with other plugins not using your virtualenv variables …

The point is to get the environment updated before any other package is loaded.
To achieve that, you’ll need to name your package in a way that ensures it is loaded “first”.
Look at 0_package_control_loader.sublime-package as an example, which is used by Package Control to extend python’s sys.path to publish dependencies which are needed by some packages.

You’d maybe need a 0_virtualenv.sublime-package to publish the environment early enough. In this case it is importand the package is zipped and placed in the Installed Packages path.

[wbond]

Here’s the thing, many ST packages require you to have installed tools like pep8, flakes, pep257… they expect them to be available on system path, well… fair enough.

This is the main issue. Those plugins are ignorant of the concept of virtualenvs, and don’t appear to provide configuration on where to look for the proper Python. Realistically, the plugins need to be updated.

I was involved with some work on (now neglected) Go package infrastructure for Sublime Text. One of the components was a way for Go plugins to easily and consistently get env var data, with the ability for users to override it within Sublime Text, per-OS and per-project. https://github.com/golang/sublime-config/blob/master/docs/user.md.

Something like this is ideally how the Python packages should find out about a user’s environment. However, that requires coordination. Otherwise, each package can provide settings to allow a user to set custom env vars. It shouldn’t take too much work to accomplish. Then when they launch the subprocess, they can use the custom PATH, etc.