What is it modifying sys.path at startup?

[BPL]

I’m puzzled here on few cases:

1. If I start ST normally (double clicking the sublime_text icon) and I query sys.path on the console i’ll get this:

   [
   ‘D:\software\SublimeText 3_x64\Data\Installed Packages\AutoPEP8.sublime-package\sublimeautopep8lib\packages_py3’,
   ‘D:\software\SublimeText 3_x64’, ‘D:\software\SublimeText 3_x64/python3.3.zip’,
   ‘D:\software\SublimeText 3_x64\Data\Packages’,
   ‘D:\software\SUBLIM~2\Data\Packages\shellenv\all’,
   ‘D:\software\SUBLIM~2\Data\Packages\coverage\ST3_WI~2’,
   ‘D:\software\SUBLIM~2\Data\Packages\newterm\all’,
   ‘D:\software\SUBLIM~2\Data\Packages\pyyaml\st3’,
   ‘D:\software\SublimeText 3_x64\Data\Packages\coverage\st3_windows_x64’
   ]

2. If I run ST from the console after activating python2.7 virtualenv I’ll get the same sys.path output than case 1, so far so good.

3. Now, the problem is when I start sublime text after activating a python3.6.2 virtualenv, where I’ll get this sys.path:

[
‘D:\software\SublimeText 3_x64\Data\Installed Packages\AutoPEP8.sublime-package\sublimeautopep8lib\packages_py3’,
‘D:\software\SublimeText 3_x64’,
‘D:\software\SublimeText 3_x64/python3.3.zip’,
‘D:\software\SublimeText 3_x64\Data\Packages’,
‘D:\software\SUBLIM~2\Data\Packages\shellenv\all’,
‘D:\software\SUBLIM~2\Data\Packages\coverage\ST3_WI~2’,
‘D:\software\SUBLIM~2\Data\Packages\newterm\all’,
‘D:\software\SUBLIM~2\Data\Packages\pyyaml\st3’,
‘D:\software\SublimeText 3_x64\Data\Packages\coverage\st3_windows_x64’,
‘D:\sources\personal\python’,
‘d:\virtual_envs\py362_32\Scripts\python36.zip’,
‘d:\virtual_envs\py362_32\DLLs’,
‘d:\virtual_envs\py362_32\lib’,
‘d:\virtual_envs\py362_32\Scripts’,
‘d:\software\python362_32\Lib’,
‘d:\software\python362_32\DLLs’,
‘d:\virtual_envs\py362_32’,
‘d:\virtual_envs\py362_32\lib\site-packages’,
‘d:\virtual_envs\py362_32\lib\site-packages\win32’,
‘d:\virtual_envs\py362_32\lib\site-packages\win32\lib’,
‘d:\virtual_envs\py362_32\lib\site-packages\Pythonwin’
]

As you can see, sys.path has been extended somehow to include the virtual_env paths and I’d like to understand why… while this is kind of cool, it’ll give visibility to the ST embedded python to the packages of my virtualenv can also be a good way to get unexpected behaviour, for instance, importing pathlib, which it’s just supported by python >= 3.4 would give you something like this:

>>> import pathlib
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "d:\software\python362_32\Lib\pathlib.py", line 382, in <module>
    class _NormalAccessor(_Accessor):
  File "d:\software\python362_32\Lib\pathlib.py", line 404, in _NormalAccessor
    scandir = _wrap_strfunc(os.scandir)
AttributeError: 'module' object has no attribute 'scandir'

Because scandir it’s not supported by St’s py 3.3. So yeah… summing up, I’d like to understand why in this case I’m not having a clean “sys.path”, why has it been extended in case 3) but not in case 2)?

Thanks in advance for the clarification.

@wbond PS: Talking about python3.6.2… is there any plan in the roadmap to upgrade the ST python embedded interpreter to 3.6.x or bigger? That’s a really major feature but it’d be a extremely awesome one . In fact, out of curiosity, how hard would be using a virtualenv as the embedded interpreter? Just thinking about how extensible would be ST this way gives me chills, ie: that would allow me to code (maybe?) pyqt5 plugins or using the whole python pypi ecosystem very easily in some cases where the whole threading stuff is not messed up

[deathaxe]

About sys.path:

I guess your virtualenv for python 3.6.2 sets the PYTHONPATH environment variable? This is what each python interpreter would use to extend its sys.path.

About python 3.6.2 in ST

This question arose recently here in the forum. Major problem is changing python version would break a lot of packages, which include compiled python files. Even those who are in the list of Top 25 with 1M users.

Upgrade embedded Python to 3.6

When I started to write my first Plugin I was kind of surprised, that Sublime was not using the Python3 version of my Ubuntu (although it makes some sense). Now what was shocking to me is that the version embedded is 3.3, which is getting really old. Now I am looking into writing more Plugins with a Server-Client architecture and using Python 3.3 for that is really no fun. So I am wondering is there a way to either tell Sublime to use the system Python or would it be possible to upgrade the embe…

[BPL]

That’s what I don’t understand, the activate.bat is just extending PATH and it’s not modifying PYTHONPATH, so I don’t get what’s going on here, look:

> cat d:\virtual_envs\py362_32\Scripts\activate.bat
@echo off
set "VIRTUAL_ENV=d:\virtual_envs\py362_32"

if defined _OLD_VIRTUAL_PROMPT (
    set "PROMPT=%_OLD_VIRTUAL_PROMPT%"
) else (
    if not defined PROMPT (
        set "PROMPT=$P$G"
    )
    set "_OLD_VIRTUAL_PROMPT=%PROMPT%"
)
set "PROMPT=(py362_32) %PROMPT%"

REM Don't use () to avoid problems with them in %PATH%
if defined _OLD_VIRTUAL_PYTHONHOME goto ENDIFVHOME
    set "_OLD_VIRTUAL_PYTHONHOME=%PYTHONHOME%"
:ENDIFVHOME

set PYTHONHOME=

REM if defined _OLD_VIRTUAL_PATH (
if not defined _OLD_VIRTUAL_PATH goto ENDIFVPATH1
    set "PATH=%_OLD_VIRTUAL_PATH%"
:ENDIFVPATH1
REM ) else (
if defined _OLD_VIRTUAL_PATH goto ENDIFVPATH2
    set "_OLD_VIRTUAL_PATH=%PATH%"
:ENDIFVPATH2

set "PATH=%VIRTUAL_ENV%\Scripts;%PATH%"

About upgrading the embedded python to 3.6, yeah, I completely understand it would break many packages straightaway. I guess if I was a ST coder I would think in terms of not coupling ST with any specific version of python and make it work for a random one. Of course, this type of generalization sounds “cool” in theory but it’s not that easy… At the moment you’ve pinned ST to a “patched/tweaked” specific version of embedded python the whole tree of plugins have already been pinned to that python dependency as well. Tricky… … But anyway, in the thread you’ve posted wbond explains clearly more about it, so I won’t go more offtopic on this one.

[deathaxe]

I would think in terms of not coupling ST

This is difficult, because some devs don’t want to provide open source and it gets impossible as soon as you include compiled python libraries such as lxml or so.

I tried virtualenv with python 3.6.2, too. Created a clean env, even set PYTHONPATH to something (was not there) and started ST. Everything is normal. No additional path is added.

Maybe any 3rd party package, which writes to sys.path?

You just start the virtualenv and then ST from console, right? No python script starts ST?

[BPL]

Mmm, thanks to give it a shot!

So yeah, I’ve download a clean ST3 and made the same test and as you, just checked out sys.path would be “clean” at startup, after that i’ve decided to copy the whole bunch of installed packages from my “corrupted” ST to this clean ST and i’ve realized one of these 3rd party package would mess up with sys.path at startup.

After discarding one by one (binary search :))… i think for some reason the faulty one which are messing up:

	"SublimeLinter",
	"SublimeLinter-flake8",

In fact, If I’m not mistaken SublimeLinter-flake8 should be the guilty one… now I’m trying to find on which part of the code is messing up with sys.path… do you see anything suspicious there?

[wbond]

@wbond PS: Talking about python3.6.2… is there any plan in the roadmap to upgrade the ST python embedded interpreter to 3.6.x or bigger?

Well, since we don’t have a public roadmap, the technical answer is no.

We will need to support 3.3 for the lifetime of Sublime Text 3, otherwise we’d be making a significant, breaking change that would affect a huge portion of our user base. One consideration is to add a second plugin host that runs a separate version of Python and that packages could opt-in to the newer version of Python. Python 3.7 would be great, but it won’t be stable until June 2018. Depending on our priority list, and if this idea pans out, it may be 3.6 or 3.7.

[deathaxe]

but it won’t be stable until June 2018

Enough time to concentrate on some of the other not so breaking requests/wishes collected in the forum/github

Not sure how hard implementing a second plugin_host and the required handling for several versions is, but as 3.7 offers some optimizations on function execution and the life with 3.3 is not too hard, it might be worth waiting, IMHO?

… and not to need to add too many different versions over the time. Each one needs to be supported for lifetime, once added.

[ThomSmith]

Python 3.7 doesn’t look that exciting compared to 3.6. The method call speedup is nice, but probably not significant for our purposes.

On the other hand, 3.6 offers:

• Enumerations
• Type annotations
• Coroutines
• Ordered kwargs
• F-strings

I have felt the lack of each of these when writing plug-ins.

In addition to the above, a truly killer feature in an upgraded python runtime would be pip integration. My YAML Macros package depends on ruamel.yaml, so I had to port it to Package Control. It would be so much easier if Sublime could grab the original package via pip.

I can’t speak to the difficulty/hassle of implementing this, of course, but it would certainly be one of my top three features.

[BPL]

@wbond Thanks for the answer, sounds awesome news, 3.7… yummy! Btw, let me make you another question, although this one will be hypothetical and just out of curiosity… but, how hard do you think it would be coding an agnostic decoupled plugin_host? I mean, let’s say there are N available virtualenvs in a system, how hard would be that this hypothetical plugin_host could use some of these available virtualenvs behind the curtains dynamically (ie: user choosing which one to use)? This fancy plugin_host obviously wouldn’t make sense in the real world, where you definitely want a rigid plugin_host working with a very specific python version so the community can work upon.

Btw, finally this thread has gone offtopic, anyone knows at which point SublimeLinter or SublimeLinterFlake8 are tweaking sys.path?

[wbond]

Btw, let me make you another question, although this one will be hypothetical and just out of curiosity… but, how hard do you think it would be coding an agnostic decoupled plugin_host?

Not really possible. There needs to be Python, of a reliable version for Sublime Text to communicate with. The C API is not stable between versions, which is why C extensions to Python only work for a single minor version. Also, pre-compiled (obfuscated) source code is tied to the minor release. Further thoughts:

1. We’d always have to provide a version of Python for people who don’t have Python 3.3+ installed (most Windows and Mac users). We’d have to ship this as a dynamic library, rather than statically linking it to plugin_host.
2. We wouldn’t have a reasonable way to support Python 2.x since unicode is fundamentally broken in many, many ways. Sublime Text uses unicode for everything.
3. As soon as Python was loaded by plugin_host, it would have to continue using that version of Python until Sublime Text (or plugin_host) was restarted.
4. It would be necessary to maintain bindings for every minor version of Python we wanted to support, which would be a ton of work since it would all be dlopen() stuff since we’d link to a dynamic library at run time.

I’m not sure what you are trying to solve by having plugin_host use one of your virtualenvs. Python can create a Python subprocess with the exact Python env from your virtualenv and run any of the code. plugin_host is really just for interacting with the Sublime API, and creating bridges between that and other programs/APIs. Sure, a newer version of Python with some of the syntactic sugar would be nice, but supporting virtualenvs doesn’t seem to make sense to me. Perhaps I am misunderstanding what you want to accomplish.