Sublime Forum

Why not link the internal Python dynamically to allow easy updating?

#1

I’m missing newer Python features like richer type annotations… As I see that the bundled 3.3 dates back to 2012, I suspect that the upgrade of it is a major problem? I just wonder… If the interpreter is just a libpython3.3.a linked statically… Why not make it libpython3.3.so and link dynamically? This way power users could switch the interpreter by overwriting the file with libpython3.7.so or …3.9.so or whatever… Which would be also a first line testing for the official upgrade. What do you think?

0 Likes
#2

Sounds like a support nightmare for ST devs and plugin authors… if you can’t have the environment in control. I.e., image your Python is not my Python.

1 Like
#3

Well I think that it wouldn’t be a ‘nightmare’ but rather an additional pressure to roll out an interpreter update more often than once in eight years (still counting…)… I think that nobody (except for power users that can provide hints to the support rather than trouble) would risk dynamic library switch if not forced to… by e.g.: 8 years old interpreter… So it would occur only if the maintainer would sleep over… I think that it would be a healthy mechanism, as the maintainer would be able to read people experiences, problems and bug reports from their preliminary, own interpreter update and this could make the upgraded release easier.

0 Likes
#4

If you are developing a plugin in ST4, then ST4 supports 2 plugin environments now, 3.3 & 3.8. The 3.3 maintained for legacy reasons and not to disrupt the existing package infrastructure. You can use 3.8 if you want type annotations.

0 Likes
#5

ABI changes make that impossible as nearly each python version introduce breaking changes which would drive a dynamically linked python interpreter useless as ABI function calls from plugin host fail.

The main reason to stick with python 3.3 for such a long time are several top 25 packages shipping in compiled form only. Changing python version brakes those for sure as byte code of python is not stable (also changed with each version).

One possible solution for faster python updates might be to forbit compiled non-open source plugins in the future.

To answer in advance, ST won’t update to python 3.9 anytime soon as it does no longer support Windows 7 which ST still does for some time.

3 Likes
#6

ABI changes make that impossible as nearly each python version introduce breaking changes which would drive a dynamically linked python interpreter useless as ABI function calls from plugin host fail.

I disagree with this statement. IMO the ABI changes are buried down the interpreter construction and would not come up that often when swapping the interpreter dynamic link libraries. The surface ABI is enough stable to aptly support great majority of the updated libraries. The embedded C API is just fairly stable, that’s all.

The main reason to stick with python 3.3 for such a long time are several top 25 packages shipping in compiled form only. Changing python version brakes those for sure as byte code of python is not stable (also changed with each version).

I think that an additional parameter could be added to the package hosting in order to resolve this issue – the version of the interpreter library. The correct binary version of the package could then be chosen at the installation.

To answer in advance, ST won’t update to python 3.9 anytime soon as it does no longer support Windows 7 which ST still does for some time.

IMO the 3.9 version of Python could still be used, as the dropping of Windows 7 support is probably mostly theoretical / symbolic, and with a small amount of effort on the ST side it could be still easily shipped with the editor. I would expect problems on some later versions of Python as it would diverge from Windows 7 more.

In general, I’m opting for a freedom of choice solution – to simply allow users take actions and responsibilities and costs of the individual interpreter upgrades, like e.g.: the binary only plugins not working properly after the upgrade. I think that any determined user would gladly accept the costs for the benefit of having a recent Python version at the plugin host up and running (because they would be aware that it’s their own decision to do so) and that it would be a quality fun time for them to work towards resolving the appearing issues, and that ST would benefit from the forged solutions. All this goodness from a basic change of the interpreter linking method. Isn’t it worth it? The little more openness in the Sublime providing method?

0 Likes
#7

What actual problem are you trying to solve? The transition to a python 3.8 runtime for ST4 allows us to use async functions, an immense step forward. Today, in the 3.3 runtime, you can accomplish asynchronous functionality with plain old callbacks. But other than that, what would 3.8 versus 3.9 accomplish?

0 Likes
#8

I want to have a general more recent Python for my plugins, to use e.g.: the richer type annotations. I’m not towards Python 3.9 in particular, just a more recent one than the 8 years old 3.3.

0 Likes
#9

You can spoof type annotations with 3.3 using a trick:

0 Likes
#10

There is actually a typing dependency for py33 plugin… I hardly see people use it though (except for myself). https://github.com/packagecontrol/typing

1 Like
#11

This sounds more like hope than arguments for a deterministic system.

See how other big programs like Kodi still rely on bundled python 2.7 and struggle hard to fix all bugs/regressions coming with an interpreter update. The jump from 2.7 to 3.8 is huge though but it shows what happens in the long term if you expect everything to work with 3.3 to 6.0 or so.

I think that an additional parameter could be added to the package hosting in order to resolve this issue

It’s not a question of hosting (only) but the need for all those closed source package developers to provide compiled binaries for all such python versions. Some of the packages seem no longer maintained and others may have there own dependencies which makes such a dynamic linking hard.

Don’t even want to think about maintenance effort to make sure all constellations work well.

… for questionable use.

0 Likes
#12

There’s already enough differences between the 3.3 and 3.8 plugin hosts to break compatibility. There does exist a stable ABI for python now, but it’s far from enough for the embedding we do.

0 Likes
#13

I just think that functions such as PyObject_GetItem, PyObject_New, Py_BuildValue, PyArg_ParseTuple, PyTuple_SetItem, etc. are just a rock-bottom that will almost certainly never change. Even if there would (somehow…) something break, then not for all available newer versions of Python, so e.g.: say that 3.9 would break but 3.8 and below not. I think that everyone would be satisfied with a 3.8 upgrade.

I regret that I don’t have enough time so that I would show the interpreter library swapping in Vim. Vim has an option pythonthreedll that specifies the interpreter library to load… Freedom and no “nightmares”, really… :slight_smile:

0 Likes
#14

Here’s vim crashing due to a minor version change from 2.7.9 to 2.7.10. Here’s vim also crashing due to a minor version change from 3.5.1 to 3.5.2. We’d much rather avoid these kinds of issues.

1 Like
#15

As the person who’s done the work of actually embedding Python 3.8 and maintaining our 3.3 environment, I can tell you it isn’t possible to make a forwards compatible embed for Python. Things change from version to version. Version 3.8 significantly revamped how initialization of the Python runtime is done. https://docs.python.org/3/c-api/init_config.html

There is a stable ABI, but it doesn’t cover the APIs used during embedding. Also, it would restrict the usable compiled modules to ones that use the stable ABI. Of the top 50 packages on PyPi, only one provided abi3 whl files.

I understand what you want, and why you want it, but it isn’t possible (in the way you described). Beyond that, it would be pragmatically unsuitable for an application API since changing Python versions would likely break all sorts of plugins.

4 Likes