Sublime Forum

Updating the bundled python

#21

It’s worth noting that a major bottleneck of an embedded plugin language is the overhead required for data exchange between the plugin host and the main application. This won’t change for a different language. You’re unlikely to do computationally expensive stuff inside an embedded environment and most of the time never hit the upper limit.

An argument could be made regarding slow initial loads, but that’s already deferred to after the main application has loaded and more likely slowed down by i/o operations anyway.

1 Like

#22

How hard is it to create two plugin_hosts, one for python 3.3 and the other for julia ?

0 Likes

#23

Probably easily two orders of magnitude (100x) more than adapting to the cpython api changes from 3.3 to 3.7.

We’d have to learn the Julia C API, write a layer for converting Julia types into our internal C++ types. We’d have to figure out how to compile Julia 5 different ways (Linux x32/x64, Windows x32/x64, macOS) and figure out what is necessary to link it into a C++ container app (plugin_host_julia). We have to work through all of the bugs and edge cases that come from new development. We’d have to write a Julia API and document it.

I see no compelling reason to do all of that work. That is way more work than the less-than-ideal nature of embedding Python due to an 18 month release cycle. Ultimately the API needs to be as accessible as possible. Python is one of the most popular languages in use today (spot 3 right now on TIOBE) whereas Julia sits at 41. So even completely disregarding backwards compatibility (which is HUGE for a commercial product people are buying and don’t want to be breaking for no good reason), Julia doesn’t seem compelling as a language to embed.

13 Likes

#24

Not to beat a dead horse but Julia is sort of a niche replacement for Matlab. Python is arguably the most versatile language in the world.

2 Likes

#25

Julia is more like a niche replacement for python, it’s focus on scientific computing does not mean it’s not suitable for general purpose programming
from the julia site

also see this topic on the julia forum.

0 Likes

#26

Despite how much I like julia, I also don’t think that Julia is a suitable language for this purpose.

First, it is still under active development. Syntax stability is very important for the ease of package development and maintenance. Secondly, plain Julia is basically useless. Additional packages are usually required for a lot of basic stuff and It is not ideal for serving as plugin host. Thirdly, Julia has a high portion of codebase written in Julia. It is usually a good thing. However, it also makes the C API very weak. For instance, there is still no way to hook into the event loop and preserve objects from GC in the C interface.

There are more issues than what I have mentioned, I just listed a few of them.

4 Likes

#27

from the 1.0 blog:

Julia manual does not highlight such problem, can you please provide further details or links.

0 Likes

#28

You will soon discover the limitations of Julia C API once you started embedding Julia in your application. For example: it is my experience: https://github.com/juliaverse/juliaapi/blob/master/src/cast.cpp

To preserve objects from GC, One way to do so is to reference them in an array in the memory. All this are not necessary if their provided something like Py_INCREF in python.

The Julia does not mention these issues because the C API does not provide ways to do so.

5 Likes

#29

There are “hundreds” of scripting engines out there. There is a good reason for python to be the defact-standard scripting engine for embedding into an application. Have a short look into the linux ecosystem and you’ll find enough examples to prove that.

That said and the desastrous breaking effect the introduction of a new plugin eco-system would have - not to mension the effort to create the core side API - the whole Julia discussion is of no value.

8 Likes

#30

About “breaking” existing packages when upgrading to modern python is easy to solve actually… IMHO you shouldn’t defer this decission to users/plugin developers at all. Said otherwise, package manager plugin should behave like pip and it should install the best choice according to the available python runtime at sublime. Haven’t looked to the packages metadata but this would probably having some python_requires field (if this field doesn’t exists you can fairly assume python==3.3.6).

About having multiple parallel python versions in Sublime… I think this woud be really a bad idea and it’d be an overly complex unnecesarily solution.

The other fair concern other than breaking backwards compatibility with existing packages is security vulnerabilities… so you’re saying than upgrading to a more modern python version would be more risky than using a version who’s reached EOL?

It’s funny though… you’re talking about breaking backwards compatibility like it’s some big deal, porting large codebases from python2.7 to >3.6 is a matter of hours/days… How long would take to upgrade existing Sublime packages (medium projects at best) from 3.3.6 to >3.6? Minutes? Hours?

In any case, I find it reasonably you want to keep backwards compatibility… but as said, don’t defer these decissions to plugin_devs/users. Just make sure “outdated” packages won’t be available if they’re not “supported”, easy peasy :slight_smile: . Ah yeah, and make sure you let know about this fact to the users when installing packages or when upgrading to this theorical new ST release, otherwise they’ll get annoyed :smiley:

0 Likes

#31

Iirc, Wbond said that there are some pre-compiled python codes in the top 100 ST plugins. Upgrading python would definitely breaking them. If those plugins are not fully open-sourced, there is nothing we could help for them. Even worse, if those developers have moved to another editor and no longer maintain their ST plugins, then those plugins are no longer usable. I believe VSCode’s thriving had caused some plugin developers to leave ST.

But anyway, sooner later than never, it’s something that ST has to overcome. :slightly_smiling_face:

Disclaimer: I don’t have any actual statistics for what I say above. It’s just what I feel/guess from what I saw on some Github repos.

1 Like

#32

Just wanted to bump this again. I agree with @ThomSmith. For me, the Python runtime being stuck on 3.3 is the biggest issue facing Sublime Text. I feel like it’s also the biggest disadvantage when comparing ST to Atom or VS Code.

People have mentioned asyncio, which would make it much easier to write powerful plugins that do network IO. mypy is worth a mention as well. Many of its newer, more powerful features (e.g. dataclasses) can’t be used in Python 3.3. Having typed code bases for complex plugins would be a huge win for developers.

0 Likes

#33

It’s true, but ST authors already explained why it isn’t possible to update the Python runtime in the current major version of the editor. Also, unless you need to handle thousands of network connections at once (and it is very unlikely for any kind of plugin) threads are an excellent way to do I/O in Python. To me the limited access to the user interface is a bigger hindrance to write great plugins than Python 3.3.

1 Like

#34

IMHO this shouldn’t be such a hard choice for sublime_hq, few thoughts:

  • It’d be convenient to support python-requires in sublime packages metadata (in case it’s not already supported, i haven’t checked the package metadata docs). That way you’d not load packages if the shipped python interpreter wasn’t supported by those packages, of course, Sublime should warn the user about this fact on the console or just creating a new view

  • It’d be great if upgrading the python interpreter became a more automated process, the fact there has been very few python upgrades in all ST life makes me think is a really pain for devs to do so (just speculating)

  • Mypy? Why? Shipped modern cpython should be more than good enough to everybody… If I wanted static typing I’d just use a system programming language in the first place instead python… Not saying mypy isn’t cool but I don’t see which advantage would bring to the table in the context of package/plugin development of sublime.

  • You said one of yours concerns is the fact there are few existing packages out there using extension binary modules you’d like to keep, that the devs are gone and package source has become closed, so… what’s the big deal here? Personally the only close source I use in SublimeText is SublimeText itself :slight_smile: … but that’s because I can configure it the way I want. Non-maintained old packages or closed source packages are basically black-listed for me.

  • I also think the packages being updated/supported should be only plugin developers responsability, Sublime should be free in this area to provide users the best more modern means that allow to create astonishing useful plugins, I do think giving us modern python would be the best feature you could give us at this point of maturity in Sublime.

  • About supporting multiple python interpreters in Sublime…IMHO I think is a really bad idea when Software overcomplex the architecture just because they want to keep backwards compatibility… even if users will start complaining a lot although you’re giving them better software

My 2 cents :slight_smile:

0 Likes

#35

Why does it need to be embedded or Python in the first place? Sure you get a bit of a performance benefit by using C bindings to Python, but then there already has to be some form of IPC between ST3 and plugin host. And it’s not like Python is fast.

Why not choose some RPC protocol, document the interface and let devs write in whatever they want? Then add a reimplementation of current sublime and sublime_plugin packages in terms of this IPC, and whichever Python version this small piece of code is compatible with, users can use to run legacy plugins.

1 Like

#36

But if Python is not embedded, ST users are destined to install Python on their machine because all plugins now are written in Python? “Forcing” users to install Python is not a good idea imo and abandoning the current ecosystem is even worse.

This historical burden is so heavy… :slightly_frowning_face:

0 Likes

#37

I’m not quite sure what you mean by abandoning the ecosystem. You could easily run existing plugins that way, provided that an appropriate interpreter version is available (installed system-wide or some private fork embedded in ST3). Sure, having to worry about Python version when installing plugins is additional burden on the users, but in a way it’s plugin dev’s problem - if they’re too hard to install, they won’t be popular.

From what I gather from this thread, the problem seems to be some plugins ship with C extensions that need to be recompiled for every Python release. When do you really need to rely on those as opposed to calling C code via ctypes? Eg. even if you need to implement a compression algorithm in C, you could provide a stable ABI for that code and talk to it using ctypes instead of making it understand internal Python data structures that change with every release.

0 Likes

#38

This means that ST “forces” users to install Python on their machine unless users are going to abandon the current ecosystem, i.e., plugins written in Python (and ST is no longer “portable”). It’s kinda weird to me that a text editor asks me to setup a Python env. It’s not a question about whether I can do it or cannot do it. It’s just weird to me.

Except that, plugins written in whatever new languages are fine since they are new.

0 Likes

#39

Except that, plugins written in whatever new languages are fine since they are new.

I wouldn’t like to install multiple interpreters (mscore, java, node, python, ruby, perl, …) just to be able to useall plugins.

When do you really need to rely on those as opposed to calling C code via ctypes?

Some python libraries like numpy (maybe not the best example) use compiled C/C++ libraries to efficiently run the backend code. So it’s a question of libraries and their way of doing things.

But more important are plugins using bytecode compiled modules *.pyc rather than plain text *.py. Those plugins would stop working after a python interpreter update as it is changed with every release.

0 Likes

#40

Why not choose some RPC protocol, document the interface and let devs write in whatever they want?

I actually considered this briefly. I even wrote a quick proof of concept: a package that spawns a Python 3.7 process and gives it access to sublime_api over pipes. This has some potential, but also some drawbacks.

I’m running the Mac OS. On this machine, Sublime is communicating with plugin_host via POSIX pipes and semaphores. I would hazard a guess that it is converting internal data to Python types in Sublime and passing them to the runtime via a standard Python serialization format. I also presume that the communication channel is platform-specific, and that the Windows implementation is different.

This works fine for a single runtime tightly controlled by Sublime HQ, but not necessarily for user-supplied runtimes. For one thing, if non-Python runtimes are supported, then the API must use language-agnostic serialization, such as JSON. This incurs overhead on both sides of the channel. For another thing, a pluggable implementation would most likely use a different channel to support multiple plugin runtimes and so that a single runtime implementation could work cross-platform. The obvious choice is a TCP socket. I haven’t run the numbers, but I would expect this to also incur significant overhead versus the current pipe implementation.

Sublime is useless without plugins; plenty of core functionality is implemented in the Default package. So Sublime HQ would still need to provide and maintain a built-in Python runtime. However, to support user-provided runtimes they would also need to maintain and document a “runtime API”. Currently, the sublime_api package is an implementation detail that Sublime HQ can change at any time. If other runtimes were to rely on it, then Sublime HQ would need to formally stabilize the API, write complete documentation, and maintain the API forever as new features are introduced.

And while user-supplied runtimes would be very convenient for developers, they could pose a substantial risk to the package ecosystem. Right now, every Sublime plugin uses the same runtime, which comes preinstalled with Sublime and is maintained by Sublime HQ. If user-provided runtimes were widely adopted, then a large number of packages would depend not only on the package author but on the runtime author. Because the 3.3 runtime is bundled with Sublime, Sublime HQ is obliged to actively maintain it (even e.g. backporting security fixes from future Python versions). Maybe the custom runtimes would be implemented, maintained and documented to the same standard as the core runtime, or maybe not. A poorly-maintained runtime could break or orphan many, many packages.

Don’t get me wrong, it would be really neat to have multiple pluggable runtimes. But I’m not sure that it would be useful enough to justify the complexity and performance costs or the risk to the ecosystem.

On the other hand, thinking through the problem of multiple runtimes does give me hope that it might not be as difficult as I’d feared to get first-party 3.3 and 3.7 runtimes working side-by-side. Of course, the hard part there is designing the developer-facing boundary that determines what runtime to use for a given plugin.

2 Likes