Sublime text 3 comes bundled with python 3.3 since 2012, but we’re in 2019 now, python 3.3 is basically dead!
I think sublime devs should consider updating it’s python interpreter once in a while, a good rate would be parallely to python versions (3.7, 3.8, 3.9, …) as the benefits are numerable:
performance and startup time improvements wich by turn means sublime text performance and startup time improvements.
fewer dependencies for packages, as more libraries now comes with python such as typing, asyncio, pathlib, …
type checks and static analysis of code with better support for this in the latest python version wich means a lot for package developpers and code maintenance.
And other benefits too many to list here.
The only problem is to choose an upgrade strategy from python 3.3 to the current version (3.7) that incurs the minimum effect on the plugin ecosystem, after that things will get easier, as package developpers will benefit from python porting mechanism in upgrading their packages.
Just to be clear, we won’t break compatibility with existing packages in the 3.x series. It is just not an acceptable change, from our perspective. So the resulting options are have a parallel install (with a separate 3.7 plugin_host) or wait until a new major/breaking release.
I assume that a major/breaking release is not expected anytime soon. Meanwhile, 3.3 has been EOL for over a year.
I’m sure that there’s no timeframe, but – to be blunt – is there a plan for this? I’m not usually a pessimist, but the Python runtime is a ticking time bomb. I know that adding another runtime will be a major effort, and that there are other priorities, but I imagine that the odds of mishaps like the TLS thing a year ago are only increasing, not to mention the inherent risk in a runtime that doesn’t receive security updates.
My company still has a couple of Flash-based features. We’ve been nagging management for years about replacing them, but that’s not a shiny new feature we can sell to customers. One day, they’re all going to blow up irreparably. If I’m still at the same company when that happens, I’ll probably just find a new job. Finding a new text editor would be considerably more painful.
I generally just smile and roll my eyes when someone posts another topic about how everyone’s going to switch to VSCode or somesuch. People point to some new feature or other, or they complain that LSP is hard to configure. These things are important, but they’re not what I worry about. When I imagine a future in which I’m not using Sublime, and I speculate as to what would have led to that, I worry about the Python runtime.
Sorry for the mini-rant; I’ve been thinking about this a lot recently. I’m sure that it’s going to happen eventually, but it’s been nearly a year and a half since EOL with no news, and it’s making me nervous.
Actually, the Python Software Foundation did not maintain their security “promises” for 3.3, so it has been effectively unmaintained since 3.3.6 (late 2014). I contacted the release manager well before 3.3.7 and he sat on security vulnerabilities for over a year, never making a release. Once 3.3.7 was released, they never even bothered to see if the patches compiled on Windows, which they didn’t. Beyond that, the patches broke functionality.
Unfortunately, IMO, Python isn’t a great fit for embedding in an application, from a release maintenance perspective. They basically only maintain the releases for 18 months, and every minor release is a breaking change for anything but pure Python packages.
For now we maintain a patched version of 3.3.6, pulling the last stable commits from the 3.3 branch and adding various other patches. We also regularly update libraries that Python depends on (OpenSSL being the most important, IMO). I imagine we are by far the largest install base of Python 3.3 (at any time in its history), and we’ll certainly work on making sure we keep updating dependencies for the sake of security.
I have a rough idea of how I’m planning on approaching it, and I’ve discussed it with Jon, so in that sense there is a “plan.” However, we haven’t slotted it into a dev release cycle yet, no.
I think I am one of the users who most had complained about Sublime Text, in every imaginable aspect. But, for you surprise, I am very happy with Sublime Text using Python 3.3. As long as Sublime Text can keep it secure with dependencies updates. And for me, Sublime Text can keep using Python 3.3 at least for the next 10 years or even more.
Of course, I am missing all those new cute things the latest Python Version has to offer. However, as language for fast development, Python 3.3 is doing very well its job me. For honesty’s sake, sometimes I even write compatible code with Python 2.7. I did not ever had much problems with the fact Sublime Text uses Python 3.3, for a simple comparison between the problems I got with Python 3.3, with the problems I got with bugs or limitations on Sublime Text Core/API:
On my option Sublime Text can/should spend time fixing its API/Segmentation Faults other than worrying about updating Python to the latest version. But let us suppose, Sublime Text updates today for Python 3.7.3. How much time until Python 3.7.3 became outdated? How much time Sublime Text have been using Python 3.3?
Python may never reach version 4.0, because they simply could not be required to. Instead, new versions of Python 3 can be released for decades, like up to 2040, before they think about releasing something like Python 4.0. Researching on the internet about Python 4.0 shows a lot of misinformations like Python 4.0 will be released after Python 3.9, which does not make any sense if you know what does the common sense of semantic versioning usually means. For example, see:
To make people stop complaining about upgrading from Python 3.3 to the latest version, we can say that Sublime Text packages are not written in Python, but is a special language called Sublime Plugin 1.0 which, was coincidentally in the beginning very similar to Python 3.3. And in the near feature, when Sublime Text API become very robust and bug free, Sublime Text team will start working into updating the Sublime Plugin language from 1.0 to 1.1 (updating “Python” from 3.3 to 3.15.0 for Sublime Text 3.x.x) or 2.0 (updating “Python” from 3.3 to 3.15.0 for Sublime Text 4.0).
As Python, even Sublime Text may never release version 4.0 because Sublime Text simply could not be required to do any breaking changes. I think that a more plausible release of Sublime Text 4.0 could be dropping support from Python 3.3 to a newer Python version like 3.20.0. Another plausible motive to release Sublime Text 4, other than introduce breaking changes, is Sublime Text run out of money to pay the bills.
Insane Ideias
But, may be you guys could manage into releasing Sublime Text with both Python 3.3 and Python 3.9, but, I wonder where this is going to end? What will be supposed to happen on the next update for Python? As Python 4.0 may never come out, would Sublime Text be launched with Python 3.3, Python 3.9 and Python 3.44.0 interpreters at the same time? I mean, running 3 versions of Python at the same time. And how it will be for the next Python update? Both 3 Python interpreters already in plus Python 3.70.7, summing up to 4 interpreters?
I guess, this is where you guys do some breaking change and decide to release Sublime Text 4 featuring both Python 3.9 and Python 3.30.3, i…e, while Sublime Text 3 had both Python 3.3 and Python 3.9, the next major version of Sublime Text, does not break completely with the older version, as both Sublime Text 3 and Sublime Text 4 will have in common Python 3.9. Then, on each new major release of Sublime Text, one of the two embed Python Interpreters are dropped out, breaking some old Sublime Text packages which did not get upgraded to run on Python 3.9 while Sublime Text was on the 3.x.x life cycle.
While running both Python 3.3 and Python 3.9 interpreters, Sublime Text could gather some information on how many packages were updated from using Python 3.3 to use Python 3.9. If only few packages get migrated, then, Sublime Text could even consider releasing a new version with both Python 3.3, Python 3.9 and Python 3.25.0. On this scenario, the next major release, i.e., Sublime Text 4.0, could have embed Python 3.9, Python 3.25.0 and Python 3.77.0. Hence, only dropping support for Python 3.3 on the next major release of Sublime Text, i.e., Sublime Text 4.0.
Another option, would be let users to run how many Python Interpreters they would like, for example, in Sublime Text 4.0, I could either run onlyPython 3.55.0, or run at the same time bothPython 3.3,Python 3.9andPython 3.55.0. I am not saying to allow users to arbitrary install anyPython Interpreter, but, to allow they to use oldPython InterpreterslikePython 3.3, only if the user would like to run some package compatible only withPython 3.3. But if the user has no need to run any package only compatible withPython 3.3, then, he can just use the latest officialPython Interpreterreleased with Sublime Text. I mean, there is no need to deletePython 3.3code from Sublime Text completely and for ever. If some user would like to run some old package, they could be able to use these legacyPython Interpreters`.
We actually have a crash reporter and regularly work on crashing bugs. Unfortunately some are extremely hard to track down! Many are fixed on a regular basis and don’t make it into the changelog. Some crashing bugs exist for months before someone reports a way they find to reliably crash the program. Partially this is because during normal every-day usage, they are extremely unlikely to be hit. Others are only triggered when a whole bunch of exact situations line up. Some crashing bugs don’t really show up until a stable release, just because there are so many more users on stable releases than dev builds that we can then get enough reports to find a pattern to identify the issue.
Just the other day I tracked down an issue where there was extreme CPU usage because a user had added their entire home dir to Sublime Text, but also that had a folder named ~/.git. This caused the new git repo functionality to be watching effectively 50%+ of their filesystem and traversing the filesystem to see how the working copy had changed. There were enough regular writes to the home dir that the git traversing was constantly running. It is probably obvious that we take performance seriously, especially in regards to new features, so time is spent trying to make sure we aren’t regressing, but sometimes it ends up being an individual situation and not a performance bug with the program.
In terms of the API, fixing one man’s bugs can be breaking another man’s features. There are certainly some things that are bona fide bugs, and some that people wish acted differently. I try to blend a mix of API-related bug fixes, syntax issues, crash fixes, visual updates and work on new features into most builds. It is a balancing act, but we try to do a reasonable job.
Just today I spent a bunch of time chasing down a report someone made indicating that Sublime Text is vulnerable to DLL hijacking. I’ve now spent quite a number of hours just trying to reproduce the issue, since it is a security problem. At this point it appears it is a vulnerability in older, unmaintained versions of Windows, and there isn’t much of any thing we can do about it. But, since someone claimed we had a security vulnerability, I took the time to try and make sure we aren’t inadvertently exposing users. There is no changelog entry for this, no new feature to show off, but nonetheless a necessary part of maintaining a mature, widely used product.
To wrap it all up, I’m certainly not spending a great deal of time worrying about the Python version we ship, but rather fixing bugs (IME issues on Linux and Windows), adding new features (git integration), refining our existing features (see recently Unicode changes and rendering fixes), keeping our OS integrations current (GTK3), and more. There is constantly work being done (largely by awesome community members) making our excellent syntax definitions even more accurate, and people taking the time to report ways to reproduce issues.
Sublime Text is an awesome product, and it is loved and used by many users, but is by no means done, nor bug free. But along with the popularity comes the desire for many users to want changes to be focused on what is important to them. We have to split that between the power users and package developers who hang out here on the forum and Discord, with the everyday users who have been using it for years to just get stuff done, to the people who just want a fast, focused text editor and not fuss around customizing 93 different settings to just start writing some Python.
but some features are desired by many users, the problem apparently is that what many users consider priority is not for devs, some of the most desired feature requests have been on hold for many years now, i’m not going to name but you can check out the most upvoted issues on the issue tracker.
For a user of ST, “New feature: git integration” is happiness, “fixed a crash” is boooring!, I think the devs should focus more on implementing new features and refining old ones, rather than waiting for the moons to line up to reproduce a bug and fix it.
They don’t just sit around if they can’t reproduce a bug. I think he was illustrating that sometimes complex bugs take time to debug. They also do refactoring that allows for new features, experimental code that sometimes doesn’t work out, researching how to add a complex feature, developing complex features that take a while to get right, etc…
Fixing bugs are always hard to plan for as you never know how long they’ll take to fix, because you never knew they were there to begin with. You have to allocate time to triage bugs as they come in or you end up with too much technical debt. If you can’t understand the bug because it doesn’t reproduce, then it goes into backlog while you do something else, until a future time when you have enough info to properly address the bug.
I will agree though that user desires and developer desires don’t always align in each other’s desired time lines. I’ll also admit that as a user, I can fixate on less important things at times as well. A developer should listen to their users, but not be ruled by them. There is a balance there. I think the Sublime team has been doing a fine job.
To avoid breaking packages existing packages it makes total sense that ST needs to continue to support Python 3.3.
And doing a bunch of work to add a separate runtime for Python 3.X doesn’t make too much sense if after 18 months it’s no longer supported and users want support for Python 3.[X+1], because soon you’ll have 3 Python runtimes, and so on.
Unfortunately, IMO, Python isn’t a great fit for embedding in an application, from a release maintenance perspective. They basically only maintain the releases for 18 months, and every minor release is a breaking change for anything but pure Python packages.
A suggestion: add support for only one other Python runtime. It could be 3.7 to start. Somehow ensure it only works for pure Python plugins. Then you can update the separate runtime as newer Python versions are released, to 3.8, 3.9 and so on, without breaking any of the plugins written against 3.7.
I feel like you can write very powerful pure Python plugins, and that these plugins would all benefit by being able to use the latest in the Python ecosystem and standard library.
That would mean you’d have to keep the old 3.3 runtime around forever for closed-source packages as they cannot switch to a more recent version.
This has been implied before, but imo the best way forward is to have a transition period between 3.3 and 3.7 where both are available, then phase out 3.3 at the next major release. A period of 6 months should be more than enough to get commercial packages on a more recent build considering they are making money with those.
The majority of packages are pure Python and will not need to be updated. I could see invalid string escapes becoming a problem in the future when they are recognized as a SyntaxError, but that’s not the case for 3.7 at least. Still, the maintenance effort for this is absolutely minimal and in no way comparable to the 2to3 situation.
Assuming ST4 comes out, I wonder if it would be a reasonable restriction to not guarantee binary compatibility for plugins between ST releases. That way ST4’s Python runtime could always track the latest Python release.
What I mean to ask is: how many big-name packages are not “pure Python”, and how many of those could not be re-written to be “pure Python”?
This thread has highlighted the difficulties encountered to update the embedded python, raising issues more than solutions, bundling two python runtimes is not ideal to say the least, and even if we managed to pump python version, two years later that python version will stop being maintained, so we’re back to square one.
Under these circumstances, getting rid of python entirely is not a crazy idea after all, especially in the presence of better alternatives from which I’ll mention Julia. Julia is a dynamic programming language with syntax very similar to python yet many times more efficient, all without relying on C extensions thus granting a (quasi) stable ABI across versions, the language can be embedded in other applications and is both JIT (for experimenting and prototyping) and AOT compiled (for deploying code).
I think Julia is a tempting option for Sublime Text 4, especially that (I think) it doesn’t take more effort than updating python from 2 to 3 (ST2 to ST3), only this time with less troubles ahead.
Julia would literally break every extension known to Sublime, and require all current developers to either learn Julia or abandon their extension. I think Julia would potentially have a disastrous effect on the plugin community. It would essentially reset the plugin landscape back to zero.
learning julia isn’t hard especially for those who know python
and postponing this rather big transition to ST4 is meant to reduce the rather big effects on the plugin community, this huge transition is inevitable as updating the embedded python will break lots of popular plugins (although I admit that it’s not this huge). my point here is rather than asking how can we update the bundled python we may ask how can we (gradually) get rid of python without much hurting the plugin ecosystem ?
even if we managed to pump python version, two years later that python version will stop being maintained, so we’re back to square one.
Not really. There is no “square one”. We’re on square 3.3, and we’d like to be on 3.7. In a couple of years, it would be nice to be on square 3.9 (or whatever), but square 3.7 would still be a lot closer to that than square 3.3.
Ideally, Sublime should stay up-to-date with the latest Python version. But this is not an ideal universe. This just isn’t an all-or-nothing issue: all of the improvements that 3.7 offers over 3.3 won’t disappear when 3.8 comes out.
how can we (gradually) get rid of python without much hurting the plugin ecosystem
