Support more than Python for package API

[Webby]

Hi Sublime folks – I’m moderately concerned about the future of Sublime Text given how old most packages in Package Control are. It’s rare to come across packages that have been updated within the last couple of years. It’s clear that Visual Studio Code and Atom have gained a lot of ground, especially for web developers. There doesn’t seem to be as much energy going into ST package development.

I prefer Sublime for its speed and built-in features, but there’s currently no Sublime counterpart to the excellent Golang, Docker, Python, F#, and REST extensions available for VS Code (the first three of those come from Microsoft directly). One idea I’ve had for sparking a lot more energy toward Sublime package development is a polyglot package API. Meaning package developers could write packages in any of several programming languages, not just Python.

How feasible is this? I realize that Sublime probably has a small team of developers, and this is a big ask, but I at least want to put it out there. A smart strategy might be to identify popular languages that are not served by any existing extension or package APIs, particularly those for text editors. I think both Atom and VS Code extensions are written in JavaScript only (maybe CoffeeScript too for Atom). Ruby strikes me as one language with lots of enthusiastic developers who currently have no outlet for their skills re: text editors. Also Lua. Also Go. And C#, Java, Kotlin, Swift, Visual Basic, etc.

A different strategy would be to support the same language those other editors do: JavaScript, preferably while still supporting Python. Of course, you could execute a master stroke and implement both strategies, adding JS, Ruby, and Go support, for example.

One thing to keep in mind is that going for the most popular languages might not get you the most package development activity. After all, Python is plenty popular and yet there isn’t a lot of ST package development lately. A language with a smaller number of users, but who are extremely passionate, might produce the best results. For example, it’s possible that Haskell support would spark more activity than Ruby or C#. I’m not sure – in this scenario maybe Haskell developers would be so excited and grateful for the support that they’d build and maintain tons of packages. Or not.

You could build an API that relies on and executes a common intermediate representation (IR), but then you’d have to build compilers for the different languages to compile to that IR. Alternatively, there just so happens to be an emerging IR: WebAssembly (Wasm). The are many, many different toolchains in the works for different languages to compile to Wasm, so it might be worth checking out.

There are at least a couple of other ways to architect a polyglot package API, including having parallel, nearly equivalent APIs for different languages, using off-the-shelf interpreters or compilers, like you do for Python. (Also, maybe now’s a good time to upgrade the Python interpreter in Sublime Text to Python 3.8.x.) You could also limit support to a strict subset of a language, making it easier to implement.

Anyway, just some thoughts I had today.

[TheSecEng]

Sublime Text given how old most packages in Package Control are. It’s rare to come across packages that have been updated within the last couple of years.

Can I ask what the concern here is ? Just because a package hasn’t been updated, doesn’t mean it’s obsolete or even needs updated. If there aren’t any changes to the API and no performance improvements to make. I don’t see an issue here.

(Also, maybe now’s a good time to upgrade the Python interpreter in Sublime Text to Python 3.8.x.)

Following the current development, this is supported. You can learn more at the Sublime Discord server

[bschaaf]

While there’s certainly things that our API limits, from looking at the golang package for VS Code it mainly uses the go language server - which you can also do in ST - and provides a bunch of shortcuts for running standard go utilities - which again is nothing exclusive to VS Code.

VS Code and Atom derive their plugin system from the fact that they’re already using a dynamic programming language in an application development environment, ie. a web browser. It’s easy for them to simply load more JavaScript for changing all kinds of behavior. Sublime Text on the other hand is written in C++. That makes it fast, but also difficult to extend. Providing a reasonable plugin experience for us means using a separate plugin process. Adding support for languages such as JavaScript, Ruby and Java/Kotlin essentially require us to run separate processes for each of these languages, massively increasing complexity and overhead.

Natively compiled languages get even worse, as you can’t simply chuck native code together into a runtime. Instead every single plugin would require it’s own process. Imagine you have 3 python plugins, 2 ruby plugins, 10 go plugins, 5 C# plugins and a javascript plugin. You’re now running 17 subprocesses, each with their own runtime, garbage collector, memory/cpu overhead, etc.

One major advantage of having a singular language for writing plugins is that it’s easy to find examples and documentation. Having multiple languages for an API means essentially splitting up the plugin development community along language barriers. You’d no longer be able to look at a plugin and copy what you need.

Another major advantage is in API design. A python API is fundamentally different to a Haskell API and even a JavaScript API. In order to design an API that works across all languages, the API would only be able to use the lowest common denominator of language features, practically limiting the API to only C functions. And calling C functions from non-C languages generally isn’t fun, so a higher level API would probably be built on top for each individual language, furthering incompatibilities.

Wasm is great, if you want to run C/C++ in a browser. Wasm doesn’t currently provide any runtime features, meaning that every non-C/C++ language needs to provide it’s own runtime, leading right back to the resource overhead problem. Except instead of 17 subprocesses you have the one process with 17 runtimes running on it.

All in all, I don’t think it makes sense to support a large range of languages for the same reason it doesn’t make sense to write an application in a large range of languages. I think we’re seeing less language specific plugin development because it’s either moved to non-editor specific implementations, such as language servers, and there’s already a mature ecosystem of plugins available.

[Webby]

My concern is multithreaded. One is that support for various programming languages is old and newer syntax and features aren’t supported. Languages evolve and packages evolve with them, if they’re maintained.

Second I think it’s a bad sign that there’s so little energy going into package development compared with say four or five years ago, or compared with other text editors. You might think it doesn’t matter – I think it definitely matters and it sends a message to users. Sublime needs to do something, maybe several things, if they don’t want to lose share or become marginalized as far as package development and community involvement/energy are concerned.

[Webby]

Good points. I think it would be wise to explore further to see if there’s a way to craft an API that supports JS, Ruby, Go, or Lua in addition to Python in a way that doesn’t strip the API down to C functions. Some packages are already using Node.

[bschaaf]

None of those languages are inter-compatible. Implementing an API for any additional languages in a similar way to the current python API means recreating those APIs for that additional language. Packages currently using other languages are doing so either through a CLI or through language specific python/C bindings, both of which are already available options for plugin developers.

[deathaxe]

Most “smart” language features provided by VS Code are implemented via LSP protocol. The python plugin of VSC for instance contains a lot of python code to drive a python-language-server running jedi for code intelligence. All of that doesn’t need VSC to support python APIs natively. The same applies to ST vice versa.

The main issue is to provide required core functions to provide rich user experience for certain features. Engineering those without bloating the GUI or introduce performance issues is a challanging task.

Python is a wide spread general purpose runtime to easily create extensions.

As you said: Python, … are provided by Microsoft driven by fulltime paid developers, while all packages/plugins of ST are purely community driven. Everybody is doing so in spare time mostly to satisfy their own maybe limited needs quickly.

[Webby]

I was just reminded of GraalVM, which seems very much related. It’s a project to provide a common target and possibly a runtime for many different PLs. Ruby and Python are in their list, but I’m not sure how far they are with them. Have you looked at this?

[Webby]

I want to reinforce something I touched on originally: Sublime Text needs to do something, or many things, to bring back package developers and/or motivate new package developers. Package development and maintenance seems to have dropped off dramatically. The theme I was using has been abandoned, as have many packages in Package Control.

VS Code and possibly Atom have grabbed a lot of share and community energy. Sublime has much better bones, so to speak, than those Node-based editors, but some additional noteworthy features might be needed to recapture more community support. A polyglot extension API would be remarkably cool and noteworthy, and might help.

[rwols]

The goal of a package is ultimately to enhance or add a feature. Can you name some examples that you need and are impossible to do with the current API?

[bschaaf]

GraalVM is far from a solution: The current python implementation is far from compatible with even most of the standard library, plus it doesn’t support python 3.3 (a requirement for backwards compatibility); while truffleruby has more compatibility but it’s still far from a simple replacement for MRI.

Instead of spending so much time and energy on adding language support and re-implementing APIs in those languages, a better focus would be to improve the API to make plugins more powerful. This is what we’ve been doing: LSP plugins had some severe limitations before, because plugins didn’t have access to required information such as what changes have been made to a file. This has changed in ST4.

[Webby]

The things I need or want are the sorts of packages that are available on VS Code and Atom, especially the Python, F# (Ionide), REST, Go, and Docker packages on VS Code.

It’s not about discrete features that could be exploited in Sublime Text, but rather features that are exploited in mature and maintained packages. My point is about the liveliness of package development for Sublime, and how it’s dropped off dramatically in recent years. I don’t necessarily have a solution. I just think a polyglot API, or better yet a unified API that consumes an IR that can be targeted by multiple popular languages, would be a sweet and noteworthy feature that VS Code and Atom lack.

[Webby]

Okay, I read you as saying that it can’t be done and that it’s not worth doing. I think there’s a non-trivial risk that this will in fact be implemented by other applications. I was a software QA manager and business analyst for a few years, before I became a social scientist, and it always bugs me when developers say something can’t be done, especially when it totally can.

[UltraInstinct05]

The packages you speak of (other than for F# and REST, I am not sure about those) are maintained by Microsoft so I am assuming that they have full time paid engineers or interns that take care of those packages (not to mention the workforce they can deploy behind those packages, it’s a gigantic tech company after all). SublimeHQ, on the other hand is a very small team (just a handful of engineers) that are working on Sublime. You can’t expect them to maintain packages and do core development at the same time without a huge compromise on rolling out essential/newer/enhanced features for the product.

My point is about the liveliness of package development for Sublime, and how it’s dropped off dramatically in recent years

There are many packages on PC that do not require any “liveliness” so to speak because they are already feature full and do not require further enhancements. Yes, it is true that the package authors can update the README atleast from time to time to give an indication that he/she is still aware of his/her package(s)

I just think a polyglot API, or better yet a unified API that consumes an IR that can be targeted by multiple popular languages

There are still a lot of issues with the current python API that are logged into the core tracker that are to be taken care of. Introducing a polyglot API would be disastrous. So is the team to spend time on core performance or to answer 100’s of bug reports generated by each language API ?

I think there’s a non-trivial risk that this will in fact be implemented by other applications

As far as I know, none of the other text editors seem to be moving towards a polyglot API. VS Code uses TypeScript/JavaScript, Brackets uses JavaScript, Atom uses CoffeeScript, Notepad++ uses C++ Win32 API’s (Window’s only). None of them have yet introduced or seem to be moving towards it.

it always bugs me when developers say something can’t be done, especially when it totally can.

The fact is that it might be possible to do it but it’s just not feasible in the long run (or even short run in this case). Just because it can be done, doesn’t necessarily mean it should be done.

[bschaaf]

I’m certainly not saying that a polyglot API can’t be done. (Though I did explain why GraalVM can’t be used as a common basis for one, unless/until massive development is done to bring graalpython in line with cpython) It’s simply that a polyglot API is a trade off. Do we spend a lot of development time and some resource usage to add a single language for writing plugins or instead spend that time adding plugin API features, core features and bug fixes.

[Webby]

The packages you speak of (other than for F# and REST, I am not sure about those) are maintained by Microsoft so I am assuming that they have full time paid engineers or interns that take care of those packages (not to mention the workforce they can deploy behind those packages, it’s a gigantic tech company after all). SublimeHQ, on the other hand is a very small team (just a handful of engineers) that are working on Sublime. You can’t expect them to maintain packages and do core development at the same time without a huge compromise on rolling out essential/newer/enhanced features for the product.

I’m not suggesting that SublimeHQ create or maintain any packages. I’m suggesting that they bolster the package API to attract more package developers and engender more and better packages.

Someone above made the excuse about Microsoft having lots of developers to build packages. That’s just an excuse, and targeted at a point I’m not making. Would the packages I mentioned exist on Sublime if they didn’t have the likes of Microsoft behind them? I’m not sure.

On the liveliness issue, updating Readmes is about what is happening right now with all the dead Sublime packages. That wouldn’t apply to my suggestion to spark more package development, since I’m saying that a bolstered API would attract substantive development.

To your point about the packages already doing what you want, well that wouldn’t apply to my point about the packages on VS Code that I really like. And I’d also make the point that lively package development leads to features and capabilities that we wouldn’t have anticipated, not to mention more bug fixes and performance/overhead improvements. It wouldn’t make sense to think that if we stopped all development and changes today that we’d be set for the next 10 years, for instance – we’d miss a lot of developments and improvements.

[Webby]

I’m certainly not saying that a polyglot API can’t be done. (Though I did explain why GraalVM can’t be used as a common basis for one, unless/until massive development is done to bring graalpython in line with cpython) It’s simply that a polyglot API is a trade off. Do we spend a lot of development time and some resource usage to add a single language for writing plugins or instead spend that time adding plugin API features, core features and bug fixes.

Got it. I see what you’re saying. It’s a bummer that such trade-offs are necessary. Unlike most people, I actually paid for Sublime, and it’s frustrating to see package development drop off so dramatically almost exactly when I paid for it. I just learned of ST4 and I hope it’s noteworthy. Making it easier to migrate packages to Python 3.8 would be a nice win, especially is it was a simple automated thing.

[hasen-judy]

Honestly when I use Sublime it’s because of its minimally.

VScode gets slower when you add packages and I don’t like the UX of editing programs in it while using LSP based plugins. They all tend to be erratic and slow.

The people who really want the plugins and autocomplete will just use VSCode. There’s no point in cometing with others on their own terms because you will always be playing catch-up.

Focus on your strengths.

[rwols]

With respect to Ionide, you may consider getting https://github.com/fsharp/FsAutoComplete to work with https://packagecontrol.io/packages/LSP