Sublime Forum

Package Distribution Ecosystem

#1

Is there any documentation somewhere that describes how Package distribution was designed to work?

I see under Preferences > Package Control several package-distribution vocabulary words like:

  • channel
  • repository

and while I can dig into the Package Control source code and figure it all out, doing so would be horrifyingly inefficient. And while I am aware of the existence of

I’m pulling my hair out trying to sort through these and figure out how they relate to one another. Has anyone written up how the Package data flows and how it was all designed to work? What it’s current state is? If it is evolving towards a different (or new) scheme, and why?

Kind regards,
Vic

0 Likes
#2

Are you looking to create and distribute your own package, or simply wish to understand the ecosystem better?

Either way, this is a good starting point:
Package Control: Usage | Sublime Text Community Documentation

0 Likes
#3

Regarding packges.sublimetext.io, there’s a bit of history documented: https://packages.sublimetext.io/about . tldr; It aims to replace packagecontrol.io with newer technology (in order to work around some existing issues with the older original packagecontrol.io) and is being maintained by a couple of community members. From a user perspective, its equivalent to packagecontrol.io, but could work better in some cases. The future plan is that SublimeHQ maintains an “official” package platform (likely with the same infrastructure that packages.sublimetext.io uses today), completely replacing packagecontrol.io. I guess no one knows, what the future of packages.sublimetext.io will be then.

Well, that was longer than expected :sweat_smile:

(Disclaimer: That’s my personal summary of the website situation)

0 Likes
#4

@gbird, good question!

I’m trying to document how it works—both for my own understanding (to better understanding how, why and when to use the commands available under Preferences > Package Control)—and I am recording what I learn in detail which I am expecting will, at some point, become a Sublime Text reference of some type for posterity (perhaps a book?).

I understand that this is really not the domain of Sublime Text proper but rather of the Package Control Package.

I also understand that the package_control_channel repository on GitHub is where new Packages are submitted (I already have one in the field and am working on 2 more with a 3rd one in the design stages). Also, from Preferences > Package Settings > Package Control > settings it is clear that the Package Control package is loading its lists from the channels in the “channels” list (namely https://packagecontrol.github.io/channel/channel_v4.json), which is not the same place.

So what is missing in my mind are 2 things:

  • the data flow that happens between the above two locations, and
  • the DESIGN INTENTION of the Package Control Package authors about how it is all intended to work. (A diagram or “data-flow plan” would be amazingly helpful, and I am willing to help document it, but I must myself have an understanding of it before I can document it.)
0 Likes
#5

Hi, @milkman!

Outstanding! Thank you! That makes sense and is what I would do: develop a parallel distribution channel while the old one is still in service, and update all the documentation after it’s all over… But I feel strongly that what you just told me is going to be helpful to all Package developers. If there is the option to do so, I recommend an additional “branch” to the Package Control Docs (sibling branch to “Usage”) like “State of Package Control” where this is explained. Probably not everyone is as interested in the details like I am, but it would provide understandings that are very helpful to Sublime Text users’ enthusiasm about Sublime Text, so they don’t get the idea that “something is broken down” or that the entity that controls packagecontrol.io hasn’t been answering their phone since COVID or something scary like that…

0 Likes
#6

The package_control_channel repository basically provides a template.

Technically, Package Control could directly use package_control_channel, but fetching required information for all packages from code hosters is rather expensive. As PC was created before asyncio arrived, it is not really designed to perform heavy parallel REST API access work.

Hence crawlers like (packagecontrol.io or thecrawl) do the hard job of fetching up-to-date package information from code hosters and convert all “release” entries to URL-based releases with directly downloadable packages. So PC just needs to load a single JSON file and can directly download requested packages.

Technically, those crawlers would just need to provide a very simple repository.json with all package entries only containing those URL based releases. We’d just need to replace “channels” setting by “repositories” to point to another URL and would be done.

A channel in it’s origninal form is just a collection of repositories. This could be achieved using “include” in a repository as well. But those probably came later.

A repository is just a collection of packages and/or libraries (aka. python packages).

The channel downloaded by PC is exactly the channel.json from package_control_channel repository with package_cache and dependencies_cache entries being added by crawlers containing package meta data with resolved URLs. Overcomplicated, don’t ask anyone why this choise was made.

0 Likes