Support (status) for Asciidoc(tor)

[gour]

Hello,

I’m aware that lot of people jumped to vscode editor since it provides good support for many languages. Personally, I’m not thrilled to use Electron app as editor and would like something better/easier to configure that Emacs/Vim (I use VIm for certain simpler things) and that’s why I bought ST3 license long-ago.

Anticipating my future needs, I might do some light PHP coding and possibly some Python development, but I do foresee lot of writing using Asciidoc(tor) markup as well as some Markdown (which is supported well in ST), so wonder about the future of Asciidoc-related packages for ST(4)?

Currently there are two:

1. Asciidoctor which is abandoned having last commit in 2015 and

2. AsciiDoc having last commit from June 2018, seemingly embraced by ST team, although in PackageControl it is stated: " Currently looking for a maintainer!" and on Github it is still stated: "AsciiDoc Package for Sublime Text2 "

So, the question is whether one can consider that ST(4) is reliable editor for those having the need to write Asciidoc(tor) markup extensively?

The situation with both Emacs & Vim is also not thrilling…only vscode seems to enjoy some active support via this extension and in that case my option would be to count on Onivim2 which is supposed to enable one to take advantage of vscode extension’s ecosystem.

Anyone?

[bschaaf]

Considering the AsciiDoc package only contains syntax highlighting, snippets and settings. It should work fine in newer versions of ST.

[tajmone]

So, the question is whether one can consider that ST(4) is reliable editor for those having the need to write Asciidoc(tor) markup extensively?

Hi @gour, I’m facing the same problems as you, since I also work with AsciiDoc (Ruby) on a daily base. None of the available AsciiDoc syntaxes work well, and they often break up when encountering patterns that they can’t handle, which ultimately breaks up most of the functionality that’s supposed to be available on a document.

The problem is not really related to any specific editor, but to the fact that the AsciiDoc syntax is very hard to handle with RegEx-based syntax definitions. To get a decent support for AsciiDoc, you’ll have to either look for a dedicated editor (e.g. Asciidoc FX or for an AsciiDoc Language Server — i.e. a syntax based on the Language Server Protocol (LSP).

Language Servers are real parsers created ad hoc for a syntax, so they are able to offer full support for any language or markup syntax, since they are not constrained by the limits of a RegEx based syntax definition. Also, Language Servers are editor/IDE agnostic applications that will work on any editor that supports LSP (ST3 does have an LSP package).

The problem is that no one has created an AsciiDoc Language Server so far — at least, none that I know of.

[gour]

Hello,

Hi @gour, I’m facing the same problems as you, since I also work with AsciiDoc (Ruby) on a daily base.

let me share how I did solve my problem…

For most of the (simple) content that I write I decide to use Markdown markup, while for the content which requires higher quality output I decided to use ConTeXt. for a long time I was considering which markup to use, but I was convinced when I got reply from Hans Hagen - main developer behind ConTeXt who wrote on the context list:

Before we started with context we uses ascii based markup (i still have
printouts of the code used for pagination, figure placement, tocs around
somewhere) … but as the input becomes more complex it makes no sense
any more to use such formats and tex (or nowadays xml) starts looking
clean and simple in comparison

writing an asciidoc parser that directly maps to context is not that
hard but i’d probably quickly start asking myself: why … esp when the
extras are to be dealt with:

[[terms]]
[glossary]
List of Terms

[“glossary”,id=“terms”]
List of Terms

[template=“glossary”,id=“terms”]
List of Terms

where

\startglossary[reference=terms,title={List of Terms}]

is not that more coding.

and he is right…to take full advantage of AsciiDoc(tor) one has to write markup which is not so simple any longer, but in that situation ConTeXt provides better typesetting engine for final output than anything available for AsciiDoc. Moreover, ConTeXt can also typeset Markdown documents.

[tajmone]

Thanks for sharing @gour!

My solution has been to edit the AsciiDoc ST package and start to tweak it to prevent braking the syntax.

Today I’ve finally pushed to GitHub my initial draft:

tajmone/ST3-Asciidoctor

I’ve ported the syntax from the old ST2 format to the new ST3 format, after having applied some of the patches in the pending pull requests of the upstream repository.

Since I need it to work every day, my approach has been to remove from the syntax definition those elements that tend to break up the document, and then gradually start to reimplement them again without the original problem. In this respect, “less is more” — i.e. I’d rather have less syntax elements covered, but prevent the document from breaking up.

Unfortunately, the repository is not yet usable as a package (the current work is not even on master), I’ll need to remove some spurious files and setup the package settings to allow third parties to clone it in the ST packages folder, so that it will automatically update via Git. Since right now I’m the only one using it, I’m in no hurry to make it a full fledged package, but of course if others ask me to, I’ll happily accelerate the procedure (having others use the package would provide more feedback, which should help isolate the problematic elements even further).

[GeraldLoeffler]

@tajmone . I’m a heavy user of AsciiDoc (in its asciidoctor implementation) and have so far been writing exclusively in sublime. I do appreciate sublime for many reasons, and its AsciiDoc support has been almost good enough for me. In 2017 i submitted this PR https://github.com/asciidoctor/sublimetext-asciidoc/pull/9 against the original GitHub project, and although it was never merged, i’ve used that patched package ever since. I see that your fork/branch (https://github.com/tajmone/ST3-Asciidoctor/tree/myST3work) incorporates an equivalent fix (and probably much more), and so i’d be happy to switch over to using your package.

Please let me know how you want me to proceed switching to your package. For example, if you want to apply additional tidying-up or documentation before i make the switch, then please do that and give me a shout when it’s ready.

[tajmone]

@GeraldLoeffler,

indeed I’ve applied to my repo a couple of fixes from the PRs that were never merged on the upstream repository. One of them was by user @bsmith-n4, which I duly credit in my integration commit:

Syntax Fixes by @bsmith-n4

the other one, I don’t remember where I took it from, which is why I haven’t yet merged to master my fork work — i.e. I’ve already updated the REAME file, but I still need to add in the credits the authors from the upstream PRs whose fixes I’ve integrated. So, I only need to go through all the commits history and find the fixes (I think they were 2, maximum 3, which I picked out from the PRs, to integrate them in the old ST3 TextMate syntax, before migrating to the .sublime-syntax format).

Once I’ve added the due credits in the README, I’m ready to merge into master (I’m fussy about crediting where credit it’s due). Then it should be possible to install the package locally via Git, directly in ST3’s Packages/ folder.

Please let me know how you want me to proceed switching to your package. For example, if you want to apply additional tidying-up or documentation before i make the switch, then please do that and give me a shout when it’s ready.

I’m really glad to hear that someone else will be benefiting from the package, which means I’ll also get some feedback on how to improve it, and contributions might flow in in the course of time.

In the weekend I’ll finish polishing the README, add the missing credits and some instructions on how to install it via Git, and then I can merge into master, since the package is already usable (and I do use it every day). When it’s done, I’ll confirm it to you here in this post, and from then on the repository Issues can be used to freely discuss any improvements suggestions, a roadmap, etc.

[GeraldLoeffler]

sounds perfect @tajmone: looking forward to it

[tajmone]

@GeraldLoeffler, I’ve update the Credits section and merged to master:

tajmone/ST3-Asciidoctor

I think the best way to install it is by forking and then cloning it via Git into ST3 “Packages” data folder, so it won’t be in a packed archive format and you can also edit it to create PRs, etc. My understanding is that the package will be auto-updated if I add new commits to master, which I only do when a fix has been tested thoroughly enough on the dev branches first.

[GeraldLoeffler]

ok, @tajmone, i’m up-and-running with your “package”. Specifically, i’ve done (on my Mac)

cd /Users/gerald/Library/Application\ Support/Sublime\ Text\ 3/Packages
git clone https://github.com/tajmone/ST3-Asciidoctor.git

restarted sublime, then ST3-Asciidoctor shows up in sublime’s Package Control: List Packages and AsciiDoc syntax is seemingly correctly highlighted. So all is good.

I’ve seen the GitHub Issues you’ve created in your repo, and will do likewise if i find anything worth improving.

Thank you!

[tajmone]

Glad to hear that!

I’ve seen the GitHub Issues you’ve created in your repo, and will do likewise if i find anything worth improving.

Great. I’ll also buzz you via repo Issues to consult you when there are major decisions to be taken the, so we’re both happy about how it will affect daily usage.

git clone https://github.com/tajmone/ST3-Asciidoctor.git

Didn’t you experience problems for omitting the subfolder name Asciidoctor after the command?

[gour]

Hello, @tajmone,

Thanks for sharing @gour!

My solution has been to edit the AsciiDoc ST package and start to tweak it to prevent braking the syntax.

Today I’ve finally pushed to GitHub my initial draft:

https://github.com/tajmone/ST3-Asciidoctor/tree/myST3work

After some more thinking and trying to configure different editors to “speak” ConTeXt (via LSP), I’ve concluded that the support is not stellar and one has to write manually a lot without much help from the editor. Moreover, ConTeXt markup by itself is not so readable and requires quite a bit of learning for which I simply do not have time. Furthermore, AsciiDoc enables one to produce HTML/epub along with nice PDF, so I’ve decided to use it.

When it comes to editor(s), I gave up on Emacs (and org-mode)…vim does require a complex setup, while Oni2 is not ready (yet), so I’ll settle on ST(4).

Unfortunately, the repository is not yet usable as a package (the current work is not even on master), I’ll need to remove some spurious files and setup the package settings to allow third parties to clone it in the ST packages folder, so that it will automatically update via Git. Since right now I’m the only one using it, I’m in no hurry to make it a full fledged package, but of course if others ask me to, I’ll happily accelerate the procedure (having others use the package would provide more feedback, which should help isolate the problematic elements even further).
[/quote]

I did install by cloning your repo and it works here and I prefer it over e.g. the VScode extension.

However, I tend to agree that it would be terrific to have proper LSP support for AsciiDoc (just asked on the list about it) and wonder if Python could be good-enough language for implementation. I’m not really a coder, but will help as much as I can - Python is not so hard to pick it up - to bring support for writing AsciiDoc in ST to a higher level.

Sincerely,
Gour

[tajmone]

I did install by cloning your repo and it works here and I prefer it over e.g. the VScode extension.

I’m glad you’re finding it useful. Right now I believe other 3 people are also using it. Now and then I try to fix some known problems, but it’s not always easy because the original syntax had ended up rather entangled, so it’s hard to work out were to make changes, especially in regard to exploiting the new ST3 syntax format, which is more flexible.

However, I tend to agree that it would be terrific to have proper LSP support for AsciiDoc (just asked on the list about it) and wonder if Python could be good-enough language for implementation.

Python would be a fine choice, and there’s already an LSP SDK for Python:

• https://github.com/openlawlibrary/pygls

Ruby might have proven a more profitable choice for it would allow reusing parts from the original Asciidoctor (Ruby) engine, but there isn’t a Ruby LSP-SDK as far as I know:

• https://microsoft.github.io/language-server-protocol/implementors/sdks/

I’m not really a coder, but will help as much as I can - Python is not so hard to pick it up - to bring support for writing AsciiDoc in ST to a higher level.

The challenge probably won’t be so much with the adopted implementation language, but rather with the need to implement a low latency and fault-tollerant parser. In other words, not only you’ll need to build a full-fledged parser which is able to build an AST and provide the semantics of the source file, but it will have to do so without affecting the editor’s usability — unlike a compiler, the Lang Server will be operating on source code which is being edited, and the editor (client) will notify the server about document changes on each key stroke!

So we’re dealing with a special type of parser here, i.e. one that is able to handle constant changes to the source, including while still parsing the previously fed source. Furthermore, it has to be a fault-tolerant parser — i.e. it has to be able to handle incomplete constructs without breaking the document. Users often type incomplete code snippets and then jump to other parts of the document while editing, and a Lang Server will have to handle in real-time all these changes.

Fault-tolerant parsers of this type are a sort of terra incognita, something that LSP has brought to attention and for which you won’t find much literature either. Here are a couple of links to articles where authors of Language Servers share they thoughts and design choices on the topic:

• https://github.com/Microsoft/tolerant-php-parser/blob/master/docs/HowItWorks.md
• http://duriansoftware.com/joe/Constructing-human-grade-parsers.html
• https://docs.microsoft.com/en-us/archive/blogs/ericlippert/persistence-facades-and-roslyns-red-green-trees

So, ultimately writing a good Lang Server package can be quite challenging, and surely more than writing a “classic” parser (which is in itself a language-engineering accomplishment). Add to that the fact that AsciiDoc has a particularly complex syntax (with Asciidoctor also adding some extra features), and the task starts to look a rather arduous and long-term one.

Surely, one might (and should) approach the task by starting with simple goals first, rather than covering the whole syntax — after all, the Lang Server server mostly as a syntax highlighter, and doesn’t necessary have to fully cover the syntax.

[Hellman]

I just returned to ST4 after a long break. Is it safe to say asciidoc is dead on ST? I just paid for the upgrade to 4 so I’d like to make asciidoc work if possible. Am I fighting a loosing battle? Should I move to Atom and cut my losses?

[jfcherng]

Neither Atom nor asciidoc user, but interested in how it works better in Atom so that you consider moving to it?

[Hellman]

They have working syntax highlighting and live preview. I used to be able to do this with OmniMarkupPreview but the last commit to either that or the asciidoc(tor) plugins were over 4 years ago, leaving me to believe the projects are dead and the maintainers have moved on to something else.

[deathaxe]

fighting a loosing battle … move to Atom

…Atom, the other dead end?

I guess most Markdown/Asciidoc users moved on to VS Code. MarkdownEditing is (nearly) dead, too, as all maintainers and most users moved to VS Code or Obsidian.

This in-app live preview thing is probably the most compelling argument for most who don’t want to fiddle around with all those external live-reload hacks. It’s just too annyoing and fragile to come up with proper solution for that.

Maybe serious improvements on those packages may bring some users back or keep others from moving.

It’s dead if noone cares about any longer.

[oootini]

I write daily in Sublime and asciidoc, with some simple tweaks it is very usable. for example, configuring an Asciidoc preview build:

1. Create a file asciidoc.sublime-build in ~/.config/sublime-text-3/Packages/User and add the following text:

{ "cmd": ["asciidoctor-preview", "$file", "$file_path/$file_base_name.html"], "selector": "text.asciidoc" }

2. Create a file asciidoctor-preview in /usr/local/bin , with the following text:

#!/usr/bin/bash
if [ $# = 0 ]; then
echo "Error: you need to provide the filename and output filename as input params in sublime build system"
else
asciidoctor -a toc $1
xdg-open $2
rm -f $2
fi

3. Make the file executable:

sudo chmod +x asciidoctor-preview

Reload Sublime. [ctrl+b] builds any asciidoc file and opens it in the browser.

[polyglot-jones]

Hey tajmone (and everyone),

Quick Intro: I’m a retired software developer who now writes books (fiction and non).I use AsciiDoc extensively. It’s my daily-driver for notetaking, journalling, etc. and I use it to self-publish (a reference encyclopedia and three of my friends’ novels, so far). I’m a loyal user of ST4 (since ST2). For one thing, writing extensions for ST (in Python) is SO much nicer than writing them for VSCode (JavaScript sucks!).

On Topic: I just submitted a pair of pull requests to aireilly’s fork of ST3-Asciidoctor https://github.com/aireilly/ST3-Asciidoctor/pulls
One advances the syntax highlighting. The other adds several smart snippets.
(Note the PR that “fails” to build will be fine after the other PR is merged.)
I believe this brings the work that aireilly started up to being worthy of merging into your fork.
Hopefully, he’ll pull them in shortly and then pass along a complete PR. If not, you might want to pull in everything anyway, without waiting for a request.

Thanks,

// Craig Jones (polyglot-jones)

[tajmone]

Thanks for these very helpful tips @polyglot-jones.

I really wish the original Asciidoctor package wouldn’t have gone stale as it has, for all these forks in the wild are unlikely to ever replace that package — partly due the fact that (unlike VSCode Marketplace) Package Control requires unique package names.

Another huge deterrent is due to the problems in keeping updated these self-hosted packages on GitHub. I initially thought that by installing my ST3-Asciidoctor fork as suggested in my instructions would result in the package being auto-updated, but it’s not so. There are workaround to this, but none of them really works the way I’d like it to.

If on the one-hand ST4’s newly added syntax features (e.g. branching) open up a lot of new possibilities for the AsciiDoc syntax, I personally think that VSCode is going to turn out to be a better editor for AsciiDoc in the long run, especially thanks to LSP.

For one thing, writing extensions for ST (in Python) is SO much nicer than writing them for VSCode (JavaScript sucks!).

I agree on this, but I also see VSCode’s enhanced HTML support as a strong point when it comes to AsciiDoc, allowing live preview in the editor (among other things).

Anyhow, I’ll look into these forks which you mentioned, and try to integrate their improvements in my fork too. Eventually, all these forks might result in a collaborative effort to create a new Asciidoctor package that might replace the original (now dead) one one Package Control (I guess we’ll have to come up with some fancy name for it).