Would it be possible to somehow override/extend the lexical analysis that Sublime Text is doing, with a native library that provides the same API to ST? If so, is there some documentation that could be shared on how to achieve this? The reason I'm asking is that, for more advanced language support, something more than the regular expressions in TextMate's language definitions allow for would be very useful. I'm asking about this in preparation for adding support for the Red (and Red/System) language (http://www.red-lang.org/) as well as for Rebol (http://www.rebol.com/ - recently open sourced.
If this is not doable today, would it be something that could be considered? The Rebol parser is an amazing thing and many ST plugin developers would benefit from using something like it (Red will also include this type of parser), over using simple regular expressions. I'm hoping that if advanced Red/Rebol support can be added by means of allowing for an external lexer, some developers might choose to use these languages over Python since resulting code would typically be more concise and expressive (esp. when dealing with more sophisticated parsing needs).