Sublime Forum

Problem developing syntax-highlight for Cottle

#1

I’m trying to build a syntax-highlighter for Cottle (which is the script language used in EDDI, a text-to-speech companion program for Elite:Dangerous).

Here’s a working script:

{_ this is a working cottle script for EDDI }

{_ this
 is 
 a 
 comment
}

{set variable to 1}											{_ this is a comment}
unquoted string with 1 {variable} inside.					{_ this is a comment}
'single-quoted string with 1 {variable} inside.'			{_ this is a comment}
"double-quoted string with 1 {variable} inside."			{_ this is a comment}
{if false:
	unquoted string with 1 {variable} inside.				{_ this is a comment}
	'single-quoted string with 1 {variable} inside.'		{_ this is a comment}
	"double-quoted string with 1 {variable} inside."		{_ this is a comment}
|elif:
	unquoted string with 1 {variable} inside.				{_ this is a comment}
	'single-quoted string with 1 {variable} inside.'		{_ this is a comment}
	"double-quoted string with 1 {variable} inside."		{_ this is a comment}
|else:
	unquoted string with a {Emphasize("this is a string_parameter")} inside.
	"double-quoted string with a {Emphasize("this is a string_parameter")} inside."
	'single-quoted string with a {Emphasize("this is a {cat("string", "_", "parameter")} ")} inside'.
	'single-quoted string with a {Emphasize("this is a {	{_ this is a comment}
		{_ this is a comment}
		cat("string", "_", "parameter")
	} ")} inside.'
}

'It\'s a "single-quoted string" with 1 {variable} inside.'	{_ this is a comment}
"It's a \"double-quoted string\" with 1 {variable} inside."	{_ this is a comment}
It\'s a \"unquoted string\" with 1 {variable} inside.		{_ this is a comment}

In this script, everything that’s not {code is treated as a string (then spoken by EDDI), even if unquoted. BUT: you can open a code block inside a string, to use variables, call functions or even call other scripts… and these code blocks can contains other strings.
This kind of nesting makes me crazy, but i think i’ve sort almost all.

(Here you may look at the syntax-highlight used by EDDI editor:
https://github.com/EDCD/EDDI/blob/develop/SpeechResponder/Cottle.xshd
It is very well commented and explains how it works better than my poor english).

To explain my problem, here i will just highlight with different color comments, code and strings.
So this is my “simplified” sublime-syntax file:

%YAML 1.2
---
file_extensions:
  - cottle
scope: source.cottle
contexts:
  main:
    - include: StringOrCode

  StringOrCode:
    - include: comment
    - include: cottle_code_block
    - include: string

  comment:
    - match: \{\_
      scope: punctuation.definition.comment.cottle
      push: comment_body

  comment_body:
    - meta_scope: comment.block.cottle
    - match: \}
      scope: punctuation.definition.comment.cottle
      pop: true

  cottle_code_block:
    - match: \{
      scope: punctuation.section.block.begin.cottle
      push: cottle_code_block_body

  cottle_code_block_body:
    - meta_scope: meta.block.cottle keyword.other.cottle
    - match: \}
      scope: punctuation.section.block.end.cottle
      pop: true
    - include: StringOrCode

  cottle_code_block_body_within_string:
    - clear_scopes: 1
    - include: cottle_code_block_body

  string:
    - include: escape_characters
    - include: double_quoted_string
    - include: single_quoted_string

  escape_characters:
    - match: \\.
      scope: constant.character.escape

  double_quoted_string:
    - match: \"
      scope: punctuation.definition.string.begin.cottle
      push: double_quoted_string_body
  double_quoted_string_body:
    - meta_scope: meta.string.cottle string.quoted.double.cottle
    - include: escape_characters
    - match: \"
      scope: punctuation.definition.string.end.cottle
      pop: true
    - include: Nested_Code_In_String

  single_quoted_string:
    - match: \'
      scope: punctuation.definition.string.begin.cottle
      push: single_quoted_string_body
  single_quoted_string_body:
    - meta_scope: meta.string.cottle string.quoted.single.cottle
    - include: escape_characters
    - match: \'
      scope: punctuation.definition.string.end.cottle
      pop: true
    - include: Nested_Code_In_String

  Nested_Code_In_String:
    - match: \{
      scope: punctuation.section.block.begin.cottle
      push: cottle_code_block_body_within_string
#    - include: StringOrCode

And this is how it looks:


(“code” is blue, “string” is brown, “comment” is grey, “source” is red)

My big, first problem is: how to catch unquoted strings???
I don’t know how should i manage them inside the syntax file.

Also, i don’t know how to solve the nesting between strings and code: if you look at line 11, the code inside the string is not highlighted as code, and i don’t know why.

Could someone give me some help?

0 Likes

#2

Here’s an attempt based on your example and assuming that the only thing allowed on a line in front of an unquoted string is whitespace and that they cannot begin with { or |.

Was done in 20 minutes or so, therefore no guarantee for correctness or to be bug-free :slight_smile:
And it’s probably not the optimal solution regarding the detection of unquoted strings and other elements, but it may help.

syntax
%YAML 1.2
---
file_extensions:
  - cottle
scope: source.cottle

contexts:
  main:
    - include: comment
    - match: (?=\{)
      push:
        - meta_scope: meta.thisscopewillbecleared
        - include: code-block
        - match: ''
          pop: true
    - include: string

  comment:
    - match: \{\_
      scope: punctuation.definition.comment.begin.cottle
      push:
        - meta_scope: comment.block.cottle
        - match: \}
          scope: punctuation.definition.comment.end.cottle
          pop: true

  string:
    - include: double-quoted-string
    - include: single-quoted-string
    - include: unquoted-string

  double-quoted-string:
    - match: (?:^\s*)?(\")
      captures:
        1 : punctuation.definition.string.begin.cottle
      push:
        - meta_scope: meta.string.cottle string.quoted.double.cottle
        - match: \"
          scope: punctuation.definition.string.end.cottle
          pop: true
        - include: escaped-character
        - include: code-block

  single-quoted-string:
    - match: (?:^\s*)?(\')
      captures:
        1: punctuation.definition.string.begin.cottle
      push:
        - meta_scope: meta.string.cottle string.quoted.single.cottle
        - match: \'
          scope: punctuation.definition.string.end.cottle
          pop: true
        - include: escaped-character
        - include: code-block

  unquoted-string:
    - match: ^\s*(?=[^|\{])
      push:
        - meta_content_scope: meta.string.cottle string.unquoted.cottle
        - match: (?=\s*(?:\n|\{\_|\}))
          pop: true
        - include: escaped-character
        - include: code-block

  code-block:
    - match: \{
      scope: punctuation.section.block.begin.cottle
      push:
        - clear_scopes: 1
        - meta_scope: meta.block.cottle
        - match: \}
          scope: punctuation.section.block.end.cottle
          pop: true
        - include: keyword
        - include: constant
        - include: function
        - include: parens
        - include: comment
        - include: double-quoted-string
        - include: single-quoted-string
        - include: unquoted-string

  escaped-character:
    - match: \\.
      scope: constant.character.escape.cottle

  keyword:
    - match: \b(?:if|elif|else)\b
      scope: keyword.control.conditional.cottle

  constant:
    - match: \b(?:true|false)\b
      scope: constant.language.cottle

  function:
    - match: (?:^\s*)?\b(Emphasize|cat)\b
      captures:
        1: support.function.cottle

  parens:
    - match: \(
      scope: punctuation.section.parens.begin.cottle
    - match: \)
      scope: punctuation.section.parens.end.cottle
0 Likes

#3

Damn, to slow.

Maybe not perfect, but that’s my interpretation of the provided link’s content.

%YAML 1.2
---
file_extensions:
  - cottle
scope: source.cottle

variables:
  builtin_functions: |-
    \b(?x:
      abs | add | call | cast | cat | ceil | char | cmp | cos | cross | default
    | defined | div | eq | except | filter | find | flip | floor | format | ge
    | gt | has | join | lcase | le | len | lt | map | match | max | min | mod
    | mul | ne | ord | pow | rand | range | round | sin | slice | sort | split
    | sub | token | type | ucase | union | when | xor | zip
    )\b

  custom_functions: |-
    \b(?x:
      BlueprintDetails | BodyDetails | CombatRatingDetails | Distance | EconomyDetails
    | Emphasize | EmpireRatingDetails | ExplorationRatingDetails | F
    | FederationRatingDetails | GalnetNewsArticle | GalnetNewsArticles
    | GalnetNewsDelete | GalnetNewsMarkRead | GalnetNewsMarkUnread | GovernmentDetails
    | Humanise | ICAO | List | Log | MaterialDetails | Occasionally | OneOf | P
    | Pause | Play | SecondsSince | SecurityLevelDetails | SetState | ShipCallsign
    | ShipDetails | ShipName | Spacialise | SpeechPitch | SpeechRate | SpeechVolume
    | StartsWithVowel | StateDetails | StationDetails | SuperpowerDetails
    | SystemDetails | TradeRatingDetails
    )\b

contexts:
  main:
    - include: body-text

  body-text:
    - include: comments
    - include: blocks
    - include: quoted-strings
    - include: unquoted-strings

  code:
    - include: comments
    - include: blocks
    - include: predicate-statements
    - include: expressions

  expressions:
    - include: groups
    - include: mappings
    - include: keywords
    - include: literals
    - include: numbers
    - include: operators
    - include: properties
    - include: function-calls
    - include: quoted-strings
    - include: variables

###[ COMMENTS ]###############################################################

  comments:
    - match: \{\s*_
      scope: punctuation.definition.comment.cottle
      push: comment-body

  comment-body:
    - meta_scope: comment.block.cottle
    - match: \}
      scope: punctuation.definition.comment.cottle
      pop: true

###[ STATEMENTS ]#############################################################

  blocks:
    - match: \{
      scope: punctuation.section.block.begin.cottle
      push: block-body
    - match: \}
      scope: invalid.illegal.stray.cottle

  block-body:
    - meta_scope: meta.block.cottle
    - match: \}
      scope: punctuation.section.block.end.cottle
      pop: true
    - include: code

  block-end-ahead:
    - match: (?=\})
      pop: true

  predicate-statements:
    - match: '\:'
      scope: punctuation.separator.statement.cottle
      push: predicate-statements-body

  predicate-statements-body:
    - match: \|
      scope: punctuation.separator.statement.cottle
      pop: true
    - include: block-end-ahead
    - include: body-text

###[ EXPRESSIONS ]############################################################

  function-calls:
    - match: '{{builtin_functions}}(?=\()'
      scope: meta.function-call.identifier.cottle support.function.cottle
      push: function-call-arguments
    - match: '{{custom_functions}}(?=\()'
      scope: meta.function-call.identifier.cottle variable.function.cottle
      push: function-call-arguments

  function-call-arguments:
    - meta_include_prototype: false
    - match: \(
      scope: punctuation.section.group.begin
      set: function-call-arguments-body

  function-call-arguments-body:
    - meta_scope: meta.function-call.arguments.cottle meta.group.cottle
    - match: \)
      scope: punctuation.section.group.end.cottle
      pop: true
    - include: expressions

  groups:
    - match: \(
      scope: punctuation.section.group.begin.cottle
      push: group-body

  group-body:
    - meta_scope: meta.group.cottle
    - match: \)
      scope: punctuation.section.group.end.cottle
      pop: true
    - include: expressions

  mappings:
    - match: \[
      scope: punctuation.section.mapping.begin.cottle
      push: mapping-body

  mapping-body:
    - meta_scope: meta.mapping.cottle
    - match: \]
      scope: punctuation.section.mapping.end.cottle
      pop: true
    - include: expressions

  keywords:
    - match: \b(if|elif|else)\b
      scope: keyword.control.conditional.cottle
    - match: \b(for|while)\b
      scope: keyword.control.loop.cottle
    - match: \b(declare|as)\b
      scope: keyword.declaration.cottle
    - match: \b(dump|echo|empty|set|to)\b
      scope: keyword.other.cottle
    - match: \b(return)\b
      scope: keyword.control.flow.cottle

  literals:
    - match: \btrue\b
      scope: constant.language.boolean.true.cottle
    - match: \bfalse\b
      scope: constant.language.boolean.false.cottle
    - match: \bvoid\b
      scope: keyword.declaration.function.cottle

  numbers:
    - match: \d*(\.)\d+
      scope: meta.number.float.decimal.cottle constant.numeric.value.cottle
    - match: \d+
      scope: meta.number.integer.decimal.cottle constant.numeric.value.cottle

  operators:
    - match: <=|>=|!=|<|=|>
      scope: keyword.operator.comparison.cottle
    - match: \&\&|\|\||!
      scope: keyword.operator.logical.cottle
    - match: \b(and|or|not|is|in)\b
      scope: keyword.operator.logical.cottle
    - match: '[-+*/%]'
      scope: keyword.operator.arithmetic.cottle
    - match: ','
      scope: punctuation.separator.sequence.cottle

  properties:
    - match: (\.)\s*(\w+)\b
      captures:
        1: punctuation.accessor.property.cottle
        2: variable.language.property.cottle

  variables:
    - match: \b\w+\b
      scope: variable.other.cottle

  quoted-strings:
    - include: double-quoted-strings
    - include: single-quoted-strings

  double-quoted-strings:
    - match: \"
      scope: punctuation.definition.string.begin.cottle
      push: double-quoted-string-body

  double-quoted-string-body:
    - meta_scope: meta.string.cottle string.quoted.double.cottle
    - match: \"|$
      scope: punctuation.definition.string.end.cottle
      pop: true
    - include: string-escapes
    - include: string-interpolations

  single-quoted-strings:
    - match: \'
      scope: punctuation.definition.string.begin.cottle
      push: single-quoted-string-body

  single-quoted-string-body:
    - meta_scope: meta.string.cottle string.quoted.single.cottle
    - match: \'|$
      scope: punctuation.definition.string.end.cottle
      pop: true
    - include: string-escapes
    - include: string-interpolations

  unquoted-strings:
    - match: (?=\S)
      push: unquoted-string-body

  unquoted-string-body:
    - meta_scope: meta.string.cottle string.unquoted.cottle
    # unquoted strings are terminated by `eol`, comments or statement delimiters
    - match: $|(?=\{\s*_|[:|}])
      pop: true
    - include: string-escapes
    - include: string-interpolations

  string-escapes:
    - match: \\.
      scope: constant.character.escape

  string-interpolations:
    - match: \{
      scope: punctuation.section.interpolation.begin.cottle
      push: string-interpolation-body

  string-interpolation-body:
    - clear_scopes: 1
    - meta_scope: meta.interpolation.cottle
    - match: \}
      scope: punctuation.section.interpolation.end.cottle
      pop: true
    - include: code
0 Likes

#4

Not too slow, it’s always nice to have an alternative. This should definitely be helpful for the thread author now. And your implementations seems to be much more complete, mine was just hacked together based on the small example posted above. I already found a bug in my suggestion, which erroneously applies the scope for quoted strings also to preceding whitespace (could probably be fixed by using meta_content_scope). And a little tweak for deathaxe’s implementation that I see is one use of pop: 1 that you’d want to replace by pop: true for ST3.

0 Likes

#5

Thanks to both of you!
I’m actually learning a lot from both your syntax files.

I don’t want to sound … lazy, but i’d have a couple of last question; i tried to solve them but my regex-fu is still not good enough, and since i need to catch all the strings in a 250-ish scripts file to translate them all, these questions are somewhat important (i mean, i’m not just pedantic :grimacing: ).

  1. @jwortmann syntax has a little bug, which is the last dot at the end of line 24 not being catched as an unquoted string
  2. how to exclude empty lines from being marked as unquoted strings?

@deathaxe syntax is very complete, and i tried it also (thanks for making it so complete). Problem is
that all the keywords for statements, loops, functions, etc are catched even in the unquoted strings, messing the highlighting.
I’ll use his code anyway, i’ll copy it in @jwortmann syntax, as it seems to not have this problem.

Meanwhile, i’m also trying to understand HOW it works, i’m sure i’ll get the logic while messing with it.

Thanks

0 Likes

#6

@jwortmann syntax has a little bug, which is the last dot at the end of line 24 not being catched as an unquoted string

My implementation regarding unquoted strings was just based on speculation, so I thought it was intended to scope characters within a {} block only as unquoted string if there is nothing other than whitespace preceding on the line. That’s why the dot after the quoted string isn’t detected as unquoted string. This way I could reuse the same context “code-block” for the {if block, as well as the { ... } blocks within strings, without having the {variable} accidentally scoped as a string too. Otherwise you probably need two separate contexts for this, one for top level code blocks, and another one for blocks within strings.

how to exclude empty lines from being marked as unquoted strings?

I haven’t tried, but I think it should work if you add \n to the disallowed starting characters into the regex for unquoted strings, i.e. replace ^\s*(?=[^|\{]) by ^\s*(?=[^|\{\n])

I would probably use deathaxe’s syntax as a starting point and try to adjust the things that you intend to work differently. I used for example a lazy workaround in my implementation to match the whitespace in front of unquoted strings (because you can’t use a lookbehind with unknown number of spaces), and this requires to also add that (?:^\s*)? pattern to quoted strings and other tokens, which is a bit ugly and error prone.

0 Likes

#7

Can you provide another script with examples? I don’t see that happening here.

grafik

0 Likes

#8

mh … i’m away now, i’ll check as soon as i return at home.
Thank :slight_smile:

0 Likes