Sublime Forum

Help with syntax for Logstash, has dictionary inner with two alternative formats

#1

I have most of the syntax working nicely.
But the Logstash config DSL has a quirk or three. It is similar to the Ruby language in a few ways.
example:

input {
  file {
    path => "/elastic/tmp/testing/logs/testing-*.txt"
    sincedb_path => "/dev/null"
    codec => json {
      charset => "CP1252"
    }
  }
}

filter {
  date {
    match => [ "message", "MMM dd, yyyy H:mm:ss a ZZZ"]
    target => "timestamp"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

It has 3 sections, input|filter|output, having a {} block after each.
In the input section at the root level there can be a number of “plugins” defined separated by whitespace, in the example there is only 1 called file. A plugin is an identifier followed by a {} block. The innards of this block are like a Ruby Hash innards except that whitespace is used instead of a comma to separate k => v elements where k is an identifier and v is a (constant|number|string|array|hash)

The quirk I am struggling with is the k = codec bit. It is a special case where the v is a plugin. The codec directive can appear anywhere in the list of k => v elements.
A further complication is Logstash will fail to parse the if the codec directive is used in a plugin in the filter section - so I can’t use one plugin syntax for everything.

My syntax snippet for this is:

  hash_inner_with_codec:
    - include: hash_inner
    - match: '\bcodec\b'
      scope: entity.other.attribute-name
    - match: "=>"
      scope: punctuation.separator.dictionary.key-value.logstash
    - include: plugin
    - include: hash_inner

The above kinda works but I want to improve it to allow alternation.
My question, then, is how to allow the syntax to switch between hash_inner and codec?

If I defined a codec context…

  codec:
    - match: '\bcodec\b'
      scope: entity.other.attribute-name
    - match: "=>"
      scope: punctuation.separator.dictionary.key-value.logstash
    - include: plugin

then I would want to use hash_inner or codec as in…

  hash_inner_with_codec:
    - include: hash_inner or codec

but I don’t think this is legal Sublime syntax yaml.

0 Likes
#2

I think I got it now. I experimented with some much simpler syntaxes and that helped a bunch.

0 Likes
#3

Actually, I still have a problem.

How do I code a syntax for

A {
    J
    K
    J
}
B {
    J
    J
}

where inside A’s {} section, J and K can occur in any order and in B’s section, K can not be allowed to appear?

0 Likes
#4

I’ll keep myself short, but for situations like these you want to push a different context that parses tokens differently. Each context has its own individual patterns (allowing for includes), which means you would make a context that you push after matching A that matches both K and J while you push a different context or B. Upon reaching a closing brace, both context pop themselves off the stack and you are back at your main context. The syntax docs should have an example that helps with this.

0 Likes
#5

Thanks for the feedback.

What you describe is the understanding I am reaching this morning.

As there is no cookbook type resource, its good to get confirmation that this is “correct” mechanism. Otherwise, it is easy to get the pushes, sets and includes all twisted up.

In the actual syntax, A occurs one level deeper than the example. I’m getting there though.

Thanks once again.

EDIT…

I have nailed it. The real sticking point was controlling the pop back to the previous context.
This helped in the contexts that don’t have a ending } to trigger the pop:

    - match: '($\n?)'
      pop: true

Also using meta_scope as the first line in each context that get pushed helps a ton when using <ctrl><shift><p> to view the scopes. Doing this allowed me to realise where my pops were not working.

0 Likes