Sublime Forum

Python Variation in ST3 -- collections unsupported?

#1

Well, I think I just ran into an actual language difference.

Does the Python version in ST3 implement the collections package? I did a rewrite of my language scanning routine for indentation and I needed a nice way to create an ordered dictionary. After some research, the collections.OrderedDict was located and studied, and I got this to work using a commandline variation on executing the method. I also ended up using collections.deque.

So the commandline version seems to work just fine. However when I move into Sublime Text 3, the priority seems to be all screwed up.

So, point of example, I have the following code (somewhat abbreviated, this got to be pretty long). Note the order of the elements in the OrderedDict.

open_rules = collections.OrderedDict(
    entity={
        'pattern': r'^entity\b',
        'indent_rule': (0, 1),
        'close_rule': (
            ('endclause', None),
        ),
        'solo_flag': False,
        'close_offset': 0
    },

    component={
        'pattern': r'(?<!end )\bcomponent\b',
        'indent_rule': (0, 1),
        'close_rule': (
            ('endclause', None),
        ),
        'solo_flag': False,
        'close_offset': 0
    },

    package={
        'pattern': r'(?<!end )\bpackage\b',
        'indent_rule': (0, 1),
        'close_rule': (
            ('endclause', None),
        ),
        'solo_flag': False,
        'close_offset': 0
    },
...
)

Lower down in the code:

    ############################################################
    # Modification Rules
    # Priority 1: Keywords
    for key, rule in open_rules.items():
        key_search = re.search(rule['pattern'], line, re.IGNORECASE)
        debug('{}: Type: {}'.format(i, key))
        if key_search:

Hopefully this is enough to get the flavor. When I execute from the command line, I get the following debug output. It correctly scanned ‘entity’ first, matched and carried on.

22: ci=0 ni=0 : entity my_entity is -- comments everywhere
22: Type: entity
22: Evaluation line: entity my_entity is 
22: Evaluation pattern: ^entity\b
22: Type: entity
22: depth=1 top=['entity', 0]
22: Evaluation line: entity my_entity is 
22: Evaluation pattern: \bend\s?(\w+)?\s?(\w+)?\s?(\w+)?\s?;
22: ci=0 ni=1 : entity my_entity is -- comments everywhere

And then the same line in Sublime Text 3 (the line number is a little different because to get debug output in ST3, I needed a shorter example than my full test file.). It doesn’t seem to start at the top of the dictionary.

1: ci=0 ni=0 : entity my_entity is -- comments everywhere
1: Type: constant
1: Type: portmap
1: Type: config
1: Type: assertion
1: Type: function
1: Type: port
1: Type: case
1: Type: casewhen
1: Type: default
1: Evaluation line: entity my_entity is 
1: Evaluation pattern: .*
1: Type: default
1: ci=0 ni=0 : entity my_entity is -- comments everywhere

One thing I ran into when switching to a dictionary style implementation of my language stuff was that dictionaries are not guaranteed to be delivered in the order you create them when iterating. Thus I needed that ordered dictionary. However I THINK ST3 either doesn’t support collections.OrderedDict or maybe there’s a bug in it.

Am I holding the wrong end of the stick here?

I could probably create a workaround for this by creating a list that has my keywords in the priority I want them to be analyzed, and use that to key into the dictionary, but I’ve already got what feels like a lot of indirection already. I’d rather not though if I don’t have to.

(Hopefully this thing formatted correctly. Also if anyone wants to see the whole shebang, that’s perfectly alright too. I tried to just present the relevant bits as much as possible.)

Also this makes me worry about the collections.deque element, because I use that for fast stack operations.

0 Likes
#2

Something that may be biting you is that the version of python built into Sublime is 3.3.6, as visible by running the following from the console:

>>> import sys
>>> print(sys.version)
3.3.6 (default, Feb  1 2017, 17:07:05) 
[GCC 4.6.3]

If you’re running your test using python 3.6 or later, you may be running into the following, as mentioned in the python 3.6 docs:

Changed in version 3.6: With the acceptance of PEP 468, order is retained for keyword arguments passed to the OrderedDict constructor and its update() method.

3 Likes
#3

I suspect you are correct. If I revert the documentation (using the drop down) to 3.3.6 it specifically says:

The OrderedDict constructor and update() method both accept keyword arguments, but their order is lost because Python’s function call semantics pass-in keyword arguments using a regular unordered dictionary.

Alas. It worked really well. Though since I didn’t find any similar notes for deque, that portion of collections should be safe.

Anyhow, off to implement a workaround. Making a list of key names should preserve order and I can iterate over that, then reference into an ordinary dictionary and maintain the bulk of the benefits. Thank you. I did not know that about the Python version embedded and that’s a pretty subtle nuance (especially since I’m relatively new, though coming up to speed I think pretty quickly.)

It also occurs to me that I could be potentially reinventing the wheel and some of what I’m doing might be able to be handled using text scope so I might take some time and look deeper into syntax definitions. I have done a brief survey and determined that the water is pretty deep there already though.

1 Like