Sublime Forum

**niosus** · August 3, 2016, 7:44am

uh, embarrassing as it is, you are totally right. It works for me too. A reminder to have no relative paths. Thanks a lot!

**niosus** · August 3, 2016, 7:45am

This is because there is a subprocess spawned and on windows it flashes a cmd window.

**rchl** · August 3, 2016, 6:22pm

So what does that prove if test works? Is it too minimalistic to expose the bug? I assume main plugin still doesn’t work?

There is a way to suppress that but I guess that is least of a problem right now.

**wbond** · August 3, 2016, 7:36pm

I believe it shows that there isn’t a problem with ctypes within Sublime Text, in the general sense.

A further data point is that Package Control has used ctypes on Windows for quite a while to integrate with WinINet and the system trust store. I imagine there are various bugs with Python 3.3 on Windows, but in the general sense ctypes seems to work fine from within Sublime Text.

If anyone has examples of ctypes working outside of Sublime Text, but not within, I can certainly try to look at them.

**niosus** · August 8, 2016, 9:37am

Could you please try the updated script from this repo?
There still seems to be a problem that only occurs from Sublime Text.

When I run the script from Power Shell (also from some other folder apart from Packages folder) it works as expected. It parses some c++ code, reparses it, completes it couple of times and then prints some error statistics, which are in this configuration plenty.

When you run it as a plugin of sublime text it crashes the plugin_host. @wbond could you try to run it with your custom version of python to narrow down the reason for this behavior? Thanks!

Yep, this is definitely not the biggest problem for now. I am sure that there is a way, but first it has to work

**niosus** · August 30, 2016, 3:00pm

Hello @wbond. Have you had the time to run the script in the repo? It is really interesting if it crashes also if you run it with the custom python from the powershell.

Hope you have time for this. Thanks anyway!

**rchl** · August 31, 2016, 10:44am

I’ve been analyzing this a bit with custom debug build of clang and while I don’t have any conclusive results, I see one problem at least, I think.

cindex38.py has a class like that:

class File(ClangObject):
    """
    The File class represents a particular source file that is part of a
    translation unit.
    """

    @property
    def name(self):
        """Return the complete file and path name of the file."""
        return conf.lib.clang_getCString(conf.lib.clang_getFileName(self)).decode('utf-8')

    def __str__(self):
        return self.name

    def __repr__(self):
        return "<File: %s>" % (self.name)

Now, getting the name property of the File instance, passes self to clang API. It seems like in Sublime’s Python this triggers __repr__ on the instance, which accesses self.name again, causing infinite loop and stack overflow.
That must happen inside Ctypes as I can’t see that loop happening when adding print’s in the __repr__. But it works if I don’t use self.name in __repr__.

conf.lib.clang_getFileName() implementation in libclang.dll is never called in this case so the infinite loop must trigger before that libclang API is invoked.

I could probably tell much more if I would have sublime’s python3.dll debug build with symbols, but it’s probably not an option. I also have custom standalone build of python3 so I could compare code paths easily then.

**rchl** · August 31, 2016, 10:57am

Actually these are not __str__ or __repr__ that cause this. I got confused with my added debug code. From what I said above, the only true part is that stack overflow is triggered on calling conf.lib.clang_getFileName(self)

**rchl** · August 31, 2016, 3:13pm

The problem is due to something going wrong when passing pointers to the clang API.

Things go wrong when calling clang’s clang_getInstantiationLocation API which has this declaration in python bindings:

("clang_getInstantiationLocation",
   [SourceLocation, POINTER(c_object_p), POINTER(c_uint), POINTER(c_uint),
    POINTER(c_uint)]),

In the python code those arguments have these memory locations:

SourceLocation: <cparam 'P' (00000000043CDFD0)>
File: <cparam 'P' (0000000004513990)>
Line : <cparam 'P' (0000000004513790)>
Column: <cparam 'P' (0000000004513690)>
Offset: <cparam 'P' (0000000004513A10)>

While clang code receives those locations:

location = {ptr_data=0x00000000043cdfd0 {0x0000000005ada3c0, 0x000000000037b890} int_data=62 }
file = python33.dll!0x000000001e58afa8 (load symbols for additional information) {0x0000000000000008}
line = 0x00000000044a0ae8 {2}
column = 0x0000000004513990 {0}
offset = 0x0000000004513790 {0}

As can be seen here, only the first and last two arguments point to the correct memory location. Rest are just random memory addresses, one even pointing to the dll code itself so it will corrupt something when clang writes into it.
Worth pointing out that memory locations for fourth and fifth arguments are actually for the second and third argument, so they got shifted right.

I think there is probably a bug in ctypes used by Sublime’s python.

**rchl** · August 31, 2016, 4:36pm

This is most likely the fix for this issue: https://hg.python.org/cpython/rev/f75b0470168b?revcount=120

**rchl** · September 2, 2016, 10:08am

@wbond, @jps: Will you consider taking that patch into sublime’s python build?

**wbond** · September 7, 2016, 2:25pm

Recently we’ve been using 3.3.6 with some changes to get a static lib on OS X and Linux, and changes to get a single .dll on Widows. We’ve also pulled in two Windows unicode patches from 3.4:

These are necessary since unicode support in 3.3 on Windows is incomplete when paths include characters not in the users single-byte locale encoding.

Moving forward, I’ll grab the latest from the 3.3 branch, which includes the patch for structs. 3.3.7 was supposed to be released back in February, and I’ve emailed the release manager, but he said it is held up by “undisclosed security vulnerabilities.” Seems like there isn’t much urgency to security issues since it has been six months now with no more updates…

**rchl** · September 14, 2016, 11:41pm

I don’t know how it’s possible but there is still exactly the same problem in 3121.

**rchl** · September 15, 2016, 11:34am

The actual fix is actually missing on 3.3 branch. For example all the parts in Modules_ctypes\libffi_msvc\ffi.c are not there.

**wbond** · September 15, 2016, 1:30pm

The patch is on the 3.3 branch. You can see it at https://hg.python.org/cpython/rev/4d33bccb59a8, which is a parent of 8e3b9bf917a7 (used in builds 3121 and 3122).

**rchl** · September 15, 2016, 1:55pm

As we’ve discussed in private, those patches are related but not the same. We are still missing the part that was relevant for this issue.

**wbond** · September 15, 2016, 3:11pm

For the record, here are two other patches for Python 3.3 ctypes on Win x64 with MSVC:

**wbond** · September 21, 2016, 10:15am

As of build 3123 we now have the patches for 64 bit Windows ctypes backported from Python 3.4.

**rchl** · September 21, 2016, 5:18pm

Confirmed fixed. Thanks.

(Whether EasyClangComplete+libclang will fully work now is a separate matter :))

**niosus** · September 25, 2016, 6:01pm

Thanks a lot @wbond
Your support in this is much appreciated!

Ctypes from sublime text 3 python broken on windows