Ctypes from sublime text 3 python broken on windows

[niosus]

Can we somehow verify that it really is this issue? How hard is it to replace python in sublime text locally?

[wbond]

Can we somehow verify that it really is this issue?

Possibly by trying to run the code launched via python.exe that has been run via subprocess (in Power Shell) with the appropriate flags to create a new process group and to ensure that no stdin or console is attached to the new process group. From looking at that docs page, using a different stdin handle should be possible, and creating a new process group is possible, but I don’t see the option to disconnect from a console. You may have to use ctypes to call FreeConsole.

How hard is it to replace python in sublime text locally?

It is not possible unless you are myself, or Jon. And if you are talking about replacing it with a different version of Python (e.g. 3.5 vs 3.3) it would possibly require a bunch of work. I don’t know if there have been any CPython API changes between the versions, or other changes that affect how plugin_host interacts with python33.dll.

[niosus]

Thanks for the answer. Doesn’t sound too good though.

Let’s see if I can make it fail from the Power Shell to verify that this is happening.

[Psyduck]

I’ve run into something before that was a bug in Python 3.3 that related to embedding it inside of a GUI app on Windows, when compiled with a specific compiler. Unfortunately these are so far down the stack that it isn’t really something we can feasibly fix, and aren’t really an issue with Sublime, but rather Python 3 itself.

This sounds suspiciously like an issue that occured with Blender two years ago. Here’s the link to the bug tracker entry:

https://developer.blender.org/T40907

The workaround involved using a modified python34.dll as a drop-in replacement. Supposedly, the issue was fixed in Python 3.4.1.

[rchl]

It would be amazing if Jon or wbond could try to reproduce with this patch:
http://bugs.python.org/file37907/python-3.3.5-fdvalidation.patch
(taken from http://bugs.python.org/issue17797 )

[wbond]

The patch is not applicable since the standard build, and for compatibility of compiled modules, ST3’s version of 3.3 are compiled with VS2010.

I have reproduced the bug and applied the patch in the past when compiling with VS2012, but that would make us incompatible with pre-compiled modules on Windows.

[rchl]

So you are saying that bug is irrelevant to this problem since you compile with VS2010?

[wbond]

So you are saying that bug is irrelevant to this problem since you compile with VS2010?

Precisely.

[niosus]

So does this mean that we can close the discussion with the result that it won’t be fixed until whenever there is the next Sublime Text coming?

I understand your motivation about breaking modules that are now working, but this issue is crucial for example to make Sublime Text the ultimate C++ editor. This is the only issue that stands between us and quick and robust code analysis using libclang working out of the box in Sublime Text.

Anyway. Thank you all guys for your time, efforts and clarifications. Hope this will eventually get sorted out some day.

[wbond]

I understand your motivation about breaking modules that are now working, but this issue is crucial for example to make Sublime Text the ultimate C++ editor.

We don’t know if a newer version of Python would fix the issue. As you said, Python 3.3 works fine from Power Shell, and so does 3.5. The bug that exists for 3.3 embedded in Sublime Text may also exist with 3.5. It may not be a bug with Python at all, but rather with how clang is doing things.

Just to be clear, upgrading the version of Python bundled with Sublime Text would likely result in hundreds of thousands to millions of developers all of a sudden having a handful of their packages broken. There would probably be hundreds of package developers fielding bug reports and trying to fix the issues. At the current point, this is not a tenable option. At some point we will likely have to update the version of Python bundled due to security support ending for 3.3. I don’t think that time is right now.

Anyway. Thank you all guys for your time, efforts and clarifications. Hope this will eventually get sorted out some day.

Did you try running your code using the environment I described? Considering the officially released version of Python 3.3 that is bundled with build 3103 had the same issue, but does not have the issue via PowerShell makes me believe there is a subtle bug that has to do with embedding. It may be that clang looks for a pty and works differently in that situation, Windows being a second-class citizen and all. There was a bug related to this issue with the silver searcher, where it would look for a pty, and not output anything to the screen if it wasn’t there. This made the user believe the exec build system in Sublime Text was broken.

I may have mentioned this before, but by default there is no “console” that a subprocess is attached to when Python is run embedded in a GUI app, because consoles are part of a command line interface. Such consoles are used for directly reading keystrokes from the user for secure password entry and other stuff. It is, however, possible to create a console and attach it to a subprocess. I have done this before and used it successfully. This is one of the few differences I know of between embedding Python and running it standalone on the command line.

[wbond]

Here is the output of script.py using our custom build of Python 3.3.6 via PowerShell:

Clang directory =  C:\Program Files\LLVM\bin
Platform:  ('64bit', 'WindowsPE')
Python version:  3.3.6
filename =  C:\Program Files\LLVM\bin\libclang.dll
YAY! Parsed TranslationUnit

[wbond]

Additionally, I noticed when loading it via Sublime Text that a console window flashes onto the screen briefly.

[wbond]

You just had a bug at https://github.com/niosus/test-complete-plugin/blob/master/script.py#L13, it should have been:

filename = path.abspath(path.dirname(__file__) + "/test.cpp")

This works fine for me on Windows 10 x64 with build 3120 and Clang 3.8.1.

[niosus]

uh, embarrassing as it is, you are totally right. It works for me too. A reminder to have no relative paths. Thanks a lot!

[niosus]

Additionally, I noticed when loading it via Sublime Text that a console window flashes onto the screen briefly.

This is because there is a subprocess spawned and on windows it flashes a cmd window.

[rchl]

So what does that prove if test works? Is it too minimalistic to expose the bug? I assume main plugin still doesn’t work?

This is because there is a subprocess spawned and on windows it flashes a cmd window.

There is a way to suppress that but I guess that is least of a problem right now.

[wbond]

So what does that prove if test works? Is it too minimalistic to expose the bug? I assume main plugin still doesn’t work?

I believe it shows that there isn’t a problem with ctypes within Sublime Text, in the general sense.

A further data point is that Package Control has used ctypes on Windows for quite a while to integrate with WinINet and the system trust store. I imagine there are various bugs with Python 3.3 on Windows, but in the general sense ctypes seems to work fine from within Sublime Text.

If anyone has examples of ctypes working outside of Sublime Text, but not within, I can certainly try to look at them.

[niosus]

Could you please try the updated script from this repo?
There still seems to be a problem that only occurs from Sublime Text.

When I run the script from Power Shell (also from some other folder apart from Packages folder) it works as expected. It parses some c++ code, reparses it, completes it couple of times and then prints some error statistics, which are in this configuration plenty.

When you run it as a plugin of sublime text it crashes the plugin_host. @wbond could you try to run it with your custom version of python to narrow down the reason for this behavior? Thanks!

There is a way to suppress that but I guess that is least of a problem right now.

Yep, this is definitely not the biggest problem for now. I am sure that there is a way, but first it has to work

[niosus]

Hello @wbond. Have you had the time to run the script in the repo? It is really interesting if it crashes also if you run it with the custom python from the powershell.

Hope you have time for this. Thanks anyway!

[rchl]

I’ve been analyzing this a bit with custom debug build of clang and while I don’t have any conclusive results, I see one problem at least, I think.

cindex38.py has a class like that:

class File(ClangObject):
    """
    The File class represents a particular source file that is part of a
    translation unit.
    """

    @property
    def name(self):
        """Return the complete file and path name of the file."""
        return conf.lib.clang_getCString(conf.lib.clang_getFileName(self)).decode('utf-8')

    def __str__(self):
        return self.name

    def __repr__(self):
        return "<File: %s>" % (self.name)

Now, getting the name property of the File instance, passes self to clang API. It seems like in Sublime’s Python this triggers __repr__ on the instance, which accesses self.name again, causing infinite loop and stack overflow.
That must happen inside Ctypes as I can’t see that loop happening when adding print’s in the __repr__. But it works if I don’t use self.name in __repr__.

conf.lib.clang_getFileName() implementation in libclang.dll is never called in this case so the infinite loop must trigger before that libclang API is invoked.

I could probably tell much more if I would have sublime’s python3.dll debug build with symbols, but it’s probably not an option. I also have custom standalone build of python3 so I could compare code paths easily then.