Ctypes from sublime text 3 python broken on windows

[niosus]

I am not 100% sure about this, but the issue that I have hints in this direction.
I am developing a plugin for c++ code completion using libclang and it works as expected on Linux and OSX. On Windows I get a very strange segfault. This segfault only happens when the code is run by sublime text 3, not when I run the same script from Power Shell.

More details can be found in a dedicated StackOverflow question.

A small quick verifiable example is in my github. Don’t forget to check out the issues of this small test project, there is a discussion there already.

Please ask if you need more details.

[wbond]

As a further point of debugging, can you see if you have the same error using beta build 3103? You can change the version number in the download URL provided at https://www.sublimetext.com/3.

Can you provide a set of steps to reproduce the error for someone unfamiliar with your project? I took a brief look, but the exception was inside of a 3k line Python file, so it seems like there is a lot of area for potential issues.

There are some differences between Python inside of Sublime Text and standalone on the command line. One of the more notable is that within Sublime Text there is not stdin file handle. I don’t know if this would affect your interaction at all.

[niosus]

Thanks for a quick answer @wbond!

The error is the same for the build 3103.

Can you elaborate more on what else can I provide? The huge file is cindex38.py and is a one-to-one copy of the original cindex.py which holds python bindings for libclang. The only difference is that it is adapted to run under Python 3 (the original bindings are python 2 only). It is hard to disassemble it as the functions in use indeed use quite a few things from it and for now it crashes only after the actions in the script.py.

The error there is that the function called to libclang.dll - clang_parseTranslationUnit returns None.
I have stripped down the file in which the error occurs to the bare minimum to still reproduce the error. It is now a file of 179 lines that binds to libclang.dll with just 2 functions. Please see updated code in the repository.

Also, I’d like to point out that this occurs on Windows only. The same code works exactly as expected both on Linux and OSX.

[wbond]

Can you elaborate more on what else can I provide?

Imagine I’m a user who has Sublime Text 3 installed on Windows 7 and nothing else. What do I need to do to reproduce this error? What software do I need to install? Is there any configuration that needs to be done?

The error is the same for the build 3103.

Sublime Text 3 version 3103 used Python 3.3.3 from official sources with a dynamic _ctypes.pyd. Newer builds use our own compile of 3.3.6 (there are only source releases for 3.3.6), which statically links ctypes. I wanted to see if that caused the issue, but apparently not.

Also, I’d like to point out that this occurs on Windows only. The same code works exactly as expected both on Linux and OSX.

Windows has a completely different compiler, with different calling conventions, and more. It doesn’t completely surprise me there could be a difference.

However, you did indicate that the code seems to work when run via standalone Python launched via Power Shell. Without know exactly what the code is doing, I don’t know if the fact that there isn’t a “console” or stdin could make a different to the library you are interacting with.

[niosus]

Imagine I’m a user who has Sublime Text 3 installed on Windows 7 and nothing else. What do I need to do to reproduce this error? What software do I need to install? Is there any configuration that needs to be done?

You need the latest clang, currently 3.8. You can get it from official website. That’s it. No more configuration needed. I tried to cover it all in the readme in the test project that I linked before.

Once you have clang installed you should be able to run the script.py in the repository from Power Shell and it should work.

It should also crash if you open sublime text, while having the test project copied to Packages directory.

I know you’re probably quite busy and it’s a lot to ask, but please refer to the readme in the test project that I have referenced before. It should tell exactly what is working and what not and what output you should expect. I tried to make it as simple as possible, having only a single script of a few lines to run.

Sublime Text 3 version 3103 used Python 3.3.3 from official sources with a dynamic _ctypes.pyd.

Yep, the version you suggested gives me version of python Python version: 3.3.3, but the same error.

And thanks a lot for your time!

[wbond]

Yep, the version you suggested gives me version of python Python version: 3.3.3, but the same error.

This makes me think the issue is something related to Python 3.3 being embedded inside of a GUI application, or some implementation detail of that embedding.

[niosus]

That’s a new idea, thanks. Do you know any way of testing it? I am willing to find out the cause, but Python is not my main language just as Windows is not the system of choice. Is there some resource you know of that I could gather more info about it from?

[facelessuser]

This makes me think the issue is something related to Python 3.3 being embedded inside of a GUI application, or some implementation detail of that embedding.

This might be the case. I know I tried to tap into some OSX libs before via ctypes. While the code worked in a terminal, inside Sublime it was broken, no error, just didn’t work. I just figured what you said was the case and moved on. It would be nice though if issues like this could be fixed, but I don’t know all the details and limitations.

[wbond]

It would be nice though if issues like this could be fixed, but I don’t know all the details and limitations.

I’ve run into something before that was a bug in Python 3.3 that related to embedding it inside of a GUI app on Windows, when compiled with a specific compiler. Unfortunately these are so far down the stack that it isn’t really something we can feasibly fix, and aren’t really an issue with Sublime, but rather Python 3 itself.

Upgrading to a newer version of Python, which make fix some very edge-case things, would break so many packages that it isn’t currently tenable.

[niosus]

Can we somehow verify that it really is this issue? How hard is it to replace python in sublime text locally?

[wbond]

Can we somehow verify that it really is this issue?

Possibly by trying to run the code launched via python.exe that has been run via subprocess (in Power Shell) with the appropriate flags to create a new process group and to ensure that no stdin or console is attached to the new process group. From looking at that docs page, using a different stdin handle should be possible, and creating a new process group is possible, but I don’t see the option to disconnect from a console. You may have to use ctypes to call FreeConsole.

How hard is it to replace python in sublime text locally?

It is not possible unless you are myself, or Jon. And if you are talking about replacing it with a different version of Python (e.g. 3.5 vs 3.3) it would possibly require a bunch of work. I don’t know if there have been any CPython API changes between the versions, or other changes that affect how plugin_host interacts with python33.dll.

[niosus]

Thanks for the answer. Doesn’t sound too good though.

Let’s see if I can make it fail from the Power Shell to verify that this is happening.

[Psyduck]

I’ve run into something before that was a bug in Python 3.3 that related to embedding it inside of a GUI app on Windows, when compiled with a specific compiler. Unfortunately these are so far down the stack that it isn’t really something we can feasibly fix, and aren’t really an issue with Sublime, but rather Python 3 itself.

This sounds suspiciously like an issue that occured with Blender two years ago. Here’s the link to the bug tracker entry:

https://developer.blender.org/T40907

The workaround involved using a modified python34.dll as a drop-in replacement. Supposedly, the issue was fixed in Python 3.4.1.

[rchl]

It would be amazing if Jon or wbond could try to reproduce with this patch:
http://bugs.python.org/file37907/python-3.3.5-fdvalidation.patch
(taken from http://bugs.python.org/issue17797 )

[wbond]

The patch is not applicable since the standard build, and for compatibility of compiled modules, ST3’s version of 3.3 are compiled with VS2010.

I have reproduced the bug and applied the patch in the past when compiling with VS2012, but that would make us incompatible with pre-compiled modules on Windows.

[rchl]

So you are saying that bug is irrelevant to this problem since you compile with VS2010?

[wbond]

So you are saying that bug is irrelevant to this problem since you compile with VS2010?

Precisely.

[niosus]

So does this mean that we can close the discussion with the result that it won’t be fixed until whenever there is the next Sublime Text coming?

I understand your motivation about breaking modules that are now working, but this issue is crucial for example to make Sublime Text the ultimate C++ editor. This is the only issue that stands between us and quick and robust code analysis using libclang working out of the box in Sublime Text.

Anyway. Thank you all guys for your time, efforts and clarifications. Hope this will eventually get sorted out some day.

[wbond]

I understand your motivation about breaking modules that are now working, but this issue is crucial for example to make Sublime Text the ultimate C++ editor.

We don’t know if a newer version of Python would fix the issue. As you said, Python 3.3 works fine from Power Shell, and so does 3.5. The bug that exists for 3.3 embedded in Sublime Text may also exist with 3.5. It may not be a bug with Python at all, but rather with how clang is doing things.

Just to be clear, upgrading the version of Python bundled with Sublime Text would likely result in hundreds of thousands to millions of developers all of a sudden having a handful of their packages broken. There would probably be hundreds of package developers fielding bug reports and trying to fix the issues. At the current point, this is not a tenable option. At some point we will likely have to update the version of Python bundled due to security support ending for 3.3. I don’t think that time is right now.

Anyway. Thank you all guys for your time, efforts and clarifications. Hope this will eventually get sorted out some day.

Did you try running your code using the environment I described? Considering the officially released version of Python 3.3 that is bundled with build 3103 had the same issue, but does not have the issue via PowerShell makes me believe there is a subtle bug that has to do with embedding. It may be that clang looks for a pty and works differently in that situation, Windows being a second-class citizen and all. There was a bug related to this issue with the silver searcher, where it would look for a pty, and not output anything to the screen if it wasn’t there. This made the user believe the exec build system in Sublime Text was broken.

I may have mentioned this before, but by default there is no “console” that a subprocess is attached to when Python is run embedded in a GUI app, because consoles are part of a command line interface. Such consoles are used for directly reading keystrokes from the user for secure password entry and other stuff. It is, however, possible to create a console and attach it to a subprocess. I have done this before and used it successfully. This is one of the few differences I know of between embedding Python and running it standalone on the command line.

[wbond]

Here is the output of script.py using our custom build of Python 3.3.6 via PowerShell:

Clang directory =  C:\Program Files\LLVM\bin
Platform:  ('64bit', 'WindowsPE')
Python version:  3.3.6
filename =  C:\Program Files\LLVM\bin\libclang.dll
YAY! Parsed TranslationUnit