New SFTP plugin needs testers: Update: Linux and OSX support

[CodeEffect]

[size=150]EDIT: UPDATE, Linux and OSX support added as of the 16th June [/size]

I’m afraid it’s windows only and SFTP only for now (it’s unlikely to ever support FTP). It uses psftp and plink to connect to servers and because it uses plink there are always likely to be issues with different app versions (ls, grep, etc) supporting (or not) different command line switches. I’m throwing this version out there to get a bit of exposure to these issues and hopefully make it as resillient and robust as possible with as little configuration from the end user as it can get away with.

It’s only just over a week old so should be considered VERY alpha software. Please make sure you have good backups of anything that you let it near. Please don’t use it on production anything. I am not responsible if it eats your dog and slaps your mother with a fish etc etc etc, you know the score

To make file browsing and the equivalent of CTRL+P as fast as possible RemoteEdit can index your server. It does this in quite a simple way by piping the output from a recursive ls to a file then pulling it down and parsing it. This obviously has serious security implications and should only be considered for development servers where it doesn’t matter if the contents were to be accidentaly shared or lost. This functionality is enabled by setting enable_cat and cat_path in the server setting file.

At the moment all functionality is available via the F5 keyboard shortcut.

Known bugs:

• All the todo’s in the RemoteEdit.py
• The order in the server list isn’t set correctly so clicking on server 1 may get you server 3, server 2 may get you server 1, etc etc. Only just noticed this this morning so I’ll get it fixed tonight.
• Another one found this morning: Grep “–exclude-dir” isn’t universally supported. It is on Debian 6, it isn’t on CentOS release 5.9 (Final).

If you do wish to have a play with it and help out with bug repots please could you give as much information as you have available. Ideally I would like to know the relevant lines of debug output from the console, the flavour and version of the remote OS you’re running (e.g. Debian Squeeze, Gentoo latest etc). Please also keep it to this thread rather than raising issues on github, at least for now.

If you have feature requests feel free to note these here too. They may not get a timely individual response for the moment but I will have noted them all down and will consider them when I get time.

I’m now at work so won’t be able to really respond to a lot until this evening or even possibly later in the week. If you get stuck the console logs quite a bit of useful info. I’ll try to help out where I can.

EDIT: Would help if I posted the URL LOL: github.com/CodeEffect/RemoteEdit

[robertcollier4]

Running Windows XP SP3 32-bit ST3 Build 3033. Console output:

[code]Process dead or never started
Waiting for output…
---------- OUT ----------
Linux servername 2.6.32-5-amd64 #1 SMP Mon Feb 24 12:26:11 UTC 2013 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

e]0;user@servername: ~auser@servername:~$

Sending command: ls -lap --time-style=long-iso --color=never “/home/username/”
Waiting for output…
---------- OUT ----------
ls -lap --time-style=long-iso --color=never “/home/username/”

total 60K
drwxr-xr-x 6 username username 4.0K 2013-04-19 22:38 ./
drwxr-xr-x 4 root root 4.0K 2013-04-02 23:10 …/
drwx------ 2 username username 4.0K 2013-04-03 13:22 .aptitude/
-rw------- 1 username username 8.0K 2013-05-28 09:03 .bash_history
-rw-r–r-- 1 username username 220 2013-04-02 23:10 .bash_logout
-rw-r–r-- 1 username username 3.5K 2013-04-03 13:44 .bashrc
drwxr-xr-x 2 username username 4.0K 2013-04-03 13:59 bin/
-rw------- 1 username username 827 2013-04-04 05:15 .mysql_history

e]0;username@servername: ~ausername@servername:~$

Traceback (most recent call last):
File “D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py”, line 220, in handle_server_select
self.start_server(self.items[selection - 2])
File “D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py”, line 265, in start_server
self.open_server()
File “D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py”, line 279, in open_server
s = self.list_directory(self.lastDir)
File “D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py”, line 1542, in list_directory
self.lastDir
File “D:\PortableApps\SublimeText\Data\Packages\RemoteEdit\RemoteEdit.py”, line 1925, in parse_ls
s = int(sl[4])
ValueError: invalid literal for int() with base 10: ‘4.0K’[/code]

I think the above is caused because it is defined in my user’s bash.rc (this is commonly defined as default in many ubuntu installs):

 alias ls='ls --color=auto -alh'

from man ls

-h, --human-readable with -l, print sizes in human readable format (e.g., 1K 234M 2G)

[CodeEffect]

Cheers for that, I think you’ve got it. Shouldn’t be too tricky to parse that back into bytes (heheh and then back into KB, MB, GB etc for display later ).

I’ll have a look a bit later and update here when it’s done.

EDIT: Fix to that issue (and the server order one) should be in commit: github.com/CodeEffect/RemoteEdi … bf5fd4a9cc

[CodeEffect]

All ssh / sftp communication code has now been moved into a separate module and we communicate with it via a queue.

A ton of other small updates have been done as well. It’s really beginning to come together now and is robust and reliable enough for me to use it at work.

If you’ve got ST3, a need for editing files over SFTP and you’re running windows I could really use your feedback

[robertcollier4]

Even with the new version, while it is connecting or getting files or doing anything over the network, my SublimeText often freezes and becomes unresponsive for 5-10 seconds. The network communication needs to be “non-blocking” to SublimeText.

[robertcollier4]

This might be helpful:

code.google.com/p/pysftp/

[CodeEffect]

Yep, am moving towards that. At first it was all inline (while I was trying to get things working), now parts of it are sort of event driven (the cataloging happens in the background now), once I’ve tidied things up the plan is for it to be all completely events driven. Most of the groundwork has now been done, it’s just a case of plumbing it all in. Should be fixed in the next few days.

The sftp functionality now works by putting your commands onto a queue and waiting for the results to come back. You can pass a callback with your command and effectively forget about things until the results come back. At the moment the results checking works by setting a timeout and checking for the results, if not there then set a timeout, check for results etc etc. My next task is to remove the timeout bits and have it be completely event driven.

Cheers for the link but I don’t think I can use pysftp as paramiko doesn’t support python 3 (as best I can tell) Would love to have it all python without needing to use psftp.exe but I don’t think it’s possible without writing the sftp code at the moment. The sftp part is now its own module so it should be relatively trivial to swap out when something is available. Same applies to getting it working on different platforms, should just be a case of writing the sftp module for that platform.

[CodeEffect]

Good to go I think

Most of the network calls have now been made non-blocking. My next task will be to re-factor things a bit and tidy up the code. Would be good to get some bug reports in for platforms where it’s not working as it should.

[neema]

I am running into some trouble when selecting a remote file. Console output:

[code]MAIN 10:30:59: Results found in callback handler, firing the callback
MAIN 10:30:59: Trying cat for folder “/work/me/src/tools/ax_lua/”
MAIN 10:31:02: root: /work/me/src/tools/ax_lua, selected: ax_lua.c
MAIN 10:31:02: run_remote_command called for sftp and cmd: “get /work/me/src/tools/ax_lua/ax_lua.c C:\Users\me\AppData\Local\Temp\RemoteEdit\Buzz\work\me\src\tools\ax_lua\ax_lua.c”
MAIN 10:31:02: …now on the queue…
MAIN 10:31:02: Using set_timeout to call the callback handler to check for results
SFTP[0] 10:31:02: Process not running: ‘NoneType’ object has no attribute ‘poll’
SFTP[0] 10:31:02: Waiting for output…
SFTP[0] 10:31:04: --------- stdout ---------
Remote working directory is /home/me
psftp>

SFTP[0] 10:31:04: Waiting for output…
error: Error downloading ax_lua.c[/code]

if I run the get command manually through the included psftp.exe exactly as above the file downloads immediately. Any thoughts?

*Edit: I am connecting to an AIX unix box, as I said the psftp.exe works fine and I use winscp regularly. One note about platform support, your ls command assumes gnu ls so I had to manually change the lsParams.

[CodeEffect]

Hi, thanks for the feedback.

Firstly, with regard to the bug you’re seeing I’ve a feeling it could be fixed by the updates I have waiting to commit. I’ve just got to tidy up 1 more method and it’s ready to go in. I’ll get this in as soon as I finish work (next few hours) and ping you a PM. If the updates don’t fix it though, could I ask fo a bit more console output please? It’s very, very verbose at the moment.

WRT BSD, good spot. Am pulling down an ISO for freebsd at the mo to do some testing against, it’s been too long!

I’ve been pondering over the best solution for handling different env’s and I now have a plan:
On first connect stay in SFTP mode until we can do some feature detection. Once the user has finished what they are doing then fire off some tests to see what functionality is supported by ls and grep. As soon as these are done we’ll save results back to the server config file so it doesn’t have to run them again. Perhaps these could be manual settings in the server config file but if you weren’t sure then RE will automatically fill in the blanks. As soon as blanks have been filled it can go into full SSH mode and fire off the catalogue too. This way it should just work for everyone from the start although for the first minute or two some features may not all be available.

EDIT: Found the bug you were experiencing and fixed it now

[CodeEffect]

Some basic feature detection added. Now it should play nice with UNIX ls and with [t]csh shells. Only tested the new changes against FreeBSD so reports against other OS’s are very welcome.

[t]csh, bash and sh should all work too. Again, reports against other shells are welcome.

I still need to add the check for grep < 2.5.2 which is where “–exclude=dir” was added but everything else should work as is including cataloging.

[CodeEffect]

[size=150]Linux and OSX support added![/size]

I’ve no test machine for OSX but I’m hoping that as it should be very, very similar to linux (using the command line ssh and sftp apps) most things should just work. Some stuff is going to be broken though so your feedback on any and all issues would be hugely appreciated. Get on it people

[bianjiang]

[quote=“CodeEffect”][size=150]Linux and OSX support added![/size]

I’ve no test machine for OSX but I’m hoping that as it should be very, very similar to linux (using the command line ssh and sftp apps) most things should just work. Some stuff is going to be broken though so your feedback on any and all issues would be hugely appreciated. Get on it people [/quote]

Nice work… I found two issues running this on OSX

1. In get_arithmetic(), it gives error, since self.get_settings().get("%s:shell" % self.serverName) can give None

shellSettings = self.get_settings().get("%s:shell" % self.serverName)
if shellSettings != None and “csh” in shellSettings:

2. OSX doesn’t have ssh-askpass, so it would fail… You’ll need to have this to get ssh-askpass.
   github.com/markcarver/mac-ssh-askpass
   But I think you can wrap it within your plugin, so we don’t have to install this to the /usr/libexec folder manually

Cheers,

Jiang

[CodeEffect]

Cheers, first one was reported as a bug at about the same time that you posted here I think and is now fixed.

With regard to the second, you shouldn’t need SSH Askpass, where I referenced it I was trying to disable (and failing) but it should work without it.

[phyllisstein]

I just wanted to say that I’ve been using this fairly frequently for a few days now and it’s working beautifully. Many thanks!

[CodeEffect]

No worries at all dude, glad it’s of use.

Also, thank you for taking time to write the bug report and assisting with debugging the issue + and of course the star.

If there are any others out there that get use out of it, star’s on github are always appreciated