Sublime Forum

Appending text to a buffer from a background thread

#1

So I’ve been playing around with building some tools for some work I’ve been doing, and I’m a bit stuck. Should I be handling my own threads, or should I be using set_timeout_async or?

The plugin I’m writing is conceptually simple. Essentially, I’m writing a command that accepts a regex pattern as input and then uses this to filter individual lines in a (big) prepared text file. I would like the lines that pass this filter to be appended to a file that is created when the command fires.

This is all more or less working, but I’m hanging the main thread quite a lot. I’ve tried a variety of implementations and I’m at a point where I thought I’d look for feedback.

my current solution is basically to have a subclass of threading.Thread() that inits with the specified pattern, and then appends passes to a list property. this thread gets passed to a run_loop() function, which looks like this:

[code] def run_loop(self, thread, lock):
lock.acquire()
if len(thread.results):
for line in thread.results:
self._output_view.run_command(‘print_filtered_line’, {‘line’: line})

        self.view.set_status('filtering lines', '%d of %d' % \
            (thread.good_lines, thread.seen_lines))
        thread.results = list()
    lock.release()

    if thread.is_alive():
        print('thread still alive')
        sublime.set_timeout_async(lambda: self.run_loop(thread, lock), 50)
    else:
        print('finished')

[/code]

which in turn passes each output string to this guy:

[code]last_cursor_position = 0

class PrintFilteredLineCommand(sublime_plugin.TextCommand):

def run(self, edit, line):
    global last_cursor_position
    self.view.insert(edit, last_cursor_position, line)
    last_cursor_position = self.view.full_line(last_cursor_position).end()

[/code]

I can see already a lot of ways that I might go about improving this (i.e. just keeping a string buffer instead of a list, and appending all new output at once) but this is all just sort of hunch-based, for me, and I thought I would see if anybody had any slightly more concrete suggestions as to how I might do what I’m trying to do, here.

0 Likes
#2

I encourage you not to make it too complicated. The Sublime API is thread safe, so you don’t need locking. Here’s an example that will filter lines in the background.

import sublime, sublime_plugin

class ExampleFilterCommand(sublime_plugin.WindowCommand):
	def run(self):
		self.window.show_input_panel("Enter pattern:", "", self.on_done, None, None)

	def on_done(self, text):
		source_view = self.window.active_view()
		# Not sure where you want your output to go.  You could use open_file,
		# or pick an existing view.
		new_view = self.window.new_file()
		def cb():
			self.filter(text, source_view, new_view)
		sublime.set_timeout_async(cb, 0)

	def filter(self, pattern, source_view, output_view):
		regions = source_view.find_all(pattern)
		for region in regions:
			line_region = source_view.full_line(region)
			text = source_view.substr(line_region)
			output_view.run_command('example_append', {'text': text})

class ExampleAppendCommand(sublime_plugin.TextCommand):
	def run(self, edit, text):
		self.view.insert(edit, self.view.size(), text)

You could even implement a fancy status bar progress indicator like what Package Control does.

However, Sublime is not efficient for this task. If you are on Mac or Linux, shelling out to grep will be thousands of times faster (or on Windows, use any grep alternative). If that is not an alternative, then just use normal Python code to directly manipulate the files (readlines, re.match, write matched lines to file). I don’t know the details of exactly what you are trying to do, so I don’t know if a simple grep is all you need. For really large files, this will work much better outside of Sublime’s API. If you need help doing it this way, I can write an example if you want.

1 Like
#3

Thanks for the quick reply! I’m currently using grep (egrep) for this, but since I need to manipulate the filtered lines within ST3 I wanted to look into basically keeping everything in one place. I’ve also been looking at using the glue plugin to run egrep and get my results output to a view, but then I have to wait for the grep to complete. An option, maybe, would just be to use smaller source files, of course. Anyway. In a lot of use cases the slowness of python isn’t a big problem for me, because I have more results then I can use anyway. I’ll play around with your suggestion and see if it’s more responsive.

and thanks for self.view.size(), that was killing me.

1 Like