Sublime Forum

Using Generators for Fun and Profit - Utility for developers

#1

TL;DR: I found a neat way to use Python’s generators to simplify using Sublime Text’s callback-based API. Templates and examples can be found here

Hey there! I’m the co-creator of the NimLime plugin for Sublime Text, and I’d like to show a useful technique that can be used to simplify code in Sublime Text plugins.
As you may or may not know, Python allows resumable functions (called ‘generators’) to be created when the ‘yield’ keyword is used in the body of a function definition:

def count_to_three():
    yield 1
    yield 2
    yield 3

c = count_to_three()
print(c.next())
print(c.next())
print(c.next())

I won’t go over the details of generator functions here as there are quite a few tutorials on the subject already, however there is a little-used feature of generator functions that isn’t mentioned very often - values don’t always have to come out of a generator function instance - they can also be sent into one via the send method, like so:

def echo():
    a = None
    while true:
        a = yield a
        print(a)

e = echo()
e.send(None) # We have to send 'None' when initially starting a generator function instance
e.send(1)
e.send(2)
e.send(3)

Since we can send any value into a generator function instance, what’s to prevent sending the generator function a reference of itself? Nothing!

def caller():
    this = yield
    output = yield callback_accepting_function(this.send)
    print(output)

c = caller()
c.send(None)
c.send(c)

Using this technique, I’ve created a decorator that sends a generator function instance a weak-referenced proxy of itself*****. This allows the generator function instance to send itself as a callback argument to any of Sublime Text’s API functions that take a callback, thereby allowing plugin writers to write their code in a linear fashion, instead of split-up into various separate functions (I’ve included a useful example as a second file in the gist).

Note that there are some caveats when using this technique:

  1. Be aware of memory allocation during a function’s lifetime. Any variables which allocate large amounts of memory will only be freed either when the generator function instance itself is garbage collected, or when the variable holding the allocated object is reassigned and the value’s reference count hits 0. If a long running function allocates large amounts of memory to a variable at a single point, then be sure the reassign or delete the variable when done with it!

  2. An empty yield must be put in every place where the function would normally return - even at the very end - otherwise the generator function instance will raise a StopIteration
    exception******. This is primarily due to the design and original use of generator functions in for loops, which use a StopIteration exception as a signal that the loop should stop. Usually nothing particularly nasty will happen if a ‘yield’ is forgotten - the exception will just be printed to the developer console - however it’s sloppy, and not a good idea to rely on this behavior.

  3. Be careful about which thread the generator function instance is running on. Whatever thread the generator function instance’s send() method was last called from is the thread the next code segment will run on. I generally avoid this by making all threaded functions run themselves via sublime.set_timeout . Also, a single generator function instance cannot directly call it’s own send() method, nor can two threads call a single generator function instance’s send() or next() methods at the same time - Doing so will result in an exception.

  4. Although generator objects have ‘send’ methods in both Python 2 and Python 3, the ‘next’ method in Python 2 was renamed to next() in Python 3. I’ve included a helper function in the gist to get the correct ‘next’ method from a generator instance.

Links:

If you have any improvements to the code, feel free to send pull requests! I’ve tried to make the code both flexible and lightweight, and have tested it for memory leaks (which, although rare in a GC’ed language, aren’t impossible). Also feel free to point out any mistakes in this post. I’ve done my best to check it for errors both factual and grammatical, but it’s always possible I missed a few.

[size=85]*To prevent circular references. Yes, Python has cyclic garbage collection, but it only activates after memory reaches a certain point[/size].
[size=85]**I could probably write a wrapper to prevent this, but my aim was for the code to be lightweight and fast. Having yet another layer of indirection wasn’t part of the planned design[/size]

2 Likes
Show Progress Bar in View
#2

Interesting.

I’ve known about the possiblity of sending data to generators and using that with yield as an expression but I never found a use for it. This is an interesting one, albeit I don’t exactly see the benefits compared to creating a separate thread. Probably because you aren’t sending anything to the example generator from externally since this is a rather complex concept. Do you have a better example?

From what I know, the built-in next() method will automatically use the correct .next or .__next__ method so you don’t need the wrapper. At least I’ve been using that before all the time.

I initially thought that instead of an _FlagObject instance you could just use a local variable, but the nonlocal keyword was added in Py3. However, it appears the termination condition of your while loop is wrong since it will terminate regardless of the return value of the callback in flag.flag.

0 Likes
#3

Well, for one thing, a thread is pre-emptively scheduled, while a coroutine isn’t - It’s easier to predict and prevent race conditions with coroutines. I also don’t see how one could create linear-style methods/functions using threads in the same way as generators - one would still have to create callbacks (and for plugins wishing the stay compatible with ST2, run those callbacks in different threads).

Well, you can look at the this method from the NimLime plugin. Normally, I would have to split that method up into at least 2 separate methods, and create object attributes to hold the shared data (not to mention change the plugin type - what if someone ran the command twice, and the runtime of each overlapped?), but with the generator passing method, I can have all the data in one tidy method.

This is true, however you can’t just pass ‘next’ in as a callback, as it has no reference to the target generator, and requires a target generator as the first argument. Passing the actual next or next method has the advantage that ‘self’ will already be passed implicitly. You could create a lambda which uses next(generator_instance), however this doesn’t play well with weak-referencing the generator instance.

Huh? I don’t quite follow. I never said that the callback’s return value in ‘flag.flag’ affects loop termination. It’s meant to be called after the loop terminates (mainly for aesthetic reasons - if you stop the loop, the inner animation loop still has to finish). I could edit the function to check for a termination value every frame, but that kind of granularity isn’t really needed, and it increases the performance impact of running a status message.

  • The termination condition of the ‘while’ loop (and there’s only one) is ‘while not flag.flag’, meaning that the loop will terminate when ‘flag.flag’ is a value for which ‘bool(flag.flag) == True’.

  • ‘flag.flag’ is initially set to ‘False’, meaning that it will run forever until ‘flag.flag’ is set to something for which ‘bool(flag.flag) == True’.

  • When the ‘stop_status_loop’ function is called, ‘flag.flag’ is set to ‘True’ (or whatever is passed in).

  • If ‘flag.flag’ is set to ‘True’, the the condition of the ‘while’ loop will be false, and the loop will stop.

  • If ‘flag.flag’ is set to ‘False’, then the condition of the ‘while’ will still be true, and the loop will continue.

  • If ‘flag.flag’ is set to a function, method, lambda, or generator instance (which are the most common callables) then the condition of the ‘while’ loop will be false, and the loop will stop. This is because by default, functions, methods, lambdas, and generator instances all cause the ‘bool’ builtin to return ‘True’

I forgot to mention, the idea for this was originally inspired by Twisted’s inlineCallbacks decorator. It uses a similar technique, using a generator’s methods to schedule running when a Deferred the generator is waiting on fires.

0 Likes
#4

Thanks for the other example,

path = yield window.show_input_panel(
    "File to check?", initial_text, this.send, None, None
)

shows it nicely. Using sublime.set_timeout is not the best example imo since it works just fine with time.sleep in a thread.

Regarding the loop termination, I just read it wrong.

I’ll see if I can put this to use at some point or maybe even improve upon since some APIs are really more usable when run synchronously.

Edit: You forgot from sys import versioninfo in the gist.

Also, wouldn’t it be a better idea to unify the next attribute of the generator function in the send_self decorator? Sounds like it should be possible.
Something like

if version_info[0] >= 3:
    setattr(generator, 'next', generator.__next__)

or the other way around after line 21.

Edit2: So, from what I understand get_next_method(this) is used when the callback should receive no parameter and this.send is used when it should. How about building a utility function that forwards a parameter if it exists or uses this.send(None) otherwise? I’ll experiment a bit.

0 Likes
#5

Okay, I got a problem now.

Basically, I wanted to simplify the usage of the yielded generator in the generator itself. The value of the first yield in the generator (=the generator itself) had basically the only usages to specify one if its method as parameter and the need to use get_next_method was annoying.

Since you can not set attributes of a generator I decided to send a wrapper of the generator instead. I decided to use a class because of two reasons:

  1. I could define call with the util function that automatically calls next(generator) or generator.send(value) it it was called with a paratemer.

  2. I could optionally catch StopIteration exceptions so that the wrapper doesn’t have to end with yield. I also suspected that this could cause an issue with the generator never being gc’ed, but read on.

However, now I have a problem with garbage collecting and the weak reference of the generator. My theory is that generator.send (and generator.next or next) reference the original generator so that any function that currently holds it as callback will keep a reference on it until it finishes. The other part of this theory is that for as long as the generator is running, Python keeps a reference to it internally so that it doesn’t just stop and get gc’ed while running. This needs to happen because otherwise your initial proposal would fail at the second yield, which is after _send_self terminates and the refcount in code is reduced to 0.

So, because I pass the (weak reference) to the wrapper I now disturb this system by introducing a wrapper to the send and next methods. The problem is that the generator seems to be gc’ed the moment when I pass the wrapper with the weak reference as a callback to a function and wait in a yield. That results in a ReferenceError once the function attempts to call the callback.

Now, I could pass the non-proxied generator to the wrapper but that would mean unless the generator itself terminates (or removes the reference to the wrapper) it won’t be gc’ed. That ultimately leads to memory a leak when a function terminates and does not call the callback.

Solution idea: I need to somehow keep a weak reference of the generator in the generator itself but create a strong reference when I pass one of its attributes as callback. Any ideas?

Here is my current code:

# Based on https://gist.github.com/Varriount/aba020b9d43c13d2794b

from weakref import proxy
from functools import wraps


class GeneratorWrapper():
    """TODOC
    """
    # An instance of this will be sent to the generator function.
    # "cont" holds a wrapper function that calls either next(gen) or gen.send(val).

    # Save some overhead here. Not exactly needed but you shouldn't mess with
    # instances of this class, only use or call.
    __slots__ = ('generator', 'catch_stopiteration')

    def __init__(self, generator, catch_stopiteration=True):
        self.generator = generator
        self.catch_stopiteration = catch_stopiteration

    def cont(self, value=None):
        try:
            if value is not None:
                return self.generator.send(value)
            else:
                return next(self.generator)
        except StopIteration:
            if self.catch_stopiteration:
                return None
            else:
                raise

    def throw(self, *args, **kwargs):
        if self.catch_stopiteration:
            try:
                return self.generator.throw(*args, **kwargs)
            except StopIteration:
                return None
        else:
            self.generator.throw(*args, **kwargs)

    def close(self):
        if self.catch_stopiteration:
            try:
                return self.generator.close()
            except StopIteration:
                return None
        else:
            self.generator.close()

    __call__ = cont


def send_self(use_proxy=True, catch_stopiteration=True):
    """Decorator that sends a generator a wrapper of itself.

    The returned function instantiates and sends a generator a wrapper of itself
    via the first 'yield' used. The wrapper is an instance of GeneratorWrapper.

    Useful for creating generators that can leverage callback-based functions
    in a linear style, by passing the wrapper as callback in a yield statement.

    The generator wrapper wraps a weakly referenced proxy by default. To
    override this set use_proxy to `False`.

    The wrapper catches StopIteration exceptions by default. If you wish to have
    them propagated, set catch_stopiteration to `False`.
    """
    # "use_proxy" needs to be the name of the first parameter. For clarity, we
    # mirror that to _first_param and override use_proxy later.
    _first_param = use_proxy

    # We either directly call this, or return it to be called by python's
    # decorator mechanism.
    def _send_self(func):
        @wraps(func)
        def send_self_wrapper(*args, **kwargs):
            nonlocal use_proxy, catch_stopiteration  # optional but for clarity

            generator = func(*args, **kwargs)
            next(generator)  # Start the generator.

            # Send generator wrapper to the generator.
            generator_ref = proxy(generator) if use_proxy else generator
            gen_wrapper = GeneratorWrapper(generator_ref, catch_stopiteration)
            generator.send(gen_wrapper)  # = gen_wrapper(gen_wrapper)
            # gen_wrapper(proxy(gen_wrapper) if use_proxy else gen_wrapper)

        return send_self_wrapper

    # If the argument is a callable, we've been used without being directly
    # passed an argument by the user, and thus should call _send_self directly.
    if callable(_first_param):
        # No arguments, this is the decorator.
        use_proxy = True
        return _send_self(_first_param)
    else:
        # Someone has called @send_self(...) with parameters and thus we need to
        # return _send_self to be called indirectly.
        use_proxy = _first_param
        return _send_self

Edit: Found a work around and getting close.

0 Likes
#6

[quote=“FichteFoll”]Okay, I got a problem now.

Basically, I wanted to simplify the usage of the yielded generator in the generator itself.
[/quote]

That’s reasonable. I didn’t try something like this primarily due to the complexity I thought it would introduce, but that’s just me. :wink:

I know your first assumption is correct - An object method maintains a reference to it’s object (so that the implicit first argument to ‘self’ can be sent). As for your second assumption, though I’m less certain about the details, it makes sense that python wouldn’t garbage collect a running function/iterator.

The fact you might be missing is that the expression on the right side of ‘yield’ is evaluated, then the generator is paused. When the generator is resumed, the sent value (if any) is passed to the right side of the expression. I don’t know much about sublime internals, but I’m guessing that this is the way things are evaluated (using set_timeout as an example):

Expression on right side of yield is evaluated -> set_timeout places callback into an internal queue -> set_timeout returns -> generator pauses -> Python GC collects generator -> Sublime Text's main loop runs -> internal code runs callback in queue -> Exception is raised, since the generator has been collected

Basically, what you want is for the wrapper to turn it’s weak reference into a real reference when the generator passes the wrapper to a yield expression (You stated you don’t want to have to call a function). Unfortunately, this can’t be done in any straightforward way***** - Python doesn’t provide a way to ‘hook’ the passing of an object as a parameter. You would have to use method, attribute, or some sort of explicit access to trigger the change from weak to real reference.

If you really don’t want to do that, a possible solution might be to just create a circular reference, and trigger GC collection manually in your code.
I would do this by having ‘send_self_wrapper’ create a looping function (possibly in another thread, or via set_timeout) that contains a weak reference to the wrapper/generator instance, and have the looping function run GC collection every 30 seconds until the weak reference is dead. How much of a performance penalty this would impose, I don’t know, though I suspect it would vary depending on the number and type of plugins the user is running.

[size=100]*****That said, there may be a non-straightforward way to do what you want. I have some ideas, however I need some time to test them out, so if you’re interested, check back in a couple days for another post.[/size]

0 Likes
#7

I found a rather simple work around that I’ve been using with great success so far: The wrapper is handed a weakref.ref instance of the generator and defines @properties that “generate” functions on-the-fly using functools.partial and a hard-reference of the generator, which keeps that hard reference to the generator. That way I can have wrapper functions for the generator’s actual functions. I also added a method to create a hard-(or strong, not sure what you call it actually)-referenced wrapper from the weak-referenced one, in case your other function would like to access more than just one function of the wrapper (i.e. throw instead of send).

I haven’t had much time lately and only did small changes whenever I found some spare time, so I still need to test a few cases (notably deferring generator operations to a sub-generator using yield from) and I need to decide on how I want to handle StopIteration and whether I want to at all. Your version solves it by an empty yield statement that will pause the generator and cause it to be garbage collected, while I catch the StopIteration exception in the wrapper.

0 Likes
#8

I currently have some testing code with lots of (more or less useful) debug output. Figured you might be interested in that. It still needs some fleshing and I have more plans for the wrapper, but it should give you an idea.

[code]# Based on https://gist.github.com/Varriount/aba020b9d43c13d2794b

import weakref
from functools import wraps, partial

import sys
import time
import threading

import sublime_plugin
import sublime

class GeneratorWrapper():
“”"TODOC

An instance of this will be sent to the generator function. `send`
holds a wrapper function that calls either next(gen) or gen.send(val).
"""

# Save some overhead here. Not exactly needed but you shouldn't mess with
# instances of this class, only use or call.
__slots__ = ('generator', 'generator_ref', 'catch_stopiteration')

def __init__(self, generator, generator_ref, catch_stopiteration=True):
    self.generator = generator
    self.generator_ref = generator_ref
    self.catch_stopiteration = catch_stopiteration

def __del__(self):
    print("Wrapper is being deleted", self)

def _get_generator(self):
    return self.generator or self.generator_ref()

@property
def send(self):
    return partial(self._send, self._get_generator())

# a wrapper around send with a default value
def _send(self, generator, value=None):
    print("send:", generator, value)
    if self.catch_stopiteration:
        try:
            return generator.send(value)
        except StopIteration:
            return None
    else:
        generator.send(value)

@property
def throw(self):
    return partial(self._throw, self._get_generator())

def _throw(self, generator, *args, **kwargs):
    print("throw:", generator, args, kwargs)
    if self.catch_stopiteration:
        try:
            return generator.throw(*args, **kwargs)
        except StopIteration:
            return None
    else:
        generator.throw(*args, **kwargs)

@property
def close(self):
    return self._get_generator().close

def with_strong_ref(self):
    if self.generator:
        return self
    else:
        return self.__class__(self._get_generator(), None,
                              self.catch_stopiteration)

__call__ = with_strong_ref

def monitor_refcounts(ref):
oldweak, oldstrong = 0, 0
print(“start minitoring”, ref())
while True:
time.sleep(0.05)

    obj = ref()
    if not obj:
        break
    newweak, newstrong = weakref.getweakrefcount(ref), sys.getrefcount(obj)
    del obj

    msg = ("weak refcount: %d - strong refcount: %d"
           % (newweak, newstrong))
    sublime.status_message(msg)

    if (newweak, newstrong) != (oldweak, oldstrong):
        oldweak, oldstrong = newweak, newstrong
        print(msg)

print("Object was garbage collected")
sublime.status_message("Object was garbage collected")

def send_self(use_weakref=True, catch_stopiteration=True):
“”"Decorator that sends a generator a wrapper of itself.

The returned function instantiates and sends a generator a wrapper of itself
via the first 'yield' used. The wrapper is an instance of GeneratorWrapper.

Useful for creating generators that can leverage callback-based functions in
a linear style, by passing the wrapper as callback in the first yield
statement.

The generator wrapper wraps a weak reference (weakref.ref) by default. To
override this set use_weakref to `False`, though you potentially need to
clean up the generator yourself afterwards in case it is not resumed because
it won't be garbage collected.

The wrapper catches StopIteration exceptions by default. If you wish to have
them propagated, set catch_stopiteration to `False`.
"""
# "use_weakref" needs to be the name of the first parameter. For clarity, we
# mirror that to first_param and override use_weakref later.
first_param = use_weakref
use_weakref = True

# We either directly call this, or return it to be called by Python's
# decorator mechanism.
def _send_self(func):
    @wraps(func)
    def send_self_wrapper(*args, **kwargs):
        nonlocal use_weakref, catch_stopiteration  # optional but for clarity

        # Create generator
        generator = func(*args, **kwargs)

        # "initial call to the generator" (=> this wrapper).
        # The first yielded value will be used as return value of the
        weak_generator = weakref.ref(generator, lambda r: print("finalized"))
        threading.Thread(target=monitor_refcounts,
                         args=[weak_generator]).start()

        ret_value = next(generator)  # Start the generator

        gen_wrapper = GeneratorWrapper(None, weak_generator,
                                       catch_stopiteration)
        # Send generator wrapper to the generator.
        generator.send(gen_wrapper)

        return ret_value

    return send_self_wrapper

# If the argument is a callable, we've been used without being directly
# passed an argument by the user, and thus should call _send_self directly.
if callable(first_param):
    # No arguments, this is the decorator.
    return _send_self(first_param)
else:
    # Someone has called @send_self(...) with parameters and thus we need to
    # return _send_self to be called indirectly.
    use_weakref = first_param
    return _send_self

def defer(callback, call=True):

def func():
    time.sleep(0.4)
    if call:
        callback()
    else:
        print("generator will be deleted (if weakref'd)")

threading.Thread(target=func).start()

def test_throw(gw, i):

def func():
    time.sleep(0.4)
    if i >= 3:
        ret = gw.throw(TypeError, "%d is greater than 2" % i)
        print("catched and returned:", ret)  # should be the above message
        gw.send()  # resume
    else:
        gw.send(i * 10)

threading.Thread(target=func).start()

def sub_generator(this):
print(“waiting in sub_generator”)
yield sublime.set_timeout(this.send, 200)
print(“resumed in sub_generator”)

try:
    yield test_throw(this(), 300)
except TypeError as e:
    print("We wanted to pass 300", e)
    yield "yeah, that was unreasonable"

class TestCommandCommand(sublime_plugin.WindowCommand):
@send_self
def run(self):
this = yield

    yield defer(this.send)
    print("one")
    yield sublime.set_timeout(this.send, 200)
    print("one.one")

    for i in range(5):
        try:
            ret = yield test_throw(this(), i)
        except TypeError as e:
            print("oops!", e)
            yield "sorry"  # we are resumed by the test_throw thread
            break
        else:
            print("result", i, ret)
    print("two")

    # Utilize a sub-generator and pass the wrapper as argument so that it
    # can have data sent to itself.
    yield from sub_generator(this)

    # text = yield self.window.show_input_panel("Enter stuff", '', this.send,
    #                                           None, None)
    # print(text)
    yield defer(this.send, False)
    print("this should not be printed")

[/code]

0 Likes
#9

Didn’t want to replace the previous version (because I don’t have this under any VCS currently), so instead have this current form that is nearly finalized.
I only need to remove all the debug stuff, clean up the example a bit and write extensive documentation because it’s really important to not accidentally create strong references in the generator itself.

[code]# Based on https://gist.github.com/Varriount/aba020b9d43c13d2794b

import weakref
from functools import wraps, partial

import sys
import time
import threading

import sublime_plugin
import sublime

class GeneratorWrapperBase(object):

"""TODOC
"""

def __init__(self, catch_stopiteration=True):
    print("new Wrapper created", self)
    self.catch_stopiteration = catch_stopiteration

def __del__(self):
    print("Wrapper is being deleted", self)

generator = NotImplemented

weak_generator = NotImplemented

def with_strong_ref(self):
    return NotImplemented

def with_weak_ref(self):
    return NotImplemented

@property
def send(self):
    return partial(self._send, self.generator)

# A wrapper around send with a default value
def _send(self, generator, value=None):
    print("send:", generator, value)
    if self.catch_stopiteration:
        try:
            return generator.send(value)
        except StopIteration:
            return None
    else:
        generator.send(value)

@property
def throw(self):
    return partial(self._throw, self.generator)

def _throw(self, generator, *args, **kwargs):
    print("throw:", generator, args, kwargs)
    if self.catch_stopiteration:
        try:
            return generator.throw(*args, **kwargs)
        except StopIteration:
            return None
    else:
        generator.throw(*args, **kwargs)

@property
def close(self):
    return self.generator.close

class WeakGeneratorWrapper(GeneratorWrapperBase):
def init(self, generator, catch_stopiteration):
self.weak_generator = weakref.ref(generator)
super(WeakGeneratorWrapper, self).init(catch_stopiteration)

@property
def generator(self):
    return self.weak_generator()

def with_strong_ref(self):
    return StrongGeneratorWrapper(self.generator, self.catch_stopiteration)

def with_weak_ref(self):
    return self

__call__ = with_strong_ref

class StrongGeneratorWrapper(GeneratorWrapperBase):
def init(self, generator, catch_stopiteration):
self.generator = generator
super(StrongGeneratorWrapper, self).init(catch_stopiteration)

@property
def weak_generator(self):
    return weakref.ref(self.generator)

def with_strong_ref(self):
    return self

def with_weak_ref(self):
    return WeakGeneratorWrapper(self.generator, self.catch_stopiteration)

__call__ = with_strong_ref  # Always return strong-referenced variant

def monitor_refcounts(ref):
oldweak, oldstrong = 0, 0
print(“start minitoring”, ref())
while True:
time.sleep(0.05)

    obj = ref()
    if not obj:
        break
    newweak, newstrong = weakref.getweakrefcount(ref), sys.getrefcount(obj)
    del obj

    msg = ("weak refcount: %d - strong refcount: %d"
           % (newweak, newstrong))
    sublime.status_message(msg)

    if (newweak, newstrong) != (oldweak, oldstrong):
        oldweak, oldstrong = newweak, newstrong
        print(msg)

print("Object was garbage collected")
sublime.status_message("Object was garbage collected")

def send_self(catch_stopiteration=True, finalize_callback=None):
“”"Decorator that sends a generator a wrapper of itself.

When a generator decorated by this is called, it gets sent a wrapper of
itself via the first 'yield' used. The wrapper is an instance of
WeakGeneratorWrapper.

Useful for creating generators that can leverage callback-based functions in
a linear style, by passing the wrapper as callback in the first yield
statement.

The wrapper catches StopIteration exceptions by default. If you wish to have
them propagated, set catch_stopiteration to `False`. Forwarded to the
Wrapper.
"""
# "catch_stopiteration" needs to be the name of the first parameter. For
# clarity, we mirror that to first_param and override catch_stopiteration
# later.
first_param = catch_stopiteration
catch_stopiteration = True

# We either directly call this, or return it to be called by Python's
# decorator mechanism.
def _send_self(func):
    @wraps(func)
    def send_self_wrapper(*args, **kwargs):
        # optional but for clarity
        nonlocal catch_stopiteration, finalize_callback

        # Create generator
        generator = func(*args, **kwargs)

        weak_generator = weakref.ref(generator, finalize_callback)
        threading.Thread(target=monitor_refcounts,
                         args=[weak_generator]).start()

        # The first yielded value will be used as return value of the
        # "initial call to the generator" (=> this wrapper).
        ret_value = next(generator)  # Start generator

        gen_wrapper = WeakGeneratorWrapper(generator, catch_stopiteration)
        # Send generator wrapper to the generator.
        generator.send(gen_wrapper)

        return ret_value

    return send_self_wrapper

# If the argument is a callable, we've been used without being directly
# passed an argument by the user, and thus should call _send_self directly.
if callable(first_param):
    # No arguments, this is the decorator.
    return _send_self(first_param)
else:
    # Someone has called @send_self(...) with parameters and thus we need to
    # return _send_self to be called indirectly.
    catch_stopiteration = first_param
    return _send_self

################################################################################

def defer(callback, call=True):

def func():
    time.sleep(0.4)
    if call:
        callback()
    else:
        print("generator was not re-called")

threading.Thread(target=func).start()

def test_throw(gw, i):

def func():
    time.sleep(0.4)
    if i >= 3:
        ret = gw.throw(TypeError, "%d is greater than 2" % i)
        print("catched and returned:", ret)  # should be the above message
        gw.send()  # resume
    else:
        gw.send(i * 10)

threading.Thread(target=func).start()

def sub_generator(this):
print(“waiting in sub_generator”)
yield sublime.set_timeout(this.send, 200)
print(“resumed in sub_generator”)

try:
    yield test_throw(this(), 300)
except TypeError as e:
    print("We passed 300, but", e)
    yield "yeah, that was unreasonable"

class TestCommandCommand(sublime_plugin.WindowCommand):

def wont_be_finished(self):
    this = yield

    print("wont_be_finished")
    yield defer(this.send)  # this is where the initial caller will be resumed
    print("middle~")
    yield defer(this.send, False)
    print("this should not be printed")

@send_self(finalize_callback=lambda x: print("finalized"))
def run(self):
    this = yield

    print("original weak-ref variant:", this)
    this = this()
    print("strong-ref variant:", this)
    this = this.with_weak_ref()
    print("new weak-ref variant:", this)

    yield defer(this.send)
    print("one")
    yield sublime.set_timeout(this.send, 200)
    print("one.one")

    for i in range(5):
        try:
            ret = yield test_throw(this(), i)
        except TypeError as e:
            print("oops!", e)
            yield "sorry"  # we are resumed by the test_throw thread
            break
        else:
            print("result", i, ret)
    print("two")

    # Utilize a sub-generator and pass the wrapper as argument so that it
    # can have data sent to itself (even exceptions).
    yield from sub_generator(this)

    # Different method to invoke a sub-generator (and less effective)
    wont_be_finished = send_self(finalize_callback=this.send)(self.wont_be_finished)

    print("launching weird sub-generator")
    old_obj = yield wont_be_finished()
    print("weakref of other sub-generator:", old_obj)

    # text = yield self.window.show_input_panel("Enter stuff", '', this.send,
    #                                           None, None)
    # print(text)

    # Now, make reference strong and cause cyclic reference
    # DON'T TRY THIS AT HOME! MEMORY LEAK!
    # this = this()
    # yield

[/code]

0 Likes
#10

If you want to make it really magical, you can have ‘send_self’ modify the generator frame’s ‘locals’ dictionary and automatically insert a ‘this’ variable.

0 Likes
#11

I would rather avoid this since it’s very implicit and linters will complain about it. I mean, self is also explicitly written in method signatures (for good reason imo).
Furthermore, currently the first yielded value will be used as the return value of the decorated generator function (i.e. send_self_wrapper).

Anyway, I just experimented with it a bit using inspect.getmembers (which appeared to me to be the method of choice) and the undocumented generator.gi_frame attribute that I found using dir, but I have been unsuccessful with trying to modify the locals dictionary. The value is set in the dict but it’s not visible in the generator itself.

Here is what I did in send_self_wrapper:

for k, v in inspect.getmembers(generator.gi_frame): if k == 'f_locals': v'that'] = "hi" print("adjusted", v)

There also seems to be a dedicated method for this since 3.2 called inspect.generatorlocals, which is however not available in ST2.

Anyway, I need to start studying now and will hopefully take a look at this again on friday. Documentation is pretty much in place now. I’d like to write send/next/throw methods that do not fail if the generator is running currently and will wait until it is suspended. They won’t be the default however since the default methods are “less forgiving” and thus design errors surface quicker.

And I should definitely write some tests because currently I’m only testing on-the-fly and based on console output. This should be mature enough to pass a test suite.

Code is now pushed to a github repo: github.com/FichteFoll/send-self

0 Likes
#12

Hm. I wonder if I should open my own repository with my version… (By the way, If nothing else, I would like a mention in your repository’s readme)

0 Likes
#13

Yes, I will certainly mention you somewhere there including a link to this thread (which is already there). There is also no license yet, partly because of that. However, the only remaining code that I borrowed from you is a few lines inside the send_self function, which is unlikely enough to warrant any copyright.

I haven’t been able to start writing tests so far, but I want to do this properly. It’s a very fragile feature and it needs to have tests in order to confirm correct functionality, imo.

If you want to upload your code as well I should consider finding a new name for mine (I actually considered that before already). I just couldn’t come up with a catchy but meaningful name for both the fuction/decorator and the module.

0 Likes
#14

Sorry for the extremely late reply - I do intend to upload my code, and was going to use the ‘send_self’ name. What were your ideas for an alternate name?

0 Likes
#15

None yet, I haven’t been able to return to this since my last reply actually. I want to have tests in place before I “release” this and will think of a name afterwards/meanwhile.

0 Likes
#16

It has been done, at last: pypi.python.org/pypi/resumeback

Can’t wait to use this in ST plugins.

0 Likes