Support for twisted in autoawait magic

[jfhbrook]

Hello!

I've been experimenting with using twisted in an ipython notebook recently, and I noticed that ipython has support for implicitly awaiting asyncio coroutines but that twisted support wasn't included.

I did a bit of research into what it might take to do it, which you can see in this gist here:

https://gist.github.com/jfhbrook/29d4076bcf43ce64a462ea7911b59393

So it seems like I have an implementation here that almost works. I think the issue I'm having is that the async runner attempts to create a new event loop, even though it's already running. I'm not 100% sure that this is the right approach, but it does seem like these magics are intended to run "synchronously" - perhaps support for awaitable methods on magics is what's needed here?

Twisted also has an added challenge in that it needs to explicitly be integrated into the event loop, which I don't think any of the other async runners have to solve for. I guess I'd be looking for advice here in terms of whether to require the user to add that integration manually or whether we would want to add some kind of optional attempted install.

Beyond these questions I think I see the general direction this would have to go, and I'd be interested in putting this together in a more official way if there's interest (and if we can figure out how to solve the problems I ran into).

Update:

I was able to solve these problems in the linked notebook by not integrating the twisted reactor and using a trick similar to twisted.internet.task#react. Given the way autoawait is currently structured I think this is the right approach, minus edge cases that react might be handling that I don't see, and minus, uh, the fact that twisted doesn't actually let you restart the reactor. 🙃

Update 2:

Using a similar strategy but with a library suggested by #twisted on irc called crochet seems to solve a lot of these problems! But I'm not sure how people would feel about an extra library.

Thanks!

[jfhbrook]

Note to self, for a thing to try later: #twisted on irc suggested using crochet: https://github.com/itamarst/crochet

[Carreau]

That is interesting;

Note that there are 2 use case: Console IPython where no event loop is running; and when using a notebook ; when tornado already have a running eventloop.

Note in the internal section of the doc; it shows you that autowait can take any valid object in the current namespace; so you don't need the "get_ipython" trick IIUC.

If you need any internal change of API I'm happy to accomodate; i've done the best I could with my time and knowledge of the ecosystem.

I would also suggest to look at #11347, #11713 (and #11726 ?) as potential future directions.

Note that we don't require trio or curio, so we'll likely not require crochet for twisted integration.

The route I would go for now is the following:

• Once your happy with your integration/prototype; create a "ipython_twisted_whatever" package, with a give function you want us to call when doing "autoawait twisted", and I'll be happy to add a "try: import... ; call your function" in IPython.

That way you can refine your experiment w/o being blocked on me/us; and once stabilized; we can even merged it into IPython.

[jfhbrook]

Note that there are 2 use case: Console IPython where no event loop is running; and when using a notebook ; when tornado already have a running eventloop.

Agreed fully. What I'd love to happen is some combination of this behavior and importing/running twisted.internet.asyncioreactor on init and auto-awaiting using a non-blocking mechanism. In fact, I've gotten quite a bit of mileage out of installing the asyncio reactor and running non-awaited code. On the face of it, it seems this would happen most easily with refactors like the issues you linked.

Note that we don't require trio or curio, so we'll likely not require crochet for twisted integration.

I feel you. I would however point out that if you're using the blocking event loop calls paradigm as seen in the autoawait magics that you'll need to pull tricks like what's in crochet, since twisted doesn't make it easy to stop and restart the reactor. If you go the full async route, maybe like in #11347, that will solve this issue, but while blocking loop execution is a concern you'd at a minimum end up writing something spiritually identical to crochet in order to get off the ground.

Once your happy with your integration/prototype; create a "ipython_twisted_whatever" package, with a give function you want us to call when doing "autoawait twisted", and I'll be happy to add a "try: import... ; call your function" in IPython.

I think we're going in the same direction here: Instead of trying to get this merged immediately, I'm likely going to build out a prototype for my own crochet-based magics in a separate repo, partially so I'm not blocked but also to serve as a proving ground. I think this will be useful because I've already discovered a number of weird edge cases, like crochet not having a good way to stop the event loop on kernel shutdown, or non-async code using the reactor still needing to be wrapped.

I'll leave this issue open because I think long-term twisted support is a reasonable ask, but in the meantime I'll try to develop this more thoroughly on my own and circle back if/when I have anything to show.

Thanks!