Jan 05, 2020

Python: Convert async_generator to generator or call it from Sync function

If you are using asyncio for asynchronous programming in Python and returning a generator for memory efficient coding from async function then the return type will be async_generator. This post will explain how to call async_generator from sync function and convert it into sync generator.

Note: This post uses Python 3.7 version.

Consider the following function:


import asyncio

async def ticker():
    for i in range(10):
        yield i
        await asyncio.sleep(2)

You can't call async_generator via asyncio.run


z = asyncio.run(ticker())

You will get error:

ValueError: a coroutine was expected, got <async_generator object ticker>

The standard way to consume async_generator is to call it in async function:


async def consume_ticker():
    data = [item async for item in ticker()]
    print(data)

asyncio.run(consume_ticker())

You will get the following output after some delay:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

We can use yield from asyncio result to convert back in generator in sync function:


async def consume_ticker():
    data = [item async for item in ticker()]
    return data

def sync_consume():
    yield from asyncio.run(consume_ticker())

print(list(sync_consume()))

It means first it will execute all async and then generate generators. The purpose of async generator is defeated.

Our goal is to consume it in sync method keeping generator behaviour for memory efficiency.

In Stackoverflow thread, there is custom function to iterate async generator:


def iter_over_async(ait, loop):
    ait = ait.__aiter__()
    async def get_next():
        try:
            obj = await ait.__anext__()
            return False, obj
        except StopAsyncIteration:
            return True, None
    while True:
        done, obj = loop.run_until_complete(get_next())
        if done:
            break
        yield obj

You can use it to call async_generator:


def sync_ticker_generator():
    loop = asyncio.get_event_loop()
    async_gen = ticker()
    sync_gen = iter_over_async(async_gen, loop)
    return sync_gen

print(list(sync_ticker_generator()))

You can see how async generator is converted into sync generator to keep generator magic. It is useful if any part of your code is asynchronous and you have to integrate it in sync method.

Hope, it helps.

Enjoy Python!!

Python