Skip to content
Search

Asyncio

lastmod 2024-07-17created 2020-06-1136 edits

Python asynchronous IO.

Categories

Exceptions

exception asyncio.CancelledError
The operation has been cancelled.
This exception can be caught to perform custom operations when asyncio Tasks are cancelled. In almost all situations the exception must be re-raised.

WARNING

Changed in version 3.8: CancelledError is now a subclass of BaseException.

Creating native coroutines

First, to use asyncio, create a native coroutine with async def as follows. In Python, coroutines created with async def are called native coroutines to distinguish them from generator-based coroutines. The async def keyword is available from Python 3.5 onward.

async def 함수이름():
    코드

Hello, World Example

import asyncio

async def hello():    # async def로 네이티브 코루틴을 만듦
    print('Hello, world!')

loop = asyncio.get_event_loop()     # 이벤트 루프를 얻음
loop.run_until_complete(hello())    # hello가 끝날 때까지 기다림
loop.close()                        # 이벤트 루프를 닫음

Running native coroutines with await

This time, here is how to run a native coroutine with await. If you put a coroutine object, future object, or task object after await as follows, it waits until that object finishes and then returns the result. True to its name, await waits until a specific object finishes.

The await keyword is available from Python 3.5 onward; in 3.4, use yield from

  • 변수 = await 코루틴객체
  • 변수 = await 퓨처객체
  • 변수 = await 태스크객체

One thing to watch out for here: await can only be used inside a native coroutine. Now let's make a native coroutine that adds two numbers, waits 1 second in the coroutine, and then returns the result.

import asyncio

async def add(a, b):
    print('add: {0} + {1}'.format(a, b))
    await asyncio.sleep(1.0)    # 1초 대기. asyncio.sleep도 네이티브 코루틴
    return a + b    # 두 수를 더한 결과 반환

async def print_add(a, b):
    result = await add(a, b)    # await로 다른 네이티브 코루틴 실행하고 반환값을 변수에 저장
    print('print_add: {0} + {1} = {2}'.format(a, b, result))

loop = asyncio.get_event_loop()             # 이벤트 루프를 얻음
loop.run_until_complete(print_add(1, 2))    # print_add가 끝날 때까지 이벤트 루프를 실행
loop.close()                                # 이벤트 루프를 닫음

Future vs Task

Future (asyncio.Future)
A class that represents work to be done in the future, used to cancel the work, check its status, and set completion and results.
Task (asyncio.Task)
A subclass of asyncio.Future that contains the functionality of asyncio.Future and the coroutine object to run.
A task is used to cancel the coroutine's execution, check its status, and set completion and results.

Synchronization Primitives

Running tasks concurrently

Use awaitable asyncio.gather(*aws, loop=None, return_exceptions=False).

import asyncio

async def factorial(name, number):
    f = 1
    for i in range(2, number + 1):
        print(f"Task {name}: Compute factorial({i})...")
        await asyncio.sleep(1)
        f *= i
    print(f"Task {name}: factorial({number}) = {f}")

async def main():
    # Schedule three calls *concurrently*:
    await asyncio.gather(
        factorial("A", 2),
        factorial("B", 3),
        factorial("C", 4),
    )

asyncio.run(main())

# Expected output:
#
#     Task A: Compute factorial(2)...
#     Task B: Compute factorial(2)...
#     Task C: Compute factorial(2)...
#     Task A: factorial(2) = 2
#     Task B: Compute factorial(3)...
#     Task C: Compute factorial(3)...
#     Task B: factorial(3) = 6
#     Task C: Compute factorial(4)...
#     Task C: factorial(4) = 24

Future

Done callbacks

Use add_done_callback(callback, *, context=None).

Adds a callback to be run when the Future is done. Parameters can be passed to the callback using functools.partial, for example:

# Call 'print("Future:", fut)' when "fut" is done.
fut.add_done_callback(
    functools.partial(print, "Future:"))

Task

Creating a task

Use asyncio.create_task(coro, *, name=None):

async def coro():
    ...

# In Python 3.7+
task = asyncio.create_task(coro())
...

# This works in all Python versions but is less readable
task = asyncio.ensure_future(coro())
...

Cancelling a task

async def cancel_me():
    print('cancel_me(): before sleep')

    try:
        # Wait for 1 hour
        await asyncio.sleep(3600)
    except asyncio.CancelledError:
        print('cancel_me(): cancel sleep')
        raise
    finally:
        print('cancel_me(): after sleep')

async def main():
    # Create a "cancel_me" Task
    task = asyncio.create_task(cancel_me())

    # Wait for 1 second
    await asyncio.sleep(1)

    task.cancel()
    try:
        await task
    except asyncio.CancelledError:
        print("main(): cancel_me is cancelled now")

asyncio.run(main())

# Expected output:
#
#     cancel_me(): before sleep
#     cancel_me(): cancel sleep
#     cancel_me(): after sleep
#     main(): cancel_me is cancelled now

async with

Now let's make a class that returns the result of an addition after 1 second.

import asyncio

class AsyncAdd:
    def __init__(self, a, b):
        self.a = a
        self.b = b

    async def __aenter__(self):
        await asyncio.sleep(1.0)
        return self.a + self.b    # __aenter__에서 값을 반환하면 as에 지정한 변수에 들어감

    async def __aexit__(self, exc_type, exc_value, traceback):
        pass

async def main():
    async with AsyncAdd(1, 2) as result:    # async with에 클래스의 인스턴스 지정
        print(result)    # 3

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

async for

import asyncio

class AsyncCounter:
    def __init__(self, stop):
        self.current = 0
        self.stop = stop

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.current < self.stop:
            await asyncio.sleep(1.0)
            r = self.current
            self.current += 1
            return r
        else:
            raise StopAsyncIteration

async def main():
    async for i in AsyncCounter(3):    # for 앞에 async를 붙임
        print(i, end=' ')

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

run_in_executor

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

if __name__ == '__main__':
    asyncio.run(main())

Returning all tasks

import asyncio
import sys

def all_tasks(loop: asyncio.AbstractEventLoop):
    if sys.version_info >= (3, 7):
        return getattr(asyncio, "all_tasks")(loop)  # nocov
    else:
        tasks = asyncio.Task.all_tasks(loop)  # type: ignore[attr-defined]
        return {t for t in list(tasks) if not t.done()}

How to give await a priority (using PriorityQueue)

There is no way to control this directly with await, so use a priority queue.

import asyncio

async def high_priority_task():
    print("High priority task started")
    await asyncio.sleep(1)
    print("High priority task finished")

async def low_priority_task():
    print("Low priority task started")
    await asyncio.sleep(2)
    print("Low priority task finished")

async def main():
    # 우선순위 큐 생성
    queue = asyncio.PriorityQueue()

    # 작업을 우선순위와 함께 큐에 추가
    await queue.put((0, high_priority_task()))
    await queue.put((1, low_priority_task()))

    # 큐에서 작업을 가져와 실행
    while not queue.empty():
        _, task = await queue.get()
        await task

asyncio.run(main())

asyncio.wait

  • asyncio.FIRST_COMPLETED - The function returns when any future finishes or is cancelled.
  • asyncio.FIRST_EXCEPTION - Returns when a future finishes by raising an exception. If no exception is raised, same as ALL_COMPLETED.
  • asyncio.ALL_COMPLETED - The function returns when all futures finish or are cancelled.

asyncio.as_completed

Returns in order of completion.

for coro in as_completed(aws):
    earliest_result = await coro
    # ...

asyncio.to_thread

Runs the function func asynchronously in a separate thread.

Take the following code:

result = requests.get('https://python.org/')

and change it as follows:

result = await asyncio.to_thread(requests.get, 'https://python.org/')

Cancelling a list of tasks from a non-async function

import asyncio
import sys

def cancel_tasks(loop: asyncio.AbstractEventLoop, *tasks) -> None:
    if not tasks:
        return

    for task in tasks:
        task.cancel()

    loop.run_until_complete(asyncio.gather(*tasks, loop=loop, return_exceptions=True))

    for task in tasks:
        if task.cancelled():
            continue
        if task.exception() is not None:
            loop.call_exception_handler(
                {
                    "message": "Unhandled exception during asyncio.run() shutdown",
                    "exception": task.exception(),
                    "task": task,
                }
            )

Subprocesses

import asyncio

async def run(cmd):
    proc = await asyncio.create_subprocess_shell(
        cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()

    print(f'[{cmd!r} exited with {proc.returncode}]')
    if stdout:
        print(f'[stdout]\n{stdout.decode()}')
    if stderr:
        print(f'[stderr]\n{stderr.decode()}')

asyncio.run(run('ls /zzz'))

Applying cmd with shlex.

app['process'] = await asyncio.create_subprocess_exec(
                           shlex.split(cmd), stdout=subprocess.PIPE)

Implementing a timeout (asyncio.wait_for)

Use asyncio.wait_for:

async def eternity():
    # Sleep for one hour
    await asyncio.sleep(3600)
    print('yay!')

async def main():
    # Wait for at most 1 second
    try:
        await asyncio.wait_for(eternity(), timeout=1.0)
    except asyncio.TimeoutError:
        print('timeout!')

asyncio.run(main())

# Expected output:
#
#     timeout!

Opening a TCP/IP socket:

fut = asyncio.open_connection( self.host, self.port, loop=self.loop )
try:
   r, w = await asyncio.wait_for(fut, timeout=self.connection_timeout)
except asyncio.TimeoutError:
   pass

Implementing a cancellable Sleep Task

# -*- coding: utf-8 -*-

from asyncio import CancelledError, Task, ensure_future
from asyncio import sleep as asyncio_sleep
from typing import Any, Dict


class Sleeper:
    def __init__(self):
        self.tasks: Dict[Any, Task] = dict()

    async def sleep(self, key: Any, seconds: float, result=None, *, loop=None) -> Any:
        coro = asyncio_sleep(seconds, result=result, loop=loop)
        task = ensure_future(coro)
        self.tasks[key] = task
        try:
            return await task
        except CancelledError:
            return result
        finally:
            del self.tasks[key]

    def cancel(self, key: Any) -> None:
        self.tasks[key].cancel()

    def cancel_all(self) -> None:
        for task in self.tasks.values():
            task.cancel()

Using asyncio in Python 3.4 and earlier

async def and await were added in Python 3.5. So they cannot be used in versions below 3.5. In Python 3.4, create a native coroutine with the @asyncio.coroutine decorator as follows.

import asyncio

@asyncio.coroutine
async def 함수이름():
    코드

In Python 3.4, use yield from instead of await.

  • 변수 = yield from 코루틴객체
  • 변수 = yield from 퓨처객체
  • 변수 = yield from 태스크객체

In Python 3.3, for asyncio, install asyncio with pip install asyncio, then use the @asyncio.coroutine decorator and yield from. However, versions below 3.3 do not support asyncio.

Example

import asyncio
import aiohttp

urls = ['http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org']

async def call_url(url):
    print('Starting {}'.format(url))
    response = await aiohttp.get(url)
    data = await response.text()
    print('{}: {} bytes: {}'.format(url, len(data), data))
    return data

futures = [call_url(url) for url in urls]

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(futures))

TCP Echo Server

import asyncio


class EchoServerProtocol(asyncio.Protocol):
    def connection_made(self, transport):
        peername = transport.get_extra_info('peername')
        print('Connection from {}'.format(peername))
        self.transport = transport

    def data_received(self, data):
        message = data.decode()
        print('Data received: {!r}'.format(message))

        print('Send: {!r}'.format(message))
        self.transport.write(data)

        print('Close the client socket')
        self.transport.close()


async def main():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    server = await loop.create_server(
        lambda: EchoServerProtocol(),
        '127.0.0.1', 8888)

    async with server:
        await server.serve_forever()


asyncio.run(main())

TCP Echo Client

A TCP echo client using the loop.create_connection() method, sends data, and waits until the connection is closed:

import asyncio


class EchoClientProtocol(asyncio.Protocol):
    def __init__(self, message, on_con_lost):
        self.message = message
        self.on_con_lost = on_con_lost

    def connection_made(self, transport):
        transport.write(self.message.encode())
        print('Data sent: {!r}'.format(self.message))

    def data_received(self, data):
        print('Data received: {!r}'.format(data.decode()))

    def connection_lost(self, exc):
        print('The server closed the connection')
        self.on_con_lost.set_result(True)


async def main():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    on_con_lost = loop.create_future()
    message = 'Hello World!'

    transport, protocol = await loop.create_connection(
        lambda: EchoClientProtocol(message, on_con_lost),
        '127.0.0.1', 8888)

    # Wait until the protocol signals that the connection
    # is lost and close the transport.
    try:
        await on_con_lost
    finally:
        transport.close()


asyncio.run(main())

ThreadPoolExecutor vs asyncio

See Python:concurrent.futures#ThreadPoolExecutor vs asyncio.

Python Concurrency Mind Maps

Python_Concurrency_Mind_Maps.jpeg

Troubleshooting

pipe closed by peer or os.write(pipe, data) raised exception

asyncio pipe closed by peer or os.write(pipe, data) raised exception.
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
...

Use drain.

proc.stdin.write(f.read())
await proc.stdin.drain()

See also

Favorite site

Tutorials

Guide

References


  1. Asynchronous_Python_-_mingrammers_note.pdf ↩

  2. Speed_Up_Your_Python_Program_With_Concurrency_-_Che1s_blog.pdf ↩

author Zer0 sitemap