Asyncio
lastmod 2024-07-17created 2020-06-1136 edits
Python asynchronous IO.
Categories
- Python:EventLoop
- grpc#asyncio
- Asyncio:AsyncPythonSubprocess - Implementation of a class that launches a Subprocess using the asyncio package of Python.
- Asyncio:TaskManager - Implementation of a Task manager using the asyncio package of Python.
Exceptions
- exception asyncio.CancelledError
- The operation has been cancelled.
- This exception can be caught to perform custom operations when asyncio Tasks are cancelled. In almost all situations the exception must be re-raised.
| WARNING |
| Changed in version 3.8: |
Creating native coroutines
First, to use asyncio, create a native coroutine with async def as follows. In Python, coroutines created with async def are called native coroutines to distinguish them from generator-based coroutines. The async def keyword is available from Python 3.5 onward.
Hello, World Example
import asyncio
async def hello(): # async def로 네이티브 코루틴을 만듦
print('Hello, world!')
loop = asyncio.get_event_loop() # 이벤트 루프를 얻음
loop.run_until_complete(hello()) # hello가 끝날 때까지 기다림
loop.close() # 이벤트 루프를 닫음
Running native coroutines with await
This time, here is how to run a native coroutine with await. If you put a coroutine object, future object, or task object after await as follows, it waits until that object finishes and then returns the result. True to its name, await waits until a specific object finishes.
The await keyword is available from Python 3.5 onward; in 3.4, use yield from
-
변수 = await 코루틴객체 -
변수 = await 퓨처객체 -
변수 = await 태스크객체
One thing to watch out for here: await can only be used inside a native coroutine. Now let's make a native coroutine that adds two numbers, waits 1 second in the coroutine, and then returns the result.
import asyncio
async def add(a, b):
print('add: {0} + {1}'.format(a, b))
await asyncio.sleep(1.0) # 1초 대기. asyncio.sleep도 네이티브 코루틴
return a + b # 두 수를 더한 결과 반환
async def print_add(a, b):
result = await add(a, b) # await로 다른 네이티브 코루틴 실행하고 반환값을 변수에 저장
print('print_add: {0} + {1} = {2}'.format(a, b, result))
loop = asyncio.get_event_loop() # 이벤트 루프를 얻음
loop.run_until_complete(print_add(1, 2)) # print_add가 끝날 때까지 이벤트 루프를 실행
loop.close() # 이벤트 루프를 닫음
Future vs Task
- Future (asyncio.Future)
- A class that represents work to be done in the future, used to cancel the work, check its status, and set completion and results.
- Task (asyncio.Task)
- A subclass of
asyncio.Futurethat contains the functionality ofasyncio.Futureand the coroutine object to run. - A task is used to cancel the coroutine's execution, check its status, and set completion and results.
Synchronization Primitives
- Synchronization Primitives — Python 3.10.5 documentation
- Lock
- Event - Plays a role similar to WaitForSingleObject (but the flag must be reset with the
clearfunction) - Condition - Plays a role similar to std::condition variable
- Semaphore
- BoundedSemaphore
Running tasks concurrently
Use awaitable asyncio.gather(*aws, loop=None, return_exceptions=False).
import asyncio
async def factorial(name, number):
f = 1
for i in range(2, number + 1):
print(f"Task {name}: Compute factorial({i})...")
await asyncio.sleep(1)
f *= i
print(f"Task {name}: factorial({number}) = {f}")
async def main():
# Schedule three calls *concurrently*:
await asyncio.gather(
factorial("A", 2),
factorial("B", 3),
factorial("C", 4),
)
asyncio.run(main())
# Expected output:
#
# Task A: Compute factorial(2)...
# Task B: Compute factorial(2)...
# Task C: Compute factorial(2)...
# Task A: factorial(2) = 2
# Task B: Compute factorial(3)...
# Task C: Compute factorial(3)...
# Task B: factorial(3) = 6
# Task C: Compute factorial(4)...
# Task C: factorial(4) = 24
Future
Done callbacks
Use add_done_callback(callback, *, context=None).
Adds a callback to be run when the Future is done. Parameters can be passed to the callback using functools.partial, for example:
# Call 'print("Future:", fut)' when "fut" is done.
fut.add_done_callback(
functools.partial(print, "Future:"))
Task
Creating a task
Use asyncio.create_task(coro, *, name=None):
async def coro():
...
# In Python 3.7+
task = asyncio.create_task(coro())
...
# This works in all Python versions but is less readable
task = asyncio.ensure_future(coro())
...
Cancelling a task
async def cancel_me():
print('cancel_me(): before sleep')
try:
# Wait for 1 hour
await asyncio.sleep(3600)
except asyncio.CancelledError:
print('cancel_me(): cancel sleep')
raise
finally:
print('cancel_me(): after sleep')
async def main():
# Create a "cancel_me" Task
task = asyncio.create_task(cancel_me())
# Wait for 1 second
await asyncio.sleep(1)
task.cancel()
try:
await task
except asyncio.CancelledError:
print("main(): cancel_me is cancelled now")
asyncio.run(main())
# Expected output:
#
# cancel_me(): before sleep
# cancel_me(): cancel sleep
# cancel_me(): after sleep
# main(): cancel_me is cancelled now
async with
Now let's make a class that returns the result of an addition after 1 second.
import asyncio
class AsyncAdd:
def __init__(self, a, b):
self.a = a
self.b = b
async def __aenter__(self):
await asyncio.sleep(1.0)
return self.a + self.b # __aenter__에서 값을 반환하면 as에 지정한 변수에 들어감
async def __aexit__(self, exc_type, exc_value, traceback):
pass
async def main():
async with AsyncAdd(1, 2) as result: # async with에 클래스의 인스턴스 지정
print(result) # 3
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
async for
import asyncio
class AsyncCounter:
def __init__(self, stop):
self.current = 0
self.stop = stop
def __aiter__(self):
return self
async def __anext__(self):
if self.current < self.stop:
await asyncio.sleep(1.0)
r = self.current
self.current += 1
return r
else:
raise StopAsyncIteration
async def main():
async for i in AsyncCounter(3): # for 앞에 async를 붙임
print(i, end=' ')
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
run_in_executor
- Event Loop — Python 3.10.12 documentation # run_in_executor
- How to call standard Python functions asynchronously in asyncio · Wireframe
import asyncio
import concurrent.futures
def blocking_io():
# File operations (such as logging) can block the
# event loop: run them in a thread pool.
with open('/dev/urandom', 'rb') as f:
return f.read(100)
def cpu_bound():
# CPU-bound operations will block the event loop:
# in general it is preferable to run them in a
# process pool.
return sum(i * i for i in range(10 ** 7))
async def main():
loop = asyncio.get_running_loop()
## Options:
# 1. Run in the default loop's executor:
result = await loop.run_in_executor(
None, blocking_io)
print('default thread pool', result)
# 2. Run in a custom thread pool:
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, blocking_io)
print('custom thread pool', result)
# 3. Run in a custom process pool:
with concurrent.futures.ProcessPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, cpu_bound)
print('custom process pool', result)
if __name__ == '__main__':
asyncio.run(main())
Returning all tasks
import asyncio
import sys
def all_tasks(loop: asyncio.AbstractEventLoop):
if sys.version_info >= (3, 7):
return getattr(asyncio, "all_tasks")(loop) # nocov
else:
tasks = asyncio.Task.all_tasks(loop) # type: ignore[attr-defined]
return {t for t in list(tasks) if not t.done()}
How to give await a priority (using PriorityQueue)
There is no way to control this directly with await, so use a priority queue.
import asyncio
async def high_priority_task():
print("High priority task started")
await asyncio.sleep(1)
print("High priority task finished")
async def low_priority_task():
print("Low priority task started")
await asyncio.sleep(2)
print("Low priority task finished")
async def main():
# 우선순위 큐 생성
queue = asyncio.PriorityQueue()
# 작업을 우선순위와 함께 큐에 추가
await queue.put((0, high_priority_task()))
await queue.put((1, low_priority_task()))
# 큐에서 작업을 가져와 실행
while not queue.empty():
_, task = await queue.get()
await task
asyncio.run(main())
asyncio.wait
- asyncio.FIRST_COMPLETED - The function returns when any future finishes or is cancelled.
- asyncio.FIRST_EXCEPTION - Returns when a future finishes by raising an exception. If no exception is raised, same as ALL_COMPLETED.
- asyncio.ALL_COMPLETED - The function returns when all futures finish or are cancelled.
asyncio.as_completed
Returns in order of completion.
asyncio.to_thread
Runs the function func asynchronously in a separate thread.
Take the following code:
and change it as follows:
Cancelling a list of tasks from a non-async function
import asyncio
import sys
def cancel_tasks(loop: asyncio.AbstractEventLoop, *tasks) -> None:
if not tasks:
return
for task in tasks:
task.cancel()
loop.run_until_complete(asyncio.gather(*tasks, loop=loop, return_exceptions=True))
for task in tasks:
if task.cancelled():
continue
if task.exception() is not None:
loop.call_exception_handler(
{
"message": "Unhandled exception during asyncio.run() shutdown",
"exception": task.exception(),
"task": task,
}
)
Subprocesses
import asyncio
async def run(cmd):
proc = await asyncio.create_subprocess_shell(
cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE)
stdout, stderr = await proc.communicate()
print(f'[{cmd!r} exited with {proc.returncode}]')
if stdout:
print(f'[stdout]\n{stdout.decode()}')
if stderr:
print(f'[stderr]\n{stderr.decode()}')
asyncio.run(run('ls /zzz'))
Applying cmd with shlex.
Implementing a timeout (asyncio.wait_for)
Use asyncio.wait_for:
async def eternity():
# Sleep for one hour
await asyncio.sleep(3600)
print('yay!')
async def main():
# Wait for at most 1 second
try:
await asyncio.wait_for(eternity(), timeout=1.0)
except asyncio.TimeoutError:
print('timeout!')
asyncio.run(main())
# Expected output:
#
# timeout!
Opening a TCP/IP socket:
fut = asyncio.open_connection( self.host, self.port, loop=self.loop )
try:
r, w = await asyncio.wait_for(fut, timeout=self.connection_timeout)
except asyncio.TimeoutError:
pass
Implementing a cancellable Sleep Task
# -*- coding: utf-8 -*-
from asyncio import CancelledError, Task, ensure_future
from asyncio import sleep as asyncio_sleep
from typing import Any, Dict
class Sleeper:
def __init__(self):
self.tasks: Dict[Any, Task] = dict()
async def sleep(self, key: Any, seconds: float, result=None, *, loop=None) -> Any:
coro = asyncio_sleep(seconds, result=result, loop=loop)
task = ensure_future(coro)
self.tasks[key] = task
try:
return await task
except CancelledError:
return result
finally:
del self.tasks[key]
def cancel(self, key: Any) -> None:
self.tasks[key].cancel()
def cancel_all(self) -> None:
for task in self.tasks.values():
task.cancel()
Using asyncio in Python 3.4 and earlier
async def and await were added in Python 3.5. So they cannot be used in versions below 3.5. In Python 3.4, create a native coroutine with the @asyncio.coroutine decorator as follows.
In Python 3.4, use yield from instead of await.
-
변수 = yield from 코루틴객체 -
변수 = yield from 퓨처객체 -
변수 = yield from 태스크객체
In Python 3.3, for asyncio, install asyncio with pip install asyncio, then use the @asyncio.coroutine decorator and yield from. However, versions below 3.3 do not support asyncio.
Example
import asyncio
import aiohttp
urls = ['http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org']
async def call_url(url):
print('Starting {}'.format(url))
response = await aiohttp.get(url)
data = await response.text()
print('{}: {} bytes: {}'.format(url, len(data), data))
return data
futures = [call_url(url) for url in urls]
loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(futures))
TCP Echo Server
import asyncio
class EchoServerProtocol(asyncio.Protocol):
def connection_made(self, transport):
peername = transport.get_extra_info('peername')
print('Connection from {}'.format(peername))
self.transport = transport
def data_received(self, data):
message = data.decode()
print('Data received: {!r}'.format(message))
print('Send: {!r}'.format(message))
self.transport.write(data)
print('Close the client socket')
self.transport.close()
async def main():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
server = await loop.create_server(
lambda: EchoServerProtocol(),
'127.0.0.1', 8888)
async with server:
await server.serve_forever()
asyncio.run(main())
TCP Echo Client
A TCP echo client using the loop.create_connection() method, sends data, and waits until the connection is closed:
import asyncio
class EchoClientProtocol(asyncio.Protocol):
def __init__(self, message, on_con_lost):
self.message = message
self.on_con_lost = on_con_lost
def connection_made(self, transport):
transport.write(self.message.encode())
print('Data sent: {!r}'.format(self.message))
def data_received(self, data):
print('Data received: {!r}'.format(data.decode()))
def connection_lost(self, exc):
print('The server closed the connection')
self.on_con_lost.set_result(True)
async def main():
# Get a reference to the event loop as we plan to use
# low-level APIs.
loop = asyncio.get_running_loop()
on_con_lost = loop.create_future()
message = 'Hello World!'
transport, protocol = await loop.create_connection(
lambda: EchoClientProtocol(message, on_con_lost),
'127.0.0.1', 8888)
# Wait until the protocol signals that the connection
# is lost and close the transport.
try:
await on_con_lost
finally:
transport.close()
asyncio.run(main())
ThreadPoolExecutor vs asyncio
See Python:concurrent.futures#ThreadPoolExecutor vs asyncio.
Python Concurrency Mind Maps
| |
Troubleshooting
pipe closed by peer or os.write(pipe, data) raised exception
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
asyncio pipe closed by peer or os.write(pipe, data) raised exception.
...
Use drain.
See also
- Python
- Python:asyncio
- Python:subprocess
- Python:multiprocessing
- Python:threading
- trio
- aiohttp
- aiortc
- libuv
Favorite site
- asyncio --- Asynchronous I/O
- [Recommended] Coroutines and Tasks — Python 3.8.2 documentation
- Event Loop — Python 3.8.2 documentation
- Asynchronous Python
Tutorials
- Python and asynchronous programming #1, what is asynchronous programming
- [Recommended] Python and asynchronous programming #2, getting started with asynchronous programming in Python
- Python and asynchronous programming #3, putting asynchronous programming to use in Python
Guide
References
- Aiobotocore
- Aiofiles
- Aiohttp
- Aioredis
- Amazon Simple Storage Service
- AnyIO
- Async-timeout
- Asyncio:AsyncPythonSubprocess
- Asyncio:TaskManager
- Asyncpg
- Boto3
- GRPC
- GRPC:Python
- Message Queuing Telemetry Transport
- PostgreSQL
- Pytest-cov
- Python
- Python:concurrent.futures
- Python:EventLoop
- Python:Libraries
- Python:multiprocessing
- Python:selectors
- Python:StandardLibrary
- Python:subprocess
- Python:threading
- Python:tkinter
- Redis-py
- Serial Communication