Unlocking my blocking API
Photo by K. Mitch Hodge on Unsplash
I’ve been hitting a performance issue in my python services development activities, so I decided to share a couple of tricks which are already well known but may be useful for you.
Asyncio is an amazing way to add concurrency into my code in a simple way.
Yet adding async on top of an exisiting codebase may require a deep refactoring which may not always be affordable in a single run
Concurrency methods in python
Please take the following descriptions as summaries: these are meant to introduce some basic concepts for practical usage only
Concurrency with Processes in python
Python processes have some memory assigned to handle function calls and their local variables (stack) and actual data (heap); the OS also assign some IO resources exclusively to a process (e.g. file handles, network connections etc.).
Python can create many processes using the multiprocessing library (which in turn uses an operating system operation called fork): each process will be an exact copy of the parent but the code will be executed from a different point.
As the parent process and its children have no memory in common, python uses pickle to serialize the data sent from the parent process to the children and vice versa. This means that only pure data can be exchanged as process IO resources cannot be shared in this way.
Concurrency with Threads in Python
Modern OS allow to create threads within a process. This allows the OS to execute different parts of the code with different cores without creating a new process. Threads have their own stack, but share the heap and resources of the process.
Python uses threading to allow creation and management of threads. As threads can access the same data at the same time they can step on each other toe and mess up the status of a program. Python offer a series of semaphores and communication channels which allow to avoid these racing conditions.
The standard python interpreter implements a Global Interpreter Lock which prevents the execution of multiple threads at the same time. There are some exception to this rule but we won’t explore them in detail now.
Moreover threads are implementing a cooperative concurrency model i.e. a thread won’t stop its execution to allow other threads to run unless explicitly ordered, e.g. by let it wait for a semaphore status or a queue value or executing an operating system call
Threads require some extra code to manage exception propagation, as each stack is independent.
A python thread may run even if the main thread is dead, thus removing any control from the main program.
Some GUI libraries (e.g. QT) have their own thread objects which allow to draw and update the graphical while other code is being executed
Concurrent Futures
python also offer a convenient way to use threads and processes with the concurrent.futures library which offers an high level interface on top of the threading and multiprocessing libraries to execute parallel computations and return results.
Concurrency with coroutines in python
coroutines are “functions whose execution can be suspended and resumed”. Python introduced them with the yield keyword whose purpose was mainly to create generators.
They do not introduce other stacks like threads, and actually only one at a time is executed. It is possible to use the yield keyword to create a system of coroutines which behave like “cooperative threads”. There should be some code to resume each one of them in turn until all are completed: this is sometime called the “executor loop”.
The syntax of python has been extended to include this concept with the async def, async for and await statements. The asyncio library provides
- executors
- IO calls which yield the control to another coroutine until the IO operation is completed.
The advantage of async coroutines and asyncio library is to easily create concurrent tasks, just extending the usual function call syntax. Exceptions are propagated natively and this does not require OS operations like fork or create new threads. Thus they are also extremely lightweight and can scale better than the other options.
Making HTTP requests async
requests is a great HTTP client library upon which many api connection are based, but its IO model is blocking.
I wanted to use async but cannot afford to refactor the whole codebase on my own, so this is the most simple solution I found
from blocking.endpoint.api import call_api import asyncio async def async_call_api(): loop = asyncio.get_running_loop() result = await loop.run_in_executor(None,call_api) return result
calling functions with complex inputs
- functools
due to its interface I had some troubles to implement this solution when keyword arguments were mandatory: the easiest solution is to use the functools.partial function
from functools import partial from blocking.endpoint.api import call_api import asyncio async def async_call_api(*args,**kargs): loop = asyncio.get_running_loop() applied_func = partial(call_api,*args,**kwargs) result = await loop.run_in_executor(None,applied_func) return result
this can be also used as a decorator
import asyncio def make_async(func): async def async_call(*args,**kwargs): loop = asyncio.get_running_loop() applied_func = partial(func,*args,**kwargs) result = await loop.run_in_executor(None,applied_func) return result return async_call @make_async def api_call(...): ...
using thread pools
Making Django ORM async for ETL scripts
The Django web framework is a very popular and feature complete system which includes
- an ASGI and WSGI interface for production (these are the modern python interfaces for web frameworks)
- an integrated Object Relational Mapping (ORM) with many database servers
- an extensible authentication and authorization system
- a great unit test system
and so much more.
I’ve been happily using it in the last 20 years and it is still my go-to choice for python based services
As the ASGI and WSGI interface spawn multiple parallel workers to manage multiple requests, checks are performed to make sure this is not going to cause inconsistencies within the framework.
One of the checks performed is about the usage of async calls: these must be isolated to avoid race conditions while responses are being served.
Django can be also used as a standalone script framework when you need to perform some Extraction Transformation and Load (ETL) process which targets your database via its ORM. If you want to take advantage of the async structure in this context, you must take special care, even if no ASGI or WSGI is actually being used.
Luckily the ORM has already an async interface
post = Post(author, content) post.asave()
but what if your ORM call are buried in a large codebase and you cannot afford to refactor all calls?
You can always call any Django synchronous part using a pattern like the following:
from asgiref.sync import sync_to_async # large old sync codebase def create_post(author, content): post = Post(author, content) post.save() return post # your async ETL code def async worker(author, post): post = await sync_to_async(create_post, thread_sensitive=True)(author=author, content=content)
A fine manual page is available here
Pros and Cons
There are multiple factors to be considered when choosing to use some tool to achieve concurrent or parallel execution of code and this short post is not intended to explore all the details.
However here are a few of the reasons why you may choose to use async:
- Async functions use lighter concurrency, with higher scalability
- Async functions are easier to write, sequences of async actions and nested async calls are easier to be created
- Async functions deal with exceptions better than Threads or Processes
Any good things come with a price so here are some of the reasons why you may not want to follow the approach described here:
- Using threads to simulate async (which is what
run_in_executordoes), does not scale well - When the main python process exits, threads keep running and this is true also for the async functions generated by
run_in_executor
Conclusions
Asyncio framework provides a great way to easily develop concurrent code, but may not be immediately usable.
The solutions described here are useful to integrate async calls when you cannot afford to refactor a large codebase: the details of each project may vary so be careful while using them.
