A simple guide on scaling and optimizing a Django application to the moon🚀

Scale & Optimize a Django application to scale

Gaurav Sharma's photo
Gaurav Sharma
·

10 min read

A simple guide on scaling and optimizing a Django application to the moon🚀

Note: If you are looking for a Full Stack Developer (Django+React), then connect with me on LinkedIn or contact me through my portfolio.

Django makes it easier to build better Web apps more quickly and with less code.

Introduction:

There are no doubts that Django is a well-structured, robust, and hell lot of maintainable web framework, which lets you focus on the real development, rather than fighting with the nitty-gritty details of a web framework and inventing the wheel again and again. Without wasting time, let's directly jump into the scalability part of Django.

What is scalability?

To understand the word "scalability", first we will take a real-life example.

Assume you have created an e-commerce store where you are selling cool printed T-Shirts, on the first day there were about 10 users who came to your website. After a week, you saw a surge in the number of users. All of a sudden there are 1 million users on the website hitting per day.

What are you going to do in this situation when the servers are on the verge of crashing and burning down?

A simple solution is to buy more resources like CPU and RAM from your cloud provider. That is called vertical scaling. But there is a catch in this solution, how long you can survive by doing this?

After 1 more week, 10 million users are coming to the website per day, are you still going to buy more RAM and CPU? But wait, there is a limitation, you can not buy after a certain amount of RAM or a certain generation of the CPU. What should you do now?

There is another option, you can buy another machine to serve the same instance of the application from two different servers and put a load balancer in between those two servers. This is called horizontal scaling.

But how long you can buy more machines to serve more users? You really need a way to optimize your application to serve millions of users without paying so much money on buying new servers. Now you have got the idea of scaling, let's understand how can you scale your Django application before any more server gets burned.

Caching:

Caching is a great mechanism to serve mostly requested content without making a query to your database or server again and again. In the case of Django, There are many ways to implement different kinds of caching mechanisms like Memcache, DB Cache, Redis Cache.

Django comes with an out-of-the-box caching mechanism, which offers different levels of cache granularity; You can cache your whole Site, a single or more Views, also templates, or a particular fragment of the template.

Django provides a simple configuration to specify the caching backend in your settings.py file, you can read more about the caching in Django and "how-to guides?" in this Part of the official documentation of Django.

But you are not limited to only Django for caching, you can also apply caching out of the Django such as using CDN, or applying server-level cache such as Varnish Cache.

You can take it to the next level by using Server level cache such as Varnish, But be very cautious before applying server-level cache for User-oriented applications, where data changes frequently depending on the user behaviors.

Last but not the least, caching is an excellent way to scale your application efficiently, but you should be careful before applying it. A simple rule of thumb is "Do not Cache everything".

Handling Middlewares:

Middlewares are the first thing at the application level that faces the HTTP request for the first time in any Django Project and each response has to go through all those middlewares on each cycle. So in the end, a middleware can either speed up your request-response cycle or it can slow it down.

Many middleware comes out-of-the-box with Django. But in many cases, you do not need them all necessarily. So why not just remove the unused middlewares one by one and instantly see the difference in performance.

Like in the case of Rest APIs, you do not need SessionMiddleware and MessageMiddleware, so just remove them from your settings.py file.

Removing some middleware sounds good, but there are some other things that you can do to reduce the response time by just adding some useful middlewares like ConditionalGetMiddleware, GZipMiddleware, and FetchFromCacheMiddleware.

Connection Pooling:

Connection pooling in the context of Database is actually a way to keep database connections open for a certain time so that they can be reused again for more requests, these types of connections are also called persistent connections.

Opening a database connection is an expensive operation, especially if the database is remote. You have to open up network sessions, authenticate, have authorization checked, and so on. Pooling keeps the connections active so that, when a connection is later requested, one of the active ones is used in preference to having to create another one.

In Django, you can control the age of any single connection by the CONN_MAX_AGE parameter in your settings.py file, which defines the maximum lifetime in seconds of a connection as an integer.

Other than integers, you can also set this value to be 0 (zero) and None. The value 0 (zero) is the default value, which simply means to close the connection after the end of each request. And None simply means to never close a connection on each request, which means unlimited persistent connections.

What is the ideal value of this parameter?

Honestly speaking, it depends on the number of users coming to your web application each hour or day. You have to figure it out by yourself but try to make a balance depending on your database capacity, otherwise, you will end up having so many connections which will eventually slow down your database performance. Try to tune it according to the usage pattern of your web application.

Query Optimization:

Django has a great tool called Django ORM, which makes database CRUD operations so much easier and you start ignoring the SQL and query optimizations.

Let's understand some optimizations that we can do to avoid slow database operations and run optimized Queries -

  1. Querysets are lazy in nature, which means the moment you run an ORM query is not the moment when SQL query gets executed. It will be executed when you are actually trying to access some data from that ORM query results. So from next time, do not forget to store the results of an ORM query into a Python variable to access the data again without running the actual SQL query.

  2. If you are using @property in any of your Django models, and you need to access that property value more than once, then instead of it, try to use @cached_property , which actually calculates the value once and then caches it in memory as a normal attribute for the life of the instance.

  3. Are you running loops to save so many data rows in the table using .create() or .update() ? Stop doing it, you are unintentionally putting so much burden on your database server, by running multiple SQL queries to save data at once. Instead of saving data one by one, use bulk_create() and bulk_update() for heavy writing operations, this will run only one SQL query to save all the data at once.

    To read more about bulk operations, read this Part of the official documentation.

  4. The last thing is the most important thing to consider, it can drastically improve the read performance of your application.

    If you have connected the database tables through Foreign Key, Many to Many, or One to One relationships and at the time of reading the related columns, usually you simply retrieve the data from the parent table and then can access its related column values using dot(.) notation given in the below example, then you are doing it the wrong way.

     """
 Assume we have a Book model connected to Author model by Many To Many relation through 'authors' column.
 and also connected to Category model by Foreign Key relation through 'category' column.
 """

 book_1 = Book.objects.get(pk=1)  # Runs an SQL query to retrieve all the books

 authors_of_book_1 = book_1.authors.all() # Runs an SQL query again to retrieve the authors

 category_of_book_1 = book_1.category.title # Runs an SQL query again to retrieve the category

 """
 This is a Bad practice, especially if you already know that you need the authors and category of the book. 
 Unintentionally you are running 3 queries to retrieve the data about book.
 """

    What is the correct method to read the related column values in Django?

    Using the select_related and prefetch_related ORM methods given in the below examples. There is simple psychology behind these two ORM methods, "Retrieve everything at once if you know you will need it".

     """
 Assume we have a Book model connected to Author model by Many To Many relation through 'authors' column.
 and also connected to Category model by Foreign Key relation through 'category' column.
 """

 # Runs an SQL query to retrieve all the books with its category value and author values
 book_1 = Book.objects.select_related('category').get(pk=1).prefetch_related('authors')

 authors_of_book_1 = book_1.authors.all() # Do not run an SQL query

 category_of_book_1 = book_1.category.title # Do not run an SQL query

 """
 This is a good practice, especially if you already know that you need the authors and category of the book. 
 You are just running a single query to retrieve the required data.
 """

    According to the official Django docs, "Hitting the database multiple times for different parts of a single 'set' of data that you will need all parts of is, in general, less efficient than retrieving it all in one query. This is particularly important if you have a query that is executed in a loop, and could therefore end up doing many database queries when only one was needed."

You can do much more to optimize the performance of your application and read-write database operations. But this is it for this section.

Distributed Task Queues:

Assume you want to implement a functionality where a user can send emails to thousands of users in your application. But the problem is that whenever a user tries to send emails, the user has to wait for a very long time to get the success status in the browser without closing the active browser tab, because behind the scenes, Django View is trying to send the email along with the request, and the user has to wait until browser gets a success response.

The solution to this problem is to use Distributed Task Queues, whenever a user will send a request to sent a thousand emails in Django View, the View will transfer that email sending load to a task queue asynchronously, and the user will immediately get a successful HTTP response with a pending task status. User can go and enjoy life, in meantime the task queue is handling the task of sending the email to thousands of users, and when it is done, the user will get an update of the success/failure.

There are many task queues out there, Celery is something that I found most intuitive and performant. You can also try Django Q which is a lighter alternative to Celery.

All the task queues leverage a separate server called Task Broker(or Message Broker) such as RabbitMQ or Redis. This is your personal choice, you can use any one of them depending on some external factors.

Distributed Task Queues can be proven magical for your application, especially if you are doing so much processing inside your view. A simple rule of thumb is "Do not run time-consuming tasks directly inside your View".

Note: You can(should) run the message broker (RabbitMQ) on a separate server to get optimal performance out of your Django server.

Some Extra Things to Consider:

Some more small things can optimize the response time in your Django application and help you scale efficiently -

  1. You can use serverless architecture having managed databases and other managed services.

  2. You can serve the Media files through an Object storage service like an AWS S3 bucket.

  3. You can also use CDN to server static files after minifying the static files.

  4. Database Sharding is another way to scale your application if the application is being used globally by different locations.

  5. Modifying the type of load balancing you are using and analyze which fits perfectly for your application.

  6. You can also try to deploy your admin app to a different subdomain and make it a separate entity.

Conclusion:

Django is great, to use it efficiently and smartly, you need to have a great in-depth understanding of many things that comes pre-built with Django.

If you are looking for more in-depth knowledge on the above things, do not forget to check out Django's official documentation from the given links.

Thanks for making it to the End of this article. If you found any mistakes in this article, please do inform me, I would appreciate it.

Reference:

For more such crispy blogs daily, follow DevJunction, subscribe to our newsletter and get notified.

Did you find this article valuable?

Support Dev.Junction by becoming a sponsor. Any amount is appreciated!

DjangocachingDeveloperWeb Developmentbackend