# A simple guide on scaling and optimizing a Django application to the moon🚀

***Note: If you are looking for a Full Stack Developer (Django+React), then connect with me on*** [***LinkedIn***](https://www.linkedin.com/in/mnamegaurav/) ***or contact me through my*** [***portfolio***](https://gaurav.devjunction.in/)***.***

> ### Django makes it easier to build better Web apps more quickly and with less code.

## Introduction:

There are no doubts that **Django** is a *well-structured*, *robust*, and hell lot of *maintainable* web framework, which lets you focus on the real development, rather than fighting with the nitty-gritty details of a web framework and inventing the wheel again and again. Without wasting time, let's directly jump into the scalability part of Django.

## What is scalability?

To understand the word "***scalability***", first we will take a real-life example.

Assume you have created an e-commerce store where you are selling cool printed T-Shirts, on the first day there were about 10 users who came to your website. After a week, you saw a surge in the number of users. All of a sudden there are 1 million users on the website hitting per day.

**What are you going to do in this situation when the servers are on the verge of crashing and burning down?**

A simple solution is to buy more resources like **CPU** and **RAM** from your cloud provider. That is called ***vertical scaling***. But there is a catch in this solution, how long you can survive by doing this?

After 1 more week, 10 million users are coming to the website per day, are you still going to buy more **RAM** and **CPU**? But wait, there is a limitation, you can not buy after a certain amount of **RAM** or a certain generation of the **CPU**. What should you do now?

There is another option, you can buy another machine to serve the same **instance** of the application from two different servers and put a **load balancer** in between those two servers. This is called ***horizontal scaling***.

But how long you can buy more machines to serve more users? You really need a way to optimize your application to serve millions of users without paying so much money on buying new servers. Now you have got the idea of scaling, let's understand how can you scale your **Django** application before any more server gets burned.

## Caching:

**Caching** is a great mechanism to serve mostly requested content without making a query to your database or server again and again. In the case of **Django**, There are many ways to implement different kinds of caching mechanisms like ***Memcache***, ***DB Cache***, ***Redis Cache***.

Django comes with an out-of-the-box caching mechanism, which offers different levels of cache granularity; You can cache your whole ***Site***, a single or more ***Views***, also ***templates***, or a particular ***fragment*** of the template.

**Django** provides a simple configuration to specify the caching backend in your `settings.py` file, you can read more about the caching in **Django** and "*how-to guides*?" in this [Part](https://docs.djangoproject.com/en/stable/topics/cache/) of the official documentation of **Django**.

But you are not limited to only **Django** for caching, you can also apply caching out of the Django such as using ***CDN***, or applying server-level cache such as ***Varnish*** Cache.

You can take it to the next level by using Server level cache such as ***Varnish***, But be very cautious before applying server-level cache for User-oriented applications, where data changes frequently depending on the user behaviors.

Last but not the least, caching is an excellent way to scale your application efficiently, but you should be careful before applying it. A simple rule of thumb is "***Do not Cache everything***".

## Handling Middlewares:

**Middlewares** are the first thing at the application level that faces the ***HTTP*** request for the first time in any **Django** Project and each response has to go through all those middlewares on each cycle. So in the end, a middleware can either speed up your ***request-response*** cycle or it can slow it down.

Many middleware comes out-of-the-box with **Django**. But in many cases, you do not need them all necessarily. So why not just remove the unused middlewares one by one and instantly see the difference in performance.

Like in the case of *Rest APIs*, you do not need `SessionMiddleware` and `MessageMiddleware`, so just remove them from your `settings.py` file.

Removing some **middleware** sounds good, but there are some other things that you can do to reduce the response time by just adding some useful **middlewares** like `ConditionalGetMiddleware`, `GZipMiddleware`, and `FetchFromCacheMiddleware`.

## Connection Pooling:

**Connection pooling** in the context of Database is actually a way to keep database connections open for a certain time so that they can be reused again for more requests, these types of connections are also called ***persistent*** connections.

Opening a database connection is an expensive operation, especially if the database is remote. You have to open up network ***sessions***, ***authenticate***, have ***authorization*** checked, and so on. Pooling keeps the connections active so that, when a connection is later requested, one of the active ones is used in preference to having to create another one.

In **Django**, you can control the age of any single connection by the `CONN_MAX_AGE` parameter in your `settings.py file`, which defines the maximum lifetime in seconds of a connection as an integer.

Other than integers, you can also set this value to be `0` (zero) and `None`. The value `0` (zero) is the default value, which simply means to close the connection after the end of each request. And None simply means to never close a connection on each request, which means unlimited persistent connections.

**What is the ideal value of this parameter?**

Honestly speaking, it depends on the number of users coming to your web application each hour or day. You have to figure it out by yourself but try to make a balance depending on your database capacity, otherwise, you will end up having so many connections which will eventually slow down your database performance. Try to tune it according to the usage pattern of your web application.

## Query Optimization:

**Django** has a great tool called **Django ORM**, which makes database **CRUD** operations so much easier and you start ignoring the **SQL** and query optimizations.

Let's understand some optimizations that we can do to avoid slow database operations and run optimized Queries -

1. **Querysets** are lazy in nature, which means the moment you run an **ORM** query is not the moment when **SQL** query gets executed. It will be executed when you are actually trying to access some data from that **ORM** query results. So from next time, do not forget to store the results of an **ORM** query into a **Python** variable to access the data again without running the actual **SQL** query.
    
2. If you are using `@property` in any of your Django models, and you need to access that property value more than once, then instead of it, try to use `@cached_property` , which actually calculates the value once and then caches it in ***memory*** as a normal attribute for the life of the instance.
    
3. Are you running loops to save so many data rows in the table using `.create()` or `.update()` ? Stop doing it, you are unintentionally putting so much burden on your database server, by running multiple **SQL** queries to save data at once. Instead of saving data one by one, use `bulk_create()` and `bulk_update()` for heavy writing operations, this will run only one **SQL** query to save all the data at once.
    
    To read more about bulk operations, read this [Part](https://docs.djangoproject.com/en/stable/topics/db/optimization/#use-bulk-methods) of the official documentation.
    
4. The last thing is the most important thing to consider, it can drastically improve the ***read*** performance of your application.
    
    If you have connected the database tables through ***Foreign Ke*y**, ***Many to Many***, or ***One to One*** relationships and at the time of reading the related columns, usually you simply retrieve the data from the parent table and then can access its related column values using **dot(.)** notation given in the below example, then you are doing it the wrong way.
    
    ```python
    """
    Assume we have a Book model connected to Author model by Many To Many relation through 'authors' column.
    and also connected to Category model by Foreign Key relation through 'category' column.
    """
    
    book_1 = Book.objects.get(pk=1)  # Runs an SQL query to retrieve all the books
    
    authors_of_book_1 = book_1.authors.all() # Runs an SQL query again to retrieve the authors
    
    category_of_book_1 = book_1.category.title # Runs an SQL query again to retrieve the category
    
    """
    This is a Bad practice, especially if you already know that you need the authors and category of the book. 
    Unintentionally you are running 3 queries to retrieve the data about book.
    """
    ```
    
    **What is the correct method to read the related column values in Django?**
    
    Using the `select_related` and `prefetch_related` ORM methods given in the below examples. There is simple psychology behind these two ORM methods, "***Retrieve everything at once if you know you will need it***".
    
    ```python
    """
    Assume we have a Book model connected to Author model by Many To Many relation through 'authors' column.
    and also connected to Category model by Foreign Key relation through 'category' column.
    """
    
    # Runs an SQL query to retrieve all the books with its category value and author values
    book_1 = Book.objects.select_related('category').get(pk=1).prefetch_related('authors')
    
    authors_of_book_1 = book_1.authors.all() # Do not run an SQL query
    
    category_of_book_1 = book_1.category.title # Do not run an SQL query
    
    """
    This is a good practice, especially if you already know that you need the authors and category of the book. 
    You are just running a single query to retrieve the required data.
    """
    ```
    
    According to the official Django docs, "*Hitting the database multiple times for different parts of a single 'set' of data that you will need all parts of is, in general, less efficient than retrieving it all in one query. This is particularly important if you have a query that is executed in a loop, and could therefore end up doing many database queries when only one was needed*."
    

You can do much more to optimize the performance of your application and read-write database operations. But this is it for this section.

## Distributed Task Queues:

Assume you want to implement a functionality where a user can send emails to thousands of users in your application. But the problem is that whenever a user tries to send emails, the user has to wait for a very long time to get the success status in the browser without closing the active browser tab, because behind the scenes, **Django View** is trying to send the email along with the ***request***, and the user has to wait until browser gets a success response.

The solution to this problem is to use **Distributed Task Queues**, whenever a user will send a request to sent a thousand emails in **Django View**, the View will transfer that email sending load to a task queue asynchronously, and the user will immediately get a successful HTTP response with a pending task status. User can go and enjoy life, in meantime the ***task queue*** is handling the task of sending the email to thousands of users, and when it is done, the user will get an update of the success/failure.

There are many task queues out there, **Celery** is something that I found most intuitive and performant. You can also try **Django Q** which is a lighter alternative to Celery.

All the task queues leverage a separate server called ***Task Broker***(or ***Message Broker***) such as **RabbitMQ** or **Redis**. This is your personal choice, you can use any one of them depending on some external factors.

**Distributed Task Queues** can be proven magical for your application, especially if you are doing so much processing inside your view. A simple rule of thumb is "***Do not run time-consuming tasks directly inside your View***".

###### *Note: You can(should) run the message broker (****RabbitMQ****) on a separate server to get optimal performance out of your Django server.*

## Some Extra Things to Consider:

Some more small things can optimize the response time in your **Django** application and help you scale efficiently -

1. You can use ***serverless*** architecture having managed databases and other managed services.
    
2. You can serve the Media files through an ***Object storage*** service like an AWS S3 bucket.
    
3. You can also use **CDN** to server static files after minifying the static files.
    
4. ***Database Sharding*** is another way to scale your application if the application is being used globally by different locations.
    
5. Modifying the type of ***load balancing*** you are using and analyze which fits perfectly for your application.
    
6. You can also try to deploy your admin app to a different subdomain and make it a separate entity.
    

## Conclusion:

**Django** is great, to use it efficiently and smartly, you need to have a great in-depth understanding of many things that comes pre-built with **Django**.

If you are looking for more in-depth knowledge on the above things, do not forget to check out Django's official documentation from the given links.

> Thanks for making it to the End of this article. If you found any mistakes in this article, please do inform me, I would appreciate it.

---

## Reference:

* **Middleware** %\[https://docs.djangoproject.com/en/stable/ref/middleware/\]
    
* **Django's cache framework** %\[https://docs.djangoproject.com/en/stable/topics/cache/\]
    
* **Performance and optimization** %\[https://docs.djangoproject.com/en/stable/topics/performance/\]
    
* **Databases** %\[https://docs.djangoproject.com/en/stable/ref/databases/\]
    

> ***For more such crispy blogs daily, follow DevJunction, subscribe to our newsletter and get notified.***

## Social Links

* **LinkedIn:** [https://www.linkedin.com/in/mnamegaurav/](https://www.linkedin.com/in/mnamegaurav/)
    
* **YouTube:** [https://www.youtube.com/c/devjunction](https://www.youtube.com/c/devjunction)
    
* **Website:** [https://gaurav.devjunction.in/](https://gaurav.devjunction.in/)
    
* **GitHub:** [https://github.com/mnamegaurav](https://github.com/mnamegaurav)
    
* **Instagram:** [https://www.instagram.com/mnamegaurav/](https://www.instagram.com/mnamegaurav/)
    
* **Twitter:** [https://twitter.com/mnamegaurav](https://twitter.com/mnamegaurav)