Designing Modules in Python is part of HashedIn’s training program for junior developers to get better at design. Today, we are making it available as a free ebook.

Download E-Book

Who is this ebook for?

This book is for all those who have some experience in an object-oriented programming language. If you wish to get better at the module or class level design, then this book is definitely for you. You will be able to identify a good design from the bad after you finish reading this book
All throughout this book, python is used to introduce various design patterns. However, the knowledge of python is not mandatory. If you can understand classes and methods, you will easily be able to understand the contents of this book

How to use this book

To make the most of this book, read the requirements at the start of each chapter and write down your solution. Then read through the rest of the chapter and critique your solution.
Good design is obvious once presented. However, arriving at that solution is difficult, especially if you are new to programming. Writing down your solution is the only way to realize the shortcomings of your design. If you read the chapter without writing, you will tell yourself “That was obvious. I would have solved it in a similar manner”

History of this book

At HashedIn, we conduct a training program for junior developers to get better at design. During one of these programs, we asked developers to design a module to send SMS/text messages. We also asked them to call the SMS module from 3 different client applications. They had 30 minutes to complete this activity.
After 30 minutes, we revealed a new set of requirements. Additionally, we set 2 hard rules for them to implement these requirements:

1. Developers were not allowed to change the way client applications invoked their SMS module
2. Developers were not allowed to modify existing code. Instead, they had to write new code to address the additional requirements, without duplicating code.

20 minutes later, we again introduced a new set of requirements. And after another 20 minutes, we introduced the final set of requirements.
Initially, the developers felt the problem was very simple. They were at loss to understand why the design was even needed. Isn’t this a simple python function?
But as new requirements emerged, and because we gave them constraints (don’t change existing code) – they began to appreciate the various design principles. Some even developed a mental model to discern good design from bad.
Over time, we got the opportunity to review designs from several developers. We learned how junior developers think when asked to solve a problem. Based on these patterns, we wrote a series of blogs on how to design modules in python.

Design Modules in Python

Python Interface Design

The first module will teach you as to how you can design a reusable python module. By the end of the post, you will get a great understanding of writing reusable and extensible modules in python.

You have been given the following requirements:

You are the developer on Blipkart, an e-commerce website. You have been asked to build a reusable SMS module that other developers will use to send SMS’es. Watertel is a telecom company that provides a REST API for sending SMS’es, and your module needs to integrate with Watertel. Your module will be used by other modules such as logistics.py and orders.py.

The first goal for a module should be good Abstraction. Your module sits between client developers on one side, and the API provided by Watertel on the other side. The module’s job is to simplify sending an SMS, and to ensure that changes in Watertel do not affect your client developers.

The essence of abstractions is preserving information that is relevant in a given context, and forgetting information that is irrelevant in that context.

Watertel exposes these concepts – username, password, access_token and expiry, phone number, message, and priority. Which of these concepts are relevant to client developers?

Client developers don’t care about username and password. They don’t want to worry about expiry either. These are things our module should handle. They only care about phone number and message – “take this message, and send it to this phone number”.

Interface Design

Since developers only care about phone_number and message, our interface design is very clear – we need a send_sms(phone_number, message) function. Everything else is irrelevant.

Now, looking from an implementation perspective. We need the URL, the username, and password. We also need to manage the access token and handle its expiry. How will we get this information? Let’s carve out an interface design for our solution.

In general, if you need some information, or if you depend on other classes – you must get it via your constructor. Your class must not go about trying to find that information on its own (say from Django settings).

Here’s how our interface looks like:

# sms.py:
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass

Using our Class in Client Code

Next, in order to use our module, our clients need an object of SmsClient. But in order to create an object, they’d need username and password. Our first attempt would be something like this:

# In orders.py
from django.conf import settings
from sms import SmsClient
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)
....
sms_client.send_sms(phone_number, message)

There are two problems with this:

1. First, orders.py shouldn’t care how SmsClient objects are constructed. If we later need additional parameters in the constructor, we would have to change orders.py, and all classes that use SmsClient. Also, if we decide to read our settings from someplace else (say for test cases), then orders.py would have to be modified.
2. The second problem is that we would have to create SmsClient objects everywhere we want to use it. This is wasteful, and also leads to code duplication.

The solution is to create SmsClient object in sms.py module [see footnote 1]. Then orders.py and logistics.py can directly import the object from SMS. Here is how it looks.

# In sms.py:
from django.conf import settings
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)

and in orders.py:

# orders.py
from sms import sms_client
...
# when you need to send an sms
sms_client.send_sms(phone_number, message)

Before you start implementing, think about how your clients will use your module. It’s a good idea to actually write down the client code (i.e. code in orders.py and logistics.py) before you start implementing your module.

Now that we have our interface defined, we can start implementing our module.

Python Dependency Injection

Next step is to learn as to how python design patterns can be benefited from dependency injection.

Here we discuss how python module designing be benefited from dependency injection. You have been given the following requirements:

You are the developer on Blipkart, an e-commerce website. You have been asked to build a reusable SMS module that other developers will use to send SMS’es. Watertel is a telecom company that provides a REST API for sending SMS’es, and your module needs to integrate with Watertel. Your module will be used by other modules such as logistics.py and orders.py.

The client modules have a dependency on our module to complete a portion of its work which is – sending an SMS. So our module’s object has to be injected in the client’s module through the respective modules do not know each other directly.

Why Dependency Injection?

The essence of dependency injection is that it allows the client to remove all knowledge of a concrete implementation that it needs to use. This helps isolate the client from the impact of design changes and defects. It promotes reusability, testability and maintainability.

Basically, instead of having your objects creating a dependency or asking a factory object to make one for them, you pass the needed dependencies into the object externally, and you make it somebody else’s problem. This “someone” is either an object further up the dependency graph, or a dependency injector (framework) that builds the dependency graph. A dependency as I’m using it here is any other object the current object needs to hold a reference to.

The client modules ie. orders.py and logistics.py can import our module’s object and then inject it where it needs the module to perform its task.

This will explain how it should look like.

# In sms.py:
from django.conf import settings
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)

and in orders.py:

# orders.py
from sms import sms_client
...
# when you need to send an sms
sms_client.send_sms(phone_number, message)

How does Dependency Injection help us?

1. 1. Dependency Injection decreases coupling between a class and its dependency.
2. 2.Because dependency injection doesn’t require any change in code behavioDependencybe applied to legacy code as a refactoring. The result is clients that are more independent and that are easier to unit test in isolation using stubs or mock objects that simulate other objects not under test. This ease of testing is often the first benefit noticed when using dependency injection.
3. 3. To free modules from assumptions about how other systems do what they do and instead rely on contract.
4. 4. To prevent side effects when replacing a module.

So in all, dependency injection is a great practice to make our modules more testable, maintainable and scalable and we should use it more often to make our lives easier as developers.

Exception Handling In Python

Now we will have a detailed discussion on the Exception Handling in Python. We will discuss in detail, the implementation of a robust SMS module.

To start sending SMS’es, we need to first login to Watertel, store the accessToken and the time when the accessToken expires.

# sms.py
class SmsClient:
    def __init__(self, url, username, password):
        self.url = url
        self.username = username
        self.password = password
        self.access_token = None
        self.expires_at = 0
    def _login(self):
        # TODO - Make HTTPS request, get accessToken and expiresAt
        # TODO - error handling, check status codes
        self.access_token = response["accessToken"]
        self.expires_at = get_current_time() + response["expiry"]

You don’t expect or want client developers to think about login, hence we make the login method private (start with the underscore).
Now we need to decide who calls the _login method. One easy (and wrong) choice, is calling the _login method from our constructor. Constructors must not do real work, but only aid in initialization. If we make network calls in the constructor, it makes the object difficult to test, and difficult to reuse.
So _login cannot be called by clients, and cannot be called in the constructor. Which leaves us with only one choice – call _login from the send_sms method. And while we are at it, we will add another wrapper to directly get us the access_token without worrying about expiry. This also gives the necessary modularity to our code.

def _get_access_token(self):
    if (get_current_time() > self.expires_at):
        self._login()
    return self.access_token
def send_sms(self, phone_number, message):
    access_token = self._get_access_token()
    # TODO: use the access_token to make an API call and send the SMS

At this point we can start writing code to send the SMS. We will assume a make_request function that returns the parsed JSON object and the http status code.

def send_sms(self, phone_number, message):
    access_token = self._get_access_token()
    status_code, response = _make_http_request(access_token, phone_number, message)
    ...

Error Handling in Python – Status Codes

What should we do with the status_code? One option is to return the status_code and let our clients handle it. And that is a bad idea.
We don’t want our clients to know how we are sending the SMS. This is important – if the clients know how you send the SMS, you cannot change how your module works in the future. When we return the HTTP status code, we are telling them indirectly how our module works. This is called a leaky abstraction, and you want to avoid it as much as possible (though you can’t eliminate it completely).
So to give our code the much-needed abstraction, we will not return the status code, but we still want our clients to do some error handling. Since our clients are calling a python method, they expect to receive errors the pythonic way – using Exception Handling in Python.
When it comes to error handling, you need to be clear whose problem it is. It’s either the client developer’s fault or your module’s fault. A problem with your module’s dependency (i.e. Watertel server down) is also your module’s fault – because the client developer doesn’t know Watertel even exists.
We will create an exception class for each possible error in the module –

• BadInputError – a base error for incorrect inputs
• InvalidPhoneNumberError – when the phone number is malformed or wrong
• InvalidMessageError – when the message is longer than 140 characters
• SmsException – when it’s our modules fault and we cannot send the SMS. This tells our clients that calling send_sms again is safe and may work.

On implementing python exception handling, here is how our code looks like:

def _validate_phone_number(self, phone_number):
    if not phone_number:
        raise InvalidPhoneNumberError("Empty phone number")
    phone_number = phone_number.strip()
    if (len(phone_number) > 10):
        raise InvalidPhoneNumberError("Phone number too long")
    # TODO add more such checks
def _validate_message(self, message):
    if not message:
        raise InvalidMessageError("Empty message")
    if (len(message) > 140):
        raise InvalidMessageError("Message too long")
def send_sms(self, phone_number, message):
    self._validate_phone_number(phone_number)
    self._validate_message(message)
    access_token = self._get_access_token()
    status_code, response = _make_http_request(access_token, phone_number, message)
    if (status_code == 400):
        # This is Watertel telling us the input is incorrect
        # If it is possible, we should read the error message
        # and try to convert it to a proper error
        # We will just raise the generic BadInputError
        raise BadInputError(response.error_message)
     elif (status_code in (300, 301, 302, 401, 403)):
         # These status codes indicate something is wrong
         # with our module's logic. Retrying won't help,
         # we will keep getting the same status code
         # 3xx is a redirect, indicate a wrong url
         # 401, 403 have got to do with the access_token being wrong
         # We don't want our clients to retry, so we raise RuntimeError
         raise RuntimeError(response.error_message)
      elif (status_code > 500):
         # This is a problem with Watertel
         # This is beyond our control, and perhaps retrying would help
         # We indicate this by raising SmsException
         raise SmsException(response.error_message)

Error Handling in Python – Exceptions

Apart from error handling of status codes, there’s one more thing that is missing – handling any exceptions that are raised by _make_request. Assuming you are using the wonderful requests library, this is the list of exceptions that can be thrown.
You can categorize these exceptions into two categories – safe to retry v/s not safe to retry. If it is safe to retry, wrap the exception in a SMSException and raise it. Otherwise, just let the exception propagate.
In our case, ConnectTimeout is safe to retry. ReadTimeout indicates the request made it to the server, but the server did not respond timely. In such cases, you can’t be sure if the SMS was sent or not. If it is critical the SMS be sent, you should retry. If your end users would be annoyed receiving multiple SMSes, then do not retry.
You can handle these exceptions inside the _make_request method.

def _make_http_request(access_token, phone_number, message):
    try:
        data = {"message": message, "phone": phone_number, "priority": self.priority}
        url = "%s?accessToken=%s" % (self.url, access_token)
        r = requests.post(url, json=data)
        return (r.status_code, r.json())
    except ConnectTimeout:
        raise SmsException("Connection timeout trying to send SMS")

This should have given you a gist of how exception handling in python can help build a cleaner and more robust code.
That covers most of the implementation of our module. We won’t get into the code that actually makes the HTTP request – you can use the requests library for that.

So,

1. Don’t leak inner workings of your module to your clients, otherwise, you can never refactor or change your implementation.
2. Raise appropriate Errors to inform your clients about what went wrong.

Open Closed Principle in Python

The open/closed principle teaches us not to modify our already working code just for adding new features.

New Requirements – Retry Failed SMS’es

We now have a feature request from the product managers:

Customers are complaining that they don’t receive SMS’es at times. Watertel recommends resending the SMS if the status code is 5xx. Hence we need to extend the sms module to support retries with exponential backoff. The first retry should be immediate, the next retry within 2s, and the third retry within 4s. If it still continues to fail, give up and don’t try further.

Of course, our product manager assumed we would take care of a few things, viz

1. We will implement this in all of the 50 modules that are sending SMS’es.
2. There would be no regression bugs due to this change.

Don’t modify the code that’s already working

Let’s start planning our change. There are two seemingly opposing views here. We don’t want client developers to change their code, not even a single character. At the same time, the SmsClient class we wrote in part IIworks great – and we don’t want to change the code that is already working either.
A little aside – let’s look at the open/closed principle. The open/closed principle tells us that we should not modify existing source code just to add new functionality. Instead, we should find ways to extend it without modifying the source code. It is a part of the SOLID Programming Principles. In fact, it stands for the ‘O’ in the SOLID principles. In our case, the open/closed principle implies that we don’t want to touch the source code for SmsClient.

Using Inheritance

This is a tricky situation, but as with everything else in software, this problem can be solved with a little bit of indirection. We need something sitting in between our clients and our SmsClient class. That something can be a derived class of SmsClient.

# This is the class we wrote in Part III
# We are not allowed to change this class
class SmsClient:
    def send_sms(self, phone_number, message):
        ...
class SmsClientWithRetry(SmsClient):
    def __init__(self, username, password):
        super(SmsClient, self).__init__(username, password)
    def send_sms(self, phone_number, message):
        # TODO: Insert retry logic here
        super(SmsClient, self).send_sms(phone_number, message)
        # TODO: Insert retry logic here
# Earlier, client was an instance of SmsClient, like this
# client = SmsClient(username, password)
# We now change it to be an instance of SmsClientWithRetry
# As a result, our client developers doesn't have to change
# They are simply importing client from sms.py
client = SmsClientWithRetry(username, password)

If you notice, using inheritance, we got ourselves a way to add retry logic without modifying the existing code that is already known to work. This is the crux of the open/closed principle. See Footnotes for a caveat on using inheritance.

Adding Retry Logic

With that done, we can now work on adding the retry logic. We don’t want to retry if our client gave us an invalid phone number or a bad message – because it is a waste of resources. Even if we retried 100 times, it won’t succeed. We also don’t want to retry if there is a logic problem in our module’s code – because our code cannot fix itself magically, and hence, retry is unlikely to help.
In other words, we only want to retry if Watertel has a problem and we believe retrying may end up delivering the message.

If you revisit our original SmsClient implementation, you will now appreciate the way we designed our exceptions. We only want to retry when we get a SmsException. We simply let the client developers deal with all the other exception types.

How do we go about writing the retry loop? The best way to write the retry logic is via a decorator. It’s been done so many times in the past, that it’s best I point you out to some libraries.

Thus, the open/closed principle tells us not to modify our already working code just to add new features. It’s okay to change the code for bug fixes, but other than that, we should look at functional/object-oriented practices to extend the existing code when we want to add new functionalities.
Thus, we saw how the application of the SOLID programming principle and the open/closed principle in specific helped us to manage retries in our system.

A Tip: In general, we should prefer composition over inheritance.

Inheritance versus Composition in Python

We will now compare the inheritance and composition to determine better ways to reuse the existing code.

We now have new requirements for our product managers:

To increase the reliability of our SMSes, we would like to add Milio to our SMS providers. To start with, we want to use Milio only for 20% of the SMSes, the remaining should still be sent via Watertel.
Milio doesn’t have a username/password. Instead, they provide an access token, and we should add the access token as a HTTP header when we make the post request.

Of course, like last time around

1. We do not want to change existing code that is already working.
2. We need to have the retry logic for Milio as well.
3. Finally, we do not want to duplicate any code.

Creating an Interface

The first obvious step is to create a new class MilioClient, with the same interface as our original SmsClient. We have covered this before, so we will just write the pseudo-code below.

class MilioClient:
    def __init__(self, url, access_token):
        self.url = url
        self.access_token = access_token
    def send_sms(self, phone_number, message):
        # Similar to send_sms method in SmsClient
        # Difference would be in the implementation of _make_request
        # We will have different JSON response,
        # and different request parameters
        print("Milio Client: Sending Message '%s' to Phone %s"
                 % (message, phone_number))

Some discussion points –

1. Should MilioClient extend SmsClient? No. Though both are making network calls, there isn’t much commonality. The request parameters are different, the response format is different.
2. Should MilioClient extend SmsClientWithRetry? Maybe. Here, the retry logic is reusable and common between the two. We can reuse that logic using inheritance, but it isn’t the right way to reuse. We will discuss this a little later in this post.

Inheritance versus Composition

The reason why we can’t inherit the SmsClientWithRetry class is that the constructor of this class expects a username and password. We don’t have that for Milio. This conflict is large because we made a mistake in the earlier chapter – we chose to build new logic using inheritance. Better late than never. We will refactor our code and this time, we will use Composition instead of Inheritance.

# We rename SmsClient to WatertelSmsClient
# This makes the intent of the class clear
# We don't change anything else in the class
class WatertelSmsClient(object):
    def send_sms(self, phone_number, message):
    ...
# This is our new class to send sms using Milio's APi
class MilioSmsClient(object):
    def send_sms(self, phone_number, message):
    ...
class SmsClientWithRetry(object):
    def __init__(self, sms_client):
        self.delegate = sms_client
    def send_sms(self, phone_number, message):
        # Insert start of retry loop
        self.delegate.send_sms(phone_number, message)
        # Insert end of retry loop
_watertel_client = SmsClient(url, username, password)
# Here, we are passing watertel_client
# But we could pass an object of MilioClient,
# and that would still give us retry behaviour
sms_client = SmsClientWithRetry(_watertel_client)

The logic to Split Traffic

Now let’s look at the next requirement – how do we split the traffic between the two implementations?
The logic is actually simple. Generate a random number between 1 and 100. If it is between 1 and 80, use WatertelSmsClient. If it is between 81 and 100, use MilioSmsClient. Since the number is random, over time we will get an 80/20 split between the implementations.

Introducing a Router class

Now that we have retry logic for Milio in place, let’s face the elephant in the room – where should we put this logic? We cannot change the existing client implementations. And of course, we can’t put this logic in MilioSmsClient or WatertelSmsClient – that logic doesn’t belong there.
The only way out is one more indirection. We create an intermediate class that is used by all existing clients. This class decides to split the logic between MilioSmsClient and WatertelSmsClient. Let’s call this class SmsRouter.

class SmsRouter:
    def send_sms(self, phone_number, message):
        # Generate random number
        # Between 1-80? Call send_sms method in SmsClient
        # Between 80-100? Call send_sms method in MilioClient
        pass

Providing Concrete Implementations to SmsRouter

Our SmsRouter needs instances of SmsClient and MilioClient. The easiest way is for SmsRouter to create the objects in the constructor.

class SmsRouter:
    def __init__(self, milio_url, milio_access_token,
                       watertel_url, watertel_username, watertel_password):
        self.milio = MilioSmsClient(milio_url, milio_access_token)
        self.watertel = WatertelSmsClient(watertel_url, watertel_username, watertel_password)

If you feel the code is ugly, you are not alone. But why is it ugly?
We are passing 6 parameters to the constructor, and are constructing objects in the constructor. If these classes require new parameters, our constructor will also change
To remove this ugliness, we pass fully constructed objects to SmsRouter, like this:

class SmsRouter:
    def __init__(self, milio, watertel):
        self.milio = milio
        self.watertel = watertel

Okay, this is a lot better, but it’s still not great. Our SmsRouter has hard coded percentages for each of the clients. We can do better.

class SmsRouter:
    def __init__(self, milio, milio_percentage, watertel, watertel_percentage):
        self.milio = milio
        self.milio_percentage = milio_percentage
        self.watertel = watertel
        self.watertel_percentage = watertel_percentage

Eeks. We again went up to four parameters. Again, why is this ugly? SmsRouter doesn’t really care about the difference between milio or watertel, but it still has variables with those names.

Generalizing SmsRouter

Let’s generalize it by passing a list of sms providers.

class SmsRouter:
    def __init__(self, providers):
        '''Providers is a list of tuples, each tuple having a provider and a percentage
           providers = [(watertel, 80), (milio, 20)]'''
        self.providers = providers
    def _pick_provider(self):
        # Pick up a provided on the basis of the percentages provided
        # For now, we will pick a random one
        return self.providers[randint(0, 1)][0]
    def send_sms(self, phone_number, message):
        provider = self._pick_provider()
        provider.send_sms(phone_number, message)

With this version of SmsRouter, we can now provide multiple implementations, and the router will intelligently delegate to the right router.

Using our SmsRouter

This is how we can now use the router:

# First, create the concrete implementations
watertel = SmsClient("https://watertel.example.com", "username", "password")
milio = MilioClient("https://milio.example.com", "secret_access_token")
# Create a router with an 80/20 split
sms = SmsRouter([(watertel, 80), (milio, 20)])
# Calling the send_sms in a loop will on an average
# send the messages using the percentages provided
for x in range(1, 10):
    sms.send_sms("99866%s" % (x * 5, ), "Test message %s" % x)

So why composition over inheritance?

Object composition has another effect on system design. Favoring object composition over inheritance helps you keep each class encapsulated and focused on one task. Your classes and class hierarchies will be small and will be less likely to grow into unmanageable monsters.

Also, we were able to add additional functionality without modifying the existing code. This is an example of composition – wrapping existing functionality in powerful ways.

Download E-Book

This is the seventh post on the Django blog series. In this post we will learn on how we can optimize the django application’s performance. The inspiration of this post has mainly been, the performance improvement of Charcha Discussion Forum. The source code for the charcha forum is available here.

For this post, like all preceding posts in this series, we are going to use Charcha’s codebase as an example and try to optimize the pages from there on out. Also, as the range for optimization is exceedingly high we would try to tackle all the domains one by one. The main focus would be to attain maximum proficiency in the backend (as it is a django application at the end of the day) and the measurement would be done through django debug toolbar

The way we are going to proceed with this is by seeing the performance of the application and see where all can we optimize it domain by domain. To perform the assessment on the site we would be using a bunch of tools, the most important one being django debug toolbar and chrome’s network tab. We are going to hit the front-end portion first and move forward from there.

Optimizing the Front-end:

Typically when you are looking to optimize the application on the front-end we should look for the entire stack to understand how our app is working. This approach should always be followed in all cases as it gives you a better perspective on how things are functioning.

Charcha uses server-side rendering instead of client-side rendering. This on its own solves a lot of problems for us. How?? When performing client-side rendering, your request loads the page markup, its CSS, and its JavaScript. The problem is that all the code is not loaded at once. Instead what happens is JavaScript loads and then makes another request, gets a response and generates the required HTML whereas, with server-side rendering, your initial request loads the page, layout, CSS, JavaScript, and content. So basically, the server does everything for you.

The moment I got to know that Charcha is rendering its template from the server, my focused turned towards looking at only caching and minification of the pages (to be honest this should be present in all applications).

* Caching Approach:

Before we start writing any code I want you to understand the working of caching in server-side rendering and how powerful it really is? As you may know, whenever there is an endpoint which returns any data, it can generally be cached. So the main question is that can we cache HTML elements? Yes, by rendering on the server we can easily cache our data.

So what’s going to happen is that the first time your client will get a response and that response will now be cached, so the next time when the same response is made not only will your browser have to do the rendering, your server will also not have to. Now that we have an approach for caching we can start implementing it. In order to perform server-side caching, we are going to use whitenoise Now to integrate whitenoise in our Django app, we just have to follow the following steps:

• Configuring static files: In Django this isa pretty common approach where you need to add a path of your static files in your settings.py file.

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.10/howto/static-files/
STATIC_ROOT = os.path.join(PROJECT_ROOT, 'staticfiles')

• Enable WhiteNoise: We need to add the whitenoise to our middleware in our settings.py. Do not forget about the hierarchy of your MIDDLEWARE_CLASSES, and whitenoise middleware should be above all middleware’s apart from Django’s SecurityMiddleware.

MIDDLEWARE_CLASSES = [
    'django.middleware.security.SecurityMiddleware',
    'whitenoise.middleware.WhiteNoiseMiddleware',
    ...
]

Now whitenoise is going to start serving all your static files but our main purpose for caching is still not done. One thing to note here is that whitenoise also supports compression which we can use.

• Whitenoise Compression and Caching support: whitenoise has a storage backend which automatically takes care of the compression of the static files and associates each file with a unique identifier so that these files can be cached forever. To exploit this feature all we have to do is add the following lines in our settings.py

# Simplified static file serving.
# https://warehouse.python.org/project/whitenoise/
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

• Handling Content Delivery Network: Although we have enabled caching we should still look at CDN to gain maximum efficiency. As whitenoise provides appropriate headers, our CDN can serve the cache files and make sure that for duplicate responses we are not hitting the application again and again, rather serve those request by itself. All we have to do is set an ENV variable as DJANGO_STATIC_HOST and access it in settings.py.

STATIC_HOST = os.environ.get('DJANGO_STATIC_HOST', '')
STATIC_URL = STATIC_HOST + '/static/'

• WhiteNoise in development: This is one of the main parts as whenever we run the runserver command, django takes over and handles all the static files on its own. So we will not be using whitenoise as effectively as we want to. For using whitenoise what we need to do is disable Django’s static file handling and allow WhiteNoise to take over the handling of all the static files. So we need to edit our settings.pyand add whitenoise.runserver_nostatic immediately above django.contrib.staticfiles.

INSTALLED_APPS = [
    ...
    # Disable Django's own staticfiles handling in favour of WhiteNoise, for
    # greater consistency between gunicorn and `./manage.py runserver`. See:
    # http://whitenoise.evans.io/en/stable/django.html#using-whitenoise-in-development
    'whitenoise.runserver_nostatic',
    'django.contrib.staticfiles',
    ...
]

• Caching images: There are a few available settings which could be used to configure whitenoise. We are going to try to cache the images for a longer duration of time so as to gain some more performance. For this all we have to do is add awhitenoise header function which would basically look for all the images and cache them for us. So we write something like:

def cache_images_forever(headers, path, url):
    """Force images to be cached forever"""
    tokens = path.split(".")
    if len(tokens) > 1:
        extension = tokens[-1].lower()
        if extension in ('png', 'jpg', 'jpeg', 'ico', 'gif'):
            headers['Cache-Control'] = 'public, max-age=315360000'
WHITENOISE_ADD_HEADERS_FUNCTION = cache_images_forever

Now our caching for the front-end is complete. Let’s move on to the second phase of front-end optimization and try to minify our files.

* Minifying the Files:

For minifying our files we are going to use spaceless template tags. Although this will not minify our files per se, it is still going to give us a better result as it would help us reduce our page weight. How?? Well, spaceless template tagsremoves whitespace between HTML tags. This includes tab characters and newlines. This is really easy to implement in Django templates. All we need to add is spaceless and close it with endspacelessin our base.html. As our base.html is going to be used everywhere as the basic skeleton, we can be sure that it will be applied to all the other templates within it as well.

Now that we are done with our front-end optimization let’s move to our back-end. This is perhaps the one place where we would be able to achieve the maximum amount of efficiency.

Optimizing the Back-end:

Ideally, the flow for optimizing your code is to move from your queries to how your Django is functioning. In queries, we need to see the number of queries, the amount of time it takes to execute the query and how our Django ORM’S are working. Internally Django already does a lot for us but we can optimize its queries even further.
We need to start scavenging the code and check the total number of query calls per page. There are a few ways to do this. The most obvious one being logging all the queries in sql.log and start reading the queries from there. There is one other way of accomplishing this, that is by using django debug toolbar.

Now django debug toolbar is a really powerful tool and extremely easy to use. It helps in debugging responses and provides us with a bunch of configurable panels that we could use. We are mainly going to use this to see how quick our queries are and if we are performing any redundant queries.

Django Debug Toolbar

So let’s first integrate the toolbar in Charcha. To install django debug toolbar we are going to follow this doc. Its pretty straightforward and easy to implement. So first we need to install the package using pip.

 $ pip install django-debug-toolbar

One thing to note here is that if you are using a virtualenv avoid using sudo. Ideally we should not be using sudo anyways if we are in a virtualenv. Now our django debug toolbar is installed. We need to now add it in our settings.py INSTALLED_APPS by adding debug_toolbar.

INSTALLED_APPS = [
    ...
    'django.contrib.staticfiles',
    ...
    'debug_toolbar',
    ...
]

Now we need to put the debug toolbar in our middleware. Since there is no preference of the django debug toolbar middleware the middleware stack we can add it wherever we see fit.

MIDDLEWARE = [
    # ...
    'debug_toolbar.middleware.DebugToolbarMiddleware',
    # ...
]

Now we need to add the url to configure the toolbar. We just need to add the url and put it in our DEBUG mode, the rest django debug toolbar will take care of, on its own.

from django.conf import settings
if settings.DEBUG:
    import debug_toolbar
    urlpatterns = [
        url(r'^__debug__/', include(debug_toolbar.urls)),
    ] + urlpatterns

There are also a few configurations that we could use but for now we are going to set it to default as its not required (with the correct version this should be done automatically).

DEBUG_TOOLBAR_PATCH_SETTINGS = False

Also, since we are using our local we need to set the following INTERNAL_IP for django debug toolbar to understand.

INTERNAL_IPS = ('127.0.0.1',)

And now we test. We can now start checking the number of queries that are being run and how much time they are taking to execute as well.

This part is actually a bit easy as all we have to do is set a benchmark on the basis of the amount of data that we have. We also need to consider the higher effect where if we have a lot of data within a lot of foreign keys, we might need to start using indexes there.

To show how we are going to refactor the code we went ahead and started seeing the performance of all the pages. Most refactoring would mostly look at how we could reduce the number of queries.

Now according to the django debug toolbar there are a lot of queries that we are making 8 queries whenever we are making our call to see the post. This would, later on, start giving us problems if we don’t eliminate a few of them. The way we are going to approach optimizing queries is in subparts as follows:

Solving N+1 queries:

We can solve the problem of N+1 queries simply by using prefetch_related and select_related in Charcha. With the two functions, we can have a tremendous performance boost as well. But first, we need to understand what exactly is it that they do and how can we implement it in Charhca.

* select_related:

select_related should be used when the object that you are selecting in a single object of a model, so something like a ForeignKey or a OneToOneField. So whenever you make such a call, select_related would do a join of the two tables and send back the result to you thereby reducing an extra query call to the ForeignKey table.
Let’s see an example to better understand how we can integrate this in Charcha.
We are going to take the example of the Post model which looks something like this:

class Post(Votable):
    ...
    objects = PostsManager()
    title = models.CharField(max_length=120)
    url = models.URLField(blank=True)
    text = models.TextField(blank=True, max_length=8192)
    submission_time = models.DateTimeField(auto_now_add=True)
    num_comments = models.IntegerField(default=0)
    ...

Do note that we have a custom manager defined, so whichever query we need to execute we can define it in our manager. Now in your first glance, you can see that our class Post is inheriting from Votable, so we now need to see what is happening in this class.

class Votable(models.Model):
    ...
    votes = GenericRelation(Vote)
    author = models.ForeignKey(User, on_delete=models.PROTECT)
    ....

Whenever we make a call to check the Author of our Post we will be doing an extra call to the database. So now we go to our custom manager and change the way are fetching the data.

class PostsManager(models.Manager):
    def get_post_with_my_votes(self, post_id, user):
        post = Post.objects\
            .annotate(score=F('upvotes') - F('downvotes'))\
            .get(pk=post_id)
        ...
        return post

If we use the django debug toolbar you would see that whenever we do a call like get_post_with_my_votes().author, we are going to be executing an extra query to the User table.

This is not required and can be rectified easily by using select_related. What do we have to do? Just add select_related to the query.

class PostsManager(models.Manager):
    def get_post_with_my_votes(self, post_id, user):
        post = Post.objects\
            .annotate(score=F('upvotes') - F('downvotes'))\
            .select_related('author').get(pk=post_id)
        ...
        return post

And that’s it. Our redundant query should be removed. Lets check it using django debug toolbar.

* prefetch_related:

We can use prefetch_related when we are going to get a set of things, so basically something like a ManyToManyField or a reverse ForeignKey. How does this help? Well, the way prefetch_related works is it makes another query and therefore reduces the redundant columns in the original object. This as of now is not really required so we are going to let it be.

* Query in a loop:

Though this is not done anywhere in Charcha but this a practise which a lot of people follow without realising its impact. Although this seems pretty basic I still feel that requires its separate section.

post_ids = Post.objects.filter(id = ?).values_list('id', flat=True)
authors = []
for post in post_ids:
    #Disaster as when we would have a lot of post_ids say around 10,000
    #we would be making that many calls, basically (n+1) calls.
    post = Author.objects.filter(id= post_ids)
    authors.append(post)

The above example is just a sample of how disasterous a query in a loop could be for a large dataset and this above example can easily be solved by the previous discussion (on select_related and prefetch_related) which we had.

Denormalization:

I recommend using denormalization only if we have some performance issues and not prematurely start optimizing it. We should check the queries before we start with our denormalization as it does not make any sense if we keep on adding more complexity if it is not impacting our performance.

The best place to explain denormalization implementation in Charcha is in the Votable model as done in the 2nd blog of this series. Why the Votable table? This is because we want to show the score i.e. upvotes – downvotes and the comments on the home page. We could make a join on the Votable table and call it from there but it would slow down our system. Instead what we are doing is, adding the fields of upvotes, downvotes and flag in the Votable model. This would in turn reduce our calls.

class Votable(models.Model):
    ...
    upvotes = models.IntegerField(default=0)
    downvotes = models.IntegerField(default=0)
    flags = models.IntegerField(default=0)
    ...

Now we can just inherit these properties in whichever models we require and from there we can move forward. Although this seems pretty easy to implement it does have a drawback. Every time there is some changes that is made, we would have to update these fields. So instead of making a JOIN we would rather have a more complex UPDATE.

WBS for Tracking Comments Hierarchy

This is also a form of denormalization. Each comment needs a reference to its parent comment and so on , so it basically makes something like the tree structure.

class Comment(Votable):
    ...
    wbs = models.CharField(max_length=30)
    ...

The problem here is that self-referential calls are exceedingly slow. So we can refactor this approach and add a new field called wbs which would help us track the comments as a tree. How would it work? Every comment would have a code, which is a dotted path. So the first comment would have the code “.0001” and the second top-level comment would have the code “.0002” and so on. Now if someone responds to the first comment it gets a code of “.0001.0001”. This would help us avoid doing self-referential queries and use wbs instead.

Now the limitation for this field is we would only allow 9999 comments at each level and the height of the wbs tree would only go till 6, which is sort of acceptable in our case. But in the case of having to go through a large database, we would have to index this field as well. We would discuss this in the next section.

Adding Indexes

Indexes is one of the many standard DB optimization techniques and django provides us with a lot of tools to add these indexes. Once we have identified which all queries are taking a long time we can use Field.db_index or Meta.index_together to add these from Django.

Before we start adding indexes we need to identify where all should we add these properly and to do that we will use django debug toolbar to see how fast we get our response. Now we are going to look at the post which we had before and we are going to track its queries. We are going to select a query which we could easily optimize indexes

SELECT ••• FROM "votes" WHERE ("votes"."type_of_vote" IN ('1', '2') AND "votes"."voter_id" = '2' AND "votes"."content_type_id" = '7' AND "votes"."object_id" = '1')

Now, this particular query is taking 1.45s to execute. Now, all we have to see is the table and which field we could add the index on. Since this query belongs to model Votes we are going to add the index on content_type_id and object_id. How?

class Vote(models.Model):
    class Meta:
        db_table = "votes"
        index_together = [
            ["content_type", "object_id"],
        ]

And that’s all we have to do. Now we run our migrations and check our performance.

Now, this query is taking only 0.4 seconds to execute and that is how easy it is to implement indexes.

Django QuerySets are LAZY

One thing to note when using django is that django querysets are lazy , what that means is queryset does not do any database activity and django will only run the query when the queryset is evaluated. So if we make a call from the Post model like

q = Post.objects.filter(...)
q = q.exclude(...)
q = q.filter(...)

This would make three separate queries which is not really required.

Caching sessions

One thing to notice using the django debug toolbar is that almost all the pages have a query made to retrieve the Django session.

The query that we are tackling is given below and since its being used in almost all the places we can simply cache it once and reap the benefits everywhere else.

SELECT ••• FROM "django_session" WHERE ("django_session"."session_key" = '''6z1ywga1kveh58ys87lfbjcp06220z47''' AND "django_session"."expire_date" > '''2017-05-03 10:56:18.957983''')

By default, Django stores the sessions in our database and expects us to occasionally prune out old entries which we would not be doing.

So on each request, we are doing a SQL query to get the session data and another to grab the User object information. This is really not required and we can easily add a cache here to reduce the load on our server. But the question still remains on which store should we use? From a higher operational point of view, introducing a distributed data store like redis is not really required. Instead, we could simply use cookie-based sessions here.

Do note that we would not be using cookie-based sessions for highly secure sites but Charcha does not need to be highly secure.

How do we implement it?

Using cookie-based sessions is very easy to do. We can simply follow the steps given in this link. All we have to do is add the following in our settings.py (or common.py as per Charcha) and see the magic happen. We will switch to storing our sessions in our cache and easily remove a SQL query from every single request to our site.

SESSION_ENGINE = "django.contrib.sessions.backends.signed_cookies"

Summary

In this post, we have discussed how one should proceed when optimizing their site. We have tackled all the domains, discussed the various techniques to optimize a Django site and talked about the tools like which django debug toolbar can be used for a sites assessment.