Designing Modules in Python - Ebook

14 Nov 2017

Designing Modules in Python (ebook)

Designing Modules in Python is part of HashedIn’s training program for junior developers to get better at design. Today, we are making it available as a free ebook.

Download E-Book

Who is this ebook for?

This book is for all those who have some experience in an object-oriented programming language. If you wish to get better at the module or class level design, then this book is definitely for you. You will be able to identify a good design from the bad after you finish reading this book
All throughout this book, python is used to introduce various design patterns. However, the knowledge of python is not mandatory. If you can understand classes and methods, you will easily be able to understand the contents of this book

How to use this book

To make the most of this book, read the requirements at the start of each chapter and write down your solution. Then read through the rest of the chapter and critique your solution.
Good design is obvious once presented. However, arriving at that solution is difficult, especially if you are new to programming. Writing down your solution is the only way to realize the shortcomings of your design. If you read the chapter without writing, you will tell yourself “That was obvious. I would have solved it in a similar manner”

History of this book

At HashedIn, we conduct a training program for junior developers to get better at design. During one of these programs, we asked developers to design a module to send SMS/text messages. We also asked them to call the SMS module from 3 different client applications. They had 30 minutes to complete this activity.
After 30 minutes, we revealed a new set of requirements. Additionally, we set 2 hard rules for them to implement these requirements:

Developers were not allowed to change the way client applications invoked their SMS module
Developers were not allowed to modify existing code. Instead, they had to write new code to address the additional requirements, without duplicating code.

20 minutes later, we again introduced a new set of requirements. And after another 20 minutes, we introduced the final set of requirements.
Initially, the developers felt the problem was very simple. They were at loss to understand why the design was even needed. Isn’t this a simple python function?
But as new requirements emerged, and because we gave them constraints (don’t change existing code) – they began to appreciate the various design principles. Some even developed a mental model to discern good design from bad.
Over time, we got the opportunity to review designs from several developers. We learned how junior developers think when asked to solve a problem. Based on these patterns, we wrote a series of blogs on how to design modules in python.

Design Modules in Python

Python Interface Design

The first module will teach you as to how you can design a reusable python module. By the end of the post, you will get a great understanding of writing reusable and extensible modules in python.

You have been given the following requirements:

You are the developer on Blipkart, an e-commerce website. You have been asked to build a reusable SMS module that other developers will use to send SMS’es. Watertel is a telecom company that provides a REST API for sending SMS’es, and your module needs to integrate with Watertel. Your module will be used by other modules such as logistics.py and orders.py.

The first goal for a module should be good Abstraction. Your module sits between client developers on one side, and the API provided by Watertel on the other side. The module’s job is to simplify sending an SMS, and to ensure that changes in Watertel do not affect your client developers.

The essence of abstractions is preserving information that is relevant in a given context, and forgetting information that is irrelevant in that context.

Watertel exposes these concepts – username, password, access_token and expiry, phone number, message, and priority. Which of these concepts are relevant to client developers?

Client developers don’t care about username and password. They don’t want to worry about expiry either. These are things our module should handle. They only care about phone number and message – “take this message, and send it to this phone number”.

Interface Design

Since developers only care about phone_number and message, our interface design is very clear – we need a send_sms(phone_number, message) function. Everything else is irrelevant.

Now, looking from an implementation perspective. We need the URL, the username, and password. We also need to manage the access token and handle its expiry. How will we get this information? Let’s carve out an interface design for our solution.

In general, if you need some information, or if you depend on other classes – you must get it via your constructor. Your class must not go about trying to find that information on its own (say from Django settings).

Here’s how our interface looks like:

# sms.py:
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass

Using our Class in Client Code

Next, in order to use our module, our clients need an object of SmsClient. But in order to create an object, they’d need username and password. Our first attempt would be something like this:

# In orders.py
from django.conf import settings
from sms import SmsClient
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)
....
sms_client.send_sms(phone_number, message)

There are two problems with this:

First, orders.py shouldn’t care how SmsClient objects are constructed. If we later need additional parameters in the constructor, we would have to change orders.py, and all classes that use SmsClient. Also, if we decide to read our settings from someplace else (say for test cases), then orders.py would have to be modified.
The second problem is that we would have to create SmsClient objects everywhere we want to use it. This is wasteful, and also leads to code duplication.

The solution is to create SmsClient object in sms.py module [see footnote 1]. Then orders.py and logistics.py can directly import the object from SMS. Here is how it looks.

# In sms.py:
from django.conf import settings
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)

and in orders.py:

# orders.py
from sms import sms_client
...
# when you need to send an sms
sms_client.send_sms(phone_number, message)

Before you start implementing, think about how your clients will use your module. It’s a good idea to actually write down the client code (i.e. code in orders.py and logistics.py) before you start implementing your module.

Now that we have our interface defined, we can start implementing our module.

Python Dependency Injection

Next step is to learn as to how python design patterns can be benefited from dependency injection.

Here we discuss how python module designing be benefited from dependency injection. You have been given the following requirements:

You are the developer on Blipkart, an e-commerce website. You have been asked to build a reusable SMS module that other developers will use to send SMS’es. Watertel is a telecom company that provides a REST API for sending SMS’es, and your module needs to integrate with Watertel. Your module will be used by other modules such as logistics.py and orders.py.

The client modules have a dependency on our module to complete a portion of its work which is – sending an SMS. So our module’s object has to be injected in the client’s module through the respective modules do not know each other directly.

Why Dependency Injection?

The essence of dependency injection is that it allows the client to remove all knowledge of a concrete implementation that it needs to use. This helps isolate the client from the impact of design changes and defects. It promotes reusability, testability and maintainability.

Basically, instead of having your objects creating a dependency or asking a factory object to make one for them, you pass the needed dependencies into the object externally, and you make it somebody else’s problem. This “someone” is either an object further up the dependency graph, or a dependency injector (framework) that builds the dependency graph. A dependency as I’m using it here is any other object the current object needs to hold a reference to.

The client modules ie. orders.py and logistics.py can import our module’s object and then inject it where it needs the module to perform its task.

This will explain how it should look like.

# In sms.py:
from django.conf import settings
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)

and in orders.py:

# orders.py
from sms import sms_client
...
# when you need to send an sms
sms_client.send_sms(phone_number, message)

How does Dependency Injection help us?

1. Dependency Injection decreases coupling between a class and its dependency.
2.Because dependency injection doesn’t require any change in code behavioDependencybe applied to legacy code as a refactoring. The result is clients that are more independent and that are easier to unit test in isolation using stubs or mock objects that simulate other objects not under test. This ease of testing is often the first benefit noticed when using dependency injection.
3. To free modules from assumptions about how other systems do what they do and instead rely on contract.
4. To prevent side effects when replacing a module.

So in all, dependency injection is a great practice to make our modules more testable, maintainable and scalable and we should use it more often to make our lives easier as developers.

Exception Handling In Python

Now we will have a detailed discussion on the Exception Handling in Python. We will discuss in detail, the implementation of a robust SMS module.

To start sending SMS’es, we need to first login to Watertel, store the accessToken and the time when the accessToken expires.

# sms.py
class SmsClient:
    def __init__(self, url, username, password):
        self.url = url
        self.username = username
        self.password = password
        self.access_token = None
        self.expires_at = 0
    def _login(self):
        # TODO - Make HTTPS request, get accessToken and expiresAt
        # TODO - error handling, check status codes
        self.access_token = response["accessToken"]
        self.expires_at = get_current_time() + response["expiry"]

You don’t expect or want client developers to think about login, hence we make the login method private (start with the underscore).
Now we need to decide who calls the _login method. One easy (and wrong) choice, is calling the _login method from our constructor. Constructors must not do real work, but only aid in initialization. If we make network calls in the constructor, it makes the object difficult to test, and difficult to reuse.
So _login cannot be called by clients, and cannot be called in the constructor. Which leaves us with only one choice – call _login from the send_sms method. And while we are at it, we will add another wrapper to directly get us the access_token without worrying about expiry. This also gives the necessary modularity to our code.

def _get_access_token(self):
    if (get_current_time() > self.expires_at):
        self._login()
    return self.access_token
def send_sms(self, phone_number, message):
    access_token = self._get_access_token()
    # TODO: use the access_token to make an API call and send the SMS

At this point we can start writing code to send the SMS. We will assume a make_request function that returns the parsed JSON object and the http status code.

def send_sms(self, phone_number, message):
    access_token = self._get_access_token()
    status_code, response = _make_http_request(access_token, phone_number, message)
    ...

Error Handling in Python – Status Codes

What should we do with the status_code? One option is to return the status_code and let our clients handle it. And that is a bad idea.
We don’t want our clients to know how we are sending the SMS. This is important – if the clients know how you send the SMS, you cannot change how your module works in the future. When we return the HTTP status code, we are telling them indirectly how our module works. This is called a leaky abstraction, and you want to avoid it as much as possible (though you can’t eliminate it completely).
So to give our code the much-needed abstraction, we will not return the status code, but we still want our clients to do some error handling. Since our clients are calling a python method, they expect to receive errors the pythonic way – using Exception Handling in Python.
When it comes to error handling, you need to be clear whose problem it is. It’s either the client developer’s fault or your module’s fault. A problem with your module’s dependency (i.e. Watertel server down) is also your module’s fault – because the client developer doesn’t know Watertel even exists.
We will create an exception class for each possible error in the module –

BadInputError – a base error for incorrect inputs
InvalidPhoneNumberError – when the phone number is malformed or wrong
InvalidMessageError – when the message is longer than 140 characters
SmsException – when it’s our modules fault and we cannot send the SMS. This tells our clients that calling send_sms again is safe and may work.

On implementing python exception handling, here is how our code looks like:

def _validate_phone_number(self, phone_number):
    if not phone_number:
        raise InvalidPhoneNumberError("Empty phone number")
    phone_number = phone_number.strip()
    if (len(phone_number) > 10):
        raise InvalidPhoneNumberError("Phone number too long")
    # TODO add more such checks
def _validate_message(self, message):
    if not message:
        raise InvalidMessageError("Empty message")
    if (len(message) > 140):
        raise InvalidMessageError("Message too long")
def send_sms(self, phone_number, message):
    self._validate_phone_number(phone_number)
    self._validate_message(message)
    access_token = self._get_access_token()
    status_code, response = _make_http_request(access_token, phone_number, message)
    if (status_code == 400):
        # This is Watertel telling us the input is incorrect
        # If it is possible, we should read the error message
        # and try to convert it to a proper error
        # We will just raise the generic BadInputError
        raise BadInputError(response.error_message)
     elif (status_code in (300, 301, 302, 401, 403)):
         # These status codes indicate something is wrong
         # with our module's logic. Retrying won't help,
         # we will keep getting the same status code
         # 3xx is a redirect, indicate a wrong url
         # 401, 403 have got to do with the access_token being wrong
         # We don't want our clients to retry, so we raise RuntimeError
         raise RuntimeError(response.error_message)
      elif (status_code > 500):
         # This is a problem with Watertel
         # This is beyond our control, and perhaps retrying would help
         # We indicate this by raising SmsException
         raise SmsException(response.error_message)

Error Handling in Python – Exceptions

Apart from error handling of status codes, there’s one more thing that is missing – handling any exceptions that are raised by _make_request. Assuming you are using the wonderful requests library, this is the list of exceptions that can be thrown.
You can categorize these exceptions into two categories – safe to retry v/s not safe to retry. If it is safe to retry, wrap the exception in a SMSException and raise it. Otherwise, just let the exception propagate.
In our case, ConnectTimeout is safe to retry. ReadTimeout indicates the request made it to the server, but the server did not respond timely. In such cases, you can’t be sure if the SMS was sent or not. If it is critical the SMS be sent, you should retry. If your end users would be annoyed receiving multiple SMSes, then do not retry.
You can handle these exceptions inside the _make_request method.

def _make_http_request(access_token, phone_number, message):
    try:
        data = {"message": message, "phone": phone_number, "priority": self.priority}
        url = "%s?accessToken=%s" % (self.url, access_token)
        r = requests.post(url, json=data)
        return (r.status_code, r.json())
    except ConnectTimeout:
        raise SmsException("Connection timeout trying to send SMS")

This should have given you a gist of how exception handling in python can help build a cleaner and more robust code.
That covers most of the implementation of our module. We won’t get into the code that actually makes the HTTP request – you can use the requests library for that.

So,

Don’t leak inner workings of your module to your clients, otherwise, you can never refactor or change your implementation.
Raise appropriate Errors to inform your clients about what went wrong.

Open Closed Principle in Python

The open/closed principle teaches us not to modify our already working code just for adding new features.

New Requirements – Retry Failed SMS’es

We now have a feature request from the product managers:

Customers are complaining that they don’t receive SMS’es at times. Watertel recommends resending the SMS if the status code is 5xx. Hence we need to extend the sms module to support retries with exponential backoff. The first retry should be immediate, the next retry within 2s, and the third retry within 4s. If it still continues to fail, give up and don’t try further.

Of course, our product manager assumed we would take care of a few things, viz

We will implement this in all of the 50 modules that are sending SMS’es.
There would be no regression bugs due to this change.

Don’t modify the code that’s already working

Let’s start planning our change. There are two seemingly opposing views here. We don’t want client developers to change their code, not even a single character. At the same time, the SmsClient class we wrote in part IIworks great – and we don’t want to change the code that is already working either.
A little aside – let’s look at the open/closed principle. The open/closed principle tells us that we should not modify existing source code just to add new functionality. Instead, we should find ways to extend it without modifying the source code. It is a part of the SOLID Programming Principles. In fact, it stands for the ‘O’ in the SOLID principles. In our case, the open/closed principle implies that we don’t want to touch the source code for SmsClient.

Using Inheritance

This is a tricky situation, but as with everything else in software, this problem can be solved with a little bit of indirection. We need something sitting in between our clients and our SmsClient class. That something can be a derived class of SmsClient.

# This is the class we wrote in Part III
# We are not allowed to change this class
class SmsClient:
    def send_sms(self, phone_number, message):
        ...
class SmsClientWithRetry(SmsClient):
    def __init__(self, username, password):
        super(SmsClient, self).__init__(username, password)
    def send_sms(self, phone_number, message):
        # TODO: Insert retry logic here
        super(SmsClient, self).send_sms(phone_number, message)
        # TODO: Insert retry logic here
# Earlier, client was an instance of SmsClient, like this
# client = SmsClient(username, password)
# We now change it to be an instance of SmsClientWithRetry
# As a result, our client developers doesn't have to change
# They are simply importing client from sms.py
client = SmsClientWithRetry(username, password)

If you notice, using inheritance, we got ourselves a way to add retry logic without modifying the existing code that is already known to work. This is the crux of the open/closed principle. See Footnotes for a caveat on using inheritance.

Adding Retry Logic

With that done, we can now work on adding the retry logic. We don’t want to retry if our client gave us an invalid phone number or a bad message – because it is a waste of resources. Even if we retried 100 times, it won’t succeed. We also don’t want to retry if there is a logic problem in our module’s code – because our code cannot fix itself magically, and hence, retry is unlikely to help.
In other words, we only want to retry if Watertel has a problem and we believe retrying may end up delivering the message.

If you revisit our original SmsClient implementation, you will now appreciate the way we designed our exceptions. We only want to retry when we get a SmsException. We simply let the client developers deal with all the other exception types.

How do we go about writing the retry loop? The best way to write the retry logic is via a decorator. It’s been done so many times in the past, that it’s best I point you out to some libraries.

Thus, the open/closed principle tells us not to modify our already working code just to add new features. It’s okay to change the code for bug fixes, but other than that, we should look at functional/object-oriented practices to extend the existing code when we want to add new functionalities.
Thus, we saw how the application of the SOLID programming principle and the open/closed principle in specific helped us to manage retries in our system.

A Tip: In general, we should prefer composition over inheritance.

Inheritance versus Composition in Python

We will now compare the inheritance and composition to determine better ways to reuse the existing code.

We now have new requirements for our product managers:

To increase the reliability of our SMSes, we would like to add Milio to our SMS providers. To start with, we want to use Milio only for 20% of the SMSes, the remaining should still be sent via Watertel.
Milio doesn’t have a username/password. Instead, they provide an access token, and we should add the access token as a HTTP header when we make the post request.

Of course, like last time around

We do not want to change existing code that is already working.
We need to have the retry logic for Milio as well.
Finally, we do not want to duplicate any code.

Creating an Interface

The first obvious step is to create a new class MilioClient, with the same interface as our original SmsClient. We have covered this before, so we will just write the pseudo-code below.

class MilioClient:
    def __init__(self, url, access_token):
        self.url = url
        self.access_token = access_token
    def send_sms(self, phone_number, message):
        # Similar to send_sms method in SmsClient
        # Difference would be in the implementation of _make_request
        # We will have different JSON response,
        # and different request parameters
        print("Milio Client: Sending Message '%s' to Phone %s"
                 % (message, phone_number))

Some discussion points –

Should MilioClient extend SmsClient? No. Though both are making network calls, there isn’t much commonality. The request parameters are different, the response format is different.
Should MilioClient extend SmsClientWithRetry? Maybe. Here, the retry logic is reusable and common between the two. We can reuse that logic using inheritance, but it isn’t the right way to reuse. We will discuss this a little later in this post.

Inheritance versus Composition

The reason why we can’t inherit the SmsClientWithRetry class is that the constructor of this class expects a username and password. We don’t have that for Milio. This conflict is large because we made a mistake in the earlier chapter – we chose to build new logic using inheritance. Better late than never. We will refactor our code and this time, we will use Composition instead of Inheritance.

# We rename SmsClient to WatertelSmsClient
# This makes the intent of the class clear
# We don't change anything else in the class
class WatertelSmsClient(object):
    def send_sms(self, phone_number, message):
    ...
# This is our new class to send sms using Milio's APi
class MilioSmsClient(object):
    def send_sms(self, phone_number, message):
    ...
class SmsClientWithRetry(object):
    def __init__(self, sms_client):
        self.delegate = sms_client
    def send_sms(self, phone_number, message):
        # Insert start of retry loop
        self.delegate.send_sms(phone_number, message)
        # Insert end of retry loop
_watertel_client = SmsClient(url, username, password)
# Here, we are passing watertel_client
# But we could pass an object of MilioClient,
# and that would still give us retry behaviour
sms_client = SmsClientWithRetry(_watertel_client)

The logic to Split Traffic

Now let’s look at the next requirement – how do we split the traffic between the two implementations?
The logic is actually simple. Generate a random number between 1 and 100. If it is between 1 and 80, use WatertelSmsClient. If it is between 81 and 100, use MilioSmsClient. Since the number is random, over time we will get an 80/20 split between the implementations.

Introducing a Router class

Now that we have retry logic for Milio in place, let’s face the elephant in the room – where should we put this logic? We cannot change the existing client implementations. And of course, we can’t put this logic in MilioSmsClient or WatertelSmsClient – that logic doesn’t belong there.
The only way out is one more indirection. We create an intermediate class that is used by all existing clients. This class decides to split the logic between MilioSmsClient and WatertelSmsClient. Let’s call this class SmsRouter.

class SmsRouter:
    def send_sms(self, phone_number, message):
        # Generate random number
        # Between 1-80? Call send_sms method in SmsClient
        # Between 80-100? Call send_sms method in MilioClient
        pass

Providing Concrete Implementations to SmsRouter

Our SmsRouter needs instances of SmsClient and MilioClient. The easiest way is for SmsRouter to create the objects in the constructor.

class SmsRouter:
    def __init__(self, milio_url, milio_access_token,
                       watertel_url, watertel_username, watertel_password):
        self.milio = MilioSmsClient(milio_url, milio_access_token)
        self.watertel = WatertelSmsClient(watertel_url, watertel_username, watertel_password)

If you feel the code is ugly, you are not alone. But why is it ugly?
We are passing 6 parameters to the constructor, and are constructing objects in the constructor. If these classes require new parameters, our constructor will also change
To remove this ugliness, we pass fully constructed objects to SmsRouter, like this:

class SmsRouter:
    def __init__(self, milio, watertel):
        self.milio = milio
        self.watertel = watertel

Okay, this is a lot better, but it’s still not great. Our SmsRouter has hard coded percentages for each of the clients. We can do better.

class SmsRouter:
    def __init__(self, milio, milio_percentage, watertel, watertel_percentage):
        self.milio = milio
        self.milio_percentage = milio_percentage
        self.watertel = watertel
        self.watertel_percentage = watertel_percentage

Eeks. We again went up to four parameters. Again, why is this ugly? SmsRouter doesn’t really care about the difference between milio or watertel, but it still has variables with those names.

Generalizing SmsRouter

Let’s generalize it by passing a list of sms providers.

class SmsRouter:
    def __init__(self, providers):
        '''Providers is a list of tuples, each tuple having a provider and a percentage
           providers = [(watertel, 80), (milio, 20)]'''
        self.providers = providers
    def _pick_provider(self):
        # Pick up a provided on the basis of the percentages provided
        # For now, we will pick a random one
        return self.providers[randint(0, 1)][0]
    def send_sms(self, phone_number, message):
        provider = self._pick_provider()
        provider.send_sms(phone_number, message)

With this version of SmsRouter, we can now provide multiple implementations, and the router will intelligently delegate to the right router.

Using our SmsRouter

This is how we can now use the router:

# First, create the concrete implementations
watertel = SmsClient("https://watertel.example.com", "username", "password")
milio = MilioClient("https://milio.example.com", "secret_access_token")
# Create a router with an 80/20 split
sms = SmsRouter([(watertel, 80), (milio, 20)])
# Calling the send_sms in a loop will on an average
# send the messages using the percentages provided
for x in range(1, 10):
    sms.send_sms("99866%s" % (x * 5, ), "Test message %s" % x)

So why composition over inheritance?

Object composition has another effect on system design. Favoring object composition over inheritance helps you keep each class encapsulated and focused on one task. Your classes and class hierarchies will be small and will be less likely to grow into unmanageable monsters.

Also, we were able to add additional functionality without modifying the existing code. This is an example of composition – wrapping existing functionality in powerful ways.

Download E-Book