In this article post, we will learn how to implement social login in a Django application by using  Social auth app Django which happens to be one of the best alternatives of the deprecated library python social auth

What is Social-Auth-App-Django?

Social auth app Django is an open source python module that is available on PyPI. This is maintained under the Python Social Auth organization. Initially, they started off with a universal python module called Python social auth which could be used in any Python-based web framework. However, at the moment they have deprecated this library.
Going further, the code base was split into several modules in order to offer better support for python web frameworks. The core logic of social authentication resides in the social core module, which is then used in other modules like – social auth app djangosocial app flask, etc.

Installation Process

Let us now create an app called authentication. This app will contain all the views/urls related to Django social authentication.
python manage.py startapp authentication
You will have to make sure that you have created a Django project and then install social-auth-app-django from PyPI.
pip install social-auth-app-django
Then, add ‘social_django’ to your installed apps.

INSTALLED_APPS = [
    ...
    'social_django',
    'authentication',
]

Now, migrate the database:

python manage.py migrate

Now, go to http://localhost:8000/admin and you should see three models under SOCIAL_DJANGO app

Django admin with social auth

Configure Authentication Backend

Now, to use social-auth-app-django’s social login, we will have to define our authentication backends in the setting.py file. Note that although we want to give our users a way to sign in through Google, we also aim to retain the normal login functionality where the username and password can be used to login to our application. In order to do that, configure the AUTHENTICATION_BACKENDS this way:

AUTHENTICATION_BACKENDS = (
    'social_core.backends.google.GoogleOAuth2',
    'django.contrib.auth.backends.ModelBackend',
)

Note that we have django.constrib.auth.backends.ModelBackend still in our AUTHENTICATION_BACKENDS. This will let us retain the usual login flow.

Template and Template Context Processors

To keep it simple, we will have a plain HTML file which has a link to redirect the user for google+ login.
Go ahead and add a templates’ folder to the authentication app and create a new index.html file in it. In this html file, we will just be putting a simple anchor tag, which will redirect to social-auth-app-django’s login URL


  
  Google+

In authentication/views.py render this template


We will now have to add context processors, which provide social auth related data to template context:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ['templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                'social_django.context_processors.backends', # this
                'social_django.context_processors.login_redirect', # and this
            ],
        },
    },
]

 

URL Configurations

So, now we have our template setup and we just need to add our index view and the Django social auth URLs in our project.
Go to the urls.py and add this code:

urlpatterns = [
    url(r'^admin/', admin.site.urls),
    url('^api/v1/', include('social_django.urls', namespace='social')),
    url('', views.index),
]

 

Obtaining Google OAuth2 Credentials

To obtain the Google OAuth2 credentials, Follow these steps:

  1. Go to Google developer console
  2. You now have to create a new project. In case you do not have any (see the drop-down at the top left of your screen)
  3. Once the project is created you will be redirected to a dashboard. You will have to click on ENABLE API button
  4. Select Google+ APIs from the Social APIs section
  5. On the next page, click the Enable button to enable Google+ API for your project
  6. To generate credentials, select the Credentials tab from the left navigation menu and hit the ‘create credentials’ button
  7. From the drop-down, select OAuth client ID.
  8. On the next screen, select the application type (web application in this case) and enter your authorized javascript origins and redirect uri

For now, just set the origin as http://localhost:8000 and the redirect URI as http://localhost:8000/complete/google-oauth2/
Hit the create button and copy the client ID and client secret

Configuration of OAuth Credentials

Now, got to setting.py and add you OAuth creadentials like this:

SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = 
SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = 

Finally, define the LOGIN_REDIRECT_URL

LOGIN_REDIRECT_URL = '//'

For setting up credentials for other OAuth2 backend, the pattern is like this:

SOCIAL_AUTH__KEY = 
SOCIAL_AUTH__SECRET = 

Usage

Now that the configuration is done, we can start off using the social login in the Django application. Go to http://localhost:8000 and click the Google+ anchor tag.
It should redirect to Google OAuth screen where it asks you to sign in. Once you sign in through your Gmail account, you should be redirected to the redirect URL.
To check if a user has been created, go to the django admin (http://localhost:8000/admin) and click on the User social auths table. You should see a new user there with the provider as goole-oauth2 and Uid as your gmail id.

Django admin social auth user data

 

Triggering The OAuth URLs from Non-Django Templates

There are chances that you want to trigger the social login URLs from a template that is not going to pass through Django’s template context processors. For instance, when you want to implement the login in an HTML template which is developed in using ReactJS.
This is not as complicated as it looks. Just mention the href in your anchor tag like this:


  Google+

The login URL for other OAuth providers is of the same pattern. If you are not sure about what the login redirect url should be, there are two ways to find the URL – either jump into their code base or try this hack –

  1. You can create a Django template and place an anchor tag with the respective URL as shown in the Templates and Template context processors section.
  2. You can remove the respective provider (in case of Google oauth2, remove social_core.backends.google.GoogleOAuth2) from the AUTHENTICATION_BACKENDS in settings.py
  3. You can click the anchor tag in a Django template. Doing so, you would see the Django’s regular error template saying “Backend not found”.
  4. Copy the current URL in the browser and use it in your non-Django template.

Summary

We learned what Social auth app Django is and that it uses social core behind the scenes. We also learned how to implement social login in Django through templates that are processed by Django and through non-Django templates, which could be developed using JS frameworks like ReactJS.

At HashedIn, we commonly deploy Django based applications on AWS Elastic Beanstalk. While EB is great, it does have some edge cases. Here is a list of things you should be aware of if you are deploying a Django application.

Aside: If you are starting a new project meant for the elastic beanstalk, Django Project Template can simplify the configuration.

Gotcha #1: Auto Scaling Group health check doesn’t work as you’d think

Elastic Beanstalk lets you configure a health check URL. This health check URL is used by the elastic load balancer to decide if instances are healthy. But, the auto scale group does not use this health check.

So if an instance health check fails for some reason – elastic load balancer will mark it as unhealthy and remove it from the load balancer. However, auto scale group still considers the instance to be healthy and doesn’t relaunch the instance.

Elastic Beanstalk keeps it this way to give you the chance to ssh into the machine to find out what is wrong. If auto scaling group terminates the machine immediately, you won’t have that option.

The fix is to configure autoscale group to use elastic load balancer based health check. Adding the following to a config file under .ebextensions will solve the problem.


Resources:
   AWSEBAutoScalingGroup:
     Type: "AWS::AutoScaling::AutoScalingGroup"
     Properties:
       HealthCheckType: "ELB"
       HealthCheckGracePeriod: "600"

Credits:

  1. EB Deployer Tips and Tricks
  2. http://www.onegeek.com.au/articles/lessons-learned-configuring-a-ha-multi-docker-container-elastic-beanstalk

Gotcha #2: Custom logs don’t work with Elastic Beanstalk

By default, the wsgi account doesn’t have write access to the current working directory, and so your log files won’t work. According to Beanstalk’s documentation, the trick is to write the log files under the /opt/python/log directory.

However, this doesn’t always work as expected. When Django creates the log file in that directory, the log file is owned by root – and hence Django cannot write to that file.

The trick is to run a small script as part of .ebextensions to fix this. Add the following content in .ebextensions/logging.config:


commands:
  01_change_permissions:
    command: chmod g+s /opt/python/log
  02_change_owner:
    command: chown root:wsgi /opt/python/log

With this change, you can now write your custom log files to this directory. As a bonus, when you fetch logs using elastic beanstalk console or the eb tool, your custom log files will also be downloaded.

[/et_pb_text][et_pb_text admin_label=”Gotcha #3: Elastic load balancer health check does not set host header” _builder_version=”3.0.86″]

Gotcha #3: Elastic load balancer health check does not set host header

Django ’s settingALLOWED_HOSTS requires you to whitelist host names that will be allowed. The problem is, elastic load balancer health check does not set hostnames when it makes requests. It instead connects directly to the private IP address of your instance, and therefore the HOST header is the private IP address of your instance.

There are several not-so-optimal solutions to the problem

Terminate health check on apache – for example, by setting the health check URL to a static file served from apache. The problem with this approach is that if Django isn’t working, health check will not report a failure

Use TCP/IP based health check – this just checks if port 80 is up. This has the same problem – if Django doesn’t work, health check will not report a failure

Set ALLOWED_HOSTS = [‘*’] – This disables Host checks altogether, opening up security issues. Also, if you mess up DNS, you can very easily send QA traffic to production.

A slightly better solution is to detect the internal IP address of the server and add it to ALLOWED_HOSTS at startup. Doing this reliably is a bit involved though. Here is a handy script that works assuming your EB environment is Linux:

def is_ec2_linux():
    """Detect if we are running on an EC2 Linux Instance
       See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
    """
    if os.path.isfile("/sys/hypervisor/uuid"):
        with open("/sys/hypervisor/uuid") as f:
            uuid = f.read()
            return uuid.startswith("ec2")
    return False
def get_linux_ec2_private_ip():
    """Get the private IP Address of the machine if running on an EC2 linux server"""
    import urllib2
    if not is_ec2_linux():
        return None
    try:
        response = urllib2.urlopen('http://169.254.169.254/latest/meta-data/local-ipv4')
        return response.read()
    except:
        return None
    finally:
        if response:
    response.close()
# ElasticBeanstalk healthcheck sends requests with host header = internal ip
# So we detect if we are in elastic beanstalk,
# and add the instances private ip address
private_ip = get_linux_ec2_private_ip()
if private_ip:
    ALLOWED_HOSTS.append(private_ip)

Depending on your situation, this may be more work than you care about – in which case you can simply set ALLOWED_HOSTS to [‘*’].

Gotcha #4: Apache Server on EB isn’t configured for performance

For performance reasons, you want text files to be compressed, usually using gzip. Internally, elastic beanstalk for python uses Apache web server, but it is not configured to gzip content.

This is easily fixed by adding yet another config file.

Also, if you are versioning your static files, you may want to set strong cache headers. By default, Apache doesn’t set these headers. This configuration file sets Cache-Control headers to any static file that has version number embedded.

Gotcha #5: Elastic Beanstalk will delete RDS database when you terminate your environment

For this reason, always create an RDS instance independently. Set the database parameters as an environment variable. In your Django settings, use dj_database_url to parse the database credentials from the environment variable.

RabbitMQ is a messaging broker, and is an excellent choice to maintain task Queues. Here is how you can configure RabbitMQ on AWS in an autoscaling load balanced environment.

Installing RabbitMQ On Ubuntu/Debian

# Add the official rabbitmq source to your apt-get sources.list
sudo sh -c "echo 'deb http://www.rabbitmq.com/debian/ testing main' >
/etc/apt/sources.list.d/rabbitmq.list";
# Install the certificate
wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
rm rabbitmq-signing-key-public.asc
# Now install the latest rabbitmq
sudo apt-get update
sudo apt-get install rabbitmq-server

The above script will download the latest version of the RabbitMQ site and install it. Here are the references for the Installation instructions.

Installing RabbitMQ on RPM-based Linux :

rpm –import https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
yum install rabbitmq-server-3.5.4-1.noarch.rpm

Install RabbitMQ-Management Plugin:

RabbitMQ management plugin is a Web UI for administration purposes. You can use it to view the queues, queue elements and a number of consumers. The management plugin is included in the RabbitMQ distribution. To enable it, use the below command.

sudo rabbitmq-plugins enable rabbitmq_management

Create an AMI of this EC2 Instance:

Once you have installed RabbitMQ and configured it, create an AMI out of it. Note the AMI ID. So that, you can use it later in the scripts to automatically scale out or scale in RabbitMQ.

Configuring RabbitMQ Cluster on AWS:

Now that we have our AMI, we will create a cluster on AWS. A cluster provides fault tolerance and load balancing.
Step 1: Create Security Groups RabbitMQ requires several ports to work. Some ports are needed for inter-node communication, others are needed between clients and RabbitMQ, and a third bucket is the HTTP based management interface.
RabbitMQ uses the following ports:

  1. 4369 for epmd
  2. 5672 and 5671
  3. 25672
  4. 15672

If you are deploying this within a VPC (which you should) – then these ports will be open to traffic from other nodes in the VPC. If you are deploying this on an EC2 classic, you will have to create a Security group that allows instances to communicate over these ports.
Step 2: Create a Load Balancer

  1. Create a new classic load balancer
  2. Set the protocol to TCP
  3. Forward port 5672 and 15672 to the instances over TCP/IP
  4. Assign security group that you created in the first step.
  5. In Health Check, ping port 5672
  6. Don’t add any instances yet, because we haven’t launched any

Step 3: Create a Launch Configuration

  1. Choose the AMI that we created earlier
  2. Expand Advanced Details, Copy paste the python script in “UserData” from the section below
  3. Note: Replace the _url value with the DNS Name of the load balancer that was created in Step 1. To get the DNS name, go to load balancer page. Then select the load balancer that is created. Below this you can see the DNS name.
  4. Click next until you reach the security group window. Then, select an existing security group and select the security group that is created in step 1.

Step 4: Create an auto-scaling Group

  1. Check “Create an Auto Scaling group from an existing launch configuration”.
  2. Choose the launch configuration that was created from the previous step.
  3. Expand Advanced Details, In Load Balancing, check to Receive traffic from Elastic Load Balancer(s). Then choose the Load balancer created in step 2.
  4. In the scaling policies, RabbitMQ machines should be scaled based on Memory Utilization.
  5. Remove 1 instances when memory utilization <= 30 for 300 seconds. Add 1 instances when memory utilization >= 70 for 60 seconds.
  6. With this configuration, you should see two instances come up in the instances page.

RabbitMQ Launch Configuration Script

This configuration, with go into the Launch Configuration as “User Data”.

Note: The variable ‘_url’ should be updated with the load balancer URL.

#!/usr/bin/env python
import json
import urllib2,base64
if __name__ == "__main__":
 prefix = ''
 from subprocess import call
 call(["rm", "-rf", "/var/tmp/aws-mon"])
 call(["rabbitmqctl", "add_vhost", "/admin"])
 call(["rabbitmqctl", "add_user", "admin", "admin"])
 call(["rabbitmqctl", "set_user_tags", "admin", "administrator"])
 call(["rabbitmqctl", "set_permissions", "-p", "/admin", "admin", ".*", ".*", ".*"])
 call(["rabbitmqctl", "set_policy", "-p", "/admin", "server-qa-cluster", ".*?", '
{"ha-mode":"all","ha-sync-mode":"automatic"}'])
 call(["rabbitmqctl", "stop_app"])
 try:
 _url = 'http://ELB-rabbitmq-QA.us-east-1.elb.amazonaws.com:15672/api/nodes'
 print prefix + 'Get json info from ..' + _url
 request = urllib2.Request(_url)
 base64string = base64.encodestring('%s:%s' % ('admin', 'admin')).replace('\n', '')
 request.add_header("Authorization", "Basic %s" % base64string)
 data = json.load(urllib2.urlopen(request))
 print prefix + 'request ok… finding for running node'
 for r in data:
 if r.get('running'):
 print prefix + 'found running node to bind..'
 print prefix + 'node name: '+ r.get('name') +'- running:' + str(r.get('running'))
 from subprocess import call
 call(["rabbitmqctl", "join_cluster",r.get('name')])
 break;
 pass
 except Exception, e:
 print prefix + 'error during add node'
 finally:
 from subprocess import call
 call(["rabbitmqctl", "start_app"])

The above code will dynamically add the upcoming instances to the cluster based on the ELB provided in the code as the URL.

What do the call methods mean?

By default, RabbitMQ stays in the reset state. We need to create users and set up the permission for the user as the administrator for future logins. The commands to set up are as follows.
The below code block creates User “Admin” with Password “Admin” – change it at your ease.

call(["rabbitmqctl", "add_vhost", "/admin"])
call(["rabbitmqctl", "add_user", "admin", "admin"])
call(["rabbitmqctl", "set_user_tags", "admin", "administrator"])
call(["rabbitmqctl", "set_permissions", "-p", "/admin", "admin", ".*", ".*", ".*"])

Once the slaves’ nodes are set up, we need to add a policy to replicate/synchronize the elements of the slave nodes. It can be done using the command in the parent node. Also, it will be applicable across all other nodes in the cluster.

call(["rabbitmqctl", "set_policy", "-p", "/admin", "server-qa-cluster", ".*?",
'{"ha-mode":"all","ha-sync-mode":"automatic"}'])

You will have to stop the service and fire a command requesting it to join one of the already running rabbitMQ services on other machines. The cluster between both the machines is automatically created when the ‘join_cluster’ request is fired.

# Stop the RabbitMQ application. As it is running as a stand-alone queue service
sudo rabbitmqctl stop_app
# Start RabbitMQ by requesting to create the cluster with the 2-machine
sudo rabbitmqctl join_cluster rabbit@<machine-2-ip> –ram
# Start the RabbitMQ application. As it is running as a stand-alone queue service
sudo rabbitmqctl start_app

To verify, go to the admin page and http://<machine-ip>:15672 you will notice 2 nodes, stating that the cluster has been created.

Authenticating REST APIs calls for selecting the right one that suits your application. There are several ways:

Use Case 1. API for Single Page Application

There are two choices for Single Page Applications:

The set of questions that need to be asked are:

  1. Should the sessions be invalidated before they expire? If Yes, Sessions must be preferred.
  2. Should the session end based on inactivity as against ending after a fixed time? If Yes, Sessions must be preferred.
  3. Should the me functionality be remembered? If Yes, Sessions must be preferred./li>
  4. Will mobile applications to use the same APIs? If yes, prefer token-based authentication (but ensure a separate API is built for these use cases)
  5. Is Your web framework protected against CSRF? Prefer token based authentication if it is a “No” or if you don’t know what CSRF is.

If token-based authentication is preferred, avoid JSON Web Tokens. JWT should be used as a short, one-time token, as against something that is reused multiple times. Alternatively, you could create a random token, store it in Redis/Memcached, and validate it on every request.

Use Case 2. API for Mobile Apps

Prefer token-based authentication. Mobile apps do not automatically maintain and send session cookies. Hence it would be far easier for a mobile app developer to set an authentication token as opposed to setting a session cookie.
Signature-based mechanisms too aren’t useful as secret keys cannot be embedded in a mobile application. Only when you have another channel, say a QR code, to pass secrets, you could use signature-based mechanisms.
A random token stored in Redis or Memcached is the most ideal approach. Ensure you use a cryptographically secure random generator in your programming.

Use Case 3. Building a Service that will be Used by Server Side Code Only

The three choices for APIs that are called from a server-side application includes:

  1. Will the API be used only internally and only by the applications you control, are the API themselves of low value? If yes, as long as you are using HTTPS, basic authentication is acceptable.
  2. Would you want the keys to be long-lived? If yes, since the keys themselves are never sent over the wire, signature-based authentication.
  3. Would you want the same APIs to be used for both mobile and web apps? If yes, since signature based auth requires clients to store secrets, prefer token-based authentication.
  4. Are you using OAuth? If yes, the decision is made for you. OAuth 1 uses signature-based authentication, whereas OAuth 2 uses token-based authentication.
  5. Do you provide a client library to access your APIs? If yes, prefer signature based auth, because you can then write the cryptography code once and provide it to all your clients.

Use Case 4. Using JSON Web Tokens

JWT works best for single use tokens. Ideally, a new JWT must be generated for each use.
Acceptable use cases:

  1. Server-to-server API calls, where the client can store a shared secret and generate a new JWT for each API call.
  2. Generate links that expire shortly – such as those used for email verification
  3. As a way for one system to provide a logged in user limited access to another system.

Use Case 5. OAuth for APIs

OAuth should be used, only if the below criteria are met:

  1. The Consumers or the user-specific data is exposed by your API
  2. Some of the user-specific data is being accessed by third-party developers that you don’t trust
  3. If You want to ask your users if they want to share their data with the third party developer

If the answer is “yes” to ALL the 3 questions, you certainly require OAuth.

Use Case 6. Compare all the Different Authentication Techniques

The Authentication Techniques for APIs Document has pros and cons for each approach.

Designing Modules in Python is part of HashedIn’s training program for junior developers to get better at design. Today, we are making it available as a free ebook.
 Download E-Book

Who is this ebook for?

This book is for all those who have some experience in an object-oriented programming language. If you wish to get better at the module or class level design, then this book is definitely for you. You will be able to identify a good design from the bad after you finish reading this book
All throughout this book, python is used to introduce various design patterns. However, the knowledge of python is not mandatory. If you can understand classes and methods, you will easily be able to understand the contents of this book

How to use this book

To make the most of this book, read the requirements at the start of each chapter and write down your solution. Then read through the rest of the chapter and critique your solution.
Good design is obvious once presented. However, arriving at that solution is difficult, especially if you are new to programming. Writing down your solution is the only way to realize the shortcomings of your design. If you read the chapter without writing, you will tell yourself “That was obvious. I would have solved it in a similar manner”

History of this book

At HashedIn, we conduct a training program for junior developers to get better at design. During one of these programs, we asked developers to design a module to send SMS/text messages. We also asked them to call the SMS module from 3 different client applications. They had 30 minutes to complete this activity.
After 30 minutes, we revealed a new set of requirements. Additionally, we set 2 hard rules for them to implement these requirements:

  1. Developers were not allowed to change the way client applications invoked their SMS module
  2. Developers were not allowed to modify existing code. Instead, they had to write new code to address the additional requirements, without duplicating code.

20 minutes later, we again introduced a new set of requirements. And after another 20 minutes, we introduced the final set of requirements.
Initially, the developers felt the problem was very simple. They were at loss to understand why the design was even needed. Isn’t this a simple python function?
But as new requirements emerged, and because we gave them constraints (don’t change existing code) – they began to appreciate the various design principles. Some even developed a mental model to discern good design from bad.
Over time, we got the opportunity to review designs from several developers. We learned how junior developers think when asked to solve a problem. Based on these patterns, we wrote a series of blogs on how to design modules in python.

Design Modules in Python

Python Interface Design

The first module will teach you as to how you can design a reusable python module. By the end of the post, you will get a great understanding of writing reusable and extensible modules in python.

You have been given the following requirements:

You are the developer on Blipkart, an e-commerce website. You have been asked to build a reusable SMS module that other developers will use to send SMS’es. Watertel is a telecom company that provides a REST API for sending SMS’es, and your module needs to integrate with Watertel. Your module will be used by other modules such as logistics.py and orders.py.


The first goal for a module should be good Abstraction. Your module sits between client developers on one side, and the API provided by Watertel on the other side. The module’s job is to simplify sending an SMS, and to ensure that changes in Watertel do not affect your client developers.

The essence of abstractions is preserving information that is relevant in a given context, and forgetting information that is irrelevant in that context.

Watertel exposes these concepts – username, password, access_token and expiry, phone number, message, and priority. Which of these concepts are relevant to client developers?

Client developers don’t care about username and password. They don’t want to worry about expiry either. These are things our module should handle. They only care about phone number and message – “take this message, and send it to this phone number”.

 

Interface Design

Since developers only care about phone_number and message, our interface design is very clear – we need a send_sms(phone_number, message) function. Everything else is irrelevant.

Now, looking from an implementation perspective. We need the URL, the username, and password. We also need to manage the access token and handle its expiry. How will we get this information? Let’s carve out an interface design for our solution.

In general, if you need some information, or if you depend on other classes – you must get it via your constructor. Your class must not go about trying to find that information on its own (say from Django settings).

Here’s how our interface looks like:

# sms.py:
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass

 

Using our Class in Client Code

Next, in order to use our module, our clients need an object of SmsClient. But in order to create an object, they’d need username and password. Our first attempt would be something like this:

# In orders.py
from django.conf import settings
from sms import SmsClient
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)
....
sms_client.send_sms(phone_number, message)

There are two problems with this:

  1. First, orders.py shouldn’t care how SmsClient objects are constructed. If we later need additional parameters in the constructor, we would have to change orders.py, and all classes that use SmsClient. Also, if we decide to read our settings from someplace else (say for test cases), then orders.py would have to be modified.
  2. The second problem is that we would have to create SmsClient objects everywhere we want to use it. This is wasteful, and also leads to code duplication.

The solution is to create SmsClient object in sms.py module [see footnote 1]. Then orders.py and logistics.py can directly import the object from SMS. Here is how it looks.

# In sms.py:
from django.conf import settings
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)

and in orders.py:

# orders.py
from sms import sms_client
...
# when you need to send an sms
sms_client.send_sms(phone_number, message)

Before you start implementing, think about how your clients will use your module. It’s a good idea to actually write down the client code (i.e. code in orders.py and logistics.py) before you start implementing your module.

Now that we have our interface defined, we can start implementing our module.

Python Dependency Injection

Next step is to learn as to how python design patterns can be benefited from dependency injection.

Here we discuss how python module designing be benefited from dependency injection. You have been given the following requirements:

You are the developer on Blipkart, an e-commerce website. You have been asked to build a reusable SMS module that other developers will use to send SMS’es. Watertel is a telecom company that provides a REST API for sending SMS’es, and your module needs to integrate with Watertel. Your module will be used by other modules such as logistics.py and orders.py.


The client modules have a dependency on our module to complete a portion of its work which is – sending an SMS. So our module’s object has to be injected in the client’s module through the respective modules do not know each other directly.

Why Dependency Injection?

The essence of dependency injection is that it allows the client to remove all knowledge of a concrete implementation that it needs to use. This helps isolate the client from the impact of design changes and defects. It promotes reusability, testability and maintainability.

Basically, instead of having your objects creating a dependency or asking a factory object to make one for them, you pass the needed dependencies into the object externally, and you make it somebody else’s problem. This “someone” is either an object further up the dependency graph, or a dependency injector (framework) that builds the dependency graph. A dependency as I’m using it here is any other object the current object needs to hold a reference to.

The client modules ie. orders.py and logistics.py can import our module’s object and then inject it where it needs the module to perform its task.

This will explain how it should look like.

# In sms.py:
from django.conf import settings
class SmsClient:
    def __init__(self, url, username, password):
        self.username = username
        self.password = password
    def send_sms(phone_number, message):
        # TODO - write code to send sms
        pass
sms_client = SmsClient(settings.sms_url, settings.sms_username, settings.sms_password)

and in orders.py:

# orders.py
from sms import sms_client
...
# when you need to send an sms
sms_client.send_sms(phone_number, message)

How does Dependency Injection help us?

1. Dependency Injection decreases coupling between a class and its dependency.

2.Because dependency injection doesn’t require any change in code behavioDependencybe applied to legacy code as a refactoring. The result is clients that are more independent and that are easier to unit test in isolation using stubs or mock objects that simulate other objects not under test. This ease of testing is often the first benefit noticed when using dependency injection.

3. To free modules from assumptions about how other systems do what they do and instead rely on contract.

4. To prevent side effects when replacing a module.

So in all, dependency injection is a great practice to make our modules more testable, maintainable and scalable and we should use it more often to make our lives easier as developers.

Exception Handling In Python

Now we will have a detailed discussion on the Exception Handling in Python.

We will discuss in detail, the implementation of a robust SMS module.

To start sending SMS’es, we need to first login to Watertel, store the accessToken and the time when the accessToken expires.

# sms.py
class SmsClient:
    def __init__(self, url, username, password):
        self.url = url
        self.username = username
        self.password = password
        self.access_token = None
        self.expires_at = 0
    def _login(self):
        # TODO - Make HTTPS request, get accessToken and expiresAt
        # TODO - error handling, check status codes
        self.access_token = response["accessToken"]
        self.expires_at = get_current_time() + response["expiry"]

You don’t expect or want client developers to think about login, hence we make the login method private (start with the underscore).
Now we need to decide who calls the _login method. One easy (and wrong) choice, is calling the _login method from our constructor. Constructors must not do real work, but only aid in initialization. If we make network calls in the constructor, it makes the object difficult to test, and difficult to reuse.
So _login cannot be called by clients, and cannot be called in the constructor. Which leaves us with only one choice – call _login from the send_sms method. And while we are at it, we will add another wrapper to directly get us the access_token without worrying about expiry. This also gives the necessary modularity to our code.

def _get_access_token(self):
    if (get_current_time() > self.expires_at):
        self._login()
    return self.access_token
def send_sms(self, phone_number, message):
    access_token = self._get_access_token()
    # TODO: use the access_token to make an API call and send the SMS

At this point we can start writing code to send the SMS. We will assume a make_request function that returns the parsed JSON object and the http status code.

def send_sms(self, phone_number, message):
    access_token = self._get_access_token()
    status_code, response = _make_http_request(access_token, phone_number, message)
    ...

Error Handling in Python – Status Codes

What should we do with the status_code? One option is to return the status_code and let our clients handle it. And that is a bad idea.
We don’t want our clients to know how we are sending the SMS. This is important – if the clients know how you send the SMS, you cannot change how your module works in the future. When we return the HTTP status code, we are telling them indirectly how our module works. This is called a leaky abstraction, and you want to avoid it as much as possible (though you can’t eliminate it completely).
So to give our code the much-needed abstraction, we will not return the status code, but we still want our clients to do some error handling. Since our clients are calling a python method, they expect to receive errors the pythonic way – using Exception Handling in Python.
When it comes to error handling, you need to be clear whose problem it is. It’s either the client developer’s fault or your module’s fault. A problem with your module’s dependency (i.e. Watertel server down) is also your module’s fault – because the client developer doesn’t know Watertel even exists.
We will create an exception class for each possible error in the module –

On implementing python exception handling, here is how our code looks like:

def _validate_phone_number(self, phone_number):
    if not phone_number:
        raise InvalidPhoneNumberError("Empty phone number")
    phone_number = phone_number.strip()
    if (len(phone_number) > 10):
        raise InvalidPhoneNumberError("Phone number too long")
    # TODO add more such checks
def _validate_message(self, message):
    if not message:
        raise InvalidMessageError("Empty message")
    if (len(message) > 140):
        raise InvalidMessageError("Message too long")
def send_sms(self, phone_number, message):
    self._validate_phone_number(phone_number)
    self._validate_message(message)
    access_token = self._get_access_token()
    status_code, response = _make_http_request(access_token, phone_number, message)
    if (status_code == 400):
        # This is Watertel telling us the input is incorrect
        # If it is possible, we should read the error message
        # and try to convert it to a proper error
        # We will just raise the generic BadInputError
        raise BadInputError(response.error_message)
     elif (status_code in (300, 301, 302, 401, 403)):
         # These status codes indicate something is wrong
         # with our module's logic. Retrying won't help,
         # we will keep getting the same status code
         # 3xx is a redirect, indicate a wrong url
         # 401, 403 have got to do with the access_token being wrong
         # We don't want our clients to retry, so we raise RuntimeError
         raise RuntimeError(response.error_message)
      elif (status_code > 500):
         # This is a problem with Watertel
         # This is beyond our control, and perhaps retrying would help
         # We indicate this by raising SmsException
         raise SmsException(response.error_message)

Error Handling in Python – Exceptions

Apart from error handling of status codes, there’s one more thing that is missing – handling any exceptions that are raised by _make_request. Assuming you are using the wonderful requests library, this is the list of exceptions that can be thrown.
You can categorize these exceptions into two categories – safe to retry v/s not safe to retry. If it is safe to retry, wrap the exception in a SMSException and raise it. Otherwise, just let the exception propagate.
In our case, ConnectTimeout is safe to retry. ReadTimeout indicates the request made it to the server, but the server did not respond timely. In such cases, you can’t be sure if the SMS was sent or not. If it is critical the SMS be sent, you should retry. If your end users would be annoyed receiving multiple SMSes, then do not retry.
You can handle these exceptions inside the _make_request method.

def _make_http_request(access_token, phone_number, message):
    try:
        data = {"message": message, "phone": phone_number, "priority": self.priority}
        url = "%s?accessToken=%s" % (self.url, access_token)
        r = requests.post(url, json=data)
        return (r.status_code, r.json())
    except ConnectTimeout:
        raise SmsException("Connection timeout trying to send SMS")

This should have given you a gist of how exception handling in python can help build a cleaner and more robust code.
That covers most of the implementation of our module. We won’t get into the code that actually makes the HTTP request – you can use the requests library for that.

So,

  1. Don’t leak inner workings of your module to your clients, otherwise, you can never refactor or change your implementation.
  2. Raise appropriate Errors to inform your clients about what went wrong.

Open Closed Principle in Python

The open/closed principle teaches us not to modify our already working code just for adding new features.

 

New Requirements – Retry Failed SMS’es

We now have a feature request from the product managers:

Customers are complaining that they don’t receive SMS’es at times. Watertel recommends resending the SMS if the status code is 5xx. Hence we need to extend the sms module to support retries with exponential backoff. The first retry should be immediate, the next retry within 2s, and the third retry within 4s. If it still continues to fail, give up and don’t try further.

Of course, our product manager assumed we would take care of a few things, viz

  1. We will implement this in all of the 50 modules that are sending SMS’es.
  2. There would be no regression bugs due to this change.

 

Don’t modify the code that’s already working

Let’s start planning our change. There are two seemingly opposing views here. We don’t want client developers to change their code, not even a single character. At the same time, the SmsClient class we wrote in part IIworks great – and we don’t want to change the code that is already working either.
A little aside – let’s look at the open/closed principle. The open/closed principle tells us that we should not modify existing source code just to add new functionality. Instead, we should find ways to extend it without modifying the source code. It is a part of the SOLID Programming Principles. In fact, it stands for the ‘O’ in the SOLID principles. In our case, the open/closed principle implies that we don’t want to touch the source code for SmsClient.

Using Inheritance

This is a tricky situation, but as with everything else in software, this problem can be solved with a little bit of indirection. We need something sitting in between our clients and our SmsClient class. That something can be a derived class of SmsClient.

# This is the class we wrote in Part III
# We are not allowed to change this class
class SmsClient:
    def send_sms(self, phone_number, message):
        ...
class SmsClientWithRetry(SmsClient):
    def __init__(self, username, password):
        super(SmsClient, self).__init__(username, password)
    def send_sms(self, phone_number, message):
        # TODO: Insert retry logic here
        super(SmsClient, self).send_sms(phone_number, message)
        # TODO: Insert retry logic here
# Earlier, client was an instance of SmsClient, like this
# client = SmsClient(username, password)
# We now change it to be an instance of SmsClientWithRetry
# As a result, our client developers doesn't have to change
# They are simply importing client from sms.py
client = SmsClientWithRetry(username, password)

If you notice, using inheritance, we got ourselves a way to add retry logic without modifying the existing code that is already known to work. This is the crux of the open/closed principle. See Footnotes for a caveat on using inheritance.

Adding Retry Logic

With that done, we can now work on adding the retry logic. We don’t want to retry if our client gave us an invalid phone number or a bad message – because it is a waste of resources. Even if we retried 100 times, it won’t succeed. We also don’t want to retry if there is a logic problem in our module’s code – because our code cannot fix itself magically, and hence, retry is unlikely to help.
In other words, we only want to retry if Watertel has a problem and we believe retrying may end up delivering the message.

If you revisit our original SmsClient implementation, you will now appreciate the way we designed our exceptions. We only want to retry when we get a SmsException. We simply let the client developers deal with all the other exception types.

How do we go about writing the retry loop? The best way to write the retry logic is via a decorator. It’s been done so many times in the past, that it’s best I point you out to some libraries.

Thus, the open/closed principle tells us not to modify our already working code just to add new features. It’s okay to change the code for bug fixes, but other than that, we should look at functional/object-oriented practices to extend the existing code when we want to add new functionalities.
Thus, we saw how the application of the SOLID programming principle and the open/closed principle in specific helped us to manage retries in our system.

A Tip: In general, we should prefer composition over inheritance.

Inheritance versus Composition in Python

We will now compare the inheritance and composition to determine better ways to reuse the existing code.

We now have new requirements for our product managers:

To increase the reliability of our SMSes, we would like to add Milio to our SMS providers. To start with, we want to use Milio only for 20% of the SMSes, the remaining should still be sent via Watertel.
Milio doesn’t have a username/password. Instead, they provide an access token, and we should add the access token as a HTTP header when we make the post request.

Of course, like last time around

  1. We do not want to change existing code that is already working.
  2. We need to have the retry logic for Milio as well.
  3. Finally, we do not want to duplicate any code.

Creating an Interface

The first obvious step is to create a new class MilioClient, with the same interface as our original SmsClient. We have covered this before, so we will just write the pseudo-code below.

class MilioClient:
    def __init__(self, url, access_token):
        self.url = url
        self.access_token = access_token
    def send_sms(self, phone_number, message):
        # Similar to send_sms method in SmsClient
        # Difference would be in the implementation of _make_request
        # We will have different JSON response,
        # and different request parameters
        print("Milio Client: Sending Message '%s' to Phone %s"
                 % (message, phone_number))

Some discussion points –

  1. Should MilioClient extend SmsClient? No. Though both are making network calls, there isn’t much commonality. The request parameters are different, the response format is different.
  2. Should MilioClient extend SmsClientWithRetry? Maybe. Here, the retry logic is reusable and common between the two. We can reuse that logic using inheritance, but it isn’t the right way to reuse. We will discuss this a little later in this post.

Inheritance versus Composition

The reason why we can’t inherit the SmsClientWithRetry class is that the constructor of this class expects a username and password. We don’t have that for Milio. This conflict is large because we made a mistake in the earlier chapter – we chose to build new logic using inheritance. Better late than never. We will refactor our code and this time, we will use Composition instead of Inheritance.

# We rename SmsClient to WatertelSmsClient
# This makes the intent of the class clear
# We don't change anything else in the class
class WatertelSmsClient(object):
    def send_sms(self, phone_number, message):
    ...
# This is our new class to send sms using Milio's APi
class MilioSmsClient(object):
    def send_sms(self, phone_number, message):
    ...
class SmsClientWithRetry(object):
    def __init__(self, sms_client):
        self.delegate = sms_client
    def send_sms(self, phone_number, message):
        # Insert start of retry loop
        self.delegate.send_sms(phone_number, message)
        # Insert end of retry loop
_watertel_client = SmsClient(url, username, password)
# Here, we are passing watertel_client
# But we could pass an object of MilioClient,
# and that would still give us retry behaviour
sms_client = SmsClientWithRetry(_watertel_client)

The logic to Split Traffic

Now let’s look at the next requirement – how do we split the traffic between the two implementations?
The logic is actually simple. Generate a random number between 1 and 100. If it is between 1 and 80, use WatertelSmsClient. If it is between 81 and 100, use MilioSmsClient. Since the number is random, over time we will get an 80/20 split between the implementations.

Introducing a Router class

Now that we have retry logic for Milio in place, let’s face the elephant in the room – where should we put this logic? We cannot change the existing client implementations. And of course, we can’t put this logic in MilioSmsClient or WatertelSmsClient – that logic doesn’t belong there.
The only way out is one more indirection. We create an intermediate class that is used by all existing clients. This class decides to split the logic between MilioSmsClient and WatertelSmsClient. Let’s call this class SmsRouter.

class SmsRouter:
    def send_sms(self, phone_number, message):
        # Generate random number
        # Between 1-80? Call send_sms method in SmsClient
        # Between 80-100? Call send_sms method in MilioClient
        pass

Providing Concrete Implementations to SmsRouter

Our SmsRouter needs instances of SmsClient and MilioClient. The easiest way is for SmsRouter to create the objects in the constructor.

class SmsRouter:
    def __init__(self, milio_url, milio_access_token,
                       watertel_url, watertel_username, watertel_password):
        self.milio = MilioSmsClient(milio_url, milio_access_token)
        self.watertel = WatertelSmsClient(watertel_url, watertel_username, watertel_password)

If you feel the code is ugly, you are not alone. But why is it ugly?
We are passing 6 parameters to the constructor, and are constructing objects in the constructor. If these classes require new parameters, our constructor will also change
To remove this ugliness, we pass fully constructed objects to SmsRouter, like this:

class SmsRouter:
    def __init__(self, milio, watertel):
        self.milio = milio
        self.watertel = watertel

Okay, this is a lot better, but it’s still not great. Our SmsRouter has hard coded percentages for each of the clients. We can do better.

class SmsRouter:
    def __init__(self, milio, milio_percentage, watertel, watertel_percentage):
        self.milio = milio
        self.milio_percentage = milio_percentage
        self.watertel = watertel
        self.watertel_percentage = watertel_percentage

Eeks. We again went up to four parameters. Again, why is this ugly? SmsRouter doesn’t really care about the difference between milio or watertel, but it still has variables with those names.

Generalizing SmsRouter

Let’s generalize it by passing a list of sms providers.

class SmsRouter:
    def __init__(self, providers):
        '''Providers is a list of tuples, each tuple having a provider and a percentage
           providers = [(watertel, 80), (milio, 20)]'''
        self.providers = providers
    def _pick_provider(self):
        # Pick up a provided on the basis of the percentages provided
        # For now, we will pick a random one
        return self.providers[randint(0, 1)][0]
    def send_sms(self, phone_number, message):
        provider = self._pick_provider()
        provider.send_sms(phone_number, message)

With this version of SmsRouter, we can now provide multiple implementations, and the router will intelligently delegate to the right router.

Using our SmsRouter

This is how we can now use the router:

# First, create the concrete implementations
watertel = SmsClient("https://watertel.example.com", "username", "password")
milio = MilioClient("https://milio.example.com", "secret_access_token")
# Create a router with an 80/20 split
sms = SmsRouter([(watertel, 80), (milio, 20)])
# Calling the send_sms in a loop will on an average
# send the messages using the percentages provided
for x in range(1, 10):
    sms.send_sms("99866%s" % (x * 5, ), "Test message %s" % x)

 

So why composition over inheritance?

Object composition has another effect on system design. Favoring object composition over inheritance helps you keep each class encapsulated and focused on one task. Your classes and class hierarchies will be small and will be less likely to grow into unmanageable monsters.

Also, we were able to add additional functionality without modifying the existing code. This is an example of composition – wrapping existing functionality in powerful ways.

 Download E-Book

In this blog post, we will discuss how we did scalability testing on Charcha, an internal discussion forum. We use JMeter – A very powerful tool capable of complex load tests, but can be very unfriendly to new users. We will start with the installation process followed by a quick introduction to the JMeter GUI and finally, we will get our hands dirty with JMeter load testing.

Configuring JMeter JUI

JMeter has a lot of things to offer but for this blog, we are going to focus on Thread groups and samplers.

Thread Groups

Thread Group is an initial stage of the test plan. The name, Thread Groups represent a group of Threads. Under this group, each thread simulates one real user requests to the server.

adding a thread group
adding a thread group

According to the figure above:

Samplers

Samplers facilitate JMeter to deliver explicit types of requests to the server. It simulates a user’s request for a page from the target server. So that, to avail POST, GET, DELETE functions on an HTTP service, the user can add HTTP Request sampler.

HTTP Request Defaults

Here we specify the configuration which we want to be applied to all the HTTP requests JMeter makes.

http request default
http request default

Listeners

Finally, we will add listeners to view the results of our test cases. There are a lot of listeners the JMeter provides for performance testing but we are going to add only two listeners – View Results Tree and Summary Report.

View result tree
Summary report

An important thing to note here is that we are going to use View Results Tree only for debugging purposes because it holds every request and response in memory. We will use the Summary Report listener for drawing load testing conclusions.

Load Testing Setup

Now that we have our GUI configured, let’s start with JMeter load testing. We will start with the scalability testing of Charcha’s home page. To do that go ahead and create some HTTP request sampler as shown below. We are going to add three HTTP requests for now – The home page, discussion page and the profile page

Profile page

Note that I have added HTTP requests for login and starting a discussion but it is disabled for now because these are POST request and we need to do some configurations for POST requests when load testing a Django application. Lets see how to do that.

CSRF Configuration for POST Requests

Authentication in JMeter is a bit tricky due to Django’s CSRF protection. To address that, JMeter provides a way to access cookies using variables.
There are two ways to enable cookies in JMeter:

  1. In the file JMeter.properties add the following line CookieManager.save.cookies=true
  2. Launch JMeter from terminal ./jmeter -JCookieManager.save.cookies=true

I would prefer the former
Now to access the cookies, JMeter provides a config element called HTTP Cookie Manager. Right click on the thread group and add the config element – HTTP Cookie Manager

Cookie Manager

Now that we have our cookie manager setup, we can save cookie variables which can be referenced like this – ${COOKIE_cookiename}
Now let’s enable our login and start discussion post request and add the csrf token so that it can be sent in the post requests.

Login

Now that we have the setup done, click on the run button and view the results in the Summary Report listener.

Things to Remember

  1. Only enable “Summary Results”. Use the treeview only for debugging purposes, because it holds every request and response in memory
  2. Always run for a fixed number of threads. Do not provide any delay / ramp-up time. Other tools also call this as “think time”
  3. You must run the tests for at least 30 minutes
  4. You only need to capture 4 things from every test run. It’s best to capture these results at a steady state – that is when you do not see the numbers changing much, usually a couple of minutes before the test ends.
    • Number of threads
    • Average response time and standard deviation
    • Throughput
    • Total duration
  5. ImportantAverage Response Time x Throughput ~= Number of Threads 10-15% deviation is okay. This is called Little’s law, and if it doesn’t hold true – something is definitely wrong in your test cases

Scalability Testing Methodology

  1. Run tests for 10 threads, 20 threads, …. 100 threads. Each run must be at least 10 minutes, and you must pass little’s law described above
  2. Throughput must increase in proportion to the number of threads. Response time must not increase as the number of threads increases
  3. If the throughput does not increase – STOP and Investigate
  4. Look at graphs for client network bandwidth, server network, I/O, CPU, Memory, database graphs, graphs for Redis/cache, number of workers configured on web servers, etc.
  5. You will find that one of these components is slowing down the entire system. Find the culprit
  6. Once you know the culprit, fix the problem
  7. Now run the tests again, and increase load till throughput becomes constant.
  8. Once you get your target throughput and an acceptable 95 percentile response time, you are done

Summary

You should now be able to create a detailed scalability test for your Django application using JMeter. Use the JMeter config elements (for example – HTTP Request Defaults) to avoid configuring repetitive elements. Remember to run your tests with a fixed number of threads for at least 30 minutes and observe the throughput and response time to find the culprit.

 

What is CAS ?

It is a single sign-on protocol for the web. Its purpose is to permit a user to access multiple applications while providing their credentials (such as user id and password) only once. It can use multiple backed identity stores like database,ldap and In- memory. It is useful when there is centralized authentication team manages the identity.

Django CAS NG

There are some available middleware with django to integrate and the most viable one django-cas-ng is a Central Authentication Service (CAS) client implementation. This project inherits from django-cas. support Single Sign Out and Can fetch Proxy Granting Ticket.

Pre Requisites

Supports CAS versions 1.0, 2.0 and 3.0. Supports Django 1.5, 1.6, 1.7 and 1.8 with User custom model Supports Python 2.7, 3.x

How to Setup Environment and Install

The project runtime needs to be setup with right dependencies. So that, project scaffold can start referencing the dependencies.
Install with pip: pip install django-cas-ng
Install the latest code: pip install https://github.com/mingchen/django-cas-ng/archive/master.zip
Install from source code: python setup.py install

Settings

Single sign-on always needs right filter type middlewares for intercepting the requests.It is always good practise to keep the configuration as a part of Django sample project. Make sure you also have the authentication middleware installed. Here’s an example:

Sample settings file for reference with configuration:


INSTALLED_APPS = ( ‘django.contrib.admin’, ‘django.contrib.auth’, ‘django.contrib.contenttypes’, ‘django.contrib.sessions’, ‘django.contrib.messages’, ‘django.contrib.staticfiles’, ‘django_cas_ng’, … )
MIDDLEWARE_CLASSES = ( ‘django.middleware.common.CommonMiddleware’, ‘django.contrib.sessions.middleware.SessionMiddleware’, ‘django.contrib.auth.middleware.AuthenticationMiddleware’, … )
AUTHENTICATION_BACKENDS = ( ‘django.contrib.auth.backends.ModelBackend’, ‘django_cas_ng.backends.CASBackend’, )

CAS_SERVER_URL needs to be defined to configure cas runtime which is a web application dpeloyed on a http java application server. Set it to the base URL of your CAS source (e.g. https://account.example.com/cas/).

Optional Settings Include:

CAS_ADMIN_PREFIX CAS_CREATE_USER CAS_LOGIN_MSG CAS_LOGGED_MSG CAS_EXTRA_LOGIN_PARAMS CAS_RENEW CAS_IGNORE_REFERER CAS_LOGOUT_COMPLETELY CAS_REDIRECT_URL CAS_RETRY_LOGIN CAS_STORE_NEXT CAS_VERSION CAS_USERNAME_ATTRIBUTE CAS_PROXY_CALLBACK CAS_FORCE_CHANGE_USERNAME_CASE CAS_APPLY_ATTRIBUTES_TO_USER

View-Wrappers Example

The settings CAS_EXTRA_LOGIN_PARAMS allows you to define a static dictionary of extra parameters to be passed on to the CAS login page. But what if you want this dictionary to be dynamic (e.g. based on user session)? Our current advice is to implement simple wrappers for our default views, like these:


from django_cas_ng import views as baseviews
@csrf_exempt def login(request, **kwargs): return _add_locale(request, baseviews.login(request, **kwargs))
def logout(request, **kwargs): return _add_locale(request, baseviews.logout(request, **kwargs))
def _add_locale(request, response): """If the given HttpResponse is a redirect to CAS, then add the proper 'locale' parameter to it (and return the modified response). If not, simply return the original response."""
if (
    isinstance(response, HttpResponseRedirect)
    and response['Location'].startswith(settings.CAS_SERVER_URL)
):
    from ourapp.some_module import get_currently_used_language
    url = response['Location']
    url += '&' if '?' in url else '&'
    url += "locale=%s" % get_currently_used_language(request)
    response['Location'] = url
return response

 

Custom Backends

The CASBackend class is heavily inspired from Django’s own RemoteUserBackend and allows for some configurability through subclassing if you need more control than django-cas-ng’s settings provide. For instance, here is an example backend that only allows some users to login through CAS:


from django_cas_ng.backends import CASBackend
class MyCASBackend(CASBackend):
    def user_can_authenticate(self, user):
        if user.has_permission(‘can_cas_login’):
            return True
    return False

 

If you need more control over the authentication mechanism of your project, then django-cas-ng’s settings provide a functionality to create your own authentication backend that inherits from django_cas_ng.backends.CASBackend and override these attributes or methods:
CASBackend.clean_username(username) CASBackend.user_can_authenticate(user) CASBackend.configure_user(user)

Summary

CAS enables single sign-on and login federation for organisations and help them centrally manage identity for multiple web applications in organisations by seamlessly integrating with them over http. This enables centralization of identity provider teams.

This is the sixth post in the Django Blog Series. In this post we will be discussing how implementing Django unit testing can assist our app Charcha, discussion forum application in Django to achieve a better quality safety net. You can find the full code for the Charcha forum here.
Before Diving into writing the unit tests for our app, we first need to understand why do we actually need to write them and what is the apt scenarios that should be taken care of with tests.

Why Should You Test Your Code?

“Quality is never an accident; it is always the result of intelligent effort.” – John Ruskin
Providing automated tests for your code is a way to repeatedly ensure, with minimal developer effort, that the code you wrote to handle a task works as advertised. I like to think of tests as my insurance policy. They generally keep me from breaking existing code & looking foolish to other people. They’re also concrete proof that the code works correctly. Without that proof, what you have is a pile of code that worked right once on your machine & that you’ll either have to hand-test again & again in the future or will break without you knowing any wiser.
This is not to say that tests solve everything. There will always be bugs in software. Maybe the tests miss a code path or a user will use something in an unexpected way. But tests give you better confidence & a safety net.

Types Of Testing

There are many different types of testing. The prominent ones this series will cover are Django unit tests and integration tests.
Unit tests cover very small, highly specific areas of code. There are usually relatively few interactions with other areas of the software. This style of testing is very useful for critical, complicated components, such as validation, importing or methods with complex business logic.
Integration tests are at the opposite end of the spectrum. These tests usually cover multiple different facets of the application working together to produce a result. They ensure that data flow is right & often handle multiple user interactions.
The main difference between these two types is not the tooling but the approach and what you choose to test. It’s also a very common thing to mix & match these two types throughout your test suite as it is appropriate. In this post, we’ll focus only on unit testing.

When should you really test?

Another point of decision is deciding whether to do test-first (a.k.a. Test Driven Development) or test-after. Test-first is where you write the necessary tests to demonstrate the proper behavior of the code Before you write the code to solve the problem at hand. Test-after is when you’ve already written the code to solve the problem, then you go back & create tests to make sure the behavior of the code you wrote is correct.
This choice comes down to personal preference. An advantage of test-driven development is that it forces you to not skimp on the tests & think about the API up front. However, it feels very unnatural at first & if you have no experience writing tests, you may be at a loss as to what to do. Test-after feels more natural but can lead to weak tests if they’re hurried & not given the proper time/effort.
Something that is always appropriate, regardless of general style, is when you get a bug report. ALWAYS create a test case first & run your tests. Make sure it demonstrates the failure, THEN go fix the bug. If your fix is correct, that new test should pass! It’s an excellent way to sanity check yourself & is a great way to get started with testing to boot.

Let’s get to it

Now that we’ve got a solid foundation on the why what & when of testing, we’re going to start diving into code. When you run python manage.py startup, this automatically creates tests.py file within your app where we can write our tests.
Before writing the test cases we need to import TestCase module and models from our models.py. Here’s the snippet for it:

from django.test import TestCase
from django.contrib.auth.models import AnonymousUser
from .models import Post, Vote, Comment, User

 

Adding Tests to Views

class DiscussionTests(TestCase):
    def setUp(self):
        self._create_users()
    def _create_users(self):
        self.ramesh = User.objects.create_user(
            username="ramesh", password="top_secret")
        self.amit = User.objects.create_user(
            username="amit", password="top_secret")
        self.swetha = User.objects.create_user(
            username="swetha", password="top_secret")
        self.anamika = AnonymousUser()
    def new_discussion(self, user, title):
        post = Post(title=title,
            text="Does not matter",
            author=user)
        post.save()
        return post

This code sets up a new test case(DiscussionTests), which you can think of as a collection of related tests. Any method written here will be run automatically & its output will be included in the testing output when we run the command.
The setUp is run before and after every test respectively. This allows you to set up a basic context or environment inside of each of your tests. This also ensures that each of your tests does not edit the data that other tests depend on. This is a basic tenet of testing, that each test should stand alone, and not affect the others. This code snippet when ran creates the respective users for User objects and a Post object. We are going to use these as the base for meaningful tests that we will write in a bit.
If at this stage we run the command: python manage.py test discussions we’ll get the result something like this:

Creating test database for alias 'default'...
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Destroying test database for alias 'default'...

Now, this is self-explanatory that no tests ran since we didn’t have any assertions in our tests right now. Before writing some of our tests we should also know about the test database that’s been created when we run our tests.
Tests that require a database (namely, model tests) will not use your “real” (production) database. Separate, blank databases are created for the tests.
Regardless of whether the tests pass or fail, the test databases are destroyed when all the tests have been executed.
You can prevent the test databases from being destroyed by using the test –keepdb option. This will preserve the test database between runs. If the database does not exist, it will first be created. Any migrations will also be applied in order to keep it up to date.

def test_double_voting(self):
        post = self.new_discussion(self.ramesh, "Ramesh's Biography")
        self.assertEquals(post.upvotes, 0)
        post.upvote(self.amit)
        post = Post.objects.get(pk=post.id)
        self.assertEquals(post.upvotes, 1)
        post.upvote(self.amit)
        post = Post.objects.get(pk=post.id)
        self.assertEquals(post.upvotes, 1)
def test_comments_ordering(self):
        _c1 = "See my Biography!"
        _c2 = "Dude, this is terrible!"
        _c3 = "Why write your biography when you haven't achieved a thing!"
        _c4 = "Seriously, that's all you have to say?"
        post = self.new_discussion(self.ramesh, "Ramesh's Biography")
        self.assertEquals(post.num_comments, 0)
        rameshs_comment = post.add_comment(_c1, self.ramesh)
        amits_comment = rameshs_comment.reply(_c2, self.amit)
        swethas_comment = rameshs_comment.reply(_c3, self.swetha)
        rameshs_response = amits_comment.reply(_c4, self.ramesh)
        comments = [c.text for c in
                    Comment.objects.best_ones_first(post.id, self.ramesh.id)]
        self.assertEquals(comments, [_c1, _c2, _c4, _c3])
        # check if num_comments in post object is updated
        post = Post.objects.get(pk=post.id)
        self.assertEquals(post.num_comments, 4)

As we discussed above, we need to determine the apt scenarios which should be handled with test cases in order to maintain quality assurance. So the above snippet demonstrates a couple of scenarios for which we wrote test cases for our discussion forum. The tests are enough self-explanatory, wherein the first one we check if there is no double voting done by a particular user. The test consists of first starting of a new discussion and then a series of upvotes and assertions checking if the result is as we expected it to be. Whereas, in the second example we are assuring the correct order and count of the comments that are being added to a discussion.
On running tests now, the result is:

Creating test database for alias 'default'...
..
----------------------------------------------------------------------
Ran 2 tests in 0.532s
OK
Destroying test database for alias 'default'...

Both of our tests passed and the test database created was destroyed.

Summary

Unit tests lay a solid foundation on which the rest of your testing process should be built. Opening your app up and taking the new feature you just developed for a test drive is always a good practice. So I think we can come down to mainly 3 reasons why unit testing our app can be a boon for us:

  1. Knowing if our code really works.
  2. It saves us a lot of pain (and time).
  3. It makes deployments a snap.

Happy unit testing folks!

Welcome to the first part of our Django blog series! If you have been looking for a way to learn-while-you-build a Django forum app, you are at the right place. This series will get you acquainted to all that’s important for you to make your very own Django app that’s also efficient. You will also learn the many how-tos of capitalizing on some Django concepts to enhance your application. The series will require you to have basic knowledge of the Django framework.
So gear up, to not only build a Django Forum app but also to build it right!


What are we building?

This series will take you through the process of building a Discussion forum, we call Charcha. The forum allows you to bring up queries/discussions on any topic and flag or vote for comments (all that should also be viable anonymously if the user wishes). Towards the end of the series, you will be able to flaunt what you build, by learning how to deploy the application via Heroku as well as Docker.

You can find the full code for the Charcha forum here.
Read on and get an overview of how you can get this app up and running!

Preview of the Django-forum Charcha
How the forum will look once we’re done with the series

What’s in store for me to learn?

In the course of the blog series, you will learn about the following concepts:

 

 

 

 

 

 

 

 

 

Summary

Read the detailed blog posts in the series and up your Django game by building this well performant discussion forum with a robust code base. In case you have any doubts or feedback for us, you can add it in the Charcha forum.
Happy coding!

Logging is one of the most crucial parts of any application. Proper logging can save countless hours and mind hassles that a developer and support team might face while debugging. Not only the developers but also the system admins and security team have a lot to gain from logging. A well-designed logging system is an important asset for any project. Django’s built-in logging system makes it very easy to set up logging for your application.

 

Django applications can easily distribute logs to multiple destinations. In an enterprise setup, it’s likely that your logs would be spread across multiple systems and you’d need to login to each of them to pull the logs.

 

This is where centralized logging shines. Following are the pros of using centralized logging:

  1. All the log data is stored in a single place, no need of pulling data from multiple servers.
  2. Log data can be searched, exported, archived without affecting the original log files.
  3. Log data can be parsed, this helps when your log data is received in multiple formats like JSON, XML, Plain Text, etc.

How to setup?

You can use a lot of services to get your logs. For the scope of this blog we’ll be focusing on:

  1. Logentries
  2. ELK Stack (Elasticsearch, Logstash, and Kibana)

Using Logentries

  1. Create a Logentries Account
  2. Choose Python in Libraries
Python Librarie
Choose Python Libraries
  1. Create a log set
Create a Log Set
Create a Log Set
  1. From pip install the python logging package for Logentries.
  2. Generate a log token
Generate a Log Token
Generate a Log Token

In your settings.py file(preferrably production settings) add the following:

LOGENTRIES_TOKEN = 'your-logentries-token'
LOGGING = {
    # ...
    'handlers': {
        # ...
        'logentries': {
            'level': 'INFO',
            'token': LOGENTRIES_TOKEN,
            'class': 'logentries.LogentriesHandler',
        },
    },
'logger': {
        # ...
        'app_name': {
            # file will accept only ERROR and higher level
            'handlers': ['file', 'logentries'],
            'level': DEBUG,
        },
    },
}

 

Using ELK Stack

ELK Stack is comprised of three products i.e. ElasticsearchLogstash, and Kibana. All of these are open source products by Elastic.

 

Elasticsearch is a distributed RESTful search engine built for the cloud and based on Lucene.

 

Logstash is an open source, server-side data processing pipeline that receives data from a multitude of sources simultaneously transforms it, and then sends it to your target stash.

 

Kibana is a browser-based analytics and search dashboard for Elasticsearch and is your window into the Elastic Stack.

 

Apart from collecting the log data, you can also generate graphical dashboards and reports using Kibana.
We will be using an Amazon EC2 instance running Ubuntu 16.04 to collect logs from a local Django app. The ELK stack would be run on a docker.

 

 

Setup EC2 Machine with ELK

  1. Launch an Amazon EC2 with Ubuntu 16.04 (with at least 3GB RAM)
  2. SSH into the machine
    sudo ssh -i ~/.ssh/ ubuntu@
    
  3. Add GPG key for official Docker repository
    sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
    
  4. Add Docker repository to APT sources
    sudo apt-add-repository 'deb https://apt.dockerproject.org/repo ubuntu-xenial main'
    
  5. Update the package database with the Docker packages
    sudo apt-get update
    
  6. Check that installation source is the Docker repo instead of the default Ubuntu 16.04 repo
    apt-cache policy docker-engine
    

    The output should be similar to

    docker-engine:
    Installed: (none)
    Candidate: 17.05.0~ce-0~ubuntu-xenial
    Version table:
       17.05.0~ce-0~ubuntu-xenial 500
          500 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
       17.04.0~ce-0~ubuntu-xenial 500
          500 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
       17.03.1~ce-0~ubuntu-xenial 500
          500 https://apt.dockerproject.org/repo ubuntu-xenial/main amd64 Packages
    
  7. Notice that docker-engine is not installed, but the candidate for installation is from the Docker repository for Ubuntu 16.04. The docker-engine version number might be different.
  8. Install Docker
    sudo apt-get install -y docker-engine
    
  9. Docker should now be installed, the daemon started, and the process enabled to start on boot. Check that it’s running
    sudo systemctl status docker
    

    The output should be similar to

    ● docker.service - Docker Application Container Engine
       Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
       Active: active (running) since Mon 2017-05-29 00:20:54 UTC; 32s ago
       Docs: https://docs.docker.com
    Main PID: 3223 (dockerd)
       CGroup: /system.slice/docker.service
             ├─3223 /usr/bin/dockerd -H fd://
             └─3228 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/li
    
  10. Docker by default requires root privileges, to avoid adding sudo to every docker command add the user to docker group
    sudo usermod -aG docker $(whoami)
    
  11. Logout from the machine and ssh again
  12. Install Docker compose
    sudo curl -o /usr/local/bin/docker-compose -L "https://github.com/docker/compose/releases/download/1.11.2/docker-compose-$(uname -s)-$(uname -m)"
    
  13. Next, we’ll set the permissions:
    sudo chmod +x /usr/local/bin/docker-compose
    
  14. Verify if the installation was successful by checking the version
    docker-compose -v
    

    Output should be

    docker-compose version 1.11.2, build dfed245
    
  15. Instead of installing Elasticsearch, Logstash, and Kibana separately we’ll be using this GitHub repo docker-elk.
  16. Setup Docker Container for ELK
    cd
    git clone https://github.com/deviantony/docker-elk
    
  17. Launch the stack
    cd docker-elk
    docker-compose up -d
    
  18. After sometime check the status of your containers
    docker ps
    

    The output should be similar to

    CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES
    8575dcd40d91        dockerelk_kibana          "/bin/sh -c /usr/l..."   10 hours ago        Up 10 hours         0.0.0.0:5601->5601/tcp                           dockerelk_kibana_1
    ef994edf6107        dockerelk_logstash        "/usr/local/bin/do..."   10 hours ago        Up 10 hours         5044/tcp, 0.0.0.0:5000->5000/tcp, 9600/tcp       dockerelk_logstash_1
    22032232767e        dockerelk_elasticsearch   "/bin/bash bin/es-..."   10 hours ago        Up 10 hours         0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   dockerelk_elasticsearch_1
    
  19. Now test if Kibana is running by going to <ec2-ip>:5601


NOTE: In case you face Elasticsearch out of memory error you would need to increase the max_map_count
vi /etc/sysctl.conf Add vm.max_map_count = 262144

Using ELK in Django

  1. In your settings.py file, add
LOGGING = {
    # ...
    'handlers': {
        # ...
        'logstash': {
            'level': 'INFO',
            'class': 'logstash.TCPLogstashHandler',
            'port': 5000,
            'host': '192.0.0.1', # IP/Name of the EC2 instance,
            'version': 1,
            'message_type': 'logstash',
            'fqdn': True,
            'tags': ['meetnotes'],
        },
    },
'logger': {
        # ...
        'app_name': {
            # file will accept only ERROR and higher level
            'handlers': ['file', 'logstash'],
            'level': DEBUG,
        },
    },
}

 

Things to Note

 

 

Summary

Centralized logging can be set up easily and it will certainly be of help when we have debugging issues. So don’t wait for the moment when you feel that life would have been easier if we had logs.