Building Scalable System for Supporting Farming Rental Business

Vinit Kumar

19 Apr 2018

Building Scalable System for Supporting Farming Rental Business

Farming equipment rental business is an unorganized sector. Suppliers (who own farming equipment and offer them on rent) mostly take orders in an ad-hoc fashion and simply maintain records in diaries.

 

This unorganized setup makes the timely availability of the required equipment almost always uncertain and thereby keeps the farmers on edge and also delay the farming process. 

 

In today’s tech-savvy world, we do have a solution for it. To tackle this problem, our client approached us with an idea which they called “Uber for farmers”.

 

We started by building a reliable and scalable cloud-based platform which can organize the farming equipment rental business. The platform aimed to bridge the gap between farmers and suppliers via technology and ensure reliable, timely and on-demand service. 

 

Taking the cue from Uber and Ola, the popular rental cab-service in India, the client came up with the thought of extending the concept to the farming sector where suppliers can declare their equipment and farmers can book this equipment online and the system can help them track the availability. 

 

Within a span of 3-6 months, we were able to come up with a platform and mobile apps for suppliers and farmers. Using the app, the farmers can search and place orders for equipment and suppliers can accept and fulfill them.

 

These apps allow suppliers to be more organized, as they can track their orders, incomes, inventories, get monthly and yearly reports on revenue etc.

 

Farmers have visibility of the available types of equipment in their neighborhood, select them from different pricing models based on duration (per hour) or dimension (per acre). Later we added more sophisticated features and apps as per the business needs.

What we have built 

Platform

Back-end application which houses the core business logic and hosts web interface and APIs to be consumed by the apps. Presently, its a Python-Django based monolith web application with various coherent subsystems like Order Management, Inventory Management, Pricing, Order Tracking, Onboarding, Notifications, Promotion etc.

 

In addition to Django, we have used plugins of Django and python, such as DRF(Django Rest Framework), Django-FSM, Arrow, jinjaSQL etc.

 

DjangoFSM

In any order management system, tracking order lifecycle is important and any transition in its state should be controlled and audited. FSM has helped here by providing a controlled and predictable way to perform order state transitions.

 

It defines Django Field and decorators to help you easily define complete state flow. The transitions between state can be authored based on user permission and some business rules. eg.

# Order can be transitioned to started state only when
# 1. It is already in a confirmed state
# 2. User who has triggered this has correct permission
@transition(field=state, source=‘confirmed’, target=‘started’,
  permission=lambda instance, user: not user.has_perm(‘order.can_start’))
def start(self):
  pass
Here is the GitHub link of the project to read more.

 

DjangoDRF

Building rest APIs requires serializing and deserializing data models in JSON format. DRF is a good Django-plugin which helps in quickly building rest APIs from Django models.

 

Celery

Celery is a python based framework for facilitating background workers. The platform uses celery to run simple and workflow-oriented tasks like Order Assignment, Notification, Breakdown Checkup. Designing workflow with celery was one pain which we have done for few things.

 

To discuss a simple collaborative workflow modeling with celery let’s take the case of order assignment and do a little peep into how we have modeled its workflow with celery. For any new order, we have to find and assign the order to a supplier. Following are the business steps for new order assignment.

 

 

  1. Find supplier (nearby to order location, order equipment, availability of order schedule, a rating of supplier etc.)
  2. Contact each supplier sequentially, wait for their response for 2 mins.
  3. If the supplier has accepted within X Mins, with “within X mins, where X can be configured from the backend”,
    then send a confirmation to the farmer and stop the process.
  4. Else continue with next supplier.
  5. If no one has confirmed then try the same process after some time.
  6. You can see, the process of assignment may require 2~10 Min(assuming 5 suppliers, 2 Min each) and shouldn’t be done in the request thread. Also, these steps require a sleep between contacts to wait for confirmation, which can block the worker thread. So, we decided to create smaller tasks and switches between them more like recursive manner. Following are the logical celery tasks;

 

  1. FindSupplierTask: Takes an order and just find the list of probable suppliers.
  2. ContactSupplierTask: Takes an order and supplier and contact them by sending new order notification.
  3. DeciderTask: Acts as a collaborator between contact and FindSupplier task. It takes an order and a list of suppliers and then contacts them one by one.

 

Implementation flow with celery tasks are;

  1. order management system submits a new order to FindsupplierTask.
  2. FindSupplierTask task then finds a list of probable suppliers and then invoke DeciderTask with order_id, suppliers found and current index to zero.
  3. DeciderTask always checks if the order has been confirmed or not and if not then if suppliers list has been exhausted then stops and update status else call ContactSupplierTask with the order, supplier and a timeout(which says when decider should be called again.)
  4. ContactSupplierTask then sends a notification to supplier and schedule Decider task to be called after ETA was given by decider in the form of timeout.

 

This way, every task has a short life during which, it performs the assigned tasks and then delegates it to the next task for subsequent action. The decider task acts as a coordinator between different task and takes the decision based on the output of the previous task.

 

These are very high-level details but it gives the basic idea. Celery is a powerful framework for running such background tasks, retrying then on failure with back-offs. Refer to celery documentation for more details.

 

Arrow

Any web application built for the global audience should handle date-time very carefully. And as a rule of thumb, it is always better to store date-time in UTC and they convert to appropriate timezone as per user preference in the presentation layer. Python date-time is too naive and verbose for such tasks. An arrow extends python date-time and provides helpful factories and fluent APIs to play with date-times and timezones.

 

JinjaSQL

Platform exposes some APIs to communicate the summary of business performance to apps. Since the data layer uses relational schema and normalized tables, a lot of information has been scattered in various tables which needs to join and extracted for these APIs.

 

Also, these data need to be processed dynamically to get digestible information. Using ORM for such dynamic and processed data is complex and time-consuming. This is where we have used JinjaSQL(in-house open source project), which allows writing raw SQL using jinja templates and hence become powerful in generating dynamic SQL.

Apps

We built the mobile apps for the suppliers, hub managers, and management and provided backend support for the farmer app. Since the target segment was rural areas, currently the mobile apps are available on Android only. 

 

Architecturally apps have been designed as the native android by utilizing its excellent Architectural Components and MVVM model with a focus on minimizing bandwidth uses by utilizing read through cache approach. It uses SQLite DB to store(cache) data on mobile devices.

 

For updates generated from the app, we have opted a simple strategy of first running the update on the server and then refresh the cached objects. This way we are able to avoid any conflict and side-effects of update which is controlled by changing business requirements.

Third Party Integrations

We have used the following third party applications/tools for various needs of the project.

  1. Exotel: Exotel provides communication services like IVR, SMS, OTP etc. To allow orders to be started, stopped, or reporting any breakdown from basic phones or from an area with low network coverage, we have integration with Exotel which allows doing IVR calls. Operators can make toll-free IVR calls, enter their order codes and select the operation they want to perform. The platform also uses Exotel for sending SMS notifications in various languages. Exotel provides a nice dashboard for easy managing your SMS templates and tracking status of sending SMSs’.
  2. Two Factor: Our apps use OTP based authentication system. The platform has integration with TwoFactor to provide OTP service.
  3. FCM (Firebase Cloud Messaging): To communicate any update like new order notification to the supplier, start, stop, breakdown, cancellation, acceptance etc. we are using google’s FCM service which provides a reliable mechanism to push updates on mobile devices.
  4. AWS (S3, RDS, EC2, etc.): We have used AWS as a cloud provider for hosting scalable and fault-tolerant application. Following are the list of few services which we have used from AWS.
    1. RDS (Relational Database Service): AWS RDS service provides fully managed relational DB service with automated backups. We have used Mysql of RDS for our DB layer.
    2. S3(Simple Storage Service): AWS S3 services provide scalable object store services with SLA of 11 9’s. We are using this service to host all the public and private static assets.
    3. ElasticCache: Caching service used by the application to store frequently used objects and as a broker for Celery Workers.
    4. Elastic Beanstalk: AWS provides short of PASS service using beanstalk which manages EB, auto-scaling, runtime-environment etc. for your application. We are using EB to host platform services.
    5. Squealy: One of the obvious business need was to have a reporting service to easily and timely generate and share reports to concerned persons. We have used an in-house solution, Squealy, for these reporting needs. Squealy is a Django + JinjaSQL + Celery based open source web application which can be deployed on your infrastructure and provides you interface for writing SQL using powerful syntaxes of jinjaSQL and generate reports from them. These reports can be generated in various forms like tables, charts etc. and can be downloaded or scheduled to email them on a timely basis.

 

 

This is just a 10000-feet highlight of integrations which we have and the things which have achieved. The system is online and functional with features being discovered and developed on the top. To discuss more on this, please feel free to reach out to us at contact@hashedin.com


Have a question?

Need Technology advice?

Connect

+1 669 253 9011

contact@hashedin.com

facebook twitter linkedIn youtube