Building Scalable System for Supporting Farming Rental Business - HashedIn Technologies

Building Scalable System for Supporting Farming Rental Business

Technology - 19 Apr 2018
Vinit Kumar

Rental business in Farming is an unorganized business sector; suppliers(who own farming equipment and offer them on rent) are managing the business in an ad-hoc manner by maintaining diaries for bookings and transactions, on-demand service, limited service areas mostly due to disconnection with farmers of different area etc. From farmer’s context, this unorganized mode causes uncertainty on the availability of required types of equipment on time and may cause the delay in their farming process.

In the era of this tech-savvy world, where even government has pushed towards digitization, we should have a solution for it. To tackle this problem domain, our client approached us and we started with the idea of building a cloud-based platform(reliable and scalable) which can organize this rental business. Platform aimed to bridge the gap between farmers and suppliers via technology and ensures to provide timely and on-demand service. We called it Uber for Farmers. Uber and Ola are popular rental cab-service in India and we thought of extending their ideas in a farming sector where suppliers can declare their types of equipment, the system can help them in keeping track of availability, and farmers can book this equipment online. This will help farmers to plan ahead and book equipment accordingly and suppliers could extend and run their rental business in more organized manner.

As an outcome, within a span of 3-6 months, we were able to come-up with platform and apps for suppliers and farmers using which farmers can search & place an order for equipment and suppliers can accept and full-fill them. These apps allow suppliers to be more organized, as they can track their orders, incomes, inventories, get monthly and yearly reports on revenue etc. Similarly, farmers can search equipment, gets the suggestion for equipment based on crop cycle, select them as per their need like on an hourly basis, or acre basis or daily basis etc, place the order, track it, get order invoices etc. Later we added more sophisticated features and apps as per the business needs.

Let’s review what we have built so far

Platform

Back-end application which houses the core business logic and hosts web interface and APIs to be consumed by the apps. Presently, its a Python-Django based monolith web application with various coherent subsystems like OrderManagement, InventoryManagement, Pricing, OrderTracking, On-boarding, Notifications, Promotion etc. Of-course using Django alone is not sufficient to achieve everything, we have used plugins of Django and python, such as DRF(Django Rest Framework), Django-FSM, Arrow, jinjaSQL etc. which helped us a lot.

DjangoFSM: In any order management system, tracking order lifecycle is important and any transition in its state should be controlled and audited. FSM has helped here by providing a controlled and predictable way to perform order state transitions. It defines Django Field and decorators to help you easily define complete state flow. The transitions between state can be authored based on user permission and some business rules. eg.

# Order can be transitioned to started state only when

# 1. It is already in a confirmed state

# 2. User who has triggered this has correct permission

@transition(field=state, source=‘confirmed’, target=‘started’,

  permission=lambda instance, user: not user.has_perm(‘order.can_start’))

def start(self):

  pass

Here is the GitHub link of the project to read more.

DjangoDRF: Building rest APIs requires serializing and deserializing data models in JSON format. DRF is a good Django-plugin which helps in quickly building rest APIs from Django models.

Celery: Celery is a python based framework for facilitating background workers. The platform uses celery to run simple and workflow-oriented tasks like Order Assignment, Notification, Breakdown Checkup. Designing workflow with celery was one pain which we have done for few things.

To discuss a simple collaborative workflow modeling with celery let’s take the case of order assignment and do a little peep into how we have modeled its workflow with celery. For any new order, we have to find and assign the order to a supplier. Following are the business steps for new order assignment.

  1. Find supplier (nearby to order location, order equipment, availability of order schedule, a rating of supplier etc.)
  2. Contact each supplier sequentially, wait for their response for 2 mins.
  3. If the supplier has accepted within 2 Mins, then send a confirmation to the farmer and stop the process.
  4. Else continue with next supplier.
  5. If no one has confirmed then try the same process after some time.
  6. You can see, the process of assignment may require 2~10 Min(assuming 5 suppliers, 2 Min each) and shouldn’t be done in request thread. Also, these steps require a sleep between contacts to wait for confirmation, which can block the worker thread. So, we decided to create smaller tasks and switches between them more like recursive manner. Following are the logical celery tasks;
  1. FindSupplierTask: Takes an order and just find the list of probable suppliers.
  2. ContactSupplierTask: Takes an order and supplier and contact them by sending new order notification.
  3. DeciderTask: Acts as collaborator between contact and FindSupplier task. It takes an order and list of suppliers and then contacts them one by one.

Implementation flow with celery tasks are;

  1. order management system submits a new order to FindsupplierTask.
  2. FindSupplierTask task then finds a list of probable suppliers and then invoke DeciderTask with order_id, suppliers found and current index to zero.
  3. DeciderTask always checks if the order has been confirmed or not and if not then if suppliers list has been exhausted then stops and update status else call ContactSupplierTask with order, supplier and a timeout(which says when decider should be called again.)
  4. ContactSupplierTask then sends notification to supplier and schedule Decider task to be called after ETA given by decider in the form of timeout.

This way, every task is short lived and just do their task and then delegates the next thing to be done by other. Having a decider task acts as a coordinator between different task and takes the decision based on the output of the previous task.

Of course these details are very high-level and a lot of edge cases has been handled in this discussion, but hopefully, this should have given the basic idea. Celery is a powerful framework for running such background tasks, retrying then on failure with back-offs. Here is the link to celery documentation.

Arrow: Any web application built for the global audience should handle date-time very carefully. And as a rule of thumb, it is always better to store date-time in UTC and they convert to appropriate timezone as per user preference in the presentation layer. Python date-time is too naive and verbose for such tasks. An arrow extends python date-time and provides helpful factories and fluent APIs to play with date-times and timezones.

JinjaSQL: Platform exposes some APIs to communicate the summary of business performance to apps. Since data layer uses relational schema and normalized tables, a lot of information has been scattered in various tables which needs to join and extracted for these APIs. Also, these data need to be processed dynamically to get digestible information. Using ORM for such dynamic and processed data is complex and time-consuming. This is where we have used JinjaSQL(in-house open source project), which allows writing raw SQL using jinja templates and hence become powerful in generating dynamic SQL.

Apps

We have built a couple of apps for end users(suppliers, farmers) which allows them to easily connect with our platform. Since target segment was rural areas currently these apps android only. Architecturally apps have been designed as the native android by utilizing its excellent Architectural Components and MVVM model with a focus on minimizing bandwidth uses by utilizing read through cache approach. It uses SQLite DB to store(cache) data on mobile devices. For updates generated from the app, we have opted a simple strategy of first running the update on the server and then refresh the cached objects. This way we are able to avoid any conflict and side-effects of update which is controlled by changing business requirements.

Third Party Integrations

Followings are some third party services which have been used in the project for different reasons.

  1. Exotel: Exotel provides communication services like IVR, SMS, OTP etc. To allow orders to be started, stopped, or reporting any breakdown from basic phones or from an area with low network coverage, we have integration with Exotel which allows doing IVR calls. Operators can make toll-free IVR calls, enter their order codes and select the operation they want to perform. The platform also uses Exotel for sending SMS notifications in various languages. Exotel provides a nice dashboard for easy managing your SMS templates and tracking status of sending SMSs’.
  2. Two Factor: Our apps use OTP based authentication system. The platform has integration with TwoFactor to provide OTP service.
  3. FCM (Firebase Cloud Messaging): To communicate any update like new order notification to the supplier, start, stop, breakdown, cancellation, acceptance etc. we are using google’s FCM service which provides a reliable mechanism to push updates on mobile devices.
  4. AWS (S3, RDS, EC2, etc.): We have used AWS as a cloud provider for hosting scalable and fault-tolerant application. Following are the list of few services which we have used from AWS.
    1. RDS (Relational Database Service): AWS RDS service provides fully managed relational DB service with automated backups. We have used Mysql of RDS for our DB layer.
    2. S3(Simple Storage Service): AWS S3 services provide scalable object store services with SLA of 11 9’s. We are using this service to host all the public and private static assets.
    3. ElasticCache: Caching service used by the application to store frequently used objects and as a broker for Celery Workers.
    4. Elastic Beanstalk: AWS provides short of PASS service using beanstalk which manages EB, auto-scaling, runtime-environment etc. for your application. We are using EB to host platform services.
    5. Squealy: Since, the application is live and being used in few states of India like Rajasthan, MP, Karnataka etc. One of the obvious business need was to have reporting service to easily and timely generate and share reports to concerned persons. We have used an in-house solution, Squealy, for these reporting needs. Squealy is a Django + JinjaSQL + Celery based open source web application which can be deployed on your infrastructure and provides you interface for writing SQL using powerful syntaxes of jinjaSQL and generate reports from them. These reports can be generated in various forms like tables, charts etc. and can be downloaded or scheduled to email them on a timely basis.

Of course, this is just a 10000-feet highlight of integrations which we have and the things which have achieved. The system is online and functional with features being discovered and developed on the top.

To discuss more on this, please feel free to reach out to us at contact@hashedin.com

 

E-book on Digital Business Transformation

Share
Tweet
+1
Share