Simple SMS Api and Collector using FastAPI, Spring Boot, Kafka and MongoDB

This article is just the proof of concept for developing various complex integration systems which you can implement in any programming language via the support of Apache Kafka.

Developing SMS API directly integrate with SMSC using SMPP protocol it’s kind of complicated. However, we can reduce system complexity by providing a separate layer of concern, we can rescue and decouple our services. The SMS delivery and triggering mostly are asynchronous, so using Kafka as a message broker is very suitable for this context. There is a java library that supports this task which we can leverage.

The important parts of the SMS system are CP, MT , MO and can be DN.

• CP: Content Provider, the actors who are using the service, mostly are sending SMS to their client, and that SMS can be freed, charged or for campaign purposes. They also need to get notifications when subscribers reply to their SMS mostly via shortcode.
• MT: Mobile Terminating, refer a message being sent to a mobile handset. It is generated by CP , sending the SMS to Subscribers.
• MO: Mobile Originating, refers to a message being sent from a mobile handset. it is generated by the subscriber to reply to SMS to a specific sender or shortcode.
• DN: delivery notification, mostly trigger back to CP

Below is the High Level Design of the system :

System HLD

System components :

• API: FastAPI to act as front web service allow CP to trigger SMS api with rest format
• Collector: Spring Boot console app, acting as daemon service keep getting data from Apache Kafka , forwarding to SMSC/Simulator using SMPP protocol and receiving MO data from SMSC/Simulator
• MongoDB : database to store logs and config data
• Apache Kafka : message broker acting as queue forwarding message to the consumer and receiving a message from producer
• SMPP Simulator : ( credit to selenium ) to act as SMSC on our computer, this is just to be sure that we can develop a system without needing to connect to real SMSC
• Sample handler: just provide a sample handler web service to simulate CP service.
• Docker : using docker-compose to make it up and running on our computer easily

Sequential diagram

As a system design process, it’s good if we have sequence diagrams so we can know the full picture of system flow.

The important sequence diagram consists of two-part , MT flow and MO flow:

MT Flow

MO Flow

Datastore structure :

As this system mostly stores the log data not containing some transactions, NoSQL is the most suitable for this system, so I use MongoDB.

Collections :

• content_provider :storing content provider info such as : username, password, secreteKey , moUrl, dnUrl
• mt_logs: storing MT logs information such as : content, src, dest , requestDate etc
• dn_logs: storing DN logs info such as : number, originator, content , signature, callBackContent, callbackStatusCode
• mo_logs : storing MO logs info such as : src, dest , content , requestDate, callBackResult, callBackStatusCode

Implementations :

FastAPI:

Since FastAPI is the high-performance API suitable for microservice, it’s fit for this system. I divided this into below project structure:

fastapi project files

config.py : define some config info like mogodb connection, kafka and topics

model.py : File defines model info such as Content Provider, Message Submit , MO etc.

main.py: API logic and some supporting function like auth verification, create call back signature and web api consumption.

Dockerfile: for running this project in docker ( compose)

Collector console application :

Springboot is the de facto , matured enterprise Java framework that provides rich features including IoC and DI allowing developers to develop flexible, maintainable enterprise-level projects.

it consists of some parts as below:

Spring Boot project structure

config :for kafka configuration based on application.properties file

data : for async data between kafka and collector

queue: Abstract class BaseAsyncQueue to create abstraction for async memory queue and it will allow us to create a concrete class for binding to SMPP like Transmitter, Receiver, Transceiver. The console application integrates with SMPP Simulator based on this lib http://smppapi.sourceforge.net

SmsService :the spring boot service to handle core logic and listen to Kafka listener

Docker file : https://gist.github.com/engleangs/b4c0222fbd040a63e31ebc0f8a8a8b00

SMPP Simulator

Simulator provided by http://www.seleniumsoftware.com To start the simulator easily , we can use below docker file: https://gist.github.com/engleangs/c4311e9322cbf78719bc50480a976989

The complete project can be found here https://github.com/engleangs/sms-api-and-collector. To make it up and running just make sure having docker setup and type docker command “ docker compose up -d “ after cloning this project into computer.

Docker images

Up and running outcome, need to create content provider first as below storing in DB :

data in Mongodb

FastAPi docs access on local computer

MT API Doc

Fast API logs

MT , DN and MO logs via Docker logs

SMPP Simulaor logs

View logs via Docker logs

Spring Boot SMPP Collector logs

Hopefully you could learn something from this basic article.

There are still more features need to add ( todo ) for this initialized project:

API :

• Auth verification : add more secure and complex security like JWT or Sha25 signature verify auth from CP
• Add log file or database logs for every APi call and other important events
• Separate api triggering from FastAPI and retry mechanism
• TPS control using RateLimit or Redis

SMPP Collector console application:

• Add more language support for SMS integration
• Add routing and filtering based on Short Code belong to specific
• Add more detail error message based on error code from SMPP

https://www.openmarket.com/docs/Content/apis/v3smpp/smpp-error-codes.htm