Implementing Webhook Handler in Python.

What is Webhook ?

Webhook is an asynchronous HTTP callback on an event occurrence. It is a simple server to server communication for reporting a specific event occurred on a server. The server on which event occurred will fire a HTTP POST request to another server on a URL which is provided by receiving server.

For example, whenever your colleague pushes code commits to github, an event has occurred on github’s server. Now if a webhook URL is provided in github settings, a webhook will be fired to that URL. This webhook will be a HTTP POST request with commit details inside the body in a specified format.  More details on github webhook can be found here.

In this post, I will share my experience of implementing webhook handler in python. For the readers, basic knowledge on implementing web application in python would be better.

Webhook Handler

A Webhook can be handled by simply providing a URL endpoint in a web application. Following is an example using Django. Add webhook url in urls.py

Now create view function in views.py which will parse the data and process it.  In most of the cases, webhook data is sent in JSON format. So lets load the webhook data and sent the data to process_webhook function.

Most of the web applications accept POST request after verifying CSRF token, but here we need to exempt it from this check. So put @csrf_token decorator above the view function. Also put an @require_post decorator to ensure the request is only POST.

The above implementation of URL endpoint will remain different for various other python web framework like Flask, tornado, twisted. But the below code  process_webhook function implementation will remain same irrespective of any framework.

Processing event

There may be different type events we need to handle. So, before proceeding to implement process_webhook function, lets create a python module named webhook_events.py, which will contain a single function for each type of event wherein will be the logic for that particular event. In other words, we are going to map event name with its function, which will handle the logic for that particular type of webhook event.

There are many ways to implement process_webhook function and how we map a webhook event with its function. We are going to discuss different implementation of process_webhook based on extendability. Most basic version of that is below.

A Better way

Now suppose, there are 10s of webhook to be served. We certainly don’t want to write repetitive code. So below is a better way of implementing process_webhook. Here we just replace dot in event name with underscore, so that we get the function name written in webhook_events.py for that event. If the function is not found that means event is not registered (not being served). In this way, no matter the number webhook to be served, just write the function to handle it, in webhook_events.py

Decorators

More robust and pythonic way of implementing process_webhook is by using decorators. Lets define a decorator in webhook_events.py which will map the event_name to its function. Here the EVENT_MAP is dictionary inside a setting module, which will contain event name as key and event function as its value.

In this case, the process_webhook will look like below:

This is the way which I prefer to implement webhook handler in python. How would you prefer ? Please feel free to comment below.

FCM – send push notifications using Python

What is FCM ?

FCM – Firebase Cloud Messaging is a cross-platform  ( Android, iOS and Chrome ) messaging solution that lets you reliably deliver messages at no cost. FCM is best suited if you want to send push notification to your app which you built to run on Android and iOS. The advantage you get is you don’t have to separately deal with GCM (Google Cloud Messaging deprecated now) and Apple’s APNS. You hand over your notification message to FCM and FCM takes care of communicating with apple’s APNS and Android messaging servers to reliably deliver those messages.

fcm-2

Using FCM we can send message to single device or multiple devices.  There are two different types of messages, notification and data. Notification messages include JSON keys that are understood and interpreted by phone’s operating system. If you want to include customized app specific JSON keys use data message. You can combine both notification and data JSON objects in single message. You can also send messages with different priority.

Note : – You need to set priority  to high  if you want phone to wake up and show notification on screen

Sending message with Python

We can use PyFCM to send messages via FCM. PyFCM is good for synchronous ( blocking ) python. We will discuss non-blocking option in next paragraph.

Install PyFCM using following command

The following code will send a push notification to

So, the PyFCM API is the pretty straight forward to use.

Sending FCM push notification using Twisted

PyFCM discussed in above paragraph is good enough if you want to send messages in blocking fashion. If you have to send high number of concurrent messages then using Twisted is a good option.

Twisted Matrix
Twisted Matrix

Network operations performed using twisted library don’t block. Thus it’s a good choice when network concurrency is required by program. We can use txFCM library to send FCM messages using twisted

Install txFCM using following command

Following code send FCM message using txFCM

txFCM is built on top of PyFCM so all the API call that are available in PyFCM are also available in txFCM.

How does email work ?

Most of people in this generation would spend their time daily to send and receive emails. E-mail plays a vital role in our daily activities. As technology evolved email became the one of major communication tool. It may not cost you to send or receive a mail but there is an intricate process involved behind. You just need to press  send button to send your message, your message goes through the complex machanism to get received by recipient.

The invention of email was started in 1961. There were more than couple of people to list as the inventors of email.

The evolution of email started from messaging users on the same computer, then message transition between computer then message among multi user and multi computer, finally email. Email has became revolutionary tool for communication. Now a days it is part of our daily life. We will see how email really works.

Terminology

There are different components involved in the email system. There are few abbreviated terms. Having idea about terminology and abbreviations  will help you to better understand the system.

MSA Mail Submission Agent
MTA Mail Transfer Agent
MDA Mail Delivery Agent
MX Mail Exchange
DNS Domain Name System

MUA is the software that user will use either to send or retrieve  the mail(message) from the server.

MSA is the piece of software which is installed on mail server. Where it is responsible to transfer the message to destination or mail server called MTA (Mail Transfer Agent)

MTA  is the Mail Transfer Agent. It is the piece software on the server which is responsible to route the mail to destination mail server. So we called it as mail router, mail server etc.                                                                        Here are few popular  MTA(Mail Server) softwares : postfixqmailCourier Mail and sendmail.

postfix is the one which is widely used, and it comes with many linux distributions.

You can find the exhaustive mail server softwares list here

Protocols involved

There are different protocol involved in email system. All of them are required to get email delivered to the recipient. They are building blocks of email system. Those protocols are,

SMTP Simple Mail Transport Protocol
IMAP Internet Message Access Protocol
POP Post Office Protocol
DNS Domain Name System (Protocol)

How does email work?

This sections will get you the idea about how these protocols work together to deliver the mail to recipient. Here is the abstract overview of email system.

How does email work abstract
How does email work abstract

The above figure will give you simple abstract overview of email system.  As described in the figure. SMTP is the protocol which is used to push or send an email to server by sender. IMAP and POP are the protocols which are used to check or retrieve message from server by recipient.  The recipient MUA is configured to use either IMAP or POP or BOTH (IMAP & POP). The protocol IMPA is bidirectional where POP is unidirectional. We will see more about POP and IMAP in later section.

        As per the above figure(abstract overview) you can sense that, the sender email client(MUA) will send message using SMPT protocol to mail server (MTA). The mail server will check for the destination if it finds the one it will connect to destination mail server, that is other MTA/MDA and will pass the message(mail) using SMTP protocol. The recipient server will store the received mail locally. Later if recipient will check the mail by using his mail client by connecting to his mail server.

Now let’s see flow of email in detail with following figure

Here Alice is sending  email to Bob(bob@domain.com) by using her email client. Where she will push her message to server using SMTP to send Bob. The mail server will determine the destination by getting the MX record for the destination server. The Alice mail server will check for the domain after in the to(recipient)  mail address to get the MX record. Once Alice(sender) mail server received the result from DNS for the DNS query to get MX record, this server will connect to destination mail server and will delivery the mail (message) using SMTP protocol. The destination mail server will store the message locally. Bob will check the newly arrived mails by using his mail client.

I would like to illustrate this process even more by using following figure.

In the above figure there are new components introduced, those are MSA, MDA and extra MX servers.

       MSA is the Mail Submission agent where it is piece of software it will receive the message from MUA. It uses the same protocol SMTP and port 25. Practically most MTAs perform the function of MSA so you may assume MTA as MSA.

       The MDA means mail delivery agent or message delivery agent  is a computer software component that is responsible for the delivery of e-mail messages to a local recipient’s mailbox. You may not need to get confused, most of MTAs perform the MDA functionality as well though there are softwares which are only designed to work as MDA.

Here, you can see that practically there will be more than 1 mail server(MX SERVER)s. All other MX servers are for backup purpose. When sender query DNS for MX record it may get more that one MX server with priorities. Here is the sample output of DNS query for gmail MX records. Gmail is having 5 MX servers with different priorities.

The above result is the out put of command line utility dig. Where we use this utility to query the DNS. In the output you would see something like below in the right side of  ANSWER SECTION.

Here you would see some numbers in front of domains. Those numbers will decide the priority of MX record. The domain which is having very lowest number associated will have highest priority. Here gmail-smtp-in.l.google.com MX record having highest priority cause it having lowest number associated, that is 5. So the sender MTA will try to connect with most prioritized MX first. If it is down or so, it will try to connect with next MX (Mail Exchange server). As a result mail will be delivered with no down time.

Once the mail is received by the destination mail server, it will store the message in the mail store. There are two types of mail stores used by various mail servers(softwares). Those are,

  • Mail DIR
  • Mail Box

Locally stored mail will be accessed(fetched) by the recipient client using either POP or IMAP.

Protocols POP and IMAP

POP and IMAP are application layer protocols, as described above POP and IMAP are the protocols which are used to fetch or access the mails by recipient email client (MUA). Both protocols are different, they server for different purpose.

The protocol POP means Post Office Protocol and POP3 is it’s  version 3.  As the name describes, this protocol is used to download the message from the server by the client. Once message is downloaded form the server, it will be removed unless you set the flag leave a copy on server just how post card is delivered to destination. If you are only the one who will access mailbox from one location the POP3 suites well. This will also save some memory on server. So, if use POP3 you can’t access the mail using different clients.

Unlike POP3 the IMAP won’t download the message by deleting it on server. It will just access the message like browser does webpages. So it is handy if you use multiple clients from different locations.

 

References:

[1] https://tools.ietf.org/html/rfc3501
[2] https://www.ietf.org/rfc/rfc1939.txt
[3] https://tools.ietf.org/html/rfc5321

InlineCallbacks

Twisted features a decorator named inlineCallbacks which allows you to work with deferreds without writing callback functions.

This is done by writing your code as generators, which yield deferreds instead of attaching callbacks.

Consider the following function written in the traditional deferred style:

using inlineCallbacks, we can write this as:

from twisted.internet.defer import inlineCallbacks

Instead of calling addCallback on the deferred returned by redis.Connection, we yield it. this causes Twisted to return the deferred‘s result to us.

Though the inlineCallbacks looks like synchronous code, which blocks while waiting for the request to finish, each yield statement allows other code to run while waiting for the deferred being yielded to fire.

inlineCallbacks become even more powerful when dealing with complex control flow and error handling.

txRedis

txredis is a non-blocking client for the redis database, written in Python. It uses twisted for the asynchronous communication with redis.

Install

Now to check if txredis is properly installed or not goto python prompt and import txredis :

Example :

Using redis-cli, add list of user’s in redis-server with hmset :

hmset, set key to value within hash name for each corresponding key and value from the mapping dict.

Now, using python twisted get user’s :

Above code do connection with redis-server.

Here, rediscreator.connectTCP return defer and register appropriate method in addCallback, addErrback

Here, define callback function onRedisConnect and errRedisConnect.

Here, hget return the value of key within the hase name and return defer

Here, define callback function onSuccess and onFailur and log the appropriate result.

It give output 123, it is correspond to user abc

WebRTC

WebRTC provides Real-Time Communications directly from better web browsers and devices without requiring plug-ins such as Adobe Flash nor Silverlight.

FreeSWITCH is a WebRTC gateway because it’s able to accept encrypted media from browsers, convert it, and exchange it with other communication networks  that use different codecs and encryptions, for example, PSTN, mobile carriers, legacy systems, and others. FreeSWITCH can be a gateway between your SIP network and applications and billions of browsers on desktops, tablets, and smartphones.

Configuration :

Look for the following in sofia profile and uncomment them:

Clients :

How it works :

By default, Sofia will listen on port 7443 for WSS clients. You may want to change this port if you need your clients to traverse very restrictive firewalls. Edit /usr/local/freeswitch/conf/sip-profiles/internal.xml and change the “wss-binding” value to 443. This number, 443, is the HTTPS (SSL) port, and is almost universally open in all firewalls.Remember that if you use port 443 for WSS, you cannot use that same port for HTTPS, so you will need to deploy your secure web server on another machine.

Example :

SIP signaling in JavaScript with SIP.js (WebRTC client)

Let’s carry out the most basic interaction with a web browser audio/video through WebRTC. We’ll start using SIP.js.

A web page will display a click-to-call button, and anyone can click. That call will be answered by our company’s PBX and routed to our employee extension (1000). Our employee will wait on a browser with the “answer” web page open, and will automatically be connected to the incoming call.

call.html :

call.js :

answer.html :

answer.js :

How it works :

Our employee (the callee, or the person who will answer the call) will sit tight with the answer.html web page open on their browser.

Our customer (the caller, or the person who initiates the communication) will visit the call.html webpage and then click on the Start Call button. This clicking will activate the JavaScript that creates the communication session using the invite method of the user agent, passing as an argument the SIP address of our employee.

use this to see if ws and wss work :

How Implement Multiservice in Twisted.

Multiservice module is service collection provided by twisted, which is useful for creating a new service and combines with two or more existing services.

The major tools that manages Twisted application is a command line utility called twistd. twistd is a cross-platform, and is the recommended tool for running twisted applications.

The core component of the Twisted Application infrastructure is the

object. which represents your application. Application acts as a container of any “Services” that your application provides. This will be done through Services.

Services manages application that can be started and stopped. In Application object can contain many services, or can even hierarchies of Services using “Multiservice” or your own custom IServiceCollection implementations.

Multiservice Implementaion:

To use multiserivce, which implements IService. For this, import internet and service module.

 

Example :

To run, Save above code in a file as serviceexample.tac . Here, “tac ” file is regular python file. Twisted application infrastructure, protocol implementations live in a module, services, using those protocols are registered in a Twisted Application Configuration(TAC) file and the reactor and configuration are managed by an external utility.

Here, I use multiservice functionality from service. agentservice create object of multiservice. Then add services using add service method. In service, you can add web servers, FTP servers and SSH clients. After this, set application name and pass application to serviceparent method.

now, add service on port 8082 as :

add another service same as above on port 8083 as:

To run serviceexample.tac file using twistd program, use command twistd -y serviceexample.tac -n. After this, open browser and enter url localhost:8082 and localhost:8083. You can see result on web page and both TCP servers are active.

Compile C program using gcc in Linux

“This post explains about using gcc to compile C program on Linux”

Compile  C program using gcc:

  • What is a compiler:

Compiler is just like translator between programing language and machine language. It converts source written in programing language to executable instructions file for computer. For different programing languages different compilers are available. Compilers differs from operating system to operating system.

  • Open text editor:

Compiling C program start with a text editor to write our C program like VI . It is generally inbuilt in Linux operating systems. By opening terminal in our system we start from there.screenshot-from-2016-12-05-13-14-11

 

 

 

 

  • Write code:

In terminal type: vi sample.c  and Enter key. Then we enter in to vi text editor with our filename given. Now type a or i key to go in to insert mode. Then type our C program in it. After typing C program press Esc key,colon(:),w,q and Enter key  respectively to save and exit from VI. Here is figure showing source code.

screenshot-from-2016-12-05-13-18-02

  • Compile using gcc command:

Now we are in terminal again.Here we type ls command to see our saved file in the list. Then type gcc sample.c -o sample and enter key. Now the gcc compiler compiles our C file and gives the output as filename we given that is executable. screenshot-from-2016-12-05-13-20-29

  • Execute output file:

If there are any mistakes or errors in the program the compiler gives warnings and error messages with line number to find out them easily, after correcting them compile once again. If it compiles successfully it gives executable file.  To check that file we use ls command and see if it is. If it is, now type ./sample in terminal to execute it. we see the result on terminal.screenshot-from-2016-12-05-13-22-54

Actually in compiling process preprocessor adds the necessary files those are included in C libraries. That we are listed in first of our program like <stdio.h> and some other files also generated by compiler . One of  those file is object file.

Email template

Coding HTML for email can be tough. If you included video, flash, javascript or image chances it won’t show up you wanted it to.

You may have had some issues with your in email clients like Gmail or Apple Mail and Outlook.

Some comman problems are :-

  • Some email client not supported all CSS
  • Syntax improperly in tables or missing tags
  • Fonts are not displaying correctly

Troubleshoot your problem :-

If you’re having issues with columns, or rows not appearing in a table, add a border to the table to identify problems and better visualize the table structure.

If you’re seeing centered text when you prefer it left-aligned, make sure you’ve included properties like valign and align in <td> cells and <p> tags.

You may have some issues with your HTML emails looking great in email clients like Gmail or Apple Mail but rendering poorly in Outlook.

Outlook does not support many features :-

  • Animated GIF.
  • Background Images.
  • Padding for <div>, <p>, and <a> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.
  • Width for <div> and <p> tags. Outlook ignores it. Use hspace or vspace attributes on images or tables to format your content.

Email Client CSS Support:-

Open below link and check which CSS support in browser, desktop and mobile in various email client’s like outlook, yahoo, gmail.

https://templates.mailchimp.com/resources/email-client-css-support/

Web-based call control with mod_httapi

The mod_httapi module was built to allow you to make your call control and IVRs dynamic. With it you can generate custom IVRs based on user input. Freeswitch mod_httapi is a simple HTTP POST operation to send various bits of information to a web application for restful way to control freeswitch call flows.

This module provides an HTTP based Telephony API using a standard FreeSWITCH application interface as well as a cached http file format interface.

HTTAPI syntax :-

mod_httapi configuration file :-

The mod_httapi configuration file is found in conf/autoload_configs and is named httapi.conf.xml . It contains several settings parameters as well as a profiles section. The example configuration contains a default HTTAPI profile or you may create your own profiles.

Inside the profile tag you will notice a number of param entries. These control things such as default settings for various work actions, permissions control (see the following sections), and the default URL to use for HTTP requests.

Example :-

Example :-

You don’t need to answer the call in the dialplan before calling into httapi Both extensions below will make httapi requests to my application:

Below example call from web :-

In above example if call is successfully bridge than it give response like :

Permissions :-
With all the control that you have in httapi , sometimes it becomes necessary to little bit with permissions on things such as variables that shouldn’t be changed, or applications and APIs that you don’t want to execute. Permissions tag you’ll find many different permissions that you can enable, with even more fine-grained control over certain aspects of some of them.

Reference link :-

https://freeswitch.org/confluence/display/FREESWITCH/mod_httapi

Asynchronous DB Operations in Twisted

Twisted is an asynchronous networking framework. Other Database API Implementations have blocking interfaces.

For this reason, twisted.enterprise.adbapi was created. It is a non-blocking interface,which allows you to access a number of different RDBMSes.

General Method to access DB API.

1 ) Create a Connection with db.

2) create a cursor.

3) do a query.

Cursor blocks to response in asynchronous framework. Those delays are unacceptable when using an asynchronous framework such as Twisted.
To Overcome blocking interface, twisted provides asynchronous wrapper for db module such as twisted.enterprise.adbapi

Database Connection using adbapi API.

To use adbapi, we import dependencies as below

1) Connect Database using adbapi.ConnectionPool

Here, We do not need to import dbmodule directly.
dbmodule.connect are passed as extra arguments to adbapi.ConnectionPool’s Constructor.

2) Run Database Query

Here, I used ‘%s’ paramstyle for mysql. if you use another database module, you need to use compatible paramstyle. for more, use DB-API specification.

Twisted doesn’t attempt to offer any sort of magic parameter munging – runQuery(query,params,…) maps directly onto cursor.execute(query,params,…).

This query returns Deferred, which allows arbitrary callbacks to be called upon completion (or failure).

Demo : Select, Insert and Update query in Database.

Here, I have used MySQLdb api, agentdata as a database name, root as a user, 123456 as a password.
Also, I have created select, insert and update query for select, insert and update operation respectively.
runQuery method returns deferred. For this, add callback and error back to handle success and failure respectively.