Twisted, Long Polling, & JSONP

Mike Milano — December 30, 2011 - 10:06pm

I recently decided to use Twisted as the framework for a new project. In short, this app listens to and parses incoming events from multiple servers, as well as issues commands back to them.

Once I had Python doing its job managing the data, the next step was to expose the data in memory from the Twisted app to an existing PHP/Drupal UI. Twisted makes it pretty simple to attach an HTTP server to your app, so I did just that.

Why JSONP?

Because Apache & the PHP app was running on port 80 and my Twisted service was running on 8000, standard AJAX requests were not going to work because of XSS restrictions. My only options were either write a PHP wrapper, or use JSONP. JQuery supports JSONP very elegantly, so I chose to move forward with that option.

Why Long Polling

Standard polling from an AJAX app will typically send a request at a given interval. Even when there's no data to return from the server, the server sends an empty response and the cycle starts all over.

Long polling cuts down on the overhead a bit by keeping the request open until there is actually data to send back. Again Twisted comes through in making it easy to serve data like this.

Example

I took the relevant pieces of code to make a very generic example to provide a starting point for anyone looking to implement something similar.

Note: You will need to modify long_poll.js to point to the destination of the twisted server. Currently it is example.com:8000

index.html (HTML file served from Apache)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <title>Twisted - Long Polling & JSONP</title>
        <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
        <script type="text/javascript" src="long_poll.js"></script>
    </head>
    <body>
        <h1>Twisted - Long Polling & JSONP</h1>
        <p>New message will appear below as they are received from the server.</p>
        <hr />
        <ul id="messages"></ul>
    </body>
</html>

long_poll.js (JavaScript file served from Apache)

// variable to keep track of the last time we received data
// a time value will be sent back with the response in a unix timestamp format
var lastupdate = 0;
 
// call getData when the document has loaded
$(document).ready(function(){
    getData(lastupdate);
});
 
// execute ajax call to server.py
var getData = function(lastupdate) {
    $.ajax({
        type: "GET",
        // set the destination for the query
        url: 'http://example.com:8000?lastupdate='+lastupdate+'&callback=?',
        // define JSONP because we're using a different port and/or domain
        dataType: 'jsonp',
        // needs to be set to true to avoid browser loading icons
        async: true,
        cache: false,
        // timeout after 5 minutes
        timeout:300000,
        // process a successful response
        success: function(response) {
            // append the message list with the new message
            var messages = response.data.messages;
            for (x in messages) {
                $('<li>'+messages[x].published+' - '+messages[x].message+'</li>').appendTo('#messages');
            }
            // set lastupdate
            lastupdate = response.timestamp;
            // call again in 1 second
            setTimeout('getData('+lastupdate+');', 1000);
        },
        // handle error
        error: function(XMLHttpRequest, textStatus, errorThrown){
            // try again in 10 seconds if there was a request error
            setTimeout('getData('+lastupdate+');', 10000);
        },
    });
};

server.py (start with: $ python server.py)

from twisted.web import server
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor, task
import json
import time
# just for simulation in getData
import random
 
class InfoServer(Resource): 
    isLeaf = True
    def __init__(self):
        # throttle in seconds to check app for new data
        self.throttle = 5
        # define a list to store client requests
        self.delayed_requests = []
        # setup a loop to process delayed requests
        loopingCall = task.LoopingCall(self.processDelayedRequests)
        loopingCall.start(self.throttle, False)
        # initialize parent
        Resource.__init__(self)
 
    def render(self, request):
        """
        Handle a new request
        """
        # set the request content type
        request.setHeader('Content-Type', 'application/json')
        # set args
        args = request.args
 
        # set jsonp callback handler name if it exists
        if 'callback' in args:
            request.jsonpcallback =  args['callback'][0]
 
        # set lastupdate if it exists
        if 'lastupdate' in args:
            request.lastupdate =  args['lastupdate'][0]
        else:
            request.lastupdate = 0
 
        # if we have data now, send it
        data = self.getData(request)
        if len(data) > 0:
            return self.__format_response(request, 1, data)
 
        # otherwise, put it in the delayed request list
        self.delayed_requests.append(request)
 
        # tell the client we're not done yet
        return server.NOT_DONE_YET
 
    def getData(self, request):
        """
        Replace this logic with code that will actually test for
        and return data your app should return.
 
        You can use request.lastupdate here if you want to pull
        data since the last time this request received data.
 
        This is just dummy logic to make this demo work.
        """
        # init data
        data = {}
 
        #simulate the chance of new data being available or not
        new_data_available = bool(random.getrandbits(1))
 
        # set some simulated data
        if new_data_available:
            # you can dynamically add any key/value pair here
            data = {'messages':[
                    {
                        'message':'Test Message',
                        'published':int(time.time())
                    },
                ]
            }
 
        return data
 
    def processDelayedRequests(self):
        """
        Processes the delayed requests that did not have
        any data to return last time around.
        """        
        # run through delayed requests
        for request in self.delayed_requests:
            # attempt to get data again
            data = self.getData(request)
 
            # write response and remove request from list if data is found
            if len(data) > 0:
                try:
                    request.write(self.__format_response(request, 1, data))
                    request.finish()
                except:
                    # Connection was lost
                    print 'connection lost before complete.'
                finally:
                    # Remove request from list
                    self.delayed_requests.remove(request)
 
    def __format_response(self, request, status, data):
        """
        Format responses uniformly
        """
        # Set the response in a json format
        response = json.dumps({'status':status,'timestamp': int(time.time()), 'data':data})
 
        # Format with callback format if this was a jsonp request
        if hasattr(request, 'jsonpcallback'):
            return request.jsonpcallback+'('+response+')'
        else:
            return response
 
#############################################       
if __name__ == '__main__':
    resource = InfoServer()
    factory = Site(resource)
    reactor.listenTCP(8000, factory)
    reactor.run()