UPDATE Sep 15th 2009 – This material needs improvement, bug-fixing and updating. Since I’m no longer working with Orbited, I will appreciate any changes (”patches”, if you will) to the article. Please email me (anirudh -at- anirudhsanjeev -dot- org) if you would like to contribute.
Note, this tutorial is highly technical and targeted to people who want to use Django with Orbited, and really know what they’re doing. I’ll give a brief introduction to comet, and how I found workarounds to my problems, but that’s it.
Django is a wonderful web framework, though like all regular web applications, you need to return a http response as soon as the request is made. But what if you want to stream HTTP, like a stock ticker, live blog, etc – when you don’t want the user to reload the page every few seconds, and you want to push data to the browser real time.
There are three standard techniques for the same.
1. Polling – Make XmlHTTPRequests (XHRs) every few seconds to the server. This is very bad as it wastes bandwidth, has high latency and isn’t scalable.
2. Long Polling – Make an XMLHttpRequest that doesn’t return until data is available on the server. This reduces latency but still is inefficient and unscalable.
3. HTTP Streaming – the best option so far – Never finishes returning a HTTP Response, but this causes a thread to lock up, and the server will run out of threads and memory before you can say “RAM”.
This paradigm of real time content delivery is called “COMET”, it’s an umbrella term similar to AJAX, and describes a concept rather than a technique. There are several comet servers available, some open source, some not. Examples are Jetty, Cometd, Meteror, Liberator, and Orbited.
I will be using Orbited, a comet server, and discussing my personal workarounds to get it to work with Django, and create a “Live Shoutbox”.
Prerequisites:
1. You know about Django, Python.
2. You have installed Orbited (orbited.org)
Recommended Reading:
1. http://ajaxpatterns.org/HTTP_Streaming
2. http://orbited.org/wiki/Documentation
3. http://cometdaily.com/2008/10/10/scalable-real-time-web-architecture-part-2-a-live-graph-with-orbited-morbidq-and-jsio/
No. 3, written by Michael Carter, a lead developer of orbited, is highly recommended. This is because my tutorial is based primarily on that. I’m going to be referring to that frequently.
Now, let’s begin:
Step 1. Create a django application.
Let’s call our application “ShoutBox”. I’m going to define two endpoints in our urls.py going to two views: xhr, and views.
$ django-admin.py startproject shoutbox
And, I create a new app – shout.
$ django-admin.py startapp shout
I add two forwarders in urls.py:
(r'^shoutbox/', include('shoutbox.shout.index')),
(r'^xhr/', include('shoutbox.shout.xhr')),
(r'^static/(?P<path>.*)$', 'django.views.static.serve', {'document_root':'./static'}),
copy all the files from “orbited/static” from the downloaded tarball to ./shoutbox/static. This will ensure that you’ll get your static files from the same port you’re serving on. Also, download the latest version of jquery and move it to static/jquery.js
Now, go and edit shoutbox/shout/views.py to look like this:
</p>
<h1>Create your views here.</h1>
<p>from django.template import Context, loader
from django.template.loader import get_template
from django.http import *</p>
<p>def index(request):
"""
handle the index request
"""
# here, we will simply fetch a template and expand it
my_template = get_template("index.html")</p>
<pre><code># no context in particular
return HttpResponse(my_template.render(Context({})))
</code></pre>
<p>def xhr(request):
"""
handle an XMLHttpRequest
"""
# see what message has been sent
message = request.POST["message"]
# for now, let's just print the message
print message</p>
<pre><code>return HttpResponse("OK")
</code></pre>
<p>
Step 2: Create a HTML Page:
<title>ShoutBox</title></p>
<script>
document.domain = document.domain; // I don't know why
// we need to do this, but we just need to
</script>
<script src="http://localhost:8000/static/Orbited.js">
</script>
<script>
// set the orbited settings and port
Orbited.settings.port = 9000;
Orbited.settings.hostname = "localhost";
//Orbited.settings.streaming = false;
TCPSocket = Orbited.TCPSocket;
</script>
<script src="http://localhost:8000/static/protocols/stomp/stomp.js">
</script>
<script src="http://localhost:8000/static/jquery.js" type="text/javascript">
</script>
<script>
var add_message = function(payload){
var message1 = payload.toString();
message_div = document.createElement("div");
message_div.innerHTML = message1;
document.getElementById("messages").appendChild(message_div);
};
// execute once the document is ready
$(document).ready(function(){
stomp = new STOMPClient();
stomp.onopen = function(){
//console.log("opening stomp client");
};
stomp.onclose = function(c){
alert('Lost Connection, Code: ' + c);
};
stomp.onerror = function(error){
alert("Error: " + error);
};
stomp.onerrorframe = function(frame){
alert("Error: " + frame.body);
};
stomp.onconnectedframe = function(){
//console.log("Connected. Subscribing");
//alert("subscribing");
stomp.subscribe("/topic/shouts");
};
stomp.onmessageframe = function(frame){
// Presumably we should only receive message frames with the
// destination "/topic/shouts" because that's the only destination
// to which we've subscribed. To handle multiple destinations we
// would have to check frame.headers.destination.
add_message(frame.body);
};
stomp.connect('localhost', 61613);
});
$(document.getElementById('send_shout')).click(function(){
// send an XMLHttpRequest to /xhr
var message_text = document.getElementById('shout_text').value;
$.ajax({
type:'POST',
url:'http://localhost:8000/xhr/',
data:{
'message':message_text,
},
});
});
</script>
<p><body></p>
<div id="shoutbox">
<input type="text" id="shout_text" /><br /><div id="send_shout"><button>Shout it</button></div>
</div>
<pre><code><div id="messages">
</div>
</code></pre>
<p></body>
Move this to templates/index.html and don’t forget to add this path in your settings.py
What the HTML does is fairly obvious if you’ve read the tutorial written by Michael Carter. I’m not going to explain what it does exactly.
Let’s test the django application. It should send an XMLHttpRequest and the console that you’re running django from should print the message that you type in the box, though we’re far from finished.
Step 2: Configure Orbited to serve stomp.
We’re going to follow nearly the same steps that was followed in Michael Carter’s tutorial,
We’re first going to create an orbited.cfg
[listen]
http://:9000
stomp://:61613</p>
<p>[access]
* -> localhost:61613</p>
<p>[static]
graph=index.html</p>
<p>[global]
session.ping_interval = 300
I don’t want to explain what this exactly does, because the other tutorial manages to do it so well.
Step 3: The key component: AN RPC SERVER
Now if you run the orbited server, and the django application you will see that there’s no way you can actually send data to the MorbidQ message queue, primarily because you cannot start up a reactor send/receive system inside your django view.
My solution? run a seperate python script, that talks to stomp, and also runs an XML-RPC server, that can be called by other applications. We’re planning to migrate our RPC to thrift, which is better for local transmission, but for our tutorial, xml-rpc will do just fine.
Now there is another problem. If you create an XML-RPC server, and try to run that and a reactor server, you won’t be able to because both functions block the thread. I worked around this by running the XMLRPC server in a separate thread.
Once you configure it, it finally looks something like this:
</p>
<h1>!/usr/bin/python</h1>
<h1>relay.py</h1>
<h1>Author: Anirudh Sanjeev (anirudh -at- anirudhsanjeev -dot- org)</h1>
<p>from stompservice import StompClientFactory
from twisted.internet import reactor
from twisted.internet.task import LoopingCall
from random import random
from orbited import json
from SimpleXMLRPCServer import *</p>
<p>from threading import Thread
import stompservice
class DataProducer(StompClientFactory):
def recv_connected(self, msg):</p>
<pre><code> print 'Connected; producing data'
# the next two lines are probably the biggest workaround
# for the weirdest bug I've seen in my entire life
# it repeatedly calls a function that absolutely does nothing
# however, if I remove them, there's a ten second delay
# between when the DataProducer transmits a message to
# when the browser actually receives the data. Me and my
# friend were mindfucked thinking about how something like
# this could possibly happen. But right now we are more worried
# about the rest of the code
self.timer = LoopingCall(self.test_data)
self.timer.start(INTERVAL/1000.0)
def send_data(self, channel, data):
print "Transmitting: ", data
# modify our data elements
self.send(channel, json.encode(data))
def test_data(self):
# WHAT THE F***?
pass
</code></pre>
<p>orbited_proxy = DataProducer()</p>
<p>class RPCServer(Thread):
def <strong>init</strong>(self, orbited):
self.orbited = orbited
Thread.<strong>init</strong>(self)
def run(self):
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
#create a server
server = SimpleXMLRPCServer(("localhost",8045),
requestHandler = RequestHandler)</p>
<pre><code> server.register_introspection_functions()
def transmit_orbited(channel, message):
"""
@param channel: The stomp channel to send to
@param message: The message that needs to be transmitted
"""
self.orbited.send_data(channel, message)
return ""
server.register_function(transmit_orbited, 'transmit')
server.serve_forever()
</code></pre>
<p>rpcthread = RPCServer(orbited_proxy)
rpcthread.start()</p>
<p>reactor.connectTCP('localhost', 61613, orbited_proxy)
reactor.run()
So this is what happens: your application calls the XMLRPC server with a message, which is pushed to the morbidq message queue, along with a channel.
Now, to do just that, we modify the xhr() function in our views.py
def xhr(request):
"""
handle an XMLHttpRequest
"""
# see what message has been sent
print "POSTDATA: ", request.POST
message = request.POST["message"]
# send the message across
import xmlrpclib
proxy = xmlrpclib.ServerProxy("http://localhost:8045")</p>
<pre><code># push the data to all clients.
proxy.transmit("/topic/shouts", message)
# That's it, thread's not locked
return HttpResponse("OK")
</code></pre>
<p>
Putting it all together
You have to run everything in this particular order:
- run orbited:
$ orbited –config=orbited.cfg
- run the relay:
$ python relay.py # warning – this is a non-daemonized thread. you won’t be able to quit it with Ctrl+C this is a bug, though you can pgrep and kill it manually.
- run the django application
$ python manage.py runserver
Open up http://localhost:8000/shoutbox in multiple tabs, browsers (though it won’t work from other computers as we hardcoded “localhost”, but if you’re smart enough to use a hostname, it would work fine).
Enter a message in one textbox, and click “Shout” and it comes up simultaneously in different windows, and you can easily scale this up to thousands of simultaneous idling users, and there still won’t be any significant load on your machine.
I’m pretty new to orbited, comet and django, and thought this tutorial is helpful. If you find errors, or have suggestions, please leave a comment.
Please please do not email me with doubts or questions, I won’t be responding. You can ask the awesome folks on #orbited at irc.freenode.net.
.
But now I’m thinking how much does it compromise on professionalism. True you can be professional in your behavior and your work and get along fine, and dress codes are pure evil – but I didn’t know where the boundaries were to what you can and cannot do, while everyone else seemed to sort of know. While I don’t remember doing anything utterly tactless and callous at work(which I’m really relieved about), there is always the looming fear that something you take for granted might really offend people and they won’t tell you.
Recent Comments