UPDATE Sep 15th 2009 – This material needs improvement, bug-fixing and updating. Since I’m no longer working with Orbited, I will appreciate any changes (”patches”, if you will) to the article. Please email me (anirudh -at- anirudhsanjeev -dot- org) if you would like to contribute.
Note, this tutorial is highly technical and targeted to people who want to use Django with Orbited, and really know what they’re doing. I’ll give a brief introduction to comet, and how I found workarounds to my problems, but that’s it.
Django is a wonderful web framework, though like all regular web applications, you need to return a http response as soon as the request is made. But what if you want to stream HTTP, like a stock ticker, live blog, etc – when you don’t want the user to reload the page every few seconds, and you want to push data to the browser real time.
There are three standard techniques for the same. 1. Polling – Make XmlHTTPRequests (XHRs) every few seconds to the server. This is very bad as it wastes bandwidth, has high latency and isn’t scalable. 2. Long Polling – Make an XMLHttpRequest that doesn’t return until data is available on the server. This reduces latency but still is inefficient and unscalable. 3. HTTP Streaming – the best option so far – Never finishes returning a HTTP Response, but this causes a thread to lock up, and the server will run out of threads and memory before you can say “RAM”.
This paradigm of real time content delivery is called “COMET”, it’s an umbrella term similar to AJAX, and describes a concept rather than a technique. There are several comet servers available, some open source, some not. Examples are Jetty, Cometd, Meteror, Liberator, and Orbited.
I will be using Orbited, a comet server, and discussing my personal workarounds to get it to work with Django, and create a “Live Shoutbox”.
Prerequisites: 1. You know about Django, Python. 2. You have installed Orbited (orbited.org)
Recommended Reading: 1. http://ajaxpatterns.org/HTTP_Streaming 2. http://orbited.org/wiki/Documentation 3. http://cometdaily.com/2008/10/10/scalable-real-time-web-architecture-part-2-a-live-graph-with-orbited-morbidq-and-jsio/
No. 3, written by Michael Carter, a lead developer of orbited, is highly recommended. This is because my tutorial is based primarily on that. I’m going to be referring to that frequently.
Now, let’s begin:
Step 1. Create a django application.
Let’s call our application “ShoutBox”. I’m going to define two endpoints in our urls.py going to two views: xhr, and views.
$ django-admin.py startproject shoutbox
And, I create a new app – shout. $ django-admin.py startapp shout
I add two forwarders in urls.py:
(r'^shoutbox/', include('shoutbox.shout.index')),
(r'^xhr/', include('shoutbox.shout.xhr')),
(r'^static/(?P<path>.*)$', 'django.views.static.serve', {'document_root':'./static'}),
copy all the files from “orbited/static” from the downloaded tarball to ./shoutbox/static. This will ensure that you’ll get your static files from the same port you’re serving on. Also, download the latest version of jquery and move it to static/jquery.js
Now, go and edit shoutbox/shout/views.py to look like this:
</p>
<h1>Create your views here.</h1>
<p>from django.template import Context, loader
from django.template.loader import get_template
from django.http import *</p>
<p>def index(request):
"""
handle the index request
"""
# here, we will simply fetch a template and expand it
my_template = get_template("index.html")</p>
<pre><code># no context in particular
return HttpResponse(my_template.render(Context({})))
</code></pre>
<p>def xhr(request):
"""
handle an XMLHttpRequest
"""
# see what message has been sent
message = request.POST["message"]
# for now, let's just print the message
print message</p>
<pre><code>return HttpResponse("OK")
</code></pre>
<p>
Step 2: Create a HTML Page:
<title>ShoutBox</title></p>
<script>
document.domain = document.domain; // I don't know why
// we need to do this, but we just need to
</script>
<script src="http://localhost:8000/static/Orbited.js">
</script>
<script>
// set the orbited settings and port
Orbited.settings.port = 9000;
Orbited.settings.hostname = "localhost";
//Orbited.settings.streaming = false;
TCPSocket = Orbited.TCPSocket;
</script>
<script src="http://localhost:8000/static/protocols/stomp/stomp.js">
</script>
<script src="http://localhost:8000/static/jquery.js" type="text/javascript">
</script>
<script>
var add_message = function(payload){
var message1 = payload.toString();
message_div = document.createElement("div");
message_div.innerHTML = message1;
document.getElementById("messages").appendChild(message_div);
};
// execute once the document is ready
$(document).ready(function(){
stomp = new STOMPClient();
stomp.onopen = function(){
//console.log("opening stomp client");
};
stomp.onclose = function(c){
alert('Lost Connection, Code: ' + c);
};
stomp.onerror = function(error){
alert("Error: " + error);
};
stomp.onerrorframe = function(frame){
alert("Error: " + frame.body);
};
stomp.onconnectedframe = function(){
//console.log("Connected. Subscribing");
//alert("subscribing");
stomp.subscribe("/topic/shouts");
};
stomp.onmessageframe = function(frame){
// Presumably we should only receive message frames with the
// destination "/topic/shouts" because that's the only destination
// to which we've subscribed. To handle multiple destinations we
// would have to check frame.headers.destination.
add_message(frame.body);
};
stomp.connect('localhost', 61613);
});
$(document.getElementById('send_shout')).click(function(){
// send an XMLHttpRequest to /xhr
var message_text = document.getElementById('shout_text').value;
$.ajax({
type:'POST',
url:'http://localhost:8000/xhr/',
data:{
'message':message_text,
},
});
});
</script>
<p><body></p>
<div id="shoutbox">
<input type="text" id="shout_text" /><br /><div id="send_shout"><button>Shout it</button></div>
</div>
<pre><code><div id="messages">
</div>
</code></pre>
<p></body>
Move this to templates/index.html and don’t forget to add this path in your settings.py
What the HTML does is fairly obvious if you’ve read the tutorial written by Michael Carter. I’m not going to explain what it does exactly.
Let’s test the django application. It should send an XMLHttpRequest and the console that you’re running django from should print the message that you type in the box, though we’re far from finished.
Step 2: Configure Orbited to serve stomp. We’re going to follow nearly the same steps that was followed in Michael Carter’s tutorial,
We’re first going to create an orbited.cfg
[listen] http://:9000 stomp://:61613</p> <p>[access] * -> localhost:61613</p> <p>[static] graph=index.html</p> <p>[global] session.ping_interval = 300
I don’t want to explain what this exactly does, because the other tutorial manages to do it so well.
Step 3: The key component: AN RPC SERVER
Now if you run the orbited server, and the django application you will see that there’s no way you can actually send data to the MorbidQ message queue, primarily because you cannot start up a reactor send/receive system inside your django view.
My solution? run a seperate python script, that talks to stomp, and also runs an XML-RPC server, that can be called by other applications. We’re planning to migrate our RPC to thrift, which is better for local transmission, but for our tutorial, xml-rpc will do just fine.
Now there is another problem. If you create an XML-RPC server, and try to run that and a reactor server, you won’t be able to because both functions block the thread. I worked around this by running the XMLRPC server in a separate thread.
Once you configure it, it finally looks something like this:
</p>
<h1>!/usr/bin/python</h1>
<h1>relay.py</h1>
<h1>Author: Anirudh Sanjeev (anirudh -at- anirudhsanjeev -dot- org)</h1>
<p>from stompservice import StompClientFactory
from twisted.internet import reactor
from twisted.internet.task import LoopingCall
from random import random
from orbited import json
from SimpleXMLRPCServer import *</p>
<p>from threading import Thread
import stompservice
class DataProducer(StompClientFactory):
def recv_connected(self, msg):</p>
<pre><code> print 'Connected; producing data'
# the next two lines are probably the biggest workaround
# for the weirdest bug I've seen in my entire life
# it repeatedly calls a function that absolutely does nothing
# however, if I remove them, there's a ten second delay
# between when the DataProducer transmits a message to
# when the browser actually receives the data. Me and my
# friend were mindfucked thinking about how something like
# this could possibly happen. But right now we are more worried
# about the rest of the code
self.timer = LoopingCall(self.test_data)
self.timer.start(INTERVAL/1000.0)
def send_data(self, channel, data):
print "Transmitting: ", data
# modify our data elements
self.send(channel, json.encode(data))
def test_data(self):
# WHAT THE F***?
pass
</code></pre>
<p>orbited_proxy = DataProducer()</p>
<p>class RPCServer(Thread):
def <strong>init</strong>(self, orbited):
self.orbited = orbited
Thread.<strong>init</strong>(self)
def run(self):
class RequestHandler(SimpleXMLRPCRequestHandler):
rpc_paths = ('/RPC2',)
#create a server
server = SimpleXMLRPCServer(("localhost",8045),
requestHandler = RequestHandler)</p>
<pre><code> server.register_introspection_functions()
def transmit_orbited(channel, message):
"""
@param channel: The stomp channel to send to
@param message: The message that needs to be transmitted
"""
self.orbited.send_data(channel, message)
return ""
server.register_function(transmit_orbited, 'transmit')
server.serve_forever()
</code></pre>
<p>rpcthread = RPCServer(orbited_proxy)
rpcthread.start()</p>
<p>reactor.connectTCP('localhost', 61613, orbited_proxy)
reactor.run()
So this is what happens: your application calls the XMLRPC server with a message, which is pushed to the morbidq message queue, along with a channel.
Now, to do just that, we modify the xhr() function in our views.py
def xhr(request):
"""
handle an XMLHttpRequest
"""
# see what message has been sent
print "POSTDATA: ", request.POST
message = request.POST["message"]
# send the message across
import xmlrpclib
proxy = xmlrpclib.ServerProxy("http://localhost:8045")</p>
<pre><code># push the data to all clients.
proxy.transmit("/topic/shouts", message)
# That's it, thread's not locked
return HttpResponse("OK")
</code></pre>
<p>
Putting it all together
You have to run everything in this particular order:
- run orbited: $ orbited –config=orbited.cfg
- run the relay: $ python relay.py # warning – this is a non-daemonized thread. you won’t be able to quit it with Ctrl+C this is a bug, though you can pgrep and kill it manually.
- run the django application $ python manage.py runserver
Open up http://localhost:8000/shoutbox in multiple tabs, browsers (though it won’t work from other computers as we hardcoded “localhost”, but if you’re smart enough to use a hostname, it would work fine).
Enter a message in one textbox, and click “Shout” and it comes up simultaneously in different windows, and you can easily scale this up to thousands of simultaneous idling users, and there still won’t be any significant load on your machine.
I’m pretty new to orbited, comet and django, and thought this tutorial is helpful. If you find errors, or have suggestions, please leave a comment.
Please please do not email me with doubts or questions, I won’t be responding. You can ask the awesome folks on #orbited at irc.freenode.net.
Very nice example! The idea of integrating Django via RPC proxy and XHR is fruitful, indeed. I guess zip would be useful too. Keep on writing.
Hi
I am not able to run the shoutbox application. I have made the following changes in the code that you have posted:
1) urls.py is modified :
(r’^shoutbox/’, include(’shoutbox.shout.index’)), (r’^xhr/’, include(’shoutbox.shout.xhr’)), (r’^static/(?P.*)$’, ‘django.views.static.serve’, {’document_root’:’./static’}),
After modification:
(r’^shoutbox/’, ’shoutbox.shout.views.index’), (r’^xhr/’, ’shoutbox.shout.views.xhr’), (r’^static/(?P.*)$’, ‘django.views.static.serve’, {’document_root’:’./static’}),
Change: ‘include’ has been removed because I was getting the ‘No module named index’ error
2) In relay.py script INTERVAL is not defined I have defined its value to 1000 referring ‘Michael Carter’ tutorial
3) In Orbited.cfg:
[static] graph=index.html
After Modification”
[static] shoutbox=index.html
/******* my orbited.cfg **********/
Example Orbited Configuration file
[global] session.ping_interval = 300
[listen] http://:9000 stomp://:61613
[static] shoutbox=index.html
[access] * -> localhost:61613
After making the above changes I am able to invoke the shoutbox application, however I am not able to see the same message entered in one browser sent to the other browser.
Following is the output in each of the console:
CONSOLE 1: ORBITED
D:\demo>orbited –config=D:\Python26\etc\orbited.cfg
CONSOLE 2: RELAY script
D:\DjangoProjects\shoutbox>python relay.py Connected; producing data *** channel: /topic/shouts *** message: hello Transmitting: hello localhost – - [01/Apr/2009 19:07:40] “POST /RPC2 HTTP/1.0″ 200 - localhost – - [01/Apr/2009 19:07:40] “POST /RPC2 HTTP/1.0″ 200 -
CONSOLE 3 DJANGO
D:\DjangoProjects\shoutbox>python manage.py runserver Validating models… 0 errors found
Django version 1.0.2 final, using settings ’shoutbox.settings’ Development server is running at http://127.0.0.1:8000/ Quit the server with CTRL-BREAK. [01/Apr/2009 19:07:13] “GET /shoutbox/ HTTP/1.1″ 200 2549 [01/Apr/2009 19:07:19] “GET /shoutbox/ HTTP/1.1″ 200 2549 *** message: hello [01/Apr/2009 19:07:40] “POST /xhr/ HTTP/1.1″ 500 70067
IMPORTANT: According to me in DJANGE console i get HTTP 500 error code that may be one of the possiblity I am not able to run the application
1) For the above operation 2 instance of firefox were used. IE was not used because it displays javascript error and after clicking shout button nothing happens.
2) If nothing is entered in the text field and shout button is clicked then in both browser instances 2 double quotes (”") are displayed
Please let me know what mistake am doing or I (as well others) will also be grateful if you post complete executable code
3) I am new to all the above tech. however before going into details i want to see how comet can be used with Django
Thanks and regards, Nilesh
Nice tutorial. I believe for versions of Orbited 0.7.8+ though, you want:
Orbited.settings.hostname = “127.0.0.1″;
or whatever the domain of your server is. I think localhost worked on 0.7.7 but I get a cross-subdomain error when I use it with 0.7.8 and 0.7.9. Go figure.
Thanks for the tutorial, i’m sure it has and will continue to serve many. That being said, I found a couple of flaws namely:-
urls.py is slightly off
It should look like:-
from django.conf.urls.defaults import *
urlpatterns = patterns(’shoutbox.shout.views’, (r’^shoutbox/’, ‘index’), (r’^xhr/’, ‘xhr’), ) urlpatterns += patterns(”, (r’^static/(?P.*)$’, ‘django.views.static.serve’, {’document_root’:’./static’}), )
Secondly, there is a Javascript Error on the page that causes IE6/IE7 to stall with the following script Error ‘Expected identifier, string or number’. More can be found at http://forumsblogswikis.com/2008/07/21/javascript-error-expected-identifier-string-or-number-in-ie/ The quick fix is:- $.ajax({ type:’POST’, url:’/xhr/’, data:{ ‘message’:message_text//, Error 1 } // }, Error 2 });
relay.py
The Value for INTERVAL is not defined so I set it to 100
Nilesh, Afrobeard,
thank you for your inputs. I’ll clean this article up and upload downloadable samples when my exams are finished.
Thanks for very nice article!
I’m wondering what I need to do to implement bidirectional communication. I guess using something like stomp.send in the client.
Although I have implemented some apps using Django, right now I’m basing my app in Micheal’s code and not using Django at all. I have managed to received notifications in the browser using your hack, but still having trouble sending messages back to the server.
Thanks.
Thanks for your good work. But I have the same problem as Nilesh, got the 500 error: POSTDATA: [23/Jun/2009 00:34:18] “POST /xhr/ HTTP/1.1″ 500 69257
Send empty no error: POSTDATA: transmit end [23/Jun/2009 00:29:09] “POST /xhr/ HTTP/1.1″ 200 2
If you look closely at the html form, the post name for the text is not message. That’s probably where you’re getting the 500 error. I’ve not tried this sample yet but I highly suspect that’s the culprit of your problems.
Thank u Lee, but I have found more Exception happened at this line: proxy.transmit(”/topic/shouts”, message)
track: POSTDATA: send: ‘POST /RPC2 HTTP/1.0\r\nHost: localhost:8045\r\nUser-Agent: xmlrpclib.py/ .0.1 (by http://www.pythonware.com)\r\nContent-Type: text/xml\r\nContent-Length: 235\r n\r\n’ send: “\n\ntransm t\n\n\n/topic/shouts</value \n\n\n123\n\n\n /methodCall>\n” reply: ‘HTTP/1.0 200 OK\r\n’ header: Server: BaseHTTP/0.3 Python/2.6 header: Date: Fri, 26 Jun 2009 12:02:58 GMT header: Content-type: text/xml header: Content-length: 315 body: “\n\n\n\n\nfaultCode\n1\n\n\nfaultString\n<type ‘exceptions.TypeError’>:Data must not be unicode\n\n\n\n\n” Traceback (most recent call last): File “J:\bjcept\Code\Others\tryorb\shout\views.py”, line 40, in xhr proxy.transmit(”/topic/shouts”, message) File “H:\Program Files\Python26\lib\xmlrpclib.py”, line 1199, in call return self.__send(self.__name, args) File “H:\Program Files\Python26\lib\xmlrpclib.py”, line 1489, in __request verbose=self.__verbose File “H:\Program Files\Python26\lib\xmlrpclib.py”, line 1253, in request return self._parse_response(h.getfile(), sock) File “H:\Program Files\Python26\lib\xmlrpclib.py”, line 1392, in _parse_respo se return u.close() File “H:\Program Files\Python26\lib\xmlrpclib.py”, line 838, in close raise Fault(**self._stack[0]) Fault: <Fault 1: “:Data must not be unicode”>
I find that if don’t encode using json, there is no err: def send_data(self, channel, data): self.send(channel, data) #just send the data The problem is not related with the xmlrpclib
Solution is: self.send(channel, json.encode(data).encode(”utf-8″)) It works ok.
At client use: var message = eval(’(’ + payload + ‘)’); to translate str to json data.
Great blog post, thanks for sharing.
I found Orbited.settings.hostname could be 127.0.0.1 or localhost, so long as the ajax POST request goes to the same url. Or bypass the whole issue by using a relative url in the POST url parameter as per Afrobeard’s js. (using FF).
I just spent a lot of time, and sanity, going through the various django comet blogs, articles, etc. For others on the same dark path, this clocks it.
Thanks dvrobj, I just went to try out this example and that was the same error that I was getting.
Hey, why use threaded RPC server, when you can have one in twisted? http://code.activestate.com/recipes/526625/
about that component — GREAT IDEA. Thank you for that, thank you!
I am looking for a specialist of ORBITED, Stomp, to finish the settings of a videochat. Price to be discussed.
Alain
The article mentions that:
“Move this to templates/index.html and don’t forget to add this path in your settings.py”
I have a directory /templates under the project which contains a file index.html with the contents posted above in the tutorial. When I run the django application, I receive an error:
ImportError: No module named index
Does anyone know how I can add the path to the settings.py file. I am new to both python and django/comet and foolishly decided to jump into the deep end
many thanks
The article is VERY GOOD but please correct the code boxes. The python code is mixed with HTML. Thanks
Gustavo:
I think a wp upgrade knocked out my code boxes. I’ll get to fixing it soon. Thanks!
Hi, Thanks for the post. Really helped!
I was wondering why you chose XMLRPC server of python over that of Twisted. Had you chosen Twisted’s, then the entire thread thing could have been avoided. (connectTCP and listenTCP in the same reactor…I dont know if its allowed)
Hey Yousuf,
The XMLRPC was a quick hack. I eventually settled for using Thrift, but never got to deploy the project to the public, so didn’t optimize it.
Oh! Cool!
BTW, which hall are you from? I was in Azad.