Blog — bruno fernandez-ruiz

Twisted Web IMPS Echo Client

January 13th, 2009

Having a twisted environment, I have been able to write a simple web resource handler. I like twisted, although the documentation is pretty weak, so you have to go down to read the code rather frequently if you want to know how to do things.

My first task was to handle an XML post for IMPS CSP. I thought I would skip the wbxml version to start with.

So first thing was to setup the test client. Just for now, I used curl to post the data:

curl http://localhost:8080/ -d @login-success.in.txt

Where login-success.in.txt is the first message the phone sends in:

<?xml version="1.0"?>
<!DOCTYPE WV-CSP-Message PUBLIC "-//OMA//DTD WV-CSP 1.2//EN" "http://www.openmobilealliance.org/DTD/WV-CSP.DTD">
<WV-CSP-Message xmlns="http://www.wireless-village.org/CSP1.1">
  <Session>
    <SessionDescriptor>
      <SessionType>Outband</SessionType>
    </SessionDescriptor>
    <Transaction>
      <TransactionDescriptor>
        <TransactionMode>Request</TransactionMode>
        <TransactionID>nok1</TransactionID>
      </TransactionDescriptor>
      <TransactionContent xmlns="http://www.wireless-village.org/TRC1.1">
        <Login-Request>
          <UserID>hermes.onesoup</UserID>
          <ClientID>
            <URL>WV:IMPEC01$00001@NOK.S60</URL>
          </ClientID>
          <Password>xxxxxxx</Password>
          <TimeToLive>86400</TimeToLive>
          <SessionCookie>wv:nokia.1789505498</SessionCookie>
        </Login-Request>
      </TransactionContent>
    </Transaction>
  </Session>
</WV-CSP-Message>

The payload is contained within the POST data, but there are not arguments on it (no a=x). So I needed to get the raw content. To achieve that, I created a hello.py file which would contain my Hello resource handler:

from twisted.web.resource import Resource

class Hello(Resource):
    allowedMethods = ('POST',)

    def render_POST(self, request):
        request.content.seek(0, 0)
        data = request.content.read()
        return data

As we can see, all this does is to send the payload in the HTTP POST request back to the client. To get this running on twisted, we write an index.rpy and save it in /Users/brunofr/Sites (adjust):

from imps.csp import hello

# comment the next line in production
reload(hello)
resource = hello.Hello()

A couple of comments about this code:

I put the code within the imps.csp module.
I call reload to allow me to edit the module during development.

And bang, we run it:

twisted -n web --path=/Users/brunofr/Sites

Point your browser to http://localhost:8080/. It does not work, since we are not accepting the ‘GET’ method. But if you use curl:

curl http://localhost:8080/ -d @login-success.in.txt

it works, as it should.

Twisted Virtual Environment Goodness

January 13th, 2009

Python 2.5.1 is bundled into Mac OS X 10.5 (Leopard), which comes also with setuptools. The breadth of easy_install packages is available pretty much at your fingertips on Leopard.

On the downside, Leopard’s python ships with old versions of some packages, and we might need to upgrade them. We are left with three choices:

Overwrite packages with new ones.
Install new ones and set PythonPath.
Use virtualenv.

I prefer to use #3. It’s cleaner, it does not change my system, it allows as many environments as we like without one contaminating the other, while at the same time not requiring a fresh python re-install for every working environment.

I want to create an environment for my imps-to-xmpp gateway, for which will be using twisted words. I want to use the latest version (8.2), but the shipped version with leopard is fairly old (2.5):

$ which twistd
/usr/bin/twistd
$ twistd --version
twistd (the Twisted daemon) 2.5.0
Copyright (c) 2001-2006 Twisted Matrix Laboratories.
See LICENSE for details.

So, we’ll start by installing virtualenv using setuptools:

$ easy_install virtualenv

And now, we’ll create a virtual environment on which we can install the updated twisted:

$ cd ~/Sites/
$ mkdir python
$ cd python
$ virtualenv imps
$ source imps/bin/activate
$ easy_install twisted

We can now check that we are running the latest version of twisted (8.2 as of writing):

$ twistd --version
twistd (the Twisted daemon) 8.2.0
Copyright (c) 2001-2008 Twisted Matrix Laboratories.
See LICENSE for details.

Will XMPP be the messaging middleware for the REST Web?

December 30th, 2008

We know the social web needs real-time event notifications, and that polling sucks. Some are wondering whether we can implement it using asynchronous messaging, ala Pub/Sub. The Publish/Subscribe is an architectural paradigm allowing asynchronous messaging from one sender (publisher) to many receivers (subscribers). PubSub is a common architecture in financial services, since it’s associated with persistence and ensured message delivery. But such qualities don’t come for free: there are complexity trade-offs.

Anyway, the conversation started in February, with Mickaël Rémond talking in his blog at ProcessOne (the maintainers of the top-notch Erlang ejbabberd XMPP server) about XMPP the application server, where he turned the PubSub XMPP extension into a public event firehose for twitter. And because Mickaël puts his money where his mouth is, he opened a public service doing just that, an XMPP gateway to twitter, or what twitter should have done to begin with.

Then came Internet veteran Jud, from upcoming Gnip, blogging about sockets, HTTP and XMPP, and how the in the original non-reliable Internet topology the stateless HTTP worked because it brought reliability, but that today’s reliable Internet topology could allow HTTP to be replaced by the stateful XMPP protocol.

It hit us again, at OSCON 2008 in July, with fellow Yahoo’s hackers Rabble and Kellan talking about data services streaming using XMPP PubSub. We know that having to do frequent HTTP polling sucks, and unfortunately that’s the best we can do many times, so they proposed to use a cut-down XMPP server (no roster, etc.) to build a public Publish/Subscribe messaging infrastructure. They worked it out for FireEagle and Flickr building external XMPP components, and connected a web stack through queues. XSF has gone further, proposing a new standard for chaining components, and why not, even a pubsub data stream for Atom.

Honestly, there is plenty of buzz. Brian Dainton also talked about XMPP for event broadcasting instead of REST. The folks at ChessPark, with Jack Moffitt, are doing just that with component bots.

It might be that XMPP eventually replaces HTTP for event notification, avoiding the need to poll. But today, HTTP and REST are patterns known to work. Let me examine why I think a REST approach still works better today:

The Internet, the firewalls, the monitoring, are not built today for persistent connections. XMPP real-time events are possible since a TCP socket is kept open. BOSH and XMPP over BOSH make it possible to run XMPP over HTTP, and provide almost real-time notification. I like BOSH, a lot, but it is still an emerging technology that needs plenty of love. XMPP metadata is still not implemented in BOSH, and the client-side implementations are weak.
Running an Apache server and setting a up a REST service is almost straight forward. We know how it works, how it scales, and how it fails. Running a XMPP server, even the excellent ejabberd, is not as easy as running Apache, and most folks are likely not to run their own XMPP server and rely on third parties like jabber.org or gtalk. The good folks at gnip cut down on XMPP because of lack of reliability of the XMPP third parties. XMPP, still, is too complex to operate.
Supporting binary data is XMPP’s achilles heel.
There are simpler HTTP only alternatives. Brad Fitzpatrick implemented a never-ending Atom feed for the LiveJournal update stream years ago. A partial solution to reduce polling, although not real-time, is Roy’s better resource mapping. And let’s not forget the straight forward callback. Oh yes, running an Apache server on your mobile phone is possible.

Will XMPP become the messaging middleware for the web? Eventually, perhaps, but in today’s environment, neither the infrastructure or the software are ready to support such a model. As importantly, there are simpler alternatives to achieve real-time event notification without requiring the publish/subscribe architecture.

Wireless Village Headed to the Deadpool

November 26th, 2008

I wrote about this header a little while ago in my onesoup blog, but since I have received various emails from folks asking me whether I’d continue or not with onesoup. Basically, they all make the point that there is a latent customer base on old phones with the native client.

So the question really becomes: even if IMPS is a lagging technology, even if operators are unfriendly, even if nobody knows how to monetize IM, … the penetration of these clients is huge. And users seem to be waking up. So, is there really a blooming market here, especially in emerging countries?

Could the financial downturn be the end of Darwinism

October 11th, 2008

Because that’s really what it is. Somebody that was not fitted and loosing big (aka Western Cultures) has just started a new game of Monopoly, and you and I are not getting any of the money in this new game. Some may call it the end of capitalism, but I would go even further, this is the end of Darwinism (and a prove that Hegel was looking at the past, and not the future like Marx wanted to see in it).

There are unknown consequences for any deep interventionist economic policy. Anybody who says so it’s either a fool or badly naive. But worse of all, intervention of this kind defeats the system as we know it, where the fittest survive.

Without intervention this crisis would have been a chance for those without debt to compete, i.e. the very poor. All that we will see is further foreign investment from large sovereign funds, and that hardly benefits the poorest nations in the World. I am very disappointed by our politicians lack of pro-activity and foresight.

We live in a culture that encourages risk, and hence competition and growth. But now we have simply wiped out the consequences of failure. The failed specimens survive, unlike what Darwin taught us …

Can search be entertaining?

August 20th, 2008

Definitely! I just got completely blown out by Oamos. The search experience is fantastic. Images, videos, transitions and music are wonderfully inspiring. Ping from Artur Ortega.

Why Video-On-Demand Streaming in Airlines Sucks

August 19th, 2008

It’s rare to take a flight from British Airways or Virgin Atlantic where the Video-On-Demand system just works. It might be a problem only with UK airlines, but I presume this is a general issue across aviation.

Sometimes you are part of the folks in the plane for whom the system does not work. The crew will normally start rebooting your seat, then the section, and then the whole plane. I guess they do so in hopes that the operating system running the streaming adheres to the same random rules as the one running on their desktop.

In case of an incident with the entertainment system, i.e. in every single flight, the crew keeps apologizing about not being able to fix the problem and continue performing reboots left and right. Reboots are seldom able to address the problem, and the only thing reboots achieve is general frustration, both from those without video, but mostly from those with video who see their programs shut down. Eventually, the crew settles down to multicasting a few movies.

Virgin Atlantic even has the courage to advertise Virgin Media Broadband and TV packages on a commercial playing before the movie. Enjoy this cool entertainment system (that does not work) at home just like you do in the plane. I am no marketing guru, but that sounds to me like a dumb ass campaign.

I’ve seen a few times a linux boot-up sequence on Virgin, and I’d be surprised if BA was not also using Linux. Most likely, they are both using VideoLAN against one or multiple streaming servers with network attached storage. For whatever reason, it’s the connectivity that fails, and the individual Linux workstations that fail to connect to the streaming server, or for the streaming server to be able to read the files from storage.

The reason to use attached storage is two-fold: cheaper, reliable and easier to update. But, guess what, wired networks don’t seem to be reliable on a plane (probably because of the high-frequency power lines). So here’s my advice folks: avoid the network (single point of failure). Here’s how.

Virgin Atlantic has around 100 movies on inventory, plus TV series, CDs and radio programs. I’d bet their total inventory at any given point in time is between 250Gb and 500Gb. Maybe the solution to use shared storage made sense years ago cost-wise, but today the cost per Gb has fallen so much that internal disk storage could make more sense. Specially since there are no availability requirements (if it fails, a simple swap fixes the issue).

They could get a 1Tb disk per seat and image (push on the ground) all disks with the OS, the video player plus and the full media library. Re-imaging will take a few hours, but the inventory only gets fully updated every 2 to 3 months. Every seat would become an autonomous and reliable processing unit with its own storage. And in case of failure, it would only cost $100 to swap disk and reboot one seat, something the crew could do in-flight. No apologies, no pissed off customers, no compensation.

So, guys, go and implement it, so that I don’t have to watch the beginning of The Oxford Murders from Alex de la Iglesia three times on my next flight.

Enterprisey Architects

July 20th, 2008

No, it’s not a typo. I have been daring to write this for ages, but somehow I always refrained myself from doing it. But here it is, this post is about the propeller heads, the skyrocket scientists designing systems in the enterprise world (mostly in the financial services and telco world). Because guys, it’s time to call it a day and move on. Time to stop building the Winchester House.

As background, I have recently been recruiting to backfil Remy, Yahoo!’s Chief Architect for Europe [and no, I am not looking any further]. It has proved to be a hell of a task. My mailbox and voicemail have been flooded with messages from agents and candidates. A pile of resumes to go through.

Most resumes are immediate rejects. Absolutely zero experience in online technologies. Do you apply for Yahoo! if you don’t have a clue about even building a site? WTF!??? And, err, where is your blog, buddy? This is not just any job.

Anyway, moving on to interviews. I called in a few guys with very solid resumes. Outstanding resumes actually. Tons of experience. Lots of technology expertise, masses of troops, managing huge budgets. Right, bring them in, I say.

On the interviews (so that you know) I don’t look at the resumes, ever. I could not care less, because you are allegedly supposed to be able to synthesize and I can read. So, expect to work hard at the interview. Rather, I’d love to see you design flickr, or Answers, or tell me how the frontpage could/would/should work, or, for fun, how you would liberalize the water supply system in the UK. I bring engineers on the room to discuss. They need your help, guidance, steering for making choices. Help them!

All I want to learn is that you can think beyond a Powerpoint and a bunch of theoretical ivory-tower diagrams. And you know, most often, candidates can’t. The large majority of candidates when going through the exercises just put a few boxes up and don’t actually get into what are the hard bits. And when challenged to think through them, they fail. I admit you might not know how to build our frontpage, but at least you should be able to appreciate the complexities during an interview.

The interesting thing is that all these candidates that fail have a common trait. They are the enterprisey architects (a few exceptions exist, like my good ol’ friend Jeremy who is truly an enterprise architect). Enterprisey architects come with certifications, waving their TOGAF and Zachman willies. They operate so high-level that they ain’t got a fucking clue.

Enterprisey architects make the argument that architects don’t need to actually know the details (I don’t code, they say). First, there is a hell of a difference between coding and designing systems and you seem not to be able to design systems. Secondly I honestly think there is something seriously wrong with anybody who wants to be a technology geek and does not like programming languages per se. It’s like being a butcher and not liking meat. I’ll tell you what: if you are not obsessed by [technology|volumes], you are not a [technology|civil] architect.

An enterprise architect is the architect that looks at solving problems across the enterprise, not the department, or the business units, to leverage the synergies and lead towards the future-state architecture. [Don't look this up, I just made this up].

An enterprisey architect is the architect that wants to be an enterprise architect but fails to engage with his customers, because he’s not capable of being detailed enough (business strategy detail, functional domain detail, applications and technology detail, and infrastructure detail).

And as I keep telling my guys, (enterprise) architecture is not about technology, it’s about people. You are expected to know the technology, mind you, but there are usually much more specialized engineers who need your steering for making choices (if they aren’t such experts in your organization, you should start by building your team), so that you can find the detail that is relevant. You are supposed to understand the bigger picture, and lead folks into doing the right choices, so that you are not only doing things right, but also doing the right things. Enterprise architects use their social networks to make technology changes happen.

Again, it’s not about technology, it’s about people. But you know your technologies, right?

what the iPhone is that Symbian isn’t

May 13th, 2008

one of the key strategic questions that developers looking into creating successful mobile applications need to answer is the choice of platform. we see an increasing consumer demand for desktop-like rich applications on the phone, but somehow the promise of such applications remains undelivered. developers can chose between the smartphone centric OSes, Windows CE and Symbian, or the desktop centric OSes, Linux and OS X. so which one should we chose to build those future desktop-like applications?

smartphone centric OSes are designed to address the limitations of the constrained mobile environment, while at the same time delivering a rich user experience. typical constraints range from input methods, battery and connectivity. let’s be clear, these are good phone OSe,s, and they make good phones. but interestingly, these are constraints that have started to go away, and are becoming largely irrelevant, if not already irrelevant.

mobile network connectivity now offers full TCP/IP stacks on always-connected phones under flat fee transfer rates. battery life on most new phones allows such stack to be always on without significantly affecting the batter life, and input devices such as the iPhone demonstrate that the paradigm can be changed for the best.

in contrast, desktop centric mobile OSes are not really designed for mobile environments, and mobile is just an add-on. where the smartphone centric OSes need to impose API limitations, the desktop centric OSes don’t impose almost any. where the smartphone centric OSes excel as phones, the desktop centric OSes tend to lack important features (the iPhone is a good computer but a really bad phone).

and the question really for developers starting now a new application is whether to develop for such phone-focused OSes, given the large market share, and hoping that they will be enhanced sometime in the future to become full OSes, or whether to go for desktop centric OSes, with currently a smaller market share, and hoping that the mobile features on the them are sufficient. in other words, what’s more likely to happen, a desktop centric mobile OS being enhanced to become better phones, or a smartphone centric OS being enhanced to become a better desktop?

the answer to this leaves within the OSes themselves. today, the iPhone is an excellent mobile computer based on the OS X, but it’s a really bad phone. however, the mach/bsd kernel, the quartz framework and the cocoa framework should make it relatively easy to add GPRS and UMTS to an iPhone. and possibly the only reason it’s not there is purely commercial, based on the current distribution deal with AT&T.

in contrast, you can’t make a good desktop OS out of the smartphone centric OSes. they are simply crippled. i find it surprising to see how analysts overrate the symbian os. technically, symbian is a really unattractive and unproductive development platform to work with. commercially, it’s a fragmented market with three competing and somehow incompatible platform variants: nokia’s s60, sony-ericsson’s uiq, and ntt docomo’s moap.

from a technical standpoint, symbian has its own memory management and exception handling mechanisms which lead to a crippled and unfriendly C++-ish variant. symbian support for string handling is as bad as it gets, and even php does a better job! finally, writing complex interaction paradigms and network access is daunting: multi-threaded applications for symbian requires rocket scientists. and finally, and most importantly, the development environment is simply non-existent, mostly because of the custom C++ memory management and complex deployment process.

very sadly, java does not make a much better symbian. although language and tool support is excellent, the lack of uniformity on the platform itself, with MIDP 1.0 and 2.0, being implemented to different degrees by different vendors and by models, has resulted in tens of different APIs supported by some but not all phone models. it’s really hard and time-consuming to develop for java applications for symbian. either you create multiple variants of your application, or you cut down to the least common denominator. neither option is attractive from a developer standpoint.

windows ce suffers similar problems to symbian. the programming model is overly complex and based on the highly developer unfriendly win32 api. alike java, the .net compact framework has not been consistently adapted, so one ends up with multiple binary targets, that require separate code, debugging, testing, distribution and installation. a nightmare from a developer’s perspective. on the plus side, windows ce development environment is superb, including remote debugging abilities on the device itself. but overall, i discount windows ce as a no-go platform.

this really cuts it down to desktop centric OSes being a better choice for development going forward. that’s really between OS X and Linux. and i am picking OS X and the iPhone. here’s why.

as with anything linux, mobile linux is really many many things. different vendors have implemented things differently and are providing different stacks. some offer Qtopia, from Trolltech, some offer Gnome. then you have android, openzaurus, openmoko, … no one phone has a significant market penetration, and to develop for linux really requires multiple binary targets and API abstraction layers. again, it’s making things more complex than it ought to be.

in terms of kernel and integrated development environment, OS X is a much better platform. it’s a highly productive environment. but it’s proprietary to apple. which is good and bad. it’s good since you know what hardware and software you need to support. it’s bad since you are locked in, including for distribution.

but here comes the twist. developing a cocoa-based application for the iPhone is extremely easy, and given that Linux and OS X (mach/BSD) share so much in common, moving an iPhone application to a linux variant would not be any harder than moving between Gnome and Qt. good OO design should make it relatively easy.

in summary, iPhones features a robust and rich OS, OS X, a highly productive development environment, an aggressive market share growth, and lots of similarities with Linux. these are all good reasons to me to pick the iPhone as the target mobile platform of the future. i am not fully discounting linux however. the current issues with lack of significant market penetration could be overturn, specially in asia, and particularly in china.

Why facebook is just a game

May 2nd, 2008

I keep saying that the web follows the pattern of the fashion industry. It’s about image, it’s about fun, it’s about entertaining. Sure, you can also buy stuff and do useful things. But that’s what you have to do, not what you want to do.

That’s why facebook is popular – it’s fun, it’s about image, it’s about entertainment.

And now, to prove it, a nice chart for the breakout of Facebook applications.

Facebook application breakdown