From a thread on multicore leverage in nodejs, Edwin Khodabakchian talking about feedly:

We have an “admin” node process which is started with as input the number of “feedly” node processes it should launch and monitor. When the admin starts, it spawns multiple “feedly” node processes, padding as input the port the feedly node should listen to 9701 for the first one,…9710 for the 10th one.

On the server we have a varnish server which load balances the traffic from m.feedly.com to 127.0.0.1:9701 — 127.0.0.1:9710.

Each “feedly” node process has a 127.0.0.1:970x/heath endpoint where it reports stats about its execution. The admin node monitor each feedly node every 2 minutes and makes sure that the feedly node is healthy and responding. If not, it kills the child and restart a new instance of the feedly node listening to the same port.

All feedly nodes point to a redis instance for shared memory/session management. This allows incoming HTTP requests to be load balances across any feedly nodes transparently. In dev/staging, we have varnish and all the node processes running on the same box.

In production, we can scale out by having varnish on one server and multiple node cell (where a cell is an admin+10 feedly). This allows us to do rolling upgrades, etc. without any interruption to the service.

I am intrigued by node.js. Others are not so much. So they ask me, why node.js, why not Jetty, or Netty, or … ? Others say, and Twisted, and EventMachine? The core of the answer lies in a simple not well understood truth: concurrent programming is really hard. Concurrent programming using threads and locks to shared memory is extremely hard. Most programmers get concurrency wrong, since they don’t quite realize that threads actually execute simultaneously.

Functional programming languages have long understood this, and designed mechanisms to avoid the risk of a programmer getting it wrong, such as immutable data structures (take a look at java.lang.String, designed by Lee Boynton, a true Lisp guy), or message passing (erlang).

Surprisingly, another line of programming has ended up in the same place. PHP can be deployed multi-threaded on Apache, but unfortunately most native libraries are not thread safe, so one ends up running processes as workers. Similarly, Python has the GIL, and Guido has no interest in fixing that. And yet, PHP and Python scale really well using processes.

So, here’s the interesting bit about node.js. It’s written on a functional programming language, Javascript, and provides a reactor server (remember POSA2?) on a single thread! I hear you say, how does it scale to multiple cores? It does not.

A single thread is only able to utilize one CPU. Even though most web apps are initially I/O bound, at some point programmers somehow manage to make them CPU bound (such as marshaling and wrangling JSON and XML). This will limit the scalability on the single process. In order to scale beyond 1 core, two possibilities exist. Right now, one would need to launch one node.js process per core, and then setup a load balancer. In the future, launching the processes needs to be done inside node (maybe this is fixed in the future?). The second option, which might also be implemented based on the node.js web site, is to adopt the Web Workers spec.

Now, because of the (single) threading model, response times are fairly unpredictable. One would expect a Java server on an async stack, such as Jetty, to provide more consistent response times, as it uses multiple threads. A test corroborates such hypothesis (look at the curve distribution for node.js vs jetty).

At the same time, because of the (single) threading model, memory usage is very low in node.js, compared with Netty, or Jetty, where as a result of being multi-threaded, memory usage will be higher for the same load and latency. A discussion on ycombinator seems to be along these lines. I am of the opinion that being able to run both on low and large memory profiles is critical to becoming successful, especially as many developers run their code in shared hosts.

If you have followed me this far, putting memory aside as that can be shared on the heap, you will then conclude that a multi-threaded Java-based async I/O server, like Jetty, should be better that node.js. And I would agree, only if it was not Java.

So here’s the deal. Except a counted number of exceptions, Java libraries carry significant legacy of the wrong kind of concurrent programming, either abusing or ignoring locks. Either they do not scale to multiple cores, or they corrupt integrity of state. And this will continue to be a problem, since Java does not prevent bad programming practices in a concurrent setting. But there is a light: using the JVM, ignoring Java and most existing Java libraries.

Building new libraries for the JVM that are truly designed for concurrency requires using the right abstractions, and this is where functional programming comes into place. There are three key functional programming language contenders to the JVM: Javascript (Rhino), Scala, and Clojure.

First, there is Javascript, with Rhino, Mozilla’s Javascript implementation for the JVM. Ringo is a nodejs alternative based on Rhino. Unfortunately, Rhino is slow and quite a memory hog. Additionally calls in Ringo are blocking, which defeats the programming model. And Ringo uses the same Java libraries that were not coded with concurrency in mind. Compared to nodejs, Ringo has no community (a quick scan reveals 9701 messages in the node.js mailing list in 12 months of usage, against only 181 in 2 years of usage).

Next, come Scala and Clojure. Both are very intriguing, perhaps Clojure being more of my taste. For Clojure, we now have Aleph, a thin Netty wrapper. On multi-core, it seems to beat node.js latency and throughput wise, at the cost of memory. And Clojure allows message passing, immutable data structures, ala Lisp, all transparently mapped onto threads. Beautiful I say.

But Scala and Clojure are not Javascript. Javascript attracts many developers from the browser world, it provides a consistent event-driven programming paradigm for both server runtime and client runtime. Eventually, this pushes most UI logic to the browser, and leaves the server as a smart data source. As much as I find Clojure extremely exciting, I don’t think it can realistically compete with Javascript.

So, realistically, only node.js is a viable alternative today. Maybe Clojure truly succeeds, and a Clojure based async I/O server appears. But for now, node.js is more interesting.

But don’t get me wrong: node.js is very young, truly bleeding edge, and needs substantial work. Some of that work will make it easy for applications to scale beyond one core. Scaling beyond one core will improve consistency in response times. But given the vibrant community and excitement (and hype) around node.js, I am fairly confident this will happen.

Tomorrow, I’ll be flying to the US and leaving the UK. But this time, it’s different. Now, it’s a one way flight. But that flight is the end of a journey that started four months. A difficult journey full of live experiences.

Back in April, Yahoo made me an offer to move to California, with my family, to take on a new role. Since then, we’ve had twins, boy and girl, moved house once, finished our ever-lasting DIY project, a little summer bungalow in the middle of Kent, only this time using contractors, sold our cars, packed everything, cancelled the school, the utilities, the direct debits, the payments, changed addresses, changed insurance policies, and finally put our lives on a plane.

And it is surprising how many things are broken, that you only realize when you get out of the usual cycle. Returns for faulty products that take weeks, P&C carriers that don’t want to take your money. Or the complexity of servicing or selling your car when you only have one car. The utter absurdity of BT, making as pay 9 months of non-fulfilled line and broadband contract. A TV license officer poking around thinking that everybody in the UK must watch TV, actually the BBC, arguing that otherwise they are weirdos. A council planning officer arguing that our carport is closer to the highway than the main elevation of the house and asking us to pay for planning permission, even though every single aerial photograph shows they are incorrect. A school asking us to pay a full term of tuition, and then denying our daughter from attending. Or well, the length of processing the legal paperwork to work in the US with a Finnish-Spanish family.

It is also saddening, because of many things aren’t broken and we have to leave a little behind. Like the goodbye photo-book our eldest got yesterday from her classmates. Or our beloved Greek-German friends that we’ll miss so much. Or my mahogany Bechstein 1980 model M. Or my new Startrek home office. Or the beautiful views at dawn over the grazing land from our conservatory.

Tomorrow, a new exciting period in our lives starts. As much as I am sad about leaving, I am also thrilled about the opportunity to work for Yahoo! Inc in its Sunnyvale headquarters doing one of what I think is one of the most exciting jobs available in Yahoo! and in the industry. And it’s not just me me. We all are excited. The wife is clearly hoping for more days of sunshine, which should not be difficult to meet, easier life with children, and most definitely, not DIY in years to come.

On my previous entry on the present and future of programming languages, I briefly covered on the reasons I think it is important to be looking at this problem now. I though I would expand the discussion.

The laws of physics that we know have stopped our ability to make chips significantly faster today, and rather hardware manufacturers now need to place more and more cores in one die. The result as software developers is that we are now faced with computers with multiple cores and multiple CPUs. It is now the norm to find computers with 2 cores, and most servers are now 4 or 8 core machines.

Historically, developers lived on the free lunch, and with time they knew code would run faster, since clock speeds would get faster. Chips will still continue to get faster in the future, but marginally what we’ve seen over the last 18 years. Today, it’s not about running faster, but having more transistors. And what once was a good programming paradigm for the single core, it’s not valid any longer for the multi-core situtation.

Here is where concurrent programming is born. Writing software that scales to multiple cores with the current programming paradigms is hard. The main mechanism we know is the usage of threads, and locks to protect shared estate between threads. It is a model somewhat understand by developers, but that it is however extremely hard to get right.

On the server-side, programming languages that can only leverage one core are going to have a really hard time to compete in the future. Programming languages that have adopted the thread-per-request model are better placed, but will nevertheless face a wall at some point because they rely on locks to operate to protect the transactionality of shared state. Multi-threaded programming and locks are not scalable, mainly because they are not composable.

Building composable software really requires us to move away from sharing state. Working with immutable data structures, and immutable containers, allows us to avoid state, hence allowing modularity and composability (the core principle of software development for the last 35 years, and the philosophy of UNIX tools). But working with immutable data goes against extensibility principles in object oriented programming.

There are a few paradigms that have been in the research arena for the last 10 years and are now finally being adopted. These paradigms take the shape of message passing, parallel programming and functional programming. Functional programming languages are naturally parallel and use immutable data. Unfortunately functional programming languages are not mainstream, and most functional languages that have been successful (Haskell and Lisp) actually allow mutable structures. Erlang however is an exception. Erlang is naturally parallel, uses only immutable data, and is based on message passing. Erlang is a good language to solve the concurrency programming paradigm.

The question though is whether adopting a language like Erlang would make sense for most organizations. Rather, some may argue that evolving the abstraction of concurrency into the current leading programming languages is the way forward. I’d disagree since this is somehow what Scala does, but given that it lives in a JVM, with threads and locks, and using libraries that use threads and locks, it’s unlikely new language constructs will allow existing code bases to be fixed. To me, Erlang looks like a much better choice. It’s a proven choice and with many years of history.

I believe being a polyglot is nothing but an advantage, and that polyglots are normally the best programmers. As a matter of fact, I challenge myself to learn a new programming language every year. Last year it was Scala, this year I’ve started learning and writing some Erlang (and yes, there is a pattern here for functional programming languages).

One of the reasons I like learning programming languages is a drive to reach the Holly Programming Grail: I find most fascinating discovering which language is better suited for which task. In other words, some languages excel at tasks where other languages fail. An organization could decide to allow all those languages to exist, and if we were doing best-of-breed in every problem domain, we would end up with a language soup. So the question then is, is such soup good or bad? In other words, when we think about developing Internet applications: which and how many languages should an organization use?

There are two factors to consider in choice of language choices (assuming the language choice is functionally the best choice for solving the problem at hand): talent and operations.

Talent is truly an interesting one. Whereas a startup can possibly afford to pick more exotic languages based on their goodness, large organizations can do less so because of talent. This is because A-class developers will excel in any language, and usually A-class developers are polyglots. But one can only hire so many A-class developers, and eventually you end up growing into B- and C-class developers. Your ability to maintain software at that point really depends on your C-class hiring pipe. Pick a rare language choice, and you are calling for trouble.

Operations is very different from talent. Here the size of the company also matters. Developer cost will be the starting cost, but as the business becomes successful operational cost becomes the key driver (specially in consumer Internet applications). In a way, you want to pick the language choices that drive your productivity higher, and later slowly move into reducing your operational cost.

If you follow me, you’ll notice talent and operations are conflicting today. Java is the mainstream programming language worldwide, i.e. A-, B- and C-class talent is widely available, yet scaling Java horizontally at linear cost is difficult. On the opposite end, Erlang is a good choice for distributed concurrency and scaling linearly, yet B- and C-class talent is difficult to find.

I used to advocate the JVM as the minimum host environment, yet allowing language choice across the stack. As long as it could run on the JVM and I could manage it, any language was fine. One could run the complete stack on a JVM. The presentation tier in JRuby, Jython, P8 or Caucho’s PHP implementation. The application logic and persistence in Java or Scala. The batch computing in Scala. But it just does not work that well. Even though Scala has the artifacts to work, the JVM and the runtime libraries don’t. Shared memory, native threads and locks means the JVM will be more expensive to scale and operate on the long run. The JVM is a feasible, stable and proven alternative, yet it is an expensive choice (in operational cost and probably lost opportunity because of slower agility).

Given this context, my language choices would be:

• Conservative stack, which gets the job done well enough, can be operated, and you can get accessible talent.
  • Frontend: PHP
  • Application Logic: Java
  • Persistence: Java and C/C++
  • Batch: Java
• Medium Risk stack, that addresses some of today’s shortcomings, but reduces the talent pool:
  • Frontend: JRuby on a JVM, with Merb
  • Application Logic: Scala on a JVM, with Jersey
  • Persistence (non-relational): Scala on a JVM
  • Batch: Scala, with Hadoop
• High Risk stack, forward looking to the highly multi-core CPU roadmaps announced by Intel and AMD, that reduces even further the talent-pool:
  • Frontend: JRuby on a JVM, with Merb
  • Application Logic: Erlang. Actually, I’d love to see a Scala-like language targeted to the Erlang VM to make it more palatable. But for now, I’d settle on Erlang.
  • Persistence (non-relational): Erlang
  • Batch: Erlang

If you have been following the media industry over the last years, you’ll have surely realized that traditional media is struggling. Even before the crisis, advertisers have started switching from offline to online, and now with the crisis we see a consolidation in a few ad agencies and a few brokers.

Many magazines and newspapers are in such position that if their debt does not get re-financed they will have to file for bankruptcy. Many music labels find themselves in a very similar position, or have already gone belly up. Video houses are struggling with costs. It’s not possible to compete with pirated content. TV channels are going bust.

But there is also a lot of smoke-n-mirrors going on.

I hear claims. Some argue that traditional media is dying. Some advocate social media has taken over. Some forget the economics of running a media business. That the article is dying. That head content is now irrelevant. That the democratization of content production will increase quality. That user-generated content is the future.

Others blame ISPs and the likes of Yahoo, Google, AOL for this. They talk about stealing content and taking away the advertiser money flows the so badly need right now to survive. Some go preaching about how policing the unregulated wild wild web with Digital Rights Management is the only way for them to remain profitable.

What is for sure is that the media industry is suffering. What is also certain is that consumers are generally rational, and make purchasing decisions based on utility. Many consumers are today rather pissed off with the traditional media. And I believe what we are seeing it’s just a consequence of the media business own behavior, being extremely zealous to protect their content rights with solutions that make the consumer experience worse, rather than better.

Is it not annoying that you can’t, legally, make a copy of a DVD for backup purposes? Is it not annoying that you can’t buy DVDs while you travel to other continents? Is it not annoying that your DRM tracks only play with proprietary technology? Is is not insane that games are region locked?

Like Lola would say, I surely think it’s extremely highly very annoying.

Long, long time ago, information traveled slowly, snail pace actually. Controlling where a film was being shown, or who had the tapes, or who was selling the newspapers was a relatively easy thing to do. Yet, already then, there was sharing. You’d share a magazine at the hairdresser’s. You’d share a VHS tape. But, for most, things were controllable since the medium was perishable, and things were not flowing the pipes, yet.

The Internet has changed it. Information is available at speed of light, at whatever bandwidth you have contracted. Sharing is easy now. And because they can, and because that’s how humans are, consumers want to share what they own. The same way they can share a book, or photos, or toys, …

Revenue starts to go down on the rich media side of the business due to pirated copies flowing on the net. So DRM moves in. Badly in. Actually completely gone wrong. It alienates consumers. Napster, eMule, Kazaa, … all appear because consumers want it. But truth is that DRM is protecting some of the revenue, and that’s why content right owners keep lobbying for it. But it’s still going down the hill. High-Definition media appears and somehow fixes partially the problem. But it won’t last long. It’s only working because of constraints in bandwidth.

Paper media businesses have trouble getting audience. You’ll find many reasons for it, but the key underlying factor is that they’ve ignored speed of light. Information can flow immediately. And if it can, it will. Those media businesses that understood and embraced this, new incumbents mostly, but also some old faces, have been doing better. But they have made the industry more competitive as a consequence,aAnd those that did not understand it, find themselves struggling.

So what do you when you are losing? For starters, you find a scapegoat. Done that. Ironically, deep inside, you realize your business model is gone and you are just trying to hang onto it for as long as possible. In reality, you are just buying time. Precious time you don’t have, to figure out the next business model.

Yet, one has to ask where did it all go wrong? Audiences are actually not spending that much time on tail content. High quality production “head” content is still where online minutes are spent. Although I put some value to UGC, I still believe good productions require good funding. Good talent needs to be compensated. Be it via ads, via subscriptions, via concerts, …

To me the issue that traditional media needs to address is how to operate their businesses at a whole new level of revenue and cost. Since revenue levels per production unit are down by one order of magnitude, one must produce significantly more, and do it a much lower cost basis. I can think of three things the media industry can do to achieve this:

• The tail, by definition, is long. Sorting out the tail is a mess. To start with, go manual, and use your editors to find the gems in the tail. Cream it out. Find the new head, at no production cost. But don’t stop there.
• In parallel, invest in algorithmic matching. Eventually you want to surface all the metadata in media to replace the editor. We can’t today. But it should the goal.
• Production costs are high and need to be shared among all players. By sharing production costs with your competitors (you hear me well) you are actually removing those cost components that should be commodities, so that you can move up the stack to solve the creative problem better. Likewise, if you can share the production costs with the long-tail of user-generated productions, all the better. You are basically getting that subsidy you need at production time. Since you have addressed costs, you can now play with content pricing. You can charge, or you can go free. You have gained that freedom back. You can design new business models. Consumers benefit. Producers benefit.

The first item is really a way to get going. The latter two are truly strategic, and are what I call the “open media infrastructure services”. Think of it like the Amazon services for media production and distribution.

Much of that infrastructure already exists for distribution, as the channels of pirated content, and they are actually almost free. Production is not there yet, and will require investment. But without taking these steps, I am doubtful head content production will last. Or if it does, it will do only out of government subsidies, using taxpayers money. And that, really, would be a pity.

Looking back at the past 20 years of the history of the internet, one can realize that what we call today the web has changed dramatically, but that one thing is certain: it will continue to evolve. Gone are the days of manually editorialized directories, of static content, of one way conversations. Today we live in a dynamic, social, interactive web. And it’s a web that is traversing the digital realm and changing how we understand our physical world, breaking communication barriers and making information and knowledge easily accessible.

Yet, although much has changed, the web remains an almost completely deregulated environment, not only on the legal side, but also on the technical side. That’s one of the web’s merits, making it easy to publish documents. Those documents are part of a forest, many sit in isolation, and only a small number of links are connecting those documents. The hyperlink is the web’s way of connecting documents, yet, because the web is a flexible environment, there are no requirements on developers as of how they structure their documents, nor as what link schemes they use.

But what is an advantage on one end, reducing the barriers of entry to publishers and developers, is also a disadvantage on the other end. We end up with billions of documents that are difficult to navigate, and the information, although there, is difficult to find and access.

Organizing the web has been a growing pain since the early days. As the web evolved, so did the technologies used to organize it. In the beginning, only a few universities and government agencies had documents, and it was feasible to maintain a directory. Later on, commercial entities entered the web at exponential rate, and business were created just to solve exactly this problem: organize the web.

The web is growing so fast, that we are not able to reasonably scale the production and maintenance of web directories. And there came web search engines, based on the information retrieval research of the late 80s and early 90s, to solve this very problem. But search engines are starting to show their age:

• We are accustomed to a web that is almost infinite, so deep and wide that we know that not even search engines can reach the very end-leafs of it. We know that the long tail is getting longer, and that the amount of disconnected content is just growing every day.
• Traditional media is struggling to find its place in the web social ecosystem. “Head” content is expensive to produce, and is becoming less and less profitable to operate. The audience is moving to tail and (social) user generated content, where it spends hours. Media businesses, as we know it, are dying slowly. Content is no longer king, and guess what, search can’t find the new king.

Today’s problem is also today’s opportunity. Interestingly, the tools necessary to fix this are almost in place. This the “Atom Web”, or a web linked by Atoms, which some would perhaps deem as a bastard child of the semantic web. The Atom Web is a web where the hyperlink is really empowered as the new king, and takes the crown away from content. It’s a web where document and link meta-data is more important than data, just like the PageRank algorithm changed search by using link (citation) flows as a measure of relevance.

The Atom Syndication Format and the Atom Publishing Protocol have the capabilities to change the web as we know it, well beyond the initial purpose to serve as a content syndication mechanism. Atom contains almost all the necessary meta-data missing in a HTML document, plus it provides a neat way to link content. But to achieve this, we need to empower Atom by introducing three new principles:

• Content is opaque. We shouldn’t accept (almost) any extensions to Atom. Atom should be reserved as an envelope containing only meta-data, with the source element containing all the content or a reference to the content resource. Perhaps in a few cases we might need extensions, like search, paging, presence and location. But rather, I see the very large majority of extensions as content.
• Post Atoms, Get Feeds. Atom entries have little meaning by themselves. Instead, they live in collections through feeds. An entry may belong to many collections. Collections are views, representations, of the underlying documents, grouped in a wide array of combinations. Feeds can represent pretty much anything: the revision history of an entry, all known content about an entity, a blend of categories and topics, related content, etc. Congruent collections are much easier for a UI consumer, as a simple request returns everything necessary to render a widget, a module, or a complete page, perhaps even as JSON directly from the browser. Notice that I am not assigning any semantics to the feed, that’s up to the implementation, but specifying that a feed is the first class citizen, followed by the entry.
• Links are type-aware. Links are the most important part of Atom, and content-types are the tools of choice to help bring semantic value to the relations (rather than custom schemas extending Atom). In a way, the Atom Web is a reduced subset, a predecessor, of the Semantic Web, since we don’t necessarily know the subject-predicate relationship for all structured data (yet). I rather think of something along the lines of <link type="application/atom+xml;content=application/foo+xml">, which would be describing the content-type inside the referred Atom document. Obviously, you may still point directly to the underlying resource, because you can, and because that’s how the web works today.

Certainly, there is a bit of web infrastructure required to make this true. The same way we have the web server for the sometimes-linked web, we need the atom server for the always-linked web. But an atom server is not just a web server adhering to the the AtomPub protocol. Atom server implementations can provide whichever machinery they want as long as the adhere to the three principles above. An atom server needs to have the ability for publishers to define collections, or more precisely, the matching rules that will define views, and which will be exposed as collections, through feed resources. Such view generation will be offloaded partially to batch computing, perhaps on the form of map/reduce jobs. Some implementations may for example decide to use schema-less document stores in order to generate those views.

One could get started today with this and build a gigantic store replicating copies of the content hosted elsewhere on the web. The store could be populated either by crawling or by allowing developers to publish into it. But eventually, this store would have problems finding all the deep end-leaf documents.

A few big players adopting Atom to expose their data, for example making the webmap available through Atom and opening the API for publishers to post onto the store, could dramatically change this game. There would still be plenty of problems to solve, ranging from the now even more important link-spam detection and moderation, to the not so obvious scaling problems, but at least we would be one step closer to organizing the web.

We know the social web needs real-time event notifications, and that polling sucks. Some are wondering whether we can implement it using asynchronous messaging, ala Pub/Sub. The Publish/Subscribe is an architectural paradigm allowing asynchronous messaging from one sender (publisher) to many receivers (subscribers). PubSub is a common architecture in financial services, since it’s associated with persistence and ensured message delivery. But such qualities don’t come for free: there are complexity trade-offs.

Anyway, the conversation started in February, with Mickaël Rémond talking in his blog at ProcessOne (the maintainers of the top-notch Erlang ejbabberd XMPP server) about XMPP the application server, where he turned the PubSub XMPP extension into a public event firehose for twitter. And because Mickaël puts his money where his mouth is, he opened a public service doing just that, an XMPP gateway to twitter, or what twitter should have done to begin with.

Then came Internet veteran Jud, from upcoming Gnip, blogging about sockets, HTTP and XMPP, and how the in the original non-reliable Internet topology the stateless HTTP worked because it brought reliability, but that today’s reliable Internet topology could allow HTTP to be replaced by the stateful XMPP protocol.

It hit us again, at OSCON 2008 in July, with fellow Yahoo’s hackers Rabble and Kellan talking about data services streaming using XMPP PubSub. We know that having to do frequent HTTP polling sucks, and unfortunately that’s the best we can do many times, so they proposed to use a cut-down XMPP server (no roster, etc.) to build a public Publish/Subscribe messaging infrastructure. They worked it out for FireEagle and Flickr building external XMPP components, and connected a web stack through queues. XSF has gone further, proposing a new standard for chaining components, and why not, even a pubsub data stream for Atom.

Honestly, there is plenty of buzz. Brian Dainton also talked about XMPP for event broadcasting instead of REST. The folks at ChessPark, with Jack Moffitt, are doing just that with component bots.

It might be that XMPP eventually replaces HTTP for event notification, avoiding the need to poll. But today, HTTP and REST are patterns known to work. Let me examine why I think a REST approach still works better today:

• The Internet, the firewalls, the monitoring, are not built today for persistent connections. XMPP real-time events are possible since a TCP socket is kept open. BOSH and XMPP over BOSH make it possible to run XMPP over HTTP, and provide almost real-time notification. I like BOSH, a lot, but it is still an emerging technology that needs plenty of love. XMPP metadata is still not implemented in BOSH, and the client-side implementations are weak.
• Running an Apache server and setting a up a REST service is almost straight forward. We know how it works, how it scales, and how it fails. Running a XMPP server, even the excellent ejabberd, is not as easy as running Apache, and most folks are likely not to run their own XMPP server and rely on third parties like jabber.org or gtalk. The good folks at gnip cut down on XMPP because of lack of reliability of the XMPP third parties. XMPP, still, is too complex to operate.
• Supporting binary data is XMPP’s achilles heel.
• There are simpler HTTP only alternatives. Brad Fitzpatrick implemented a never-ending Atom feed for the LiveJournal update stream years ago. A partial solution to reduce polling, although not real-time, is Roy’s better resource mapping. And let’s not forget the straight forward callback. Oh yes, running an Apache server on your mobile phone is possible.

Will XMPP become the messaging middleware for the web? Eventually, perhaps, but in today’s environment, neither the infrastructure or the software are ready to support such a model. As importantly, there are simpler alternatives to achieve real-time event notification without requiring the publish/subscribe architecture.

Because that’s really what it is. Somebody that was not fitted and loosing big (aka Western Cultures) has just started a new game of Monopoly, and you and I are not getting any of the money in this new game. Some may call it the end of capitalism, but I would go even further, this is the end of Darwinism (and a prove that Hegel was looking at the past, and not the future like Marx wanted to see in it).

There are unknown consequences for any deep interventionist economic policy. Anybody who says so it’s either a fool or badly naive. But worse of all, intervention of this kind defeats the system as we know it, where the fittest survive.

Without intervention this crisis would have been a chance for those without debt to compete, i.e. the very poor.  All that we will see is further foreign investment from large sovereign funds, and that hardly benefits the poorest nations in the World.  I am very disappointed by our politicians lack of pro-activity and foresight.

We live in a culture that encourages risk, and hence competition and growth. But now we have simply wiped out the consequences of failure. The failed specimens survive, unlike what Darwin taught us …

One of the most useful recent additions to my Gnome desktop has been Beagle, Nat’s personal desktop search daemon. I also discovered that in Ubuntu Feisty there is a deskbar that traces all the actions you do from your desktop, including your web activity and beagle searches. The more you use it, the more relevant it becomes since humans are repetitive.

Then, it happened: Tim pointed me this morning to Beagle++, a semantic desktop search engine based on Beagle, and that triggered the thought.

There ain’t a semantic web. No, you read me well, there won’t be one. It won’t be the Web 3.0. No. The next revolution will be on the OS, not on the web.

Web 2.0, user generated content, and data mashups present a taxonomy of data available from an almost infinite variety of sources. And that will continue to grow. But understanding the meaning of data is something that belongs to our brains. The ability of a server to interact with a human being is very limited and it would require a tremendous flow of data between us and many servers out there. Whereas our desktops currently track much more information which can be used to understand the meaning of for example, a search query.

Eventually, the desktop and the OS will capture our facial expressions, our mood, our feelings. Really, it’s not that hard.

What this will mean is that the applications that deliver the semantic web will actually reside on our desktops, not on the server. There won’t be a need for processing on the server. The semantic analysis will happen on the desktop.

When you really think about it, it sort of makes sense. People are reluctant to leave a behavioral trail on the web, let alone have things such as feelings captured on somebody’s database. You want to keep that privacy, and ensure that sort of data lives somewhere where we have much better control of it: our desktops.

Web 3.0 will be about client-side applications, powered on Javascript, Flash, Java Web Start, .NET Smart clients, etc. The current browsers will change to integrate better with the desktop. The OS will change to feed a lot of behavioural data to the desktop. The desktop will control our semantic experience. The OS will become a browser, and the browser will become an OS. The OS will become a search engine, and the search engine will be our OS.

Don’t believe me? Well, I am not asking you to do so … my good friends at MIT have already proven it:

• Re:Search, a search engine on past search results: http://people.csail.mit.edu/teevan/rse/
• Exhibit, a CSS+Javascript mashup technology with no server-side processing: http://simile.mit.edu/exhibit/
• The Universal Information Client, a semantic office/browser/…: http://haystack.csail.mit.edu/home.html

Who said Gnome wasn’t cool?