On my previous entry on the present and future of programming languages, I briefly covered on the reasons I think it is important to be looking at this problem now. I though I would expand the discussion.

The laws of physics that we know have stopped our ability to make chips significantly faster today, and rather hardware manufacturers now need to place more and more cores in one die. The result as software developers is that we are now faced with computers with multiple cores and multiple CPUs. It is now the norm to find computers with 2 cores, and most servers are now 4 or 8 core machines.

Historically, developers lived on the free lunch, and with time they knew code would run faster, since clock speeds would get faster. Chips will still continue to get faster in the future, but marginally what we’ve seen over the last 18 years. Today, it’s not about running faster, but having more transistors. And what once was a good programming paradigm for the single core, it’s not valid any longer for the multi-core situtation.

Here is where concurrent programming is born. Writing software that scales to multiple cores with the current programming paradigms is hard. The main mechanism we know is the usage of threads, and locks to protect shared estate between threads. It is a model somewhat understand by developers, but that it is however extremely hard to get right.

On the server-side, programming languages that can only leverage one core are going to have a really hard time to compete in the future. Programming languages that have adopted the thread-per-request model are better placed, but will nevertheless face a wall at some point because they rely on locks to operate to protect the transactionality of shared state. Multi-threaded programming and locks are not scalable, mainly because they are not composable.

Building composable software really requires us to move away from sharing state. Working with immutable data structures, and immutable containers, allows us to avoid state, hence allowing modularity and composability (the core principle of software development for the last 35 years, and the philosophy of UNIX tools). But working with immutable data goes against extensibility principles in object oriented programming.

There are a few paradigms that have been in the research arena for the last 10 years and are now finally being adopted. These paradigms take the shape of message passing, parallel programming and functional programming. Functional programming languages are naturally parallel and use immutable data. Unfortunately functional programming languages are not mainstream, and most functional languages that have been successful (Haskell and Lisp) actually allow mutable structures. Erlang however is an exception. Erlang is naturally parallel, uses only immutable data, and is based on message passing. Erlang is a good language to solve the concurrency programming paradigm.

The question though is whether adopting a language like Erlang would make sense for most organizations. Rather, some may argue that evolving the abstraction of concurrency into the current leading programming languages is the way forward. I’d disagree since this is somehow what Scala does, but given that it lives in a JVM, with threads and locks, and using libraries that use threads and locks, it’s unlikely new language constructs will allow existing code bases to be fixed. To me, Erlang looks like a much better choice. It’s a proven choice and with many years of history.

I believe being a polyglot is nothing but an advantage, and that polyglots are normally the best programmers. As a matter of fact, I challenge myself to learn a new programming language every year. Last year it was Scala, this year I’ve started learning and writing some Erlang (and yes, there is a pattern here for functional programming languages).

One of the reasons I like learning programming languages is a drive to reach the Holly Programming Grail: I find most fascinating discovering which language is better suited for which task. In other words, some languages excel at tasks where other languages fail. An organization could decide to allow all those languages to exist, and if we were doing best-of-breed in every problem domain, we would end up with a language soup. So the question then is, is such soup good or bad? In other words, when we think about developing Internet applications: which and how many languages should an organization use?

There are two factors to consider in choice of language choices (assuming the language choice is functionally the best choice for solving the problem at hand): talent and operations.

Talent is truly an interesting one. Whereas a startup can possibly afford to pick more exotic languages based on their goodness, large organizations can do less so because of talent. This is because A-class developers will excel in any language, and usually A-class developers are polyglots. But one can only hire so many A-class developers, and eventually you end up growing into B- and C-class developers. Your ability to maintain software at that point really depends on your C-class hiring pipe. Pick a rare language choice, and you are calling for trouble.

Operations is very different from talent. Here the size of the company also matters. Developer cost will be the starting cost, but as the business becomes successful operational cost becomes the key driver (specially in consumer Internet applications). In a way, you want to pick the language choices that drive your productivity higher, and later slowly move into reducing your operational cost.

If you follow me, you’ll notice talent and operations are conflicting today. Java is the mainstream programming language worldwide, i.e. A-, B- and C-class talent is widely available, yet scaling Java horizontally at linear cost is difficult. On the opposite end, Erlang is a good choice for distributed concurrency and scaling linearly, yet B- and C-class talent is difficult to find.

I used to advocate the JVM as the minimum host environment, yet allowing language choice across the stack. As long as it could run on the JVM and I could manage it, any language was fine. One could run the complete stack on a JVM. The presentation tier in JRuby, Jython, P8 or Caucho’s PHP implementation. The application logic and persistence in Java or Scala. The batch computing in Scala. But it just does not work that well. Even though Scala has the artifacts to work, the JVM and the runtime libraries don’t. Shared memory, native threads and locks means the JVM will be more expensive to scale and operate on the long run. The JVM is a feasible, stable and proven alternative, yet it is an expensive choice (in operational cost and probably lost opportunity because of slower agility).

Given this context, my language choices would be:

• Conservative stack, which gets the job done well enough, can be operated, and you can get accessible talent.
  • Frontend: PHP
  • Application Logic: Java
  • Persistence: Java and C/C++
  • Batch: Java
• Medium Risk stack, that addresses some of today’s shortcomings, but reduces the talent pool:
  • Frontend: JRuby on a JVM, with Merb
  • Application Logic: Scala on a JVM, with Jersey
  • Persistence (non-relational): Scala on a JVM
  • Batch: Scala, with Hadoop
• High Risk stack, forward looking to the highly multi-core CPU roadmaps announced by Intel and AMD, that reduces even further the talent-pool:
  • Frontend: JRuby on a JVM, with Merb
  • Application Logic: Erlang. Actually, I’d love to see a Scala-like language targeted to the Erlang VM to make it more palatable. But for now, I’d settle on Erlang.
  • Persistence (non-relational): Erlang
  • Batch: Erlang

If you have been following the media industry over the last years, you’ll have surely realized that traditional media is struggling. Even before the crisis, advertisers have started switching from offline to online, and now with the crisis we see a consolidation in a few ad agencies and a few brokers.

Many magazines and newspapers are in such position that if their debt does not get re-financed they will have to file for bankruptcy. Many music labels find themselves in a very similar position, or have already gone belly up. Video houses are struggling with costs. It’s not possible to compete with pirated content. TV channels are going bust.

But there is also a lot of smoke-n-mirrors going on.

I hear claims. Some argue that traditional media is dying. Some advocate social media has taken over. Some forget the economics of running a media business. That the article is dying. That head content is now irrelevant. That the democratization of content production will increase quality. That user-generated content is the future.

Others blame ISPs and the likes of Yahoo, Google, AOL for this. They talk about stealing content and taking away the advertiser money flows the so badly need right now to survive. Some go preaching about how policing the unregulated wild wild web with Digital Rights Management is the only way for them to remain profitable.

What is for sure is that the media industry is suffering. What is also certain is that consumers are generally rational, and make purchasing decisions based on utility. Many consumers are today rather pissed off with the traditional media. And I believe what we are seeing it’s just a consequence of the media business own behavior, being extremely zealous to protect their content rights with solutions that make the consumer experience worse, rather than better.

Is it not annoying that you can’t, legally, make a copy of a DVD for backup purposes? Is it not annoying that you can’t buy DVDs while you travel to other continents? Is it not annoying that your DRM tracks only play with proprietary technology? Is is not insane that games are region locked?

Like Lola would say, I surely think it’s extremely highly very annoying.

Long, long time ago, information traveled slowly, snail pace actually. Controlling where a film was being shown, or who had the tapes, or who was selling the newspapers was a relatively easy thing to do. Yet, already then, there was sharing. You’d share a magazine at the hairdresser’s. You’d share a VHS tape. But, for most, things were controllable since the medium was perishable, and things were not flowing the pipes, yet.

The Internet has changed it. Information is available at speed of light, at whatever bandwidth you have contracted. Sharing is easy now. And because they can, and because that’s how humans are, consumers want to share what they own. The same way they can share a book, or photos, or toys, …

Revenue starts to go down on the rich media side of the business due to pirated copies flowing on the net. So DRM moves in. Badly in. Actually completely gone wrong. It alienates consumers. Napster, eMule, Kazaa, … all appear because consumers want it. But truth is that DRM is protecting some of the revenue, and that’s why content right owners keep lobbying for it. But it’s still going down the hill. High-Definition media appears and somehow fixes partially the problem. But it won’t last long. It’s only working because of constraints in bandwidth.

Paper media businesses have trouble getting audience. You’ll find many reasons for it, but the key underlying factor is that they’ve ignored speed of light. Information can flow immediately. And if it can, it will. Those media businesses that understood and embraced this, new incumbents mostly, but also some old faces, have been doing better. But they have made the industry more competitive as a consequence,aAnd those that did not understand it, find themselves struggling.

So what do you when you are losing? For starters, you find a scapegoat. Done that. Ironically, deep inside, you realize your business model is gone and you are just trying to hang onto it for as long as possible. In reality, you are just buying time. Precious time you don’t have, to figure out the next business model.

Yet, one has to ask where did it all go wrong? Audiences are actually not spending that much time on tail content. High quality production “head” content is still where online minutes are spent. Although I put some value to UGC, I still believe good productions require good funding. Good talent needs to be compensated. Be it via ads, via subscriptions, via concerts, …

To me the issue that traditional media needs to address is how to operate their businesses at a whole new level of revenue and cost. Since revenue levels per production unit are down by one order of magnitude, one must produce significantly more, and do it a much lower cost basis. I can think of three things the media industry can do to achieve this:

• The tail, by definition, is long. Sorting out the tail is a mess. To start with, go manual, and use your editors to find the gems in the tail. Cream it out. Find the new head, at no production cost. But don’t stop there.
• In parallel, invest in algorithmic matching. Eventually you want to surface all the metadata in media to replace the editor. We can’t today. But it should the goal.
• Production costs are high and need to be shared among all players. By sharing production costs with your competitors (you hear me well) you are actually removing those cost components that should be commodities, so that you can move up the stack to solve the creative problem better. Likewise, if you can share the production costs with the long-tail of user-generated productions, all the better. You are basically getting that subsidy you need at production time. Since you have addressed costs, you can now play with content pricing. You can charge, or you can go free. You have gained that freedom back. You can design new business models. Consumers benefit. Producers benefit.

The first item is really a way to get going. The latter two are truly strategic, and are what I call the “open media infrastructure services”. Think of it like the Amazon services for media production and distribution.

Much of that infrastructure already exists for distribution, as the channels of pirated content, and they are actually almost free. Production is not there yet, and will require investment. But without taking these steps, I am doubtful head content production will last. Or if it does, it will do only out of government subsidies, using taxpayers money. And that, really, would be a pity.

Looking back at the past 20 years of the history of the internet, one can realize that what we call today the web has changed dramatically, but that one thing is certain: it will continue to evolve. Gone are the days of manually editorialized directories, of static content, of one way conversations. Today we live in a dynamic, social, interactive web. And it’s a web that is traversing the digital realm and changing how we understand our physical world, breaking communication barriers and making information and knowledge easily accessible.

Yet, although much has changed, the web remains an almost completely deregulated environment, not only on the legal side, but also on the technical side. That’s one of the web’s merits, making it easy to publish documents. Those documents are part of a forest, many sit in isolation, and only a small number of links are connecting those documents. The hyperlink is the web’s way of connecting documents, yet, because the web is a flexible environment, there are no requirements on developers as of how they structure their documents, nor as what link schemes they use.

But what is an advantage on one end, reducing the barriers of entry to publishers and developers, is also a disadvantage on the other end. We end up with billions of documents that are difficult to navigate, and the information, although there, is difficult to find and access.

Organizing the web has been a growing pain since the early days. As the web evolved, so did the technologies used to organize it. In the beginning, only a few universities and government agencies had documents, and it was feasible to maintain a directory. Later on, commercial entities entered the web at exponential rate, and business were created just to solve exactly this problem: organize the web.

The web is growing so fast, that we are not able to reasonably scale the production and maintenance of web directories. And there came web search engines, based on the information retrieval research of the late 80s and early 90s, to solve this very problem. But search engines are starting to show their age:

• We are accustomed to a web that is almost infinite, so deep and wide that we know that not even search engines can reach the very end-leafs of it. We know that the long tail is getting longer, and that the amount of disconnected content is just growing every day.
• Traditional media is struggling to find its place in the web social ecosystem. “Head” content is expensive to produce, and is becoming less and less profitable to operate. The audience is moving to tail and (social) user generated content, where it spends hours. Media businesses, as we know it, are dying slowly. Content is no longer king, and guess what, search can’t find the new king.

Today’s problem is also today’s opportunity. Interestingly, the tools necessary to fix this are almost in place. This the “Atom Web”, or a web linked by Atoms, which some would perhaps deem as a bastard child of the semantic web. The Atom Web is a web where the hyperlink is really empowered as the new king, and takes the crown away from content. It’s a web where document and link meta-data is more important than data, just like the PageRank algorithm changed search by using link (citation) flows as a measure of relevance.

The Atom Syndication Format and the Atom Publishing Protocol have the capabilities to change the web as we know it, well beyond the initial purpose to serve as a content syndication mechanism. Atom contains almost all the necessary meta-data missing in a HTML document, plus it provides a neat way to link content. But to achieve this, we need to empower Atom by introducing three new principles:

• Content is opaque. We shouldn’t accept (almost) any extensions to Atom. Atom should be reserved as an envelope containing only meta-data, with the source element containing all the content or a reference to the content resource. Perhaps in a few cases we might need extensions, like search, paging, presence and location. But rather, I see the very large majority of extensions as content.
• Post Atoms, Get Feeds. Atom entries have little meaning by themselves. Instead, they live in collections through feeds. An entry may belong to many collections. Collections are views, representations, of the underlying documents, grouped in a wide array of combinations. Feeds can represent pretty much anything: the revision history of an entry, all known content about an entity, a blend of categories and topics, related content, etc. Congruent collections are much easier for a UI consumer, as a simple request returns everything necessary to render a widget, a module, or a complete page, perhaps even as JSON directly from the browser. Notice that I am not assigning any semantics to the feed, that’s up to the implementation, but specifying that a feed is the first class citizen, followed by the entry.
• Links are type-aware. Links are the most important part of Atom, and content-types are the tools of choice to help bring semantic value to the relations (rather than custom schemas extending Atom). In a way, the Atom Web is a reduced subset, a predecessor, of the Semantic Web, since we don’t necessarily know the subject-predicate relationship for all structured data (yet). I rather think of something along the lines of <link type="application/atom+xml;content=application/foo+xml">, which would be describing the content-type inside the referred Atom document. Obviously, you may still point directly to the underlying resource, because you can, and because that’s how the web works today.

Certainly, there is a bit of web infrastructure required to make this true. The same way we have the web server for the sometimes-linked web, we need the atom server for the always-linked web. But an atom server is not just a web server adhering to the the AtomPub protocol. Atom server implementations can provide whichever machinery they want as long as the adhere to the three principles above. An atom server needs to have the ability for publishers to define collections, or more precisely, the matching rules that will define views, and which will be exposed as collections, through feed resources. Such view generation will be offloaded partially to batch computing, perhaps on the form of map/reduce jobs. Some implementations may for example decide to use schema-less document stores in order to generate those views.

One could get started today with this and build a gigantic store replicating copies of the content hosted elsewhere on the web. The store could be populated either by crawling or by allowing developers to publish into it. But eventually, this store would have problems finding all the deep end-leaf documents.

A few big players adopting Atom to expose their data, for example making the webmap available through Atom and opening the API for publishers to post onto the store, could dramatically change this game. There would still be plenty of problems to solve, ranging from the now even more important link-spam detection and moderation, to the not so obvious scaling problems, but at least we would be one step closer to organizing the web.