First, in Mail.app, I export my mailbox. In my case, after I select my Archive mailbox, I get:

$ ls -1
Inbox2007.mbox

From Lion onwards, the Apple mbox format is actually a folder that contains a few files inside:

$ ls -1 Inbox2007.mbox
Info.plist
mbox
table_of_contents

Where mbox is the real mbox file we need to import into Outlook, but if we try now (Import -> Contacts or messages from a text file -> Import messages from an MBOX-format text file) we’ll see the mbox file in the Finder is greyed out. This is because the FSTypeCode is not set:

$ mdls Inbox2007.mbox/mbox | grep 'FSTypeCode'
kMDItemFSTypeCode = ""

What we need to do is to change the code to 'TEXT'. We can do this with a simple command (I used find since I exported several mboxes at once):

$ find . -name '*.mbox' -exec SetFile -t 'TEXT' {}/mbox \; -print
./Inbox2007.mbox

If we now test for the FSTypeCode, we verify it’s correctly set as 'TEXT'.

$ mdls Inbox2007.mbox/mbox | grep 'FSTypeCode'
kMDItemFSTypeCode              = "TEXT"

We can now finally go back to Outlook and import the mbox file: Import -> Contacts or messages from a text file -> Import messages from an MBOX-format text file.

We are dealing with the problem of connection channels, an abstraction that allows a producer distribute the message to many connected consumers. Our challenge is to design a distributed channel delivery mechanism that can scale out to millions of connected consumers. Throughout, our assumption is that this is a stateless delivery system, i.e. messages are either delivered or dropped and no persistence guarantees exists; if a consumer is not connected, it will miss the message.

The naïve approach is to perform consistent hashing by channel. In this model, each channel and all its consumers are in the same server. Since the channel identifier is part of the URI, the load balancer can effectively perform this operation, and we can add servers as required without requiring re-balancing. When we have many channels per server, the distribution is eventually uniform. Problems arise however as some channels have an order of magnitude more consumers than other channels. There is also a problem if a channel has more consumers than a server can sustain.

To solve the limitations of hashing by channel, we can instead perform consistent hashing by channel and connection (consumer). In this model, each consumer is consistently assigned to a pool of servers and we can add servers without having to re-balance consumers among servers. The channel stores a list of all the consumer identifiers and channels are consistently hashed across servers. To deliver a message, the load balancer will find the server holding the channel, and dispatch the request. The channel will lookup the list of consumer identifiers and again apply the consistent hashing algorithm to reach all the consumers.

Although the hashing by channel and connection is conceptually simple, it presents significant operability challenges. First, the loss of the server holding the channel metadata and list of connected consumers will require a watchdog cleaning up all the stale consumer connections. Second, as consumers join in and disappear, the channel server would need to maintain a consistent view of the list of consumers by the means of locks, with the incurred performance degradation of very large number of consumers. Third, as more consumers connect uniformly across the nodes, the more chattiness that will occur. At some point, all nodes will have consumer connections for a given channel. In order to to fulfill every operation, we must issue N requests to all nodes, where N is the number of nodes in the cluster. For the cluster to be able to process and deliver M messages, every node must be capable of processing N*M messages. This design will be limited in the number of connections it can hold, because of the centralized channel-consumer tracking problem, and will also only scale to the maximum request processing capacity of an individual node.

We can solve some of the operability challenges by removing the channel management of consumer connections, and instead of keeping a list, keeping the visibility of the peer nodes. Here the thinking is that since we will asymptotically reach the point where all nodes hold consumer connections for a given channel, all we really need to do is keep a list of all nodes in the cluster. Some centralized agent keeps a directory of all active peers holding consumer connections, e.g. a Zookeeper ensemble.

Consumers get uniformly connected to the nodes in the cluster by a “good” load-balancing scheme. Since any node can hold connections to consumers on any channel, there is therefore no snapshot of a channel’s consumers, and to be able to identify all consumers connected to a channel it is necessary to interrogate all nodes in the cluster. Whereas this design improves the previous ones in that it allows scaling to an infinite number of connections, it will still only scale to the message processing throughput of an individual node.

A popular alternative to the directory of nodes is the tree of nodes. In this model, we start with a single node. As we reach the maximum number of connections the node can hold, we add two new nodes. The original node still accepts messages from publishers, but brokers the delivery to the two new nodes. As those nodes themselves become saturated, we add a new layer of four nodes. And so forth. This approach has the same limitation as the one using a directory of nodes, i.e. the maximum throughput is bound to that of the individual node.

We’ve seen how to hold the connections to an infinite number of consumers, but not how to deliver an infinite number of messages. These solutions scale very well to tens of thousands of messages and millions of active connected consumers, but have an upper limit. For most producers out there, that upper limit is probably high enough to be fine. But such limit exists in push-based systems.

Both the messaging literature and the messaging praxis have historically preferred using pull-based models rather than push-based ones. In a pull model, consumers come back to the broker to fetch messages, at each consumer’s own rate, and the problem is therefore no longer dispatching across millions of connections. Pull-based messaging systems chose to store the messages until consumers come back to fetch them. In fact, the only scalable messaging system to millions of messages and millions of consumers that we know of uses store- and-forward: SMTP.

As much as I may think that the techniques that enable web push-models such as HTTP streaming, long-poll and WebSockets are genuinely useful to solve point problems, they are not techniques we can use to implement Internet-scale push- based web services, as they are fundamentally based on a PubSub model. The scalability of PubSub under high load remains an unresolved research question and as such is not a paradigm we should apply at Internet-scale.

In fact, I am now almost convinced that we’ve been looking at this in the wrong way, and that the right solution to this problem is a store-and-forward solution, where web consumers connect at their own rate to fetch messages and intermediaries throttle concurrent connection rates in order to achieve linear scalability. Essentially, this is a web of partially connected store-and- forward almost real-time async data peers. And that’s a mouthful, but a really exciting one.

I posted this diagram, without justification, yesterday evening, in an attempt to gauge the reactions of the community, in twitter. Thank you to all of you that commented. With the experiment done, let me now provide my thesis and hopefully address most of the feedback so far. I intend to make this post fluid and keep updating as the conversation evolves.

Now, let me qualify the thesis. First, this is a thesis applicable to multiple domains, whether that’s a low-level network switch appliance, a http proxy gateway or a web application. As a consequence, this is not a framework comparison, it’s not about Rails vs Django vs Play vs Express vs …. This is not an argument about dynamic vs statically typed programming languages, although that definitely plays a partial role in the thesis.

Mine is a thesis about the complete team productivity by programming language and across the full product life-cycle from inception to retirement. The thesis is purely based on my observations throughout the years. There is not data to back it up, except anecdotal evidence. In summary, mine is a qualitative statement, not a quantitative one.

If the thesis were true, this would mean that teams using high-performance and functional programming languages can iterate more quickly from idea to concept. Such quick turnaround allows a constant validation of ideas in the code. Because it is quickly possible to see the idea running, we can afford to have more ideas. The quicker we are able to iterate between idea and concept, the more we are innovating.

Such quick turnaround has a real trade-off and a false trade-off.

Moving these concepts to production is difficult. Whether it’s Scheme, or Lisp, or Clojure, or JavaScript, it becomes clear that we are “naked”. Except in “safe but useless” languages, such as Haskell, in most “useful but unsafe” programming languages, the developer has to compensate for the dynamic nature of the language. The developer has to provide the strictness, and to a degree perform the job of the compiler. In statically-typed and imperative programming languages, the compile infrastructure helps the developer ensure the correctness of the code. In dynamically-typed and functional programming languages –by the way, yes these are orthogonal concepts but usually associated–, the developer needs to structure code and provide the necessary checks, assertions, unit tests, regressions tests and modular design. The addition of type hinting and type inference does not change this, it only shifts effort between “before concept” and “after concept”.

High-performance functional programming languages trade strictness for innovation potential. Without the additional investment in checks and tests, these systems are not predictable. The time we gained upfront, we pay later for. I’d argue such trade-off is good, and allows quick iteration to find the right idea.

The second trade-off is about tech debt. The time to production is identical across languages, but because the developer in high-performance functional programming languages had to compensate for the lack of compiler infrastructure and runtime guardrails by adding completeness of tests and checks, the system is easier to maintain and has overall a better architecture. This is almost counter intuitive and I think most people don’t think this way because they move too quickly from concept to production, and don’t implement the necessary strictness. There is a tendency in the developer to move quickly from concept to production, and therefore accruing overall technical debt.

Finally, I believe server-side JavaScript and Google V8 provide such high-performance functional programming language, in the skin of an imperative one, and enable this quick turnaround from idea to concept. This possibly one of the key reasons we are seeing such explosion in adoption. I only hope developers start to realize the trade-off they are doing, and that in order to avoid accrual of technical debt, they better invest in checks and tests before they rush into production.

Historically I’d been using a WD My Book 2×1 TB RAID1 array connected to an Apple Airport Extreme and shared via AirDisk (afp). We had Time Machine backup all the computers in the house in the AirDisk. We’d also store our photos and videos, etc. in the AirDisk. Aside running out of space, we had no good offsite strategy. Given that our digital picture collection is continuously becoming more and more valuable as a family history artifact, I though it was time to address this properly, rather than just adding more space.

I started by classifying the risk associated with each data type:

• Personal files: document scans, photos, videos, etc. This is the critical part of the equation. In case of fire or theft we’d have no replacement and the loss would be HUGE.
• Shared Media: movies, music, etc. These are regrettable if lost, but can be restore. I know how to find them back, even though it might be painful or costly.
• Backups: the time machine backups from our home computers. Since we have the laptops plus the backup, I was not too worried about losing this.

Since no all data is born equal, the required reliability levels vary. For personal files, I want to be covered against hardware failures. For the other kinds, I don’t mind that much.

Additionally, for videos, I want to have the ability to connect the storage unit directly to my computer while editing (my experience editing over the network has not been great).

Finally, I really wanted an offsite backup for our personal data.

Given these constrains, I struggled to find a good (cost-effective) combination, and since I think I’ve found something that works well for me, I thought I’d share with others just in case this helps:

• Airport Extreme Station 802.11n 2nd Generation (100/1000) (any 1 Gbps switch would do).
• Buffalo LinkStation Pro Duo 2×3 TB (NAS), connected to AES via 1 Gbps ethernet. Setup as RAID1 and exposing two afp shares (Personal, Media) and a Time Machine ‘Backup’ share that each computer uses to create a separate sparsebundle.
• 2x WD My Book 2×1 TB on RAID0, mounted ‘usbdisk’, connected via USB to LinkStation and used to backup incrementally every night from the NAS internal disks. Every week I rotate the USB WD drive units between office and home.
• If I want to edit video, I unmount the USB drive and connect via Firewire to my computer for doing video edits. When I mount it back, I manually copy the edits back to the NAS internal drives. This is the only manual step in the process.
• The media files are exposed through the embedded server in the LinkStation via UPnP and iTunes Server. This allows me to use Boxee on any computer (and a patched Apple TV 2nd gen) to watch our videos. We also share iTunes music this way without having to share libraries.

I ended up putting all the data under RAID1, but I think the cost is low since we mostly only read from the array (except the backups). With this setup:

• We have Time Machine for all computers in the house, happening over the network and transparently.
• We can access all data both via afp mounts as well as directly via USB/FW for intensive reads / edits.
• We have an offsite copy.

Overall, I am happy with the setup. I was initially worried about using a home-grade NAS, but even though the LS is slow for writes, it’s actually fast for reads, so I am happy.

• If you do more CPU than I/O, use threads.
• If you do more I/O than CPU, use more threads.

which would allow us to conclude with the following corollary:

at full utilization, threads and events have the same theoretical throughput.

Such argument ignores praxis — it is a purely theoretical debate disconnected from the reality of scaling services –.

Yahoo! serves over 20 billion daily requests through it’s edge services (remote proxies and caches throughout the world). These intermediate servers are doing pure IO workloads, handling slow client IO and handing connections off to the origin servers through Yahoo’s pipes. It is critical that we minimize the CPU cost per connection to be able to max the CPU at the max number of connections per host.

The hosts on Yahoo!s edge network run exclusively event loops, and have been doing so for over a decade, originally with Inktomi Traffic Server then with Yahoo Traffic Server, and now with Apache Traffic Server, etc. The design throughout is the same: a few “master” event loop threads, usually one per core, and a small pool of worker threads. In total, a handful of 20~50 threads per server. With this design, Yahoo! is able to scale to hundreds of thousands of connections per server. It is currently still impossible, in practice, to run a server with so many threads and still serve data.

Another practical need for event loops occurs at the other end of the serving stack. Resolving a search query follows a general pattern of parsing and rewriting the query, followed by fetching potential search results, and finally doing document re-ranking. The first and last phases are CPU intensive. The fetch operation is purely an IO workload that performs a scatter-gather operation which fans out to hundreds to thousands of back servers holding the search index across tens of columns. As a consequence, for every client connection, it’s possible to require one thousand upstream connections. When the upstream index servers become slow, which is a common failure situation, or perhaps in scenarios where we have to fetch data from a remote data center, the number of connections in the system grows to tens of thousands. It is also important that we keep all three phases running on the same process to avoid serialization and transfer costs, essentially forcing us to mix CPU and IO intensive workload. It is currently still impossible, in practice, to perform this type of data intensive processing without using event loops.

Unlike Yahoo’s services, which combine an event loop with a handful of threads per core, Node.JS design is a single-threaded event-loop per core. For pure IO workloads this ensures the necessary simplicity required to be able to design software that scales to thousands of concurrent active connections, as long as nothing is blocking. I find it unfortunate that some developers have not internalized this and are trying to run CPU intensive applications using Node.JS. Inferring that because these badly designed applications are a failure, therefore Node.JS is a failure is an unnecessary and unfair generalization. Node.JS has a field of practical applicability, and like any tool, a seasoned practitioner should know when, and when not, to use it.

I am happy with this process. I use a standard editor I find in any operating system I happen to work with. I use a standard document format I know I will be able to open in years to come: plain text. I use source control to version my text files and synchronize between computers. This works everywhere I have git and I am not locked-in into any particular cloud or tool or format or …

As an editor, I use emacs markdown mode and it does me well. And it’s not about emacs: you may use vi, or TextMate, or whatever. The point is that your favorite text editor, whichever it is, is probably good enough. I have also recently started using Marked as a convenience for previewing the transformed markdown output without having to continuously switch back to the browser or do C-c C-c p all the time.

I also use Jekyll to transform some of my markdown text files onto a static site where git is the glue here. It works everywhere I have git.

Recently I have had a need to bring my markdown files to the iPhone and the iPad. And although one may find some specialized git iOS clients for things like github, there is no general git client for iOS, integrated with a text viewer, as far as I know.

Given the lack of shared file system in iOS, whichever app I use must have both text editor, ideally with support for markdown preview, as well as sync capabilities. Nocs is exactly that. It uses Dropbox to sync your files, and it has an embedded text editor with markdowns support. It fits perfectly into my workflow. I continue to use emacs on the laptop, and Nocs on the iPad. Dropbox syncs between laptop and tablet, and git between computers.

I am sure there are ways I could simplify the flow. But for now, this meets my requirements.

Finally, I am not keen at all to use tools like Evernote, since as far as I am concerned I am losing both freedom and the future-proof aspects of plain text. I have literally thousands of text notes accumulated over the years, and I have the reassurance that I will always be able to access and edit my notes in years to come. I don’t want to risk putting my documents onto a proprietary platform.

• Ebnf2ps (Haskell). This is the only tool I have not been able to get to work. I seem to be missing AFM fonts in my TeX installation and I am not sure I want to spend time figuring out how to generate the AFM files.
• SQLite bubble generator (Tk/Tcl). Strictly this tool does not consume EBNF grammars, but a custom DSL. If I didn’t care about EBNF, this would be the best tool
• node-ebnf-diagram (Javascript). Although it works, I have to issues with it. One is that it can only generate PNG files. That would not be too bad if it weren’t for the second issue: the tool does not automatically resize the canvas, and it requires explicit width and height input. If I don’t find anything else, I’ll probably end up using it.
• ebnf2ps.el (Emacs Lisp). It works as advertised. The only issue I found is that the diagrams generated have a small white gap on the lines on the right hand side.
• ANTLRWorks (Java). Bundled with antlr, it fits the task. Once you are in the game of defining the grammar in Java, why not just go ahead and use the same tool to generate not only the parser but the diagrams? This is what the tool does. Even if you are not doing a Java parser/lexer, this is a good tool to use for documentation purposes.

I am using ANTLRWorks, generating all the diagrams from the command line as part of my markdown transform pipeline:

java -cp antlrworks-1.1.4.jar org.antlr.works.Console -f yql.g -o output/ -sd eps

It works very well.

We have continued and extended our investment in Node.JS. I can tell you that what the teams are doing is transformative and pure awesomeness. Unfortunately this is as much as I can tell you right now, but really soon you’ll start hearing what we’ve done. In every single demo, internal and external, we have done of the technology, the feedback has been fantastic. I am very happy and fortunate to have a team of super-stars working on this.

As per my concern on being locked into Google V8 and not being able to support the software, things have changed since I last blogged. First, Google has been very, very, supportive addressing V8 bugs whenever they existed.

Secondly, here at Yahoo! we donated some code to the fine folks at Mozilla to wrap the Spidermonkey API with the V8 API. Mozilla went on to create a full implementation of Node.JS on Spidermonkay and provide the architectural re-assurance we needed. I am still surprised however from the discussion on the blogosphere and twitterspace months ago about how many developers don’t seem to pay attention to open source governance. When you have the responsibility for a multi-million dollar technology investment, you want to make sure you cover all your bases.

Finally, we have partnered closely with the fine folks at Joyent and are working through the final stages of negotiation for how we’ll work together going forward.

Let me say that I am really excited about what the technology innovation that Node.JS is bringing to the industry, whether that is to create a new generation of I/O intensive servers, or to bring Javascript to the server-side. I see Node.JS’ future is bright.

Jason Hoffman (Chief Scientist, Founder at Joyent) has posted some good questions to me, based on my original nodejs and V8 post. Let me summarise Jason’s questions and comments into three key messages:

It’s Joyent’s responsibility that NodeJS runs well period. We’re not afraid of a language VM. [...] actual node.js committers (who all work at Joyent) know quite a bit and have pretty good relations with the V8 team.

That’s a very honourable goal and perfectly within Joyent’s capabilities and track record. I am not debating that.

If there are actual technical problems with V8′s reliability or ??? and these affect the use of node.js in production then I’d like to see details.

The discussion is not really whether I have technical production problems or not with V8 (within NodeJS); all software has bugs. My issue is about project governance. Let’s say such a problem affecting reliability exists and that Joyent fixes it by developing a critical patch for V8, a patch that the upstream maintainer does not deem it necessary to merge. For how long would Joyent maintain and test such patches?

Another aspect of governance: intellectual property rights.

My point: it’s not whether Google’s V8 project is open or not, it’s that since governance of V8 may become a problem in the future, it requires a solution now so that large organisations can put their full weight behind NodeJS. Joyent’s weight behind NodeJS is necessary, but not sufficient. I prefer addressing the problems in the code.

node.js is only implemented on V8 but that’s only because we’re going to focus on making node.js awesome first. [...] After all, we are still working on getting it to 1.0

Open source software is commonly designed to make it easy to replace parts or components that may be at risk of being legally tainted or patent encumbered (e.g. mono’s architecture as an example of isolation). I would put V8 in the same bucket of “risky parts”, although for different reasons. Given the lack of definition on Google’s V8 project governance, and the current tight coupling between node and V8, I wonder how realistic it will be to implement a different VM post-1.0 if we haven’t got the right layer of abstraction pre-1.0. Or perhaps the problem (the reassurance of governance) can be addressed by working under the umbrella of a foundation that ensures such governance.

Let me end by saying that I am still as committed as ever to NodeJS and that I really believe in Joyent’s engineering strength to make NodeJS 1.0 a reality. Joyent is putting their money where their mouth is. We know they are working hard making node awesome.

If you have not watched Douglas Crockford’s video lecture on server-side Javascript, I recommend you do that first before reading further into this post.

I have been saying for a while that server-side Javascript matters. We, at Yahoo!, see a bright future in server-side Javascript and are making a big investment in it. But if you follow me on twitter, you’ll know that I am also looking into ensuring high-availability of server-side Javascript-based services on production. Which really comes down to something like: to V8 or not to V8.

NodeJS is currently tightly coupled to Google’s V8 engine. V8 was not designed as a server-side engine, but as a browser-based engine. Furthermore, V8 was designed squarely to run in Chrome’s multi-process model. As much as I think V8 is a brilliant piece of engineering, it’s software that was not designed to run on a server.

More to the point, it’s really up to Google to work with the community to make V8 work on the server-side. Sometimes Google is responsive, but sometimes it might not. It varies as it depends how fixing a bug or applying a patch may align with Google’s product roadmap and plans. I don’t know Google’s plans, and I suspect most NodeJS committers don’t know either.

Maybe for some folks this might not seem like a big problem. And probably, if you are running a site with a few thousand daily page views, it might actually not be a big deal. But to Yahoo!, and to others, it’s a big deal and we believe this is fundamental to the success of the NodeJS project.

Writing a Javascript runtime that does not fail is hard. Even the really smart V8 folks have explicitly designed V8 for failures to happen and safeguard the browser. In a browser, a JS engine failure is an inconvenience to the user: damn, you lost a tab. In a thread-per-request blocking I/O server design it’s also not a big deal, you lose one in-flight request. But in an event-driven web server, it’s a major flaw, you lose thousands of in-flight requests you have already accepted.

For NodeJS to scale to billions of page views like Yahoo!’s, we need to make sure the Javascript engine / VM behind Node is rock-solid for server-side loads, i.e. it fails extremely rarely.

Google may invest on supporting V8 on the server-side, just like the Mozilla folks do. Or somebody else might invest and ensure V8 is rock-solid on the server and Google may merge the patches nicely. Or maybe not, and the community may need to fork V8. Or something else … Nobody really knows.

Either way, it’s time for the NodeJS community to realise there is a roadblock and discuss it openly.