olympum

Problem: on the web, enable a large number of message producers send a very large number of messages to a much larger number of message consumers. Example: allow 100,000 publishers send a total of 1 million messages per second to 100 million concurrently connected consumers.

We are dealing with the problem of connection channels, an abstraction that allows a producer distribute the message to many connected consumers. Our challenge is to design a distributed channel delivery mechanism that can scale out to millions of connected consumers.…

For years I’ve been continuously fighting with backups. I have not been particularly good or consistent at it. We’ve been okay with Time Machine and Carbon Copy Cloner, but the recent addition of a digital SLR to our gadget collection has meant running out of space on our shared home drive.

Historically I’d been using a WD My Book 2×1 TB RAID1 array connected to an Apple Airport Extreme and shared via AirDisk (afp). We had Time Machine backup all the computers in the house in the AirDisk.…

On a theoretical world, given the ability for a processor to run an infinite amount of threads, we could prove the following statements (no attribution purposely given):

• If you do more CPU than I/O, use threads.
• If you do more I/O than CPU, use more threads.

which would allow us to conclude with the following corollary:

at full utilization, threads and events have the same theoretical throughput.

Such argument ignores praxis — it is a purely theoretical debate disconnected from the reality of scaling services –.…

I’ve been using Markdown since 2006, taking all my notes at work using a simple text editor and using the Markdown text markup format. I also use Markdown for writing down thoughts and posting to this blog. I rarely, if ever, use Word or even TeX/LaTeX. I treat markdown as my source format and I generate all my target formats using multimarkdown: PDF, HTML, RTF, etc.

I am happy with this process. I use a standard editor I find in any operating system I happen to work with.…

I am playing with a new query language. I am defining the grammar as EBNF, but I want to show railroad diagrams for those readers that are more graphical and less familiar with BNF. I’ve found limited support for generating syntax diagrams from EBNF. I’ve found a few tools, some working better than others:

• Ebnf2ps (Haskell). This is the only tool I have not been able to get to work. I seem to be missing AFM fonts in my TeX installation and I am not sure I want to spend time figuring out how to generate the AFM files.
…

Update (2011/9/11): this post is picking up again 8 months later, I’ve written an update as of where we stand.

Jason Hoffman (Chief Scientist, Founder at Joyent) has posted some good questions to me, based on my original nodejs and V8 post. Let me summarise Jason’s questions and comments into three key messages:

It’s Joyent’s responsibility that NodeJS runs well period. We’re not afraid of a language VM. [...] actual node.js committers (who all work at Joyent) know quite a bit and have pretty good relations with the V8 team.

…

Update: Jason Hoffman (Chief Scientist, Founder of Joyent) has written a very good response to this post. Obviously I owe him some responses, which is on a separate post.

If you have not watched Douglas Crockford’s video lecture on server-side Javascript, I recommend you do that first before reading further into this post.

I have been saying for a while that server-side Javascript matters. We, at Yahoo!, see a bright future in server-side Javascript and are making a big investment in it.…