Bruno Fernandez-Ruiz

No, it’s not a typo. I have been daring to write this for ages, but somehow I always refrained myself from doing it. But here it is, this post is about the propeller heads, the skyrocket scientists designing systems in the enterprise world (mostly in the financial services and telco world). Because guys, it’s time to call it a day and move on. Time to stop building the Winchester House.

As background, I have recently been recruiting to backfil Remy, Yahoo!’s Chief Architect for Europe [and no, I am not looking any further]. It has proved to be a hell of a task. My mailbox and voicemail have been flooded with messages from agents and candidates. A pile of resumes to go through.

Most resumes are immediate rejects. Absolutely zero experience in online technologies. Do you apply for Yahoo! if you don’t have a clue about even building a site? WTF!??? And, err, where is your blog, buddy? This is not just any job.

Anyway, moving on to interviews. I called in a few guys with very solid resumes. Outstanding resumes actually. Tons of experience. Lots of technology expertise, masses of troops, managing huge budgets. Right, bring them in, I say.

On the interviews (so that you know) I don’t look at the resumes, ever. I could not care less, because you are allegedly supposed to be able to synthesize and I can read. So, expect to work hard at the interview. Rather, I’d love to see you design flickr, or Answers, or tell me how the frontpage could/would/should work, or, for fun, how you would liberalize the water supply system in the UK. I bring engineers on the room to discuss. They need your help, guidance, steering for making choices. Help them!

All I want to learn is that you can think beyond a Powerpoint and a bunch of theoretical ivory-tower diagrams. And you know, most often, candidates can’t. The large majority of candidates when going through the exercises just put a few boxes up and don’t actually get into what are the hard bits. And when challenged to think through them, they fail. I admit you might not know how to build our frontpage, but at least you should be able to appreciate the complexities during an interview.

The interesting thing is that all these candidates that fail have a common trait. They are the enterprisey architects (a few exceptions exist, like my good ol’ friend Jeremy who is truly an enterprise architect). Enterprisey architects come with certifications, waving their TOGAF and Zachman willies. They operate so high-level that they ain’t got a fucking clue.

Enterprisey architects make the argument that architects don’t need to actually know the details (I don’t code, they say). First, there is a hell of a difference between coding and designing systems and you seem not to be able to design systems. Secondly I honestly think there is something seriously wrong with anybody who wants to be a technology geek and does not like programming languages per se. It’s like being a butcher and not liking meat. I’ll tell you what: if you are not obsessed by [technology|volumes], you are not a [technology|civil] architect.

An enterprise architect is the architect that looks at solving problems across the enterprise, not the department, or the business units, to leverage the synergies and lead towards the future-state architecture. [Don’t look this up, I just made this up].

An enterprisey architect is the architect that wants to be an enterprise architect but fails to engage with his customers, because he’s not capable of being detailed enough (business strategy detail, functional domain detail, applications and technology detail, and infrastructure detail).

And as I keep telling my guys, (enterprise) architecture is not about technology, it’s about people. You are expected to know the technology, mind you, but there are usually much more specialized engineers who need your steering for making choices (if they aren’t such experts in your organization, you should start by building your team), so that you can find the detail that is relevant. You are supposed to understand the bigger picture, and lead folks into doing the right choices, so that you are not only doing things right, but also doing the right things. Enterprise architects use their social networks to make technology changes happen.

Again, it’s not about technology, it’s about people. But you know your technologies, right?

We, as engineers, are frequently faced with solving business problems, or sometimes technology problems, that really move the needle. We, as engineers, are usually never happy with any of the technologies in our tool chest. There is a place for every technology, but it always seems like our favorite technologies fall behind the cool kids in the block.

It is attractive to bet on emerging technologies. They are sexy, cool, promising high productivity gains, and in a way, good technologies are disruptive, not only to the technology landscape, but to the business landscape, since they change what can be achieved with technology.

And so, we feel inclined to risk and pick emerging technologies, without perhaps taking the time and consciously analyze and understand the impact. To list a few things:

• How many developers can I find with the required skills to support this technology? Where are they located? Are those locations where I can attract and retain talent?
• How secure and proven is the technology? Is there a good track record of fixing security bugs in very short time-frames?
• How complex is it to deploy this technology? How stable is it and what is the monitoring and analytics required to operate it?
• How long would it take to migrate everybody in my company/group/division/org from the “old” stuff to this “new” stuff? Does it pay off?
• How long would it take to migrate my project from this “new” stuff to the “old” stuff? Does it make sense that I even start doing it?

Unfortunately, many times the answer to these questions leads to a categoric deception: we can do with the old technology, and that there is no need to introduce “new” stuff. While we should be trained to accept this outcome, we are not, and end up choosing the new cool and sexy stuff instead.

Picking stable, mature technology, despite its known shortcomings, should be the most traveled route for engineers, but unfortunately it is not. Specially in the context of startups, there is a desire to play with new technologies, somehow associating business innovation with technology risk.

Unfortunately, these risky technology choices later become heavy burdens that cost the venture either big bucks for paying star “talent” developers, big bucks for hardware and operations, or very low availability. In some cases, the technology might even become a dead-end and a re-engineering exercise is required anyway.

So, next time around, let’s try to think beyond our desk, and into the long-term implications whenever we chose a technology.

Have you have ever been part of the inception of a software system that later became big and complex? Have you later felt the frustration of not being able to make further changes to the core architecture? Did you end up being taken hostage by the software? I did.

We hope as developers to be able to adapt software as requirements and bugs arise, and to be able to organize our software releases accordingly. By doing incremental releases we hope to work with a stable code-base where we can release often. We then receive feedback more often and we are therefore agile and responsive to business needs.

Until one day we become legacy. A competitor starting from scratch has been able to innovate and quickly implement these new cool features the market really wants. We can’t catch up with them, because they don’t have the legacy we have: the other thirty thousand features we need for all revenue-generating customers we have.

We set ourselves to large regeneration plans, greenfield development. “This time around, we’ll do it right”, we say. Two to three years later, after again a cycle of incremental releases, we become again legacy.

And this is how the software industry is run. Competition between those entering the legacy stage and those on the startup curve is what keeps us employed. And yet this continuous hostage situation by incremental releases is what annoys me most about software.

There are two realistic approaches to solving the problem. You could plan upfront for a software life of 2 to 3 years and not 5 to 10 as some would hope. Or you could take the red pill.

I would rather see more people taking the red pill. It ain’t easy, I admit it, but it is possible. Architectural refactoring is extremely painful - you are basically breaking the system as hard as it can be broken before pulling it up together again -. I have done it, but don’t take my word for it, and look at Linux going from 2.4 to 2.6, at Windows going from 98 to 2000, at Mac OS going to X. They took the pain, and it paid off.

Let’s stop building software and let’s start thinking of software reconstruction. After a few incremental releases, do an architectural release. I know you need it. You know you need it. It’s your choice, the red pill.

We have recently moved into a new house. The house is pre-wired with all sort of things one can imagine, for sound, video, network, motion, alarms, etc. It’s really cool! But guess what, all wiring is behind plasterboards, and we don’t have any instructions as of what and where it is. One end of all wires end up in one of the rooms, so I could put a high frequency pulse generator and trace down the wires behind the drywall. Sorted.

Well, not really.

You see, whoever put that wiring in while the house was being built had an idea in mind. The wiring would fit into a particular set of sensors, alarms, and speakers. The builder was also conditioned by what was available in the market, so it planned wiring suitably for a multi-room A/V controller. The thing is, the builder left it “pre-wired”, never finished it. Sort of an optional thing you could do whenever you moved in.

7 years later. Multi-room A/V and control systems have evolved so much that the technology the builder had in mind has been rendered obsolete. 1Gbit network now allows video streaming over CAT5e/CAT6 cables. So you don’t need anymore expensive multi-room controller. I just need good network around the house, and commodity amplifiers in each room where you need sound/image/alarm, etc.

You see, all that wiring the builder put in? Useless.

Moral for us software engineers. Whenever you write a piece of code or implement a new feature always remember to make it obvious. And the best way to make it obvious is to make sure it is used. Don’t write software for the just-in-case scenario.

You see documentation is good, very good, but there is no point documenting a feature that is not obvious. By the time you get to use actually it, it will be obsolete and too complex to make it worth understanding it.

In summary, it will always be easier to find out how things work, than how things could work. Things that work are self-explanatory: they are obvious.

Any small startup that succeeds and grows to become a big corporation will suffer from technologitis, this is, the inflamation of technology and detachment from the business body.

Seriously, the startup mode, where technology governance is not necessarily as important as time-to-market, allows for dynamic environments that allow quick growth. But at the cost of increasing operational costs due to technologitis: most of the IT budget being spent in operations, in keeping the power on, and only a little fraction of the budget is actually spent in new business initiatives. Certainly not the place the business would like to be in.

Over the last couple of years, a lot of folks have been switching their strategy in chip technology, going from Sun SPARC to Intel Xeon, IBM Power5 to AMD Opteron, Intel Xeon to SUN Niagara, and what else. The end result is that as much as they might have solved a point problem, on the long term they have increased their operational complexity.

There is nothing wrong in switching technology, but one must be aware of the technology cycles inherent with any technology. Whereas changing an application framework is something that theoretically one can do every 12 to 18 months, given that major version upgrades tend to be quite dramatic it would not be that much additional cost to ditch the framework altogether (but beware, project managers will tell you otherwise).

But CPUs stay in the data center for at least 3 years, mostly 5 years, and sometimes even 6 or 7 years. A choice of CPU technology should be long-lived (some may argue CPUs are a commodity, but once you start facing bugs traceable to specific architectures, you wonder whether you would not be better off standardising on a single architecture). A particular vendor may not have at all times over 5 years the fastest, coolest, higher bandwidth chip out there, but good vendors will come regularly come back to take the first place

Before the release of Sun Niagara, Sun failed (repeatedly) to meet its own objectives and bring out its Rock processor, up to the point that it cancelled the whole project. The PR impact was damaging, as well as the effect on the books, and the stock plummeted. After the release of Niagara, a lot of folks came back looking at Sun, hoping all trouble was over. The chip is good, but the question is not whether is good, but whether Sun is able to keep innovating over the next 10 years to ensure the chip remains good.

Apple was using the IBM PowerPC chip on their laptop, desktop and server product line. But IBM was focused on its own server business, and engineering the Power5+ chip. Apple needed a low power high clock device that would make the Apple truly competitive with Intel’s and AMDs’ offering, otherwise their laptop sales would never go higher than PCs. The switch to Intel is surely costing Apple a lot of money in development, marketing and lost opportunities (not all software is ready for Intel). This switch it has however also served as a good marketing effort to bring it on front of typical PC users, and the Intel chip is still faster and cooler than a PowerPC. But what is really important as a lesson is that Apple switched vendors, not technologies.

IBM just released yesterday a note on Power6. Power6 chips will be coming up in mid-2007, at clocks reaching 5Ghz, with double the clock, double the bandwidth, and most importantly the same power profile. This is an amazing individual breakthrough, but nothing you would not expect out of one of the best research labs in the world, IBM’s Almaden Research Center.

But while IBM keeps on delivering new technology breakthroughs, Sun has failed repeatedly in the past. Sun will have to prove itself repeatedly to stablish its credibility back. On the x86 side, AMD has shown historically its ability to innovate in the server space, and, to a lesser degree, Intel.

That’s why for servers I will keep on recommending AMD/Opteron (Linux) and IBM/Power (AIX). They are cool, they keep your data center cool, and make you look cool! But whatever you happen to chose, make a good choice of vendor, not just technology, and stick to it for as long as the technology life is. Otherwise you will end up with technologitis.

Reading an article about the usual failures of Enterprise Architecture in Skyscrapr I draw the following summary about what successful enterprise architectures have in common:

• Decrease complexity by partitioning problems into smaller non-overlapping problems.
• Increase probability of success by using fast iterations (rather than long iterations focusing too much on quality). The keyword here is fast. The faster the iteration in OOPA (observe, orientate, plan, act) the higher the likelihood of success.
• Create business architecture design, technical architecture design, implementation, testing and deployment for each partition. Don’t move to another partition until the last one was completed. This is what the author calls iterative partitioning.
• Prioritise the iterative partitions considering Time-to-Value, Return-on-Investment. Focus first on “low-hanging fruit” to establish credibility.
• Think big, start small.
• Stay away from application architectures and focus on interoperability (ie don’t try to standardise on the implementation).

I highly recommend the article. Straight to the point — it’s definitely not the typical abstract enterprise framework discussion.

Update (2007-11-19): Microsoft has eaten up the site and it’s not providing the customary HTTP 302 response code, but I could dig the article again here.

No, they never do. Here is the deal.

A centralized Enterprise Architecture (EA) function represents a compromise between the distributed functions in the enterprise. The distributed functions give up some control to invest in the enterprise technology strategy. As with any good investment, the returns need to outweighed the risk profile, otherwise the business functions will pull out of the whole EA thingy.

If the Enterprise Architecture function does not clearly articulate the returns on investment in technology, it will have trouble to enforce architecture governance. As much as the return may seem obvious to those situated in the realm of the enterprise — nevertheless detached from the actual business functions — returns on investment in technology need to be articulated, forecasted and measured on an ongoing basis to provide adequate architectural steering.

Even if EA is the right thing to do for most mature-growth organizations, the whole EA initiative is doomed to fail in the long-term without following this basic principle. Without it, it is hard to ascertain the required degree of adoption of the EA principles (how much EA do we need?), and the lack of clarity and transparency will eventually end up with the disengagement of the business functions.

Until an EA function is able to clearly articulate these returns, it should refrain itself from becoming a policing and dictatorial authority: The Ends Don’t Justify the Means.

I was reading yesterday the tutorial for the Globus Toolkit 4 (GT4), especifically this and this page. It highly disturbs me. Something is not right. It reminds me of the CORBA, and early day’s of EJBs. 90% systems programming, for 10% application programming. Thankfully AOP, Spring, Hibernate and friends kicked in to help, and introduced Inversion of Control to the masses. Hollywood’s principle for everybody.

In GT4, you code manually the WSDL, and your Java classes reference directly the framework and XSD namespaces.

I believe writing the WSDL/XSD by hand is a good thing when you are writing truely reusable (web) services, to ensure they are not just doing RPC over a pretty crappy protocol (HTTP), and you are focusing on the message. However, if Globus is to be used for parallel computing, ala my beloved PVM, you actually want RPC-style, and Globus should be generating the WSDL automatically upon deployment without developer’s intervention.

Additionally, when it comes to the Java side, the code looks like an EJB in the 90s. I don’t care if you could generate that code, it simply sucks as a class.

I look forward to the Spring of GT4 to come out really soon!

SOA means a lot of different things to different people. Now, if I go by the W3C’s definition, it’s about processes that assemble services and an organisation that owns the processes and the services.

However, to me, SOA is simply distributed computing over web services. I have been building distributed component applications since 2000 that were exposed as business processes. We did not have web services, but the facade was a process interface, and the execution was managed by a simple workflow engine. BPEL and SOAP were not yet mainstraim, still our application was based on a SOA.

So what makes a SOA different from good practice in doing distributed components (the service facade, service locator and state machine have been published design patterns for a while now). I think the only difference is the message encoding protocol (SOAP).

So, what’s with SOA scaling?

Many organisations are adopting SOA as almost the holly grail that would enable them to achieve reuse. For a SOA to succeed, one needs to think about processes and data.

Organisations that start now modeling on the process side, and construct processes assembling services exposing existing legacy systems, will hit a big performance wall. You need to think about data: ensuring a given entity ownership sits in one or few known parties. Otherwise, each service part of a larger process will access records which are not necessary for the ‘whole’. A little performance overhead summed up over a bunch of services that constitute a process results in possibly non acceptable overall service performance.

Those organisations that decide to focus on the data side encounter the reversed problem - you can’t work in your information architecture if you don’t know who is using you. So if you just build services and processes that expose existing data, you are doing it in a best-effort basis. It may, or may not, fit the interface and data needs of the service consumers. If they don’t, they will need to go against your datasources directly, and this will defeat the whole reason for an SOA.

I believe the following development methodology for services could work:

1. Model your processes, analyse the required data and use those requirements to create a data model.
2. Create a true component architecture, with proper entity relationships and ownership to encapsulate and abstract your data model.
3. Create services and assemble services according to your process design (which you may/should revisit by now).
4. Iterate the previous steps, starting from the core business processes, and adding processes to the SOA.

Interesting enough, this is how agile web 2.0 development is happening! Start off with HTML screens, then model your DB, then construct your model, controller and revisit the views. Which brings me back to the web 2.0 == SOA. We’ll eventually get there …

AJAX is the new pink. In fact, Web 2.0 could be the internet revolution that makes even Bill Gates wonder about the future of software. The enhanced usability in AJAX implies a couple of things for web applications:

• Some of the controller logic is moved from the server to the client. Less CPU cycles on the server per page, more on the client.
• Clients make more frequent requests to the server, but with significantly smaller payloads. The business logic on the server is broken up into smaller, more self-contained XML services (SOAP, REST). In essence, same aggregate CPU, more I/O.

Add to that a higher level of concurrency. More frequent, less expensive requests, same user, results in higher transaction rates, which will require more concurrent threads. More threads will consume more resources, both CPU and I/O.

During a simple AJAX revamping exercise, for the same amount of functionality, user CPU time is likely to be reduced on the server, and network I/O to go up. The extreme case with an AJAX MVC, and the server providing only web services, will actually reduce the CPU requirements on the server.

However, there are a few catches software architects should really start to consider:

1. AJAX will increase bandwidth requirements on the server, and reduce the ability to scale up hardware to CPU saturation, increasing the average hardware cost per CPU cycle. In other words, AJAX is not a server efficient technology unless you revisit the data going back and forth.
2. Once the AJAX-night fever comes down, a next wave of web design will start to put more emphasis on orchestration and collaboration. Web services transaction management will start to be more mainstream due to additional client requirements, and stateful web services will appear (sadly?).
3. Current server tuning rules will no longer apply, given to changes in usage patterns. Multi-core architectures, such as Niagara and Power5 will be best placed to serve the increase in threading. Niagara being a lower power, high bandwidth chip, but not necessarily a fast CPU, will be one of the best architectures to run AJAXed applications.