Beyond number of servers, another factor is rate of infrastructure growth. Bottom line is that investing in a provisioning & monitoring infrastructure early (say going from 50 to 100 nodes) would pay off in a 100 to 500 surge and beyond. My postulation is that, the flatter the growth curve, the longer you can avoid building custom infrastructure.

Sure, the argument can be made that growth rates are not a hard science, but one can improve that by monitoring growth trends and intangibles such as word-of-mouth, press & infosphere sentiment. Sometimes, one can just sense the impending detonation; and the sooner the investment is made, the faster the return and the smoother the experience. That’s assuming whatever is built actually works :).

The other factor that must be taken seriously is that the anomalies one experiences @ 10/100/1000/10000/100000/1000000 node levels are different. I don’t think anybody beyond Yahoo/MS/Google have experienced all the circles of hell.

What you say in your final paragraph is much like what the primary message of my latest column was, which is to be aware that there are other ways of doing these things that are often better — both convenient *and* correct — so developers should do themselves a favor and learn those ways, rather than just sticking with what’s convenient and ignoring the “correctness” part, whether ignorantly or willfully.

Yep, it should be a pragmatic exercise. You should test, and automation would be nice but TDD cannot be allowed to drive your architecture to significant degree. Testing is an important facet but there are many other forces that influence architecture and determine it’s success in the longer term.

As for the TDD YAGNI implications on architecture, that works but only if you know you are building something architecturally simple (such as a simple layered application). If I had to guess, that’s probably the type of application many of the extremeist YAGNI folks are building (and I’m sure it works great for them).

Those who have dealt with more complex systems know that to not think about architecture is really just gambling with the success of the project.

Of course, for the simple layered system, doing a lot of “architectural” work up front is just as bad as not doing architectural work on the more complex system (think: both scenarios may double the cost of the app).

In short, it’s all about context. If anything is true in software these days, it’s that there is no universal right or wrong way to do software development. Context is king.

(4) It’s all about code - we don’t need a stinkin’ architecture.

3) Automated Testing is too expensive at the moment, we’ll add it in later if there are any problems

Back around 98 or 99, I went on a dive trip in West Palm Beach with several friends; I drove my car. I live in Tampa Florida and so taking trips like this are easy. Well, we did 4 dives that day. The dives were all good, and afterwards we all had a great dinner and talked about the cool stuff we experienced during the dives. Well, all of us had to work the next day and so we ending up leaving West Palm Beach at midnight. The drive is about 3 hours long, plus the time to drop everyone off. Well, I was driving and everyone else passed out….pretty soon I was starting to fade. So, to keep myself awake, I started driving with the lights off; the roads were lightly traveled at that time of night. The rush of driving without the lights forced me to focus and kept me from falling asleep; I was pretty damn wiped out so I needed to do something extreme to stay alert. Well pretty soon one of my friends Barbara woke up and saw me doing this. She immediately decided to switch places with the other guy and talk to me the rest of the way. I drove with the lights on the rest of the trip…

I can imagine using Javaspaces to help implement the co-ordination part of extreme data manipulation but I wouldn’t imagine using them to hold or transport the data itself unless the chunks of data are small (I wouldn’t use a JavaSpace for passing around huge log files).

Generally, if I advocate the use of a JavaSpace it will be with a specific context and I’ll present some analysis. I appreciate some of the vendors in this vertical like to portray their software as being very broadly applicable in all sorts of “extreme” circumstances and thus it can be confusing.

The above was not intended as a JavaSpaces piece just a line of thinking I’d been pursuing in respect of “do it all” type products, their inherent flexibility problems and how that impacts architecture, design and code.

HTH,

Dan.

are the “javaspaces” products a good fit in those certain circumstances?, especially, in extreme data manipulation applications?.

Out Of The Crisis - Deming
Why Things Go Wrong: Deming Philosophy in a Dozen Ten-Minute Sessions - Fellers

As we’re talking philosophy, the tricky thing can be understanding how to apply it. One of the things I focus on a great deal these days is ensuring that I can measure and grasp the problems I’m dealing with. Note for example how much software goes out the door these days with little support for useful measurement - the closest one typically gets is to dig through log files such that when things aren’t working right or you need to re-design the information you’d like to guide your efforts is seldom available.