Alright, we’ve previously established that at least some enterprises have a substantial software investment outside of the classic business process arena. We’ve also seen an example of advice that fails to take account of this class of enterprise. Now it’s time to talk architecture.

The more traditional enterprises (those that only need software for business process automation and support) made a mistake in their past which needs undoing. They focused on building applications not systems. Each application was designed to tackle some individual aspect of their business processes which when they needed to be integrated caused much pain. The result has been a trend (SOA) to break up all those application silo’s into a collection of shared services on top of which appropriate applications can be built by the enterprises themselves or others. In the latter case shared services must in some way be exposed to others outside the firewall.

Creating services as above is merely one method of partitioning code and data. This method does not apply so well outside of business processes so we must find another model. ROA presents one such model, one that is data centric and works well in the world of the Web where we desire ad-hoc (chaotic) assembly of resources (mashups). The nature of the Web is such that it can be difficult to know who your users are (there are too many) and to manage transitions from one version of an API to another. Basically you don’t know what your dependencies are and it can be difficult therefore to measure the impact of changes. Some have suggested that it’s not appropriate to retire old APIs or resources but this can have significant impact in terms of maintenance such that at least some organisations do deprecate old APIs and retire them eventually. ROA like any other architecture has it’s limitations.

ROA is of course derived from REST. REST includes a set of constraints which are essentially just useful architectural patterns. Some would claim they deliver scalability but I prefer to state that they are “scalability enabling”, they don’t inhibit scaling. However it’s important to realise that behind the Web layer one must still build scalable infrastructure. Building this infrastructure the right way (architecturally) yields scalability not REST itself. Many websites rely on running multiple copies of their web application, scaling via their database and caching solution which will often be more than enough.

However there is a class of enterprise website for which this approach fails, because the consistency models provided by databases and the like actually can’t scale as far as is required. A further complication is that managing everything as a single web application becomes impossible. Each part of the application has its own unique demands in respect of tuning, configuring, monitoring and maintenance:

• Tuning for one part of the application has adverse effects on other parts.
• Configuration becomes a nightmare because there are so many different settings to worry about. Something that works well in testing isn’t appropriate for production leading to separate profiles that must be maintained and kept in sync leading to forgotten changes etc.
• Monitoring produces so much of a mixture of data that it becomes a major exercise to filter out just what you need.
• Maintenance becomes an exercise in chasing down long chains of dependencies to make a simple change.

It becomes necessary to break up the application, storage, caching and so on into more manageable pieces that can run separately as a distributed system. Each element provides a service but not as would generally fit with the “classic” definition of SOA. Our requirements for partitioning are driven by multiple forces and thus the decision as to how exactly to break up the application must be determined on a case by case basis. Such a decision could be driven by amongst other things business model (web applications are still surrounded by business processes), scaling needs, specific storage requirements of underlying data or provision of a specific feature for the website.

One might choose to expose such a service using WS-*, messaging, a Web/Resource approach, CORBA or even some form of custom service invocation layer. Perhaps surprisingly there’s a growing number of examples where the custom service invocation layer option is used. I believe this is because all other approaches represent a compromise achieved through limitation of architectural options in such a manner as to be inappropriate for these demanding cases.

We cannot call this architectural approach SOA, ROA, EDA or anything else, it is simply about creating isolated, independent elements and minimising dependencies. It is something we’ve been doing inside of our programs for years. It also allows us to construct a working, manageable system at large scale. It is common sense. CSA anyone?

Technorati Tags: architecture, distributed systems, scalability, software

Check out this article from Computing. It is apparent good advice for SOA implementation but as mentioned in my previous post, something has been forgotten - some enterprises provide software as a web app that is their product and revenue generator. This software could be rendered into services behind the firewall, yet is not about business processes and must be treated differently.

A quote from the article:

Mistake No. 3: Leaving SOA to the techies

When the SOA process is left mostly with the IT side of the organisation, services risk being designed to optimise software performance and reliability, but may not fully reflect the business requirements.

Clarity of business interfaces is essential for cross-application integration or multi-organisation use.

What about an interface that provides a specific website feature and is a service in it’s own right? Such an interface is unlikely to be exposed across organizations because it provides a business specific feature we do not wish to share with others. Further such an interface probably has few business requirements though the underlying service may need to support auditing or customer tracking tools.

A further quote:

Mistake No. 1: Irrational SOA exuberance

Excessive numbers of services ­ those that cannot be readily matched to the business model of the application ­ are a sign of an SOA environment where applications need to be checked as they are completed.

Such environments may feature repositories full of services, volumes of documentation and an impressive collection of new tools and middleware, but what they will not have is agility, incremental software versioning or reuse.

Again let’s consider a service that provides a website feature such as recommendations. How much does it have to match the business model? One might argue that SOA is only concerned with business processes but surely we can model other things as services?

So what exactly is a service, what is SOA and where does REST fit in? I’ll cover that next….

Technorati Tags: architecture, distributed systems, soa

Let’s say you sell pottery. Your core business is about making that pottery, perhaps to order but there’s a whole load of additional work you must do around accounting and the like. Perhaps you also ship your pottery via surface carrier, this introduces further work tracking customer details (watch our for pesky privacy regulations), working with couriers etc.

Almost from the get go these days you’ll automate as much of this drudgery as possible using computers, essentially creating a one person IT department that supports your business processes. Obviously the bigger the business gets the more automation you will require, perhaps developing custom software to support the execution of your business processes (all that drudge you have to do but isn’t core). Your IT department will have to grow accordingly, hiring developers, project managers etc.

This standard tale of the relationship between business and technology plays out in many an enterprise every day. IT and in particular software development is a necessary evil, an undesirable cost to be carefully managed.

However for a certain class of enterprise, software is the product. Maybe it’s a shrink wrap product or a set of services provided to customers via the web. There will still be a need to have an IT department to support the business processes and it might well develop software of it’s own. There’s a fundamental difference between these two types of software because one of them is the business’ money generator. This software must be nurtured and tended to carefully, one can’t just add features, one must care about stability, performance, availability, scalability and so on. Money put into this effort is not a cost it’s an investment. Fundamentally, this software is not enterprise IT as most know it.

This key difference is oft-missed by management, vendors and analysts leading to confusion and unnecessary mistakes. I’ll cover one such mistake (related to SOA) in a follow-up post.

Technorati Tags: business, engineering, software

Going to be at the Google London Open Source Jam on Thursday 29th November 2007, 6pm - 9.30pm for an evening of distributed systems hackery.

Much of what Betfair does must remain secret, however this stuff is a matter of public record so I can share a little with you - couple of excerpts that put a smile on my face:

JUDGE: You will explain to us how you find a matching bet?

OUR QC: I will, yes. That brings me to the little demonstration, your Honours. Your Honours ought have a bundle of material which is entitled “Demonstration of Online Betting”.

JUDGE: How much of this is on the CD?

OUR QC: This is all on the CD, your Honour. On the CD it is - - -

JUDGE: We had a Playstation shown to us in Sony and it was very exciting. Why did you not try that?

OUR QC: This is more fun.

JUDGE: It is one of the most exciting things that has happened in my time here.

…let’s talk about the Internet.

JUDGE: I do not think I have ever seen in my 12 years here longer written submissions.

OUR QC: I do not know whether to take that as a compliment or not, your Honour.

JUDGE: Take it as you will. I mean, the danger of very, very long submissions is that the detail swamps the appreciation of the structure and essence of the point which the whole point of oral advocacy ought to be to try and refine and press upon our minds in the brief time that is available.

OUR QC: Yes, I know all that and that is the way I hope to use the oral advocacy, your Honour.

JUDGE: It is just a response to your threat that you are going to be very long in some of your oral submissions.

…you should see the average system requirements spec.

…to Steve McConnell for distilling out the key issues around technical debt:

“The reason most often cited by technical staff for avoiding debt altogether is the challenge of communicating the existence of technical debt to business staff and the challenge of helping business staff remember the implications of the technical debt that has previously been incurred. Everyone agrees that it’s a good idea to incur debt late in a release cycle, but business staff can sometimes resist accounting for the time needed to pay off the debt on the next release cycle. The main issue seems to be that, unlike financial debt, technical debt is much less visible, and so people have an easier time ignoring it.”

I would quibble with the “easier to ignore” aspect though as I think for the most part both kinds of debt attract the same behaviour - sticking our heads in the sand allowing things to get worse…..

So we’re building an Internet service not a Web service, decide on an address, allocate a port, write our daemon (yes, I like Unix) job done. One might think so but there’s a killer deployment issue lurking in the background - the Firewall.

The average corporate security policy really doesn’t like opening ports on external firewalls (and often the same goes for internal ones). Best case we’ll have to wade through masses of red tape, worst case we’ll be given a flat no to our request for an open port. What to do?

Find an open port, find a way to tunnel through it. Which is the port most likely to be open? 80 and we all know which protocol runs over that. Sure as night follows day, we end up building a solution that tunnels over http.

Do we want to be “of the web”? No.

Would we by desire tunnel over http? No. It’s not designed for that purpose and it’ll likely let us know come implementation time.

Should we re-design our service to fit with ROA? No. Hopefully we did the research and looked at this option before choosing to implement an Internet service as opposed to a Web service.

So Internet and Web services are different but can end up looking similar enough to lead to confusion. Ain’t reality ugly? Tradeoffs must be made, the results are often less than pretty and there might well be a lot of complaining.

Technorati Tags: architecture, networks, systems