Pragmatic Dictator

So we’re building an Internet service not a Web service, decide on an address, allocate a port, write our daemon (yes, I like Unix) job done. One might think so but there’s a killer deployment issue lurking in the background – the Firewall.

The average corporate security policy really doesn’t like opening ports on external firewalls (and often the same goes for internal ones). Best case we’ll have to wade through masses of red tape, worst case we’ll be given a flat no to our request for an open port. What to do?

Find an open port, find a way to tunnel through it. Which is the port most likely to be open? 80 and we all know which protocol runs over that. Sure as night follows day, we end up building a solution that tunnels over http.

Do we want to be “of the web”? No.

Would we by desire tunnel over http? No. It’s not designed for that purpose and it’ll likely let us know come implementation time.

Should we re-design our service to fit with ROA? No. Hopefully we did the research and looked at this option before choosing to implement an Internet service as opposed to a Web service.

So Internet and Web services are different but can end up looking similar enough to lead to confusion. Ain’t reality ugly? Tradeoffs must be made, the results are often less than pretty and there might well be a lot of complaining.

Technorati Tags: architecture, networks, systems

Everybody’s pitch is about the great things that can be done with their technology, method, architectural approach etc. They’ll sing it from the roof tops, put it up on websites, spam us with email and so on.

But they rarely (if ever) discuss the flip-side – what their stuff is not good for. Sometimes:

• selecting an appropriate solution is more about what something is bad at than good at
• the easiest way to understand something is to know it in terms of what it can’t do

It’s amazing how often asking someone about the negatives of their stuff results in silence.

Technorati Tags: architecture, design, systems

There’s a lot of noise about transactional memory, thought I should do a bit of research. Having read a number of papers I’m left wondering just what all the racket is for. At least for me the benefits are unclear.

Let’s consider this paper which discusses amongst other things “transactifying” Berkeley Db (a piece of software I know quite well). It contains a comparison of the original version of Db’s locking system (which used a global lock) and the paper’s authors’ modified version. Initial changes were to replace all uses of the global lock with a set of transactions. A test was run and the transactional version was worse all around than the original – ooops.

The root cause boiled down to three issues:

1. False sharing – a problem which occurs when variables accessed by different threads happen to fall in the same cacheline – this was solved with a traditional approach known as padding.
2. Statistics collection – Db collected a bunch of statistics keeping them accurate by using the global lock. Rather than address what is surely a common problem, the authors simply turned this feature off.
3. Object pooling – the pooling associated with lock descriptors and their related objects had to be changed from single linked-lists to collections of linked-lists to improve potential for concurrent access.

The tests were re-run and beyond a certain level of scale the transactional memory version was now better but wait, there’s a problem. Notice that all the work being done to make the transactional version better is broadly the same as the work one would do to make the locking version better. How much of the scalability gain is due to better concurrent structure and how much is down to transactional memory? Is the work we’ve just done any simpler than what we already have to do for conventional thread/lock based systems?

Another under-discussed factor across many papers in this area is related to the assertion that transactional memory is better than locking due to it being more efficient in the non-conflict case. However many modern lock primitives are now also optimized for this circumstance.

What about the fact that, one must make sure to correctly isolate the atomic actions in a system and bound them appropriately with transactions just as one currently does with locking? We still have to make sure we do that consistently across the entire system or risk the usual concurrency debugging nightmares.

Many of the transactional memory systems appear to be based on optimistic approaches – does that make sense for all algorithms and systems we might build? Other transactional systems have evolved to provide both optimistic and pessimistic options (in an attempt to cover all design possibilities) and the programmer must make the appropriate choice for their application. Will transactional memory systems also need to move this way and if so, how will the programmer work with that?

Asserting Order

I’m not going to write-off transactional memory but it seems that should it turn out to be more scalable than the conventional lock-based approaches we use:

1. It’s really not much simpler to program with.
2. It’s no use in the distributed case.

Meanwhile:

1. There are other approaches around that do work across both multi-core and multi-box/distributed cases with little change (some would argue the amount of change is zero but I don’t buy that).
2. Dealing with concurrency is about much more than whether you use locks or transactions.

Technorati Tags: concurrency, distributed systems, scalability, systems

The question is, can we see it? If this article is anything to go by the answer would be no.

SOA is an approach to building systems, it certainly couldn’t be called a style (much to the annoyance of some) but it sure isn’t a technology.

And this is the problem – so many view everything to do with building systems as being about deploying the right technologies rather than adopting an approach and driving technology selection from there.

No surprise then that SOA adoption is isolated to small parts of various businesses – that’s the maximum level of use that can be achieved whilst it is treated as a technological shift. Change across the entire business is essential for SOA to get real traction – systems shouldn’t be viewed as necessary evils that cost, rather they should be considered as means for delivering enhanced business value. Processes and culture are the real challenge not hardware and software.

IT/Systems Development needs to be considered a first class citizen within an organization rather than simply the poor cousin that mops the floors and cleans the toilets. Fewer conversations like "here’s what we want" and more discussion around "here’s what we’re trying to do, how can you help?". Switched on readers may notice an interesting parallel with Web vs Enterprise….

Web is (amongst other things) about enabling users and agents to do interesting (interconnected/social) stuff more effectively, whilst Enterprise is often treated as little more than automating laborious tasks with strict controls. Loosely coupled versus tightly coupled, granular and cooperative versus monolithic and uncooperative.

Technorati Tags: enterprise, soa, technology, web

The debate about all these many core processors continues to circle the blogosphere. Tim Bray had this to say which set me thinking (always a bad thing):

Any time we have a piece of state that needs to be accessed concurrently we hit problems. One can hide this problem using messaging (or similar) but the key aspect in these solutions is that we can partition operations into streams against discrete elements of data (a discrete element could be a group of things) that don’t interfere with each other. Partitioning however can be problematic:

1. Our data has to be amenable to partitioning via hashing or some other method.
2. It gets tricky when we need to deal with availability and disaster recovery.
3. Getting the correct granularity of partitioning can be challenging.

Which is interesting because whilst we’ve eliminated the concurrency issue, we’re now faced with a different one (partitioning) which could be just as hard to cope with and requires just as much thought from a developer and/or architect. Coincidentally, Werner Vogels (Amazon) is going to be talking about an internal data store (HASS) at the Google Scability Conference and specifically the problems of partitioning and consistent hashing (my original interest with respect to this talk was in the context of the CAP conjecture).

Another means of avoiding all these concurrency issues is to push them somewhere else. More often than not this becomes an exercise in creating a supposedly stateless system which in reality simply puts all the state in one place, usually the database. The argument is that this is acceptable because it’s only the likes of databases that should deal with these hard issues.

The rub with having the database handle it is that the concurrency model it uses will only scale across so many processors (more if you’re read mostly, less if your not) and cope with so many concurrent accesses from the stateless component. Once again to get our database layer to scale, we’ll need to partition our data into shards across multiple databases (an approach adopted by a number of top-line websites) or find some other way to reduce concurrent load on the database instance.

The act of partitioning can mean we reach a point where we can no longer expect to have atomic updates because the mechanisms for achieving it (e.g. two-phase commit) stop us scaling. When this happens we must construct complex or at least exotic solutions such as that proposed by Pat Helland.

Okay we got rid of our concurrency problem and swapped it for a partitioning problem which then turned into something of an exotic problem. Are we any better off? It seems no matter which way we go we end up with some tough problems to solve.

Perhaps there’s a sweet-spot tradeoff where the combination of a CMT box, with data partitioned across a number of processes and each process containing a simple concurrency model covers most situations. Even if that’s the case it seems developers will have to learn a few new tricks.

Technorati Tags: architecture, concurrency, software, systems

Update: A good comment over on Reddit.