Pragmatic Dictator

I mentioned a while back that one could exploit DNS to ease some of the common static configuration issues around hostnames, ports etc. What follows is a simple outline solution, we’ve moved a long way beyond this at Betfair but the details will have to remain secret for now (sorry).

Let’s assume that we have several different releases in testing at any one time such that we wish to segment our development/testing systems into separate enclaves (each handling a separate release) and may wish to add more enclaves over time. Assume also that production is an enclave in its own right.

Firstly we define a set of logical hostnames that refer to the significant components of our system such as databases, file servers etc. Other elements such as webservers are probably independent and not referenced from other parts of the system and thus do not need names. These logical hostnames are what feature in our configuration files and do not need to change from enclave to enclave because we are going to use DNS to map from these logical hostnames to real physical machines.

Thus we want is a separate namespace for hosts in each of these enclaves so as to prevent leakage. To that end we map each namespace onto a separate domain within our DNS setup.

[Note our DNS setup would typically consist of a set of servers that maintain records for our own internal domains and possibly forward other requests for say external web address to other servers.]

Each enclave therefore has:

1. A separate namespace represented as a unique domain
2. A set of services deployed onto physical machines
3. A mapping from logical machine names to physical machine names (or IP addresses)
4. A collection of configuration files all referencing logical machine names

Each domain (namespace) contains the logical to physical mapping of machines for its associated enclave. Each domain can be a separate zone and is thus kept in a separate file read by our DNS master. This allows us to maintain a template file which can be quickly edited to create a new domain (namespace). Thus whenever we wish to create a new enclave we setup a new zone, containing the definition of a new domain which is the namespace for that enclave.

To actually resolve a logical hostname we must ensure that it is concatenated with the domain appropriate to the enclave’s namespace. Before discussing options, note that each machine will be allocated to an enclave and must be configured accordingly which we can exploit to our advantage:

1. Simple configuration – ensure that the application has access to the domain to concatenate. This could be done via command-line argument but better is to source it from a well-known file on the machine which could be setup as part of allocating it to an enclave.
2. Default search domain – any name not fully qualified has the default search domain appended to it. This default is typically part of the resolver configuration of the operating system and again can be setup as part of allocating a machine to an enclave.

Missing from the above is the handling of ports which might change from one enclave to the next. This can be tackled with a similar logical/physical mapping approach but must be based on the use of DNS SRV records rather than simple hostname mappings. The JDK provides little help out of the box for querying these records so something like dnsjava will be required.

Technorati Tags: distributed systems, dns, testing

I commented in my previous post on the fact that working with message brokers can lead to tensions where we’re forced into letting the broker do more than we’d like.

I suspect that the bigger the list of features provided in a single broker the greater the probability we’ll be forced to comply with (and attempt to work around) undesirable models and behaviours that aren’t core to our requirements. Thus under certain circumstances a specific solution that addresses one core problem whilst asserting a minimum of additional constraints can be more attractive than a jack-of-all-trades solution.

It also occurred to me that brokers might not be the only piece of software to exhibit this characteristic. RPC systems tend to suffer similar problems having a tendency to tightly bind endpoint addressing, transport and argument marshalling. I also wonder if these issues contribute to some of the backlash against the behemoth that is an RDBMS implementation. For example whilst we might like our RDBMS to look after our data, we might not wish to put up with the querying, concurrency, deployment and management models it also asserts even with a myriad of configuration options.

Technorati Tags: architecture, design, technology

Say the word messaging to a subset of developers and for some reason the immediate knee-jerk is to assume that means using some kind of message broker (Tibco, ActiveMQ or whatever). Utter the term “asynchronous communication” and that is typically equated to messaging and thus also implies use of a message broker.

I find this strange because messaging is possible via a myriad of methods including carrier pigeon, pony express, tcp, multicast, http and many other transports. As for supporting asynchronous operations well that’s governed by the API(s) provided by the transport. In fact this is only partially true because a lower-level synchronous API can be wrapped in an additional layer to produce an asynchronous API. Transports and layering are often seen together, for example it allows us to introduce support for reliable delivery if the underlying transport is not robust (often it isn’t).

There’s no denying that all (or most) of these features (asynchronous APIs, messaging, reliable delivery) are provided in various of the message broker implementations but rarely in the form of a set of composable layers. The preferred approach is usually a myriad of configuration options leaving us at the mercy of the vendor to provide and support just the right combination of configuration possibilities to match our design challenge.

The indivisible nature of these brokers hampers us in other ways too. It can be difficult to use the broker purely for messaging without being forced to work with say its models for routing control and security. I’m sure this is appealing to the vendors but it doesn’t seem like such a great deal to me.

Technorati Tags: architecture, design, messaging

It never ceases to amaze me how often supposedly professional software people (developers, designers and architects) choose to trust in blind luck. They become rabbits in the headlights of pressure be it deadlines, expectation or lack of insight resorting to chucking code out the door as quickly as possible hoping it’ll lead to something good.

If it does lead to something good, how would we recognise it? If it’s going badly, how do we recognise that and formulate a new approach? What is the cost of taking such a speculative action? What’s the best thing that might come out of the action?

Many of these people would claim to be brilliant problem solvers and yet they are lacking grasp of some of the fundamentals.

Technorati Tags: engineering, process, software