Pragmatic Dictator

The release of the Dynamo paper has generated a lot of interest around the net. That’s more than appropriate because I don’t think there can be any doubt that Dynamo is a great piece of work.

It seems there might be a further bonus that’s largely gone unmentioned (even Greg seems to have missed it) but has been hinted at by Werner at various points in the past. Read carefully and you’ll find some details of a custom invocation infrastructure:

“Both get and put operations are invoked using Amazon’s infrastructure-specific request processing framework over HTTP. There are two strategies that a client can use to select a node: (1) route its request through a generic load balancer that will select a node based on load information, or (2) use a partition-aware client library that routes requests directly to the appropriate coordinator nodes. The advantage of the first approach is that the client does not have to link any code specific to Dynamo in its application, whereas the second strategy can achieve lower latency because it skips a potential forwarding step.”

Notice how they have support for both smart and dumb clients with the smart client setup being somewhat akin to a pattern that’s been seen in Google’s software including Chubby. The choice to reuse http would give them an option to leverage many a load balancer’s capability to apply custom routing by URL which would assist in service invocation routing.

Other interesting tidbits include:

1. A mention at Google Scalability Conference of a lightweight rendering engine that might invoke upwards of 150 requests per page. Given some of the latencies discussed in the dynamo paper I am wondering if this custom framework might have some support for making collections of requests in parallel.
2. Common service types are stateless aggregator services that can perform a lot of caching (wondering how much the use of http helps here) or stateful services.
3. A statement from a past interview with Vogels:

   “The first category is the services that make up the Amazon platform. There we use interface specifications such as WSDL but we use optimized transport and marshalling technology to ensure efficient use of CPU and network resources.”.

   See the mention of the custom framework again but also a possible hint that they make use of a variety of interface specifications (perhaps including something homebrew).

Food for thought?

Technorati Tags: distributed systems, performance, amazon, scalability

This entry was posted on Thursday, October 11th, 2007 at 9:02 pm and is filed under Distributed Systems. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.