We don’t have such a limit obviously being applied in software/systems. I think it’s possible to do such a thing but requires a radically different approach to interviewing, the junking of the various certifications we have etc.

Happy New Year!

I hadn’t thought about blogging that stuff but, given I’ve had some prompting from my audience, I’ll try and cover some more of that ground in the future.

Thanks for the feedback,

Dan.

I’m going to post it to the River dev list ’cause we’ve just started talking about way to run the QA test suite in way that lets us simulate some of these kinds of failures.

Could you explain how much remote intercommunication there is between your actors? Are you for example spreading trades across a bunch of actors that are spread across a collection of machines – basically a fan-out/tree model or something more complicated?

Presumably you’re using a JMS cluster or similar? What happens if that becomes unavailable (they aren’t totally reliable and networks fail)? How do you partition your actor comms to ensure you don’t overwhelm a single messaging cluster with too much load (they don’t scale forever)?

Do you have all your actors whether they are local or remote use the JMS instance? If not you’ll have variable latency which can cause queues to fill etc unless you account for that in your algorithm or you’re in the fortunate situation where your peak load doesn’t outstrip queue throughput.

Where do you keep the state each actor is in possession of? Clustered database? (They fail as do the likes of Coherence if you’re using a cache). What happens if an actor receives a message and cannot then update the state?

Guaranteed messaging of course only means “deliver eventually” – when might that be?

There’s a really basic basic I didn’t mention

Our environment is well enough setup that we can make use of basic math to predict what we need come production hardware wise but some of that is a function of the overall side of environment we require right now. As we grow that will become more difficult to do and probably require us to focus more on “dark releases”.

“But if you’ve set-up your environment to maximise application performance for some performance work you might eliminate all of your scheduled jobs so you can focus on the KPI you are interested in. You need to remember to turn all of these back on when you are doing proper load testing, as you need to be as like live as possible.”

Another good bit of advice – automation of course is key to making that “turn off the jobs, turn on the jobs” thing work well. Oh and our perennial favourite monitoring can help too

Thanks for the feedback and the encouragement – been busy of late and taken some time to figure out what’s “safe” to blog about commercially speaking. Let’s see if I can’t push up that blog post rate….

You also need to make sure that you’ve properly recreated your production environment, if you are running the batch jobs – it sounds like you are pretty much there.

But if you’ve set-up your environment to maximise application performance for some performance work you might eliminate all of your scheduled jobs so you can focus on the KPI you are interested in. You need to remember to turn all of these back on when you are doing proper load testing, as you need to be as like live as possible.

This is another good blog entry Dan, you should post more often !

Alex