How should you build distributed sytems: API or messages? This post by Eric Armstrong says messages -- http://www.artima.com/forums/flat.jsp?forum=106&;thread=120669 -- and I couldn't agree more.

APIs create a tight a binding between the protocol functionality and your code. If they use a certain libary then so do you. If their API conflicts with your code then you are out of luck. If their API corrupts the heap then you are out of luck. If their API can't handle interrupts, semaphores, queuing, memory contraints, priority, or a host of other issues properly then you are out of luck. Your code can't evolve independently from the API which makes your system less functional, more dependent, and more brittle than when a protocol is used. That's too high a price to pay when your butt is on the line to get something working and to keep it working.

Another key advantage of using a protocol is that it can be implemented in any language. You don't have to wait for a vendor to create a language binding for your language version and operating version. This can take forever. If you are on an unconventional OS like VxWorks, you'll probably never see an API. You don't have to wait for bug fixes either. And you don't have to worry about how to include their code in your build system.

When using a a protocol as long you can stuff rightly formatted bits down the TCP, UDP or whatever socket then you are in. For example, I was able to quickly make a sweet load test system in Perl for a set-top box network because a protocol was used and it was easy to compose bits together in Perl. If I had to wait for a Perl binding to the API I would still be waiting and the load testing system may never have been attempted.

I think it is. I discover a way to get from A to B using code, much like I discover a route from point A to B using roads. You could say the roads already exist, so it's not a discovery to figure out a route. But our notion of discovery is really to make plain that which already exists. Columbus discovered America and it already existed. DNA already existed. Gravity already existed. Many mathematicians are at bottom Platonists. Invention is something much more rare. We rarely invent anything in software, while we constantly make discoveries. That's why software is so fun!

Out of my hundreds and hundreds of thousands of lines of code how many real inventions have I made? I've been clever. I've been effective. But I don't know if I've ever invented anything, yet virtually all of my brain farts are patentable by current standards.

So I stand along side Donald Knuth (http://www.groklaw.net/article.php?story=20050724140820490) who sayeth:

My personal opinion is that algorithms are like mathematics, i.e. inherently non-patentable. It worries me that most patents are about simple ideas that I would expect my students to develop them as part of their homework. Sometimes there are exceptions, e.g. something as refined as the inner point method of linear programming, where one can really talk about a significant discovery. Yet for me that is still mathematics.
I come from a mathematical culture where we don't charge money from people who use our theorems. There is the notion that mathematics is discovered rather than invented.

"In rites at large, it is always better to be too simple rather than too lavish.
In funeral rites, it is more important to have the real sentiment of sorrow
than minute attention to observances."

-- Confucius in the Analects

I was struck recently by the similarity of the role of ritual in Confucian philosophy and the role of ritual in Agile projects. The system of thought put forth by Confucius tries to create a world where people can live together in harmony without resorting to uncivilized behaviour. You think I am stretching here? We'll see...

Confucius was motivated by his times. He lived in what was called the Warring States Period. Battling warlords made life difficult for your average person. For simplicity, this era will play the waterfall methodology "big process up front" role in my analogy :-)

Deeply concerned about creating a better world in which to live, Confucius proposed ritual as a key civilizing influence. So does Agile.

What you say? We are programmers, we don't need no stinkin' rituals! Hold on now. Don't get your editor in undo mode.

Let's consider just what ritual is: ritual can be thought of as respect for the the deepest sense of proper way of doing things.

Isn't that a fine Agile principle?

On a project you could could consider ritual the pattern of disciplined behavior which governs each moment in the life of a project.

Now does that sound bad?

Without the bonding of ritual there isn't order. Why? Because people learn proper relationships through ritual. Ritual promotes harmony by showing everyone how they should act and interact without using a heavy hand. He who governs best governs least. Using ritual everything just sort of happens and fits together. In a world where everyone is looking to kill each other you can see how you might want ritual to coordinate social situations. Modern software projects aren't so different at their base.

Ritual is all around us.

One saying i really like is: Only Say Things That Can be Heard

What this means is that everyone and every organization is in a different place in their lives. If you show up at a company and tell them everything they do stinks then you won't be heard. You will just make enemies and nothing will get done as everyone sharpens their fangs for the next fight. You get the same result when "talking" with your family too :-) Somehow you must find a way to talk to someone in a way that makes sense to them.

I must admit this is a hard bit of wisdom for me to take out in the real world. I tend to be too direct at times. I remember one particular meeting where I realized just how destructive the direct only the facts matter approach can be. In this meeting from hell this one particular ubergeek was coming hard and fast. He was questioning everything. He was disagreeing with everything. He was saying everything I wanted to do wasn't going to work because it didn't work when he did it at company X. He wasn't giving an inch on anything. And he was doing it all with that infuriating "I am just trying to understand" attitude. In short, it was like looking at me at my worst.

Now I don't much like looking in the mirror anyway, but when that's what I see staring back I want to banish all mirrors, just like they did in the movie The Skeleton Key.

At that time I made a solid pledge to change may ways and try to Only Say Things That Can be Heard. Am I 100% successful? Unfortunately, no. But I am much better now. Through my career I have consistently witnessed this same movie replayed, only with different actors and a new sound track. Software is only partly technical. Building software is mostly social.

Believe it or not, there's an agile style to doing the laundry and BDUF style to doing the laundry.

My beautiful wife, Linda, does laundry BDUF style. The laundry piles up and until it just has to be done. Then she sorts. Every load is pre-sorted into its own pile before it can be washed. If you look at our laundry room it looks like a field of hay stacks ready to be bailed. The piles are formed by some set of rules that I have never been able to master, after 20 years of trying. When I try to do the laundry her way I can never quite get it right. So everything is well organized. We have lots of nice piles optimized into the correct size for our washer. There are no left over clothes that don't get washed. We don't have small loads, which I am told are a waste, even with our water efficient washer.

What could go wrong?

There are lots of piles. So many piles that they can't possibly get done in one night or even one day. Now let's add to that the remarkable ability to be distracted and very few piles actually ever get washed. So the piles just hang around, sometimes for days.

What is happening while the piles go unwashed? We use and dirty more clothes. Part of the problem is we may have too many clothes, but it doesn't seem like it. Anyway, with new clothes being added all the time the piles no longer make sense. If we just added the clothes to existing piles then they wouldn't fit in the washer anymore.

The piles need to be rebalanced. Wouldn't you just split the large piles in half? No. That would be wasteful. We want the washer to be maximally full because that is the most efficient way. Just splitting the piles would leave us with too many small piles. To make larger better piles we have to rethink what clothes can be washed together. Maybe the jeans and the towels can be washed together after all. Like I said, I've never been able to figure out the rules, let alone when the rules can be ignored.

Projects are full of Sham Customers. If I use something you are building on a project we'll all subscribe to the collective delusion that I am your "customer." But am I really a customer? Am I a real customer?

No I am not. Why?

Because I am only a real customer when I am paying you to do something and you really want my money. Otherwise you have no incentive to do what I want. But wait, why is money even part of the equation? I am the customer, doesn't that mean you'll do what I want as long as it makes sense and is reasonable?

Oh no.

If you aren't dependent on me in some direct way--not some metaphysical way like we are all in this together, or we are all working for the same company, we are all working to the same purpose--then you can tell me to get stuffed and there's nothing I can do about it. Nothing!

Let's say I need a a couple of tables in a database. You are the database group and I have to go through you for database stuff. For organizational reasons I just can't make my own database. I am using a database example, but these relationships are everywhere in companies.

So I am your customer because you are implementing a service for me. If I were a real customer then I could make sure you would do what I need because I wouldn't pay you otherwise. If you didn't want to provide me a service then I would go elsewhere.

Yet in this scenario I am a Sham Customer. I am forced to go through you and I am nothing but a burden you wish would go away. If you don't want to make my tables what can I do? Whine to management. Try persuasive argumentation. Plead. Get pissed. I'll often cycle through all these strategies hoping something sticks. Most often nothing sticks and I am stuck.

any other code I write.

I firmly believe in constantly improving my code and I see a difference in quality because of that. Seemingly this would put me firmly in the refactoring camp. But it doesn't. Why? Refactoring says you can't break interfaces. That puts me in an awkward position.

According to the rules of refactor I have to not touch over 50% of my code when that 50% needs just as much improvement as any of the other code I write. But I want to keep improving my system. That often means removing classes because I have figures out a better way. I may need to change the signature of a method because a I have figured out a better way. I may need to drop a method because I have figured out a better way. I may need to make a lot of changes of all types to keep improving the system.

Because I want to constantly improve all my code I can't refactor. I am more extreme than that. If I see an improvement I make it. I don't care if it breaks interfaces or not.

I hear we shouldn't be afraid of changing software, isn't the whole idea of story based iterative development that evolves software? So why shouldn't I break interfaces too?

No reason. Unit tests keep you safe from interface changes too. Certainly an interface change may break a particular layer of software. But interface changes should just impact that one layer. All your tests above and below that layer should remain stable. If they don't then that's probably a smell that should be taken care of anyway.

I've seen more time wasted because of the fear of change code than I can shake a bit at. Interfaces aren't in any more sacred a position than the code behind the interfaces.

So bee extreme. Don't refactor. Improve all of your system all of the time.

on Jeffries wrote on scrumdevelopment@yahoogroups.com :

I suppose most boys want to be Superman or Spiderman when they are little. Fact is, we'll have more impact on the world by working with people than by working alone. The sooner the young genius figures that out, the better.

Another way to understand the super hero myth is that each of us is capable of being great, but the normal pressures of tribal life encourage us to hide our capabilities so we won't stand out.

Standing out is a sure way to get hammered down. So the super hero who mythos is a reminder to us all the we are capable of reaching down, overcoming obstacles, and doing great things.

That the super hero image has become iconic speaks to how much we need to balance the message of conformity we get on a daily basis.

Personally I want Einstein thinking the big thoughts. And I want 1000s of teams working all around the world on cancer too. We can have the lone genius, we can have the Manhattan project, we can have the Justice League, we can have the team member that does great things but may not fit in 100%. On a system wide basis the power of progress is a function of the interplay of them all.

The simple rule that teams are most important is too simple.

In a group I enjoy spending the most time on that chaotic border of individual and team creativity. I hate being in a group where people are passive and wait to take direction. That's what happens when people forget they too can be superheros.

I also hate being on team where everyone thinks everyone else is an idiot. That's what happens when people are like Lex Luthor and need to control everything to feel safe.

Fortunately we don't need go to either extreme.

I think Joel has an interesting take on this subject in his High Notes article at http://joelonsoftware.com/articles/HighNotes.html .

Companies are a hive of different development activities. Occassionally you hear about a project and you think hey, we could use that. You contact the developers and they say sure, no problem, but talk to our manager first. It's with the manager where you splat against a stone wall. And the wall has graffitied on it "we can't support that." Or "you can fork the code but it's yours after that."

The manager is just playing it safe. Nobody is paying them to make a general software package everyone can use. That would take more people, more time, and divert them away from the project that they really got funded for and against which they will be judged a success or failure.

But wait, the company is paying for development, is it too much to ask for people to work together so we can leverage syngeristic and other manegerial aphorisms? Apparently it is. Locally maximizing a group's outcome does not lead to global company maximization.

The manager is not being irrational. The manager wants to minimize risk and telling people to buzz off is an easy way to minimize risk.

Does it really take so much effort to internally support software? And if it does, doesn't that indicate there's a problem? The easiest way around a problem is to deny, deny, deny. But what's the big deal? What can go wrong?

Somone Has a Question

Yes, lots of questions can "waste" time locally. There are ways around this however.
* Create a community. Try an email list so everyone using a program can help with technical support. Create a wiki so common topics can be covered.
* Document all answers to questions. Everytime a question is answered try to improve the system so it doesn't need to be asked. If it is asked then people can just say RTFM and here's the link. That takes about 30 seconds.

Someone Has a Bug

How do you configure your system? I've worked on configuration many many times, but I've never done it right. I have come closer this time, but it's not perfect yet.

Ok, so what?

Here's what you'll get if you open up the prize hidden inside:

1. You get a highly configurable system where every stakeholder in the system gets a chance to configure the system in a rational well defined order using a powerful general mechanism. You can configure the same thing from the command, line, from the environment, from a configuration file, or over a command port at runtime.

2. You have an automatic way to provide an interactive interface to every library in an application over multiple interfaces (telnet, http, command line, etc). This means you can configure anything in an application at runtime and you can access all of an application "debug" interfaces at runtime.

3. You get a system is describable at runtime by applications in their own code. I like this better than building meta systems from configuration files. It's simple enough that programmers might just consider doing it because the benefits are many and the costs are few. The tricks are mostly in how everything fits and works together, not in the amount of programmer work that is needed.

Just what is configuration? Anything in a program that can be set or got.

Configuration is all the really sexy stuff in programming like setting log levels for different parts of an application, setting port numbers, setting host names, setting max queue sizes, max error thresholds, turning on and off features, injecting faults, getting application metrics, running tests, invoking functions, and getting values from different layers of a program so you can tell what is going on.

This is an interesting discussion you can find at: http://www.theserverside.com/news/thread.tss?thread_id=38287. It's an issue that comes up all the time inside a project and is ultimately the primary driver for project success or failure.

The key point of why time-boxes are more effective is:

To make a project successful you must change how every decision in a project is made. That's what time-boxes do. Time-boxed releases using a short time horizon force every decision made in a project. A time-box is like a black whole sucking in everything that gets close. For every decision branch a shorter implementation strategy selection is forced which results in features getting implemented in less time.

When implementing anything you always have a choice of how to implement it. You can choose a way that gets the job in done with high quality, but takes a shorter period of time. Or you can choose a way, that you may prefer, but will take a longer period of time. This pattern applies fractally to the highest architectural decision to the lowest down and dirty programming implementation detail.

Software is the result of thousands of decisions that trade time for other qualities like elegance, quality, power, generality, familiarity, and excitement.

With a short time-box you are forced to make every decision with the time-box in mind. If you make 100 decisions everyday and everyone one of them results in a shorter implementaiton time, you can see how over a succession of decisions you will implement more stories. Now multiply this same process by everyone in your project. The chances are better your project will meet its goals.

This posting (http://discuss.joelonsoftware.com/default.asp?joel.3.301232.15) on Joel On Software discussion group poses an interesting design question.

The first approach:
class Car
{
public void Wash() { /*code to wash the car*/}
}

Car car;
car.Wash();

The second approach:
class Car : public IWashable
{
//private members. This class just acts as a business object that is manipulated by "verb" objects. This maps directly to a db entity.
//no methods. just props and constructors and a dispose pattern
}

class Washer
{
public void Wash(IWashable ptr) {/*code to wash thingie*/
}

IWashable myCar = new Car();
Washer w = new Washer();
w.Wash(myCar);

From a C++ perspective Washer is nice because you don't need to change Car to add washing capabilities. This is solid pragmatic reason, though I think a more philisophical discussion was desired.

Given all the objects that would be needed to wash a car, what makes Car the right place to know how car washing works? Should a car know about how to find the nearest and cheapest car wash, for example? To answer this question you have to somehow be able to decide which objects are natural for Car to know about.

In this case, Washer is really a relation between all objects involved in washing a car. Washer is an algorithm abstraction represented by a class. This could easily be a function in a functional language, though in my mind classes and functions are pretty much the same thing.

There's an unexpected and often fatal type of error that happens when you add resources to a horizontally scaled architecture. When the new resource comes online all traffic can be immediately redirected to the new resource, because it has the least load, and it just folds up and dies. You are left wondering WTF (what the f*ck) and it is really hard to track to down and even harder to fix.

The idea behind a scaling out horizontally is that you can add new resources to handle load. This sounds great and it works, but it has some subtle and surprising error conditions that you may want to keep in mind.

Imagine you have a load balancing appliance using the least load metric. Now you add same some new slave MySql servers to handle the load. Your load balancer will redirect traffic to the new slaves, but the slaves are trying to sync, yet they can't sink because they are getting hammered by the new traffic. Deadlock.

Imagine you have storage network that is full to the gills with data. You add a new appliance to give yourself more storage. Now what happens? All the new data goes to the new appliance. That means all users are hitting the same appliance for their data. Your performance slows to a crawl because the appliance can't handle that. You sort of expected parallel IO among all your appliances to handle the load. Now you say WTF?

A related idea is the dark side of partitioning. You partition data to get high performance via parallelization. For example, you hash on the user name to a cluster dedicated to handle those users. Unless your system is very flexible you can't scale anymore by adding resources because you can't repartition the data. All users are handled by their cluster. If you want a different organization you would have to redistribute data across all the clusters. Most systems can't handle that and you end not being able to scale out as easily as you wished.

This is much shorter version of a long argument I have been having with myself about the role of software development as an abstract expressive art form. We developers are often seen as just little code robots. But software can be a lot more than that and if it was allowed to more I think we could create a lot more truly great software instead of heaping more dumptrucks full of uninspired trash in the bitbin.

There's an interesting parallel between the western musical tradition and the agile tradition.

If you would have asked Bach the purpose of music he would have said music serves to glorify god. His perspective was a carry over from the middle ages when music was primarily spiritual in purpose.

With the enlightenment a more secular view of music developed. We see, for example, in Mozart the classical emphasis on the individual which lead to clean, tuneful, and entertaining music.

The classical ideal soon evolved into the romantic ideal which valued individual self expression. In Beethoven and Liszt and in 20th century composers we see self expression as the primary driver behind their music.

Bach was undeniable creative, but his music was not intended as a vehicle of self expression. The polyphony of the fugue in many ways resembles a software project. The complex and intricate dance of the many fugue voices is held together by a strong underlying structure. Bach perfected "counterpoint", the "combination of independent melodies to form a harmonically and rhythmically vital union."

In fact, the word "baroque", Bach's musical style, originally means "irregularly shaped pearl", which I take to be in spirit with Wabi Sabi, an idea that has inspired many in the agile world.

Someone reading my C++ coding standard recommendation for using doxygen to automatically generate documentation from source code, asked a great question:

I've often considered using doxygen, I always ask myself - is this
really useful? Would I use it if I were new to a project? Would
programmers working on the project use it?

I'll rephrase their question to more conveniently express a point I've thought a lot about:

Why do companies put so little effort into automating their own development
process to make development easier?

It's like the hair stylist whose own hair looks like someone cut it using a late night infomercial vacuum cleaner attachment. Or it's like the interior decorator whose own house looks like a monk's cell.

Software organizations rarely build software to make developing software easier. Why is that?

There are three ways changes are made in an organization: