It's an interesting question that comes up from time to time. I don't really know of course. I feel strongly that the next evolution has to move humans out of the programming loop. Anything else is just really the same thing different day. OOP, functional, logic, AOP, relational, etc is all really the same stuff. It is the human that is the programmer, it doesn't really matter what we use as materials. An artist creates art using paint, marble, fabric, glass, twigs. Programmers create programs. The results are different using different materials, but it is still fundamentally the same creation.

From a process perspective some combination of SCRUM/XP is probably as productive as human programmers can be. That will be our funadmental limit of production as programmers.

We can use model driven architecture to systemetize the production of code we already know how to build. That gives us another multiple of productivity.

Human political/social infrastructure is the true limit of productivity. We will always be limitted by our institutions, which tends to be the most regressive part of any system.

Then we are limitted by basic skills. Not everyone can be a good programmer. We could scale the system by making more programmers, but that doesn't really work as you have be good to do good work. And doing good work requires keeping small highly interactive teams.

The small incremental wins we make make in software development with improved languages, processes, tools, IDEs, skills, etc are good, but they fundamentally limit how much software can be produced to a small unimpressive are under the curve.

Nothing original in all of this meandering, but it is the reason I don't get very excited on debates about what's next afterr OOP, or if functional is better than OO, or if dynamic is better than static. It's all close enough to the same thing that it doesn't matter much.

Having been crunched many times, i found this an interesting take on why crunch mode is counter productive (http://www.igda.org/articles/erobinson_crunch.php).

A crunch mode that extends into a death march is as bad as it is common. It uses up and spits out people while not providing a good product. It's lose lose lose.

Crunch mode is not always bad, it can be the best experience ever for a group and a product. For me crunch mode is just another word for having a clear focus and a common fixed set of priorities. Crunch mode is a prism for project focus. You can use a prism to scatter focus or bring the scatter back into one coherent laser beam of energy.

A group of people working with a strong focus and the same priorities is an incredible force. Great things can happen in crunch mode. Most of the time we work with splattered focus and we get very little done.

A good example is adding a complex feature across different groups. In normal mode each group will have to schedule their part of the feature. The schedules will never match up so the feature is pushed out or pushed in so everyone knows it won't happen according to plan. And of course everyone is working on eleventy-seven other features at the same time plus a continual stream of critical bug fixes. Plus you have status meetings, meeting meetings, and a 1000 other distractions. Most of the time you just give up on the feature saying its too risky or you have to scale it back into nothingness. Focus has been split into a thousand different colours.

Project Greenlight is a program that covers directors pitching their films for funding and then following the film getting made. I got a serious sense of deja vu from watching the show. It was so like many software projects i have worked on. There was a lot of uncertainty about which project to fund. Nobody was quite sure which movie to pick. The guy who won was the guy who talked the best. A lot like software. The director who finally got funding was a little unsure now about how to deliver the film. A lot like software. The day after the project was greenlighted they went throught the script and decided the movie could only be done for about triple the original budget. A lot like software.

Now here's where the parallels to software were amazing. It's in the horse trading about how to bring the project back under budget. The industries may seem very different but the process was oh so familiar. Can you do without that feature? Can you change this or that? I know the movie is set in Chicago in 1976 but can we move it to LA and set it in the present?

An executive on a phone conference played the we-want-what-you-want-as-long-as-it's-under-budget card. The execs would say that they would love to have X in the movie as long as they could fit it in the budget. Classic. Just classic. That's reality though. Someone is putting up a lot of money. They have a right to expect certain things, especially if you told them what they should expect. Unfortunately a lot things can't be known until you actually try to do them. That doesn't make the precision planning people very happy.

Watching Project Greenlight the parallels between movie making in the studio system and corporate software development keep popping out. For those of you not watching Project Greenlight, it chronicles the making of a movie from
script selection through a contest to picking the team to the making of
the movie.

The parallels between the studio system for making movies and the
"studio" system for producing software are amazing. The director is
saying stuff like "how can we know the shot until we see the scene?" The
director of photography is pissed about every change. Team work and
personal issues come to dominate the whole process. All the actors want
to be told exactly what to do and when to do it. You have to make your
daily shots or the world ends.

And a week or so later you look at a rough cut of what you shot and then you are hosed when you don't like it. All the management types are running around having angst-filled meetings about how everything is off-track and unplanned and the budget shot and they'll never make the schedule. All along the director never really got to do things the way he wanted and even when they get to the shooting of the movie the whole infrastructure and momentum of the movie is just about getting the shot done and moving on to the next shot.

The measure of success is not making any changes to what someone decided ages ago in pre-production so the daily schedule can be made. The director wants to explore a little and gets nothing but grief for his efforts. What the actors and the crew seem to admire is a director that knows exactly what they want to do all the time and can tell exactly what to do.

I never realized before how similar software was to movie making and how
the role of director is much like a software developer. Only in software

How do you layout your office space to optimize software development? It's a question I don't think has been seriously considered at very many places I have worked. Mostly it's just cubes farms of one variety or another. Certainly there are hybrid varieties, but it comes down to cubes most of the time.
I had the opportunity to seriously consider and create my ideal office layout for a software development team. I read lots of different papers and talked to lots of different people. Here's what I came up with. It's a subject without a objectively correct answer, so there is plenty of room for disagreement, but it may prove interesting in your research.
This is a bullet list of my recommendations.

Organize software developers in a war room that is dedicated to the software group.
Separate phone heavy groups like marketing and admin from developers. All "distractions" in an area should be project related.
Create offices and conference rooms for privacy and larger meetings.
Make space for those people who represent the customer to the team.
Have the hardware group in the next room.
Arrange desktops so people are not looking directly at each other.
Pay close attention to is the traffic pattern. Do not arrange desktops around the edge of the room, by bathrooms, by the kitchen, by noisy groups, etc.
The initial idea is to use wireless development machines so people can move around easily if they wish.
The desk layout should leave enough room to support pair programming; it should have lots of horizontal space for documents, monitors, and books.
Have the QA group in the next room or the same room depending on the size of the team.
Keep the team to 12 or fewer people if possible.
Keep the hardware being developed on in the same room as the developers.
Purchase good headphones for engineers.
Cell phones need to be on vibrator mode.
Phone calls must to be handled in one of the private areas, not in the war room. No speaker phones.

On the Yahoo Scrum email list there was an interesting discussion on how Scrum and XP should be combined. There's a lot of synergy between the two agile methodologies. I advocate using Scrum for the project methodology while adopting many of the XP programming practices for the development team. Many people seem to agree. But how do you do that?

One person said you should adopt XP and Scrum separately, then combine them. Experience each on their own so you'll know best best how to combine them. I didn't think much about the suggestion and at first I thought this was a sensible approach. Then there was some discussion about if it was really necessary to practice them separately first?

Then the analogy was used to say you don't use more than one detergent in your laundry at the same time, do you? So don't mix your methodologies either. And again, without thinking much about it, I thought yes, using more than two soaps in a load of laundry would be silly.

So far it doesn't sound like I do much thinking...and sometimes that's true, but on more reflection I realized I do use more than one soap when I do the laundry. In fact, I use a whole bunch of products in the laundry depending on the effect I want to get. I realized I do the same with methodologies as well.

Sorting: Laundry Metaphor Number One

Sorting is the most critical part of doing laundry. If you don't properly sort then your whites get dingy, colors fade, stains get set, fabrics fray, and bulky items don't get clean. By the magik of metaphorical extension I'll say sorting is also the most important step for selecting methodologies. Different projects are different. They have different people, different political environment, different customers, different funding, different goals, and so on. Applying the same approach to every project shouldn't work. Use each project as a chance to take a fresh look and sort things out.

Use Products for Effect: Laundry Metaphor Number Two

I wonder if internal prediction markets (http://commerce.net/publications/CN-TR-05-02.pdf) could be used on a project to more accurately determine project milestone and release dates? There are markets for other events, like who will win the presidental election, it seems project dates could be better predicted by some sort of market gestalt derived from everyone involved on a project.

We know large project deadlines are pretty much a joke. How do you get more realistic numbers? Can you imagine Microsoft setting up an internal stock market in which people can buy and sell shares on Longhorn milestone and release dates? I bet the result would be pretty accurate. Around release times I imagine a flury of buying and selling as people make their bets on what will happen.

How would feature reduction/time boxing fit into the market? People would have to make a bet on if they thought the schedule dates mattered more than features or if features mattered more than dates. Perhaps the internal calculations of all the people would predict what management would do. But the market wouldn't predict what management should do, it would predict what people thought management would do, so using market information to set policy might not be a good idea. That would be the ideal though, figuring out a way to get group input on the most likely release dates.

I think it would be interesting anyway. Usually all the scheduling negotiations happen elsewhere and developers don't get much input. A market mechanism would give everyone some input and a way to make a very complex aggregate calculation.

How about this for a definition of algorithm:
A set of rules that protects the programmer from too much information.

When presented with too much information we have a hard time making decisions. That's a problem we face in programming all the time. Solution spaces are almost always infinite. We can do anything and that's the problem. Algorithms and their relative, patterns, reduce the solution space to something where better decisions can be made.

It turns out the more choices we have the more indecisive we become. More choices causes the accumulation of opportunity costs and fill us with regret from the choices we did not make. Having more data doesn't mean we'll make better decisions either. We'll usually do worse because the more data we have means the more alternatives we can think of which means the more doubt we have which means our chances of picking the right solution is small.

Asking people to make decisions between many options when their knowledge is shallow means their chances of picking the correct options are no better than chance. With experience you learn the crucial minimal set of facts you need to make the best decisions. Until you get that experience your chances of success are small.

That's why books of algorithms and patterns are so valuable. If your knowledge is shallow, which most of ours is in new domains, then you need help in minimizing the number of good solution alternatives.

When people say they don't like patterns I always cringe a little. If you are an expert then patterns aren't necessary because you will naturally use them as appropriate in crafting solutions. But if you are not you are forcing people to endure a blank canvas with the instruction to just paint something. Talk about brain lock.

How should you build distributed sytems: API or messages? This post by Eric Armstrong says messages -- http://www.artima.com/forums/flat.jsp?forum=106&;thread=120669 -- and I couldn't agree more.

APIs create a tight a binding between the protocol functionality and your code. If they use a certain libary then so do you. If their API conflicts with your code then you are out of luck. If their API corrupts the heap then you are out of luck. If their API can't handle interrupts, semaphores, queuing, memory contraints, priority, or a host of other issues properly then you are out of luck. Your code can't evolve independently from the API which makes your system less functional, more dependent, and more brittle than when a protocol is used. That's too high a price to pay when your butt is on the line to get something working and to keep it working.

Another key advantage of using a protocol is that it can be implemented in any language. You don't have to wait for a vendor to create a language binding for your language version and operating version. This can take forever. If you are on an unconventional OS like VxWorks, you'll probably never see an API. You don't have to wait for bug fixes either. And you don't have to worry about how to include their code in your build system.

When using a a protocol as long you can stuff rightly formatted bits down the TCP, UDP or whatever socket then you are in. For example, I was able to quickly make a sweet load test system in Perl for a set-top box network because a protocol was used and it was easy to compose bits together in Perl. If I had to wait for a Perl binding to the API I would still be waiting and the load testing system may never have been attempted.

I think it is. I discover a way to get from A to B using code, much like I discover a route from point A to B using roads. You could say the roads already exist, so it's not a discovery to figure out a route. But our notion of discovery is really to make plain that which already exists. Columbus discovered America and it already existed. DNA already existed. Gravity already existed. Many mathematicians are at bottom Platonists. Invention is something much more rare. We rarely invent anything in software, while we constantly make discoveries. That's why software is so fun!

Out of my hundreds and hundreds of thousands of lines of code how many real inventions have I made? I've been clever. I've been effective. But I don't know if I've ever invented anything, yet virtually all of my brain farts are patentable by current standards.

So I stand along side Donald Knuth (http://www.groklaw.net/article.php?story=20050724140820490) who sayeth:

My personal opinion is that algorithms are like mathematics, i.e. inherently non-patentable. It worries me that most patents are about simple ideas that I would expect my students to develop them as part of their homework. Sometimes there are exceptions, e.g. something as refined as the inner point method of linear programming, where one can really talk about a significant discovery. Yet for me that is still mathematics.
I come from a mathematical culture where we don't charge money from people who use our theorems. There is the notion that mathematics is discovered rather than invented.

"In rites at large, it is always better to be too simple rather than too lavish.
In funeral rites, it is more important to have the real sentiment of sorrow
than minute attention to observances."

-- Confucius in the Analects

I was struck recently by the similarity of the role of ritual in Confucian philosophy and the role of ritual in Agile projects. The system of thought put forth by Confucius tries to create a world where people can live together in harmony without resorting to uncivilized behaviour. You think I am stretching here? We'll see...

Confucius was motivated by his times. He lived in what was called the Warring States Period. Battling warlords made life difficult for your average person. For simplicity, this era will play the waterfall methodology "big process up front" role in my analogy :-)

Deeply concerned about creating a better world in which to live, Confucius proposed ritual as a key civilizing influence. So does Agile.

What you say? We are programmers, we don't need no stinkin' rituals! Hold on now. Don't get your editor in undo mode.

Let's consider just what ritual is: ritual can be thought of as respect for the the deepest sense of proper way of doing things.

Isn't that a fine Agile principle?

On a project you could could consider ritual the pattern of disciplined behavior which governs each moment in the life of a project.

Now does that sound bad?

Without the bonding of ritual there isn't order. Why? Because people learn proper relationships through ritual. Ritual promotes harmony by showing everyone how they should act and interact without using a heavy hand. He who governs best governs least. Using ritual everything just sort of happens and fits together. In a world where everyone is looking to kill each other you can see how you might want ritual to coordinate social situations. Modern software projects aren't so different at their base.

Ritual is all around us.

One saying i really like is: Only Say Things That Can be Heard

What this means is that everyone and every organization is in a different place in their lives. If you show up at a company and tell them everything they do stinks then you won't be heard. You will just make enemies and nothing will get done as everyone sharpens their fangs for the next fight. You get the same result when "talking" with your family too :-) Somehow you must find a way to talk to someone in a way that makes sense to them.

I must admit this is a hard bit of wisdom for me to take out in the real world. I tend to be too direct at times. I remember one particular meeting where I realized just how destructive the direct only the facts matter approach can be. In this meeting from hell this one particular ubergeek was coming hard and fast. He was questioning everything. He was disagreeing with everything. He was saying everything I wanted to do wasn't going to work because it didn't work when he did it at company X. He wasn't giving an inch on anything. And he was doing it all with that infuriating "I am just trying to understand" attitude. In short, it was like looking at me at my worst.

Now I don't much like looking in the mirror anyway, but when that's what I see staring back I want to banish all mirrors, just like they did in the movie The Skeleton Key.

At that time I made a solid pledge to change may ways and try to Only Say Things That Can be Heard. Am I 100% successful? Unfortunately, no. But I am much better now. Through my career I have consistently witnessed this same movie replayed, only with different actors and a new sound track. Software is only partly technical. Building software is mostly social.

Believe it or not, there's an agile style to doing the laundry and BDUF style to doing the laundry.

My beautiful wife, Linda, does laundry BDUF style. The laundry piles up and until it just has to be done. Then she sorts. Every load is pre-sorted into its own pile before it can be washed. If you look at our laundry room it looks like a field of hay stacks ready to be bailed. The piles are formed by some set of rules that I have never been able to master, after 20 years of trying. When I try to do the laundry her way I can never quite get it right. So everything is well organized. We have lots of nice piles optimized into the correct size for our washer. There are no left over clothes that don't get washed. We don't have small loads, which I am told are a waste, even with our water efficient washer.

What could go wrong?

There are lots of piles. So many piles that they can't possibly get done in one night or even one day. Now let's add to that the remarkable ability to be distracted and very few piles actually ever get washed. So the piles just hang around, sometimes for days.

What is happening while the piles go unwashed? We use and dirty more clothes. Part of the problem is we may have too many clothes, but it doesn't seem like it. Anyway, with new clothes being added all the time the piles no longer make sense. If we just added the clothes to existing piles then they wouldn't fit in the washer anymore.

The piles need to be rebalanced. Wouldn't you just split the large piles in half? No. That would be wasteful. We want the washer to be maximally full because that is the most efficient way. Just splitting the piles would leave us with too many small piles. To make larger better piles we have to rethink what clothes can be washed together. Maybe the jeans and the towels can be washed together after all. Like I said, I've never been able to figure out the rules, let alone when the rules can be ignored.

Projects are full of Sham Customers. If I use something you are building on a project we'll all subscribe to the collective delusion that I am your "customer." But am I really a customer? Am I a real customer?

No I am not. Why?

Because I am only a real customer when I am paying you to do something and you really want my money. Otherwise you have no incentive to do what I want. But wait, why is money even part of the equation? I am the customer, doesn't that mean you'll do what I want as long as it makes sense and is reasonable?

Oh no.

If you aren't dependent on me in some direct way--not some metaphysical way like we are all in this together, or we are all working for the same company, we are all working to the same purpose--then you can tell me to get stuffed and there's nothing I can do about it. Nothing!

Let's say I need a a couple of tables in a database. You are the database group and I have to go through you for database stuff. For organizational reasons I just can't make my own database. I am using a database example, but these relationships are everywhere in companies.

So I am your customer because you are implementing a service for me. If I were a real customer then I could make sure you would do what I need because I wouldn't pay you otherwise. If you didn't want to provide me a service then I would go elsewhere.

Yet in this scenario I am a Sham Customer. I am forced to go through you and I am nothing but a burden you wish would go away. If you don't want to make my tables what can I do? Whine to management. Try persuasive argumentation. Plead. Get pissed. I'll often cycle through all these strategies hoping something sticks. Most often nothing sticks and I am stuck.

any other code I write.

I firmly believe in constantly improving my code and I see a difference in quality because of that. Seemingly this would put me firmly in the refactoring camp. But it doesn't. Why? Refactoring says you can't break interfaces. That puts me in an awkward position.

According to the rules of refactor I have to not touch over 50% of my code when that 50% needs just as much improvement as any of the other code I write. But I want to keep improving my system. That often means removing classes because I have figures out a better way. I may need to change the signature of a method because a I have figured out a better way. I may need to drop a method because I have figured out a better way. I may need to make a lot of changes of all types to keep improving the system.

Because I want to constantly improve all my code I can't refactor. I am more extreme than that. If I see an improvement I make it. I don't care if it breaks interfaces or not.

I hear we shouldn't be afraid of changing software, isn't the whole idea of story based iterative development that evolves software? So why shouldn't I break interfaces too?

No reason. Unit tests keep you safe from interface changes too. Certainly an interface change may break a particular layer of software. But interface changes should just impact that one layer. All your tests above and below that layer should remain stable. If they don't then that's probably a smell that should be taken care of anyway.

I've seen more time wasted because of the fear of change code than I can shake a bit at. Interfaces aren't in any more sacred a position than the code behind the interfaces.

So bee extreme. Don't refactor. Improve all of your system all of the time.