A thread on The Server Side (http://www.theserverside.com/news/thread.tss?thread_id=29012)
turned to talking about a topic of special interest to me, namely application architectures
for high performance high load situations.

Here are some thoughts on the subject of
architecture: http://www.possibility.com/epowiki/Wiki.jsp?page=AppBackplane.

They are the result of years of conversations with some very smart people
working in one of the most difficult environments possible, a Class 5 core telecom
switch.

Comming to the java application server world of servlets, hibernate, struts, spring,
etc, i was confused at first by how these frameworks dictated the threading
architecture of applications by using ThreadLocal and a single threaded approach
for all requests.

I am curious if people are inerested in other approaches to application architectures?

Anyway...

> From the thread:

>don't you break with the common one thread per request
> scenario that us developers have come to depend on?

It needs to be broken. These frameworks force
an application architecture. Your application architecture
shouldn't be determined by a servlet or a database or
anything but the needs of your application.

Sure, a single threaded approach may work fine for
a stateless web back end.

But what if you are doing a real application on the
backend like handling air traffic control or a
manufacturing process?

In these cases a single threaded approach makes
no sense because a web page is just one of a thousand
different events an application will be handling.

All events are not created equal. Threads, queues,
priorities, CPU limits, batching, etc are all tools
you can use to handle it all.

It took me a while to figure out why i was having problems
with certain frameworks. It is because they hard code a
threading architecture into your apps.

If i want an object to participate in transactions from

A recent thread in comp.object has helped
me realize what has bugged me about extreme programming
is not actually XP itself. In my mind i kept thinking
XP should be about building systems. It isn't. XP is actually
about just what it says: programming.

XP addresses the programming part of any project and
that's it.

I am largely in agreement with the primary XP practices.
Some of the secondary practices, like using a separate
integration machine, are just, well, kind of silly.

But most of the other XP practices are sound. And I won't
say they are all just stuff i already did. That's not
true. I have learned a lot from XP.

Yet XP doesn't address developing a system and it
never said it did. But that's always the context
in which i evaluated XP and always found it wanting.

A major part of systems work are things like creating a
products requirements definition (PRD); complex hardware and
software codependencies; stringent high availability
requirements; stringent performance requirements; stringent
interop requirements; specifying hardware, much of which
has to be built; being compliant with many complex standards;
buy or build decions; predicting staffing, budgets, and
costs; and so on.

None of this is programming. It certainly impacts programming.
And you'll only find out some of it when you start programming. But
much of it must happen before any programming happens
because it is the kind of information that needs to be fed into the
planning game.

This is why the customer role in XP is by far the hardest
role. Much harder than programming because the system has
largely been figured out by the time the programmers
see it.

And figuring out the system is the semi mystical act of creation
that seems to defy systematization. Techniques like JAD seem inadequate,
but they are probably the best you can do.

Just-In-Time-Requirements are good for many things, but when you

It is recommended to always "release as soon as it is possible"
where this is taken to mean early and often.

My reply is to release ASAP, but no sooner.

To be my usual tiresome self i would like to interject a little "it
depends" here.

Define an ideal and come up with a rubric for pattern variation.

These absolute rules always frustrate me because it does depend.

The ideal is release as often and soon as possible.

What that means in each context is something different.

For example, in one of my favorite projects i was working on a large
internal web site that had over 100 simultaneous active heavy users. It used
perl and CGI so i made live changes continually. There was never
a real release of anything. This worked 99% of the time and it
was exciting.

Training was an issue and i tried not to break features, but that's
not always possible or even desirable. You can't make stuff
better if you can't break it.

On another project each release cost millions of dollars because an
entire nationwide network had to be upgraded and we could cut all
data traffic in large regions of north America. This customer treated
each release like a nuclear attack so releases were infrequent to
say the least. Yet other customers in a similar situation
didn't care and wanted releases much faster.

On another project the software was more your traditional enterprise
software that was installed using installshield or whatever. Typically
everyone was always very busy so releases were more of
an annoyance to them.

> Comments lie. Code doesn't.

This sentiment is used as justification for having very few if any comments in your code. I just don't buy it for many reasons.

Code lies like a dog under a shade tree in summer. Code lies because the variable names, class names, and method names don't match what the code does. People will use lame names or they will insert new code into a method or new methods into a class that change the nature of the thing. That is a lie in my book. And it happens all the time.

You may say use good names and you won't have a problem. I agree to a large extent. But i can't make people program well no more than i can make people comment well. If you can accept that people must use good names then you can also accept that people must make good comments. Trust cuts both ways.

And the lie continues because a name is flat. It relates only to one aspect of a thing. Things are multidemensional and can't be mapped meaningfully to a single name in all its contexts. To the government i am social security number. To my dog i am a pat and a meal. To those i disagree with i am an idiot. To my doctor i am a series of stats. What is my name?

I have been mislead by comments, but have been helped far more than i have been misled, so that's a win in my book.

I have yet to see any of this code that is self-documenting so i am unwilling to do away with comments on that assumption.

XP assumes a continuous chain of oral tradition to make the use of comments less necessary. Perhaps on an XP project this make sense. But much of my experience is in large distributed teams with lots of churn so i don't think this is a generally applicable rule to do away with comments. No more than i would get rid of jails everywhere just because there is almost no crime in my house.

article titled Roads Gone Wild by Tom McNichol that reminds
me a lot of the spirit of agile software development.

The article is about a new kind of traffic engineering
advocated by Holland's Hans Monderman. And by traffic
engineering we are talking about roads, sidewalks,
interestions, etc, not TCP/IP.

The article lead in starts:
No street signs. No crosswalks. No accidents. Surprise:
Making driving seem more dangerous could make it safer.

Another graphic has the title:
How to Build a Better Intersection: Chaos = Cooperation

Step 1: Remove Signs - The architecture of the road, not signs and
signals dictates traffic flow.

Step 2: Install Art - The height of the fountain indicates how
congested the interstate is.

Step 3: Share the Spotlight - Lights illuminate not only the roadbed,
but also the pedestrian areas.

Step 4: Do it in the Road - Cafes extend to the edge of the street,
further emphasizing the idea of shared space.

Step 5: See Eye to Eye - Right-of-way is negotiated by human interaction
rather than commonly ignored signs.

Step 6: Elimanate Curbs Instead of a raised curb, sidewalks are denoted
by texture and color.

Some interesting quotes:
* Hans Monderman is a traffic engineer who hates traffic signs. ...
To him, they are an admission of failure, a sign - literally -
that a road designer somewhere hasn't done his job. The trouble
with traffic engineers is that when there's a problem with a
road, they always try to add something. To my mind it's much better
to remove things.
* Monderman ripped out all the traditional instruments used by traffic
engineers to influence driver behaviour - traffic lights, road markings,
and some pedestrian crossings - and in their place created a traffic
circle. The circle is remarkable for what it doesn't contain: signs
or signlas telling drivers how fast to go, or curbs separating the street and

In C++ the friend keyword makes writing unit test code easy and clean.

The question is how do you keep your test code separate from your "real" code while having a minimal public interface and allowing seperate test classes access to the internals of the classes being tested?

I want code separation so my code is clean and the final image doesn't include test code. As the test code is usually larger than the code being tested this is important. Please, no separate compilation using macros.

I believe in testing everything that can break so i don't just test public interfaces. Public interfaces often use a common private interface that i want to be able to test directly so it doesn't have to be retested for each public interface. This requires another class to have non-public access to the innards of another class.

In C++ this is what friend does for you, rather cleanly. Test classes can be put in another package. And with a forward declaration and the friend keyword, a class can be put under test by any number of other test classes.

class TestClass;
class ClassTested
{
private:
friend TestClass;
};

The implementation source file would include the path to the full TestClass.

I allow all the code in a package to touch the privates of other classes in the same package. This makes for a minimum public display of behaviour. I assume all code in the same package goes together somehow so there's no need for a class to protect itself from code in the same package.

An interesting article in slashdot (http://it.slashdot.org/article.pl?sid=04/12/26/052212):
30,000 people have had their flights cancelled by Comair this weekend thanks to
a computer system shutdown

A couple of posters said they didn't think it could be the software or shouldn't
be the software. This post was a good example:
> Computers don't freak out or get depressed
> when work piles up. Backlogs mean nothing;
> they just keep processing one piece at a
> time until the pieces run out. I think
> someone was speaking imprecisely.

In my experience, it's just the opposite. Systems usually only seriously break when scale increases. That's why unit testing is never even close to good enough coverage. To find scale problems you need to test at scale, and few people want to pay for that. So all hell breaks loose when scale starts happening.

Increases in backlogs may make queues sizes too small which causes drops which causes retransmissions which makes the problem spiral worse. Maybe a OS network stack queue gets full, a queue which you can't control, and you are in a downward spiral.

Or the queues may not be flow protected and your memory use sky rockets which causes a cascade of failures including out-of-memory conditions that may reassert themselves even after a reboot which causes continuous failure.

Any algorithms based on size X are now way too slow for 10X which can cause scaling problems everywhere else or pathologically slow times for certain algorithms.

CPU time is sucked up which again causes push back and scaling problems everywhere else. Priorities that worked with a certain workload may now cause too much work to be done which kills responsiveness and starves other parts of the system which spirals into more problems.

There was an interesting thread on comp.object related to how XP would code up scoring for a bowling game (http://www.xprogramming.com/xpmag/acsBowling.htm). One of the thread particpants was trying to say something interesting, but didn't do a very good job at it, so I think I'll take a crack at it. They were trying to talk about Fact Based Architectures, which I think is a better up front design for the bowling game solution.

For the complete article please take a look at: http://www.possibility.com/epowiki/Wiki.jsp?page=FactBasedArchitectures

It's an interesting question that comes up from time to time. I don't really know of course. I feel strongly that the next evolution has to move humans out of the programming loop. Anything else is just really the same thing different day. OOP, functional, logic, AOP, relational, etc is all really the same stuff. It is the human that is the programmer, it doesn't really matter what we use as materials. An artist creates art using paint, marble, fabric, glass, twigs. Programmers create programs. The results are different using different materials, but it is still fundamentally the same creation.

From a process perspective some combination of SCRUM/XP is probably as productive as human programmers can be. That will be our funadmental limit of production as programmers.

We can use model driven architecture to systemetize the production of code we already know how to build. That gives us another multiple of productivity.

Human political/social infrastructure is the true limit of productivity. We will always be limitted by our institutions, which tends to be the most regressive part of any system.

Then we are limitted by basic skills. Not everyone can be a good programmer. We could scale the system by making more programmers, but that doesn't really work as you have be good to do good work. And doing good work requires keeping small highly interactive teams.

The small incremental wins we make make in software development with improved languages, processes, tools, IDEs, skills, etc are good, but they fundamentally limit how much software can be produced to a small unimpressive are under the curve.

Nothing original in all of this meandering, but it is the reason I don't get very excited on debates about what's next afterr OOP, or if functional is better than OO, or if dynamic is better than static. It's all close enough to the same thing that it doesn't matter much.

Having been crunched many times, i found this an interesting take on why crunch mode is counter productive (http://www.igda.org/articles/erobinson_crunch.php).

A crunch mode that extends into a death march is as bad as it is common. It uses up and spits out people while not providing a good product. It's lose lose lose.

Crunch mode is not always bad, it can be the best experience ever for a group and a product. For me crunch mode is just another word for having a clear focus and a common fixed set of priorities. Crunch mode is a prism for project focus. You can use a prism to scatter focus or bring the scatter back into one coherent laser beam of energy.

A group of people working with a strong focus and the same priorities is an incredible force. Great things can happen in crunch mode. Most of the time we work with splattered focus and we get very little done.

A good example is adding a complex feature across different groups. In normal mode each group will have to schedule their part of the feature. The schedules will never match up so the feature is pushed out or pushed in so everyone knows it won't happen according to plan. And of course everyone is working on eleventy-seven other features at the same time plus a continual stream of critical bug fixes. Plus you have status meetings, meeting meetings, and a 1000 other distractions. Most of the time you just give up on the feature saying its too risky or you have to scale it back into nothingness. Focus has been split into a thousand different colours.

Project Greenlight is a program that covers directors pitching their films for funding and then following the film getting made. I got a serious sense of deja vu from watching the show. It was so like many software projects i have worked on. There was a lot of uncertainty about which project to fund. Nobody was quite sure which movie to pick. The guy who won was the guy who talked the best. A lot like software. The director who finally got funding was a little unsure now about how to deliver the film. A lot like software. The day after the project was greenlighted they went throught the script and decided the movie could only be done for about triple the original budget. A lot like software.

Now here's where the parallels to software were amazing. It's in the horse trading about how to bring the project back under budget. The industries may seem very different but the process was oh so familiar. Can you do without that feature? Can you change this or that? I know the movie is set in Chicago in 1976 but can we move it to LA and set it in the present?

An executive on a phone conference played the we-want-what-you-want-as-long-as-it's-under-budget card. The execs would say that they would love to have X in the movie as long as they could fit it in the budget. Classic. Just classic. That's reality though. Someone is putting up a lot of money. They have a right to expect certain things, especially if you told them what they should expect. Unfortunately a lot things can't be known until you actually try to do them. That doesn't make the precision planning people very happy.

How do you layout your office space to optimize software development? It's a question I don't think has been seriously considered at very many places I have worked. Mostly it's just cubes farms of one variety or another. Certainly there are hybrid varieties, but it comes down to cubes most of the time.
I had the opportunity to seriously consider and create my ideal office layout for a software development team. I read lots of different papers and talked to lots of different people. Here's what I came up with. It's a subject without a objectively correct answer, so there is plenty of room for disagreement, but it may prove interesting in your research.
This is a bullet list of my recommendations.

Organize software developers in a war room that is dedicated to the software group.
Separate phone heavy groups like marketing and admin from developers. All "distractions" in an area should be project related.
Create offices and conference rooms for privacy and larger meetings.
Make space for those people who represent the customer to the team.
Have the hardware group in the next room.
Arrange desktops so people are not looking directly at each other.
Pay close attention to is the traffic pattern. Do not arrange desktops around the edge of the room, by bathrooms, by the kitchen, by noisy groups, etc.
The initial idea is to use wireless development machines so people can move around easily if they wish.
The desk layout should leave enough room to support pair programming; it should have lots of horizontal space for documents, monitors, and books.
Have the QA group in the next room or the same room depending on the size of the team.
Keep the team to 12 or fewer people if possible.
Keep the hardware being developed on in the same room as the developers.
Purchase good headphones for engineers.
Cell phones need to be on vibrator mode.
Phone calls must to be handled in one of the private areas, not in the war room. No speaker phones.

On the Yahoo Scrum email list there was an interesting discussion on how Scrum and XP should be combined. There's a lot of synergy between the two agile methodologies. I advocate using Scrum for the project methodology while adopting many of the XP programming practices for the development team. Many people seem to agree. But how do you do that?

One person said you should adopt XP and Scrum separately, then combine them. Experience each on their own so you'll know best best how to combine them. I didn't think much about the suggestion and at first I thought this was a sensible approach. Then there was some discussion about if it was really necessary to practice them separately first?

Then the analogy was used to say you don't use more than one detergent in your laundry at the same time, do you? So don't mix your methodologies either. And again, without thinking much about it, I thought yes, using more than two soaps in a load of laundry would be silly.

So far it doesn't sound like I do much thinking...and sometimes that's true, but on more reflection I realized I do use more than one soap when I do the laundry. In fact, I use a whole bunch of products in the laundry depending on the effect I want to get. I realized I do the same with methodologies as well.

Sorting: Laundry Metaphor Number One

Sorting is the most critical part of doing laundry. If you don't properly sort then your whites get dingy, colors fade, stains get set, fabrics fray, and bulky items don't get clean. By the magik of metaphorical extension I'll say sorting is also the most important step for selecting methodologies. Different projects are different. They have different people, different political environment, different customers, different funding, different goals, and so on. Applying the same approach to every project shouldn't work. Use each project as a chance to take a fresh look and sort things out.

Use Products for Effect: Laundry Metaphor Number Two

I wonder if internal prediction markets (http://commerce.net/publications/CN-TR-05-02.pdf) could be used on a project to more accurately determine project milestone and release dates? There are markets for other events, like who will win the presidental election, it seems project dates could be better predicted by some sort of market gestalt derived from everyone involved on a project.

We know large project deadlines are pretty much a joke. How do you get more realistic numbers? Can you imagine Microsoft setting up an internal stock market in which people can buy and sell shares on Longhorn milestone and release dates? I bet the result would be pretty accurate. Around release times I imagine a flury of buying and selling as people make their bets on what will happen.

How would feature reduction/time boxing fit into the market? People would have to make a bet on if they thought the schedule dates mattered more than features or if features mattered more than dates. Perhaps the internal calculations of all the people would predict what management would do. But the market wouldn't predict what management should do, it would predict what people thought management would do, so using market information to set policy might not be a good idea. That would be the ideal though, figuring out a way to get group input on the most likely release dates.

I think it would be interesting anyway. Usually all the scheduling negotiations happen elsewhere and developers don't get much input. A market mechanism would give everyone some input and a way to make a very complex aggregate calculation.

How about this for a definition of algorithm:
A set of rules that protects the programmer from too much information.

When presented with too much information we have a hard time making decisions. That's a problem we face in programming all the time. Solution spaces are almost always infinite. We can do anything and that's the problem. Algorithms and their relative, patterns, reduce the solution space to something where better decisions can be made.

It turns out the more choices we have the more indecisive we become. More choices causes the accumulation of opportunity costs and fill us with regret from the choices we did not make. Having more data doesn't mean we'll make better decisions either. We'll usually do worse because the more data we have means the more alternatives we can think of which means the more doubt we have which means our chances of picking the right solution is small.

Asking people to make decisions between many options when their knowledge is shallow means their chances of picking the correct options are no better than chance. With experience you learn the crucial minimal set of facts you need to make the best decisions. Until you get that experience your chances of success are small.

That's why books of algorithms and patterns are so valuable. If your knowledge is shallow, which most of ours is in new domains, then you need help in minimizing the number of good solution alternatives.

When people say they don't like patterns I always cringe a little. If you are an expert then patterns aren't necessary because you will naturally use them as appropriate in crafting solutions. But if you are not you are forcing people to endure a blank canvas with the instruction to just paint something. Talk about brain lock.