Ten Weeks at Pillar
Margaret Mead amongst the Agilists
First released draft: 9/18/2009.
I began work as a Pillar employee on 6/8/2009, on-site at Gale Research,
jumping into the middle of the Career Transitions ("CT") project.
The work has been fun and educational, and I hope to continue
with it, if and when CT resumes.
I also saw individual and group behaviours that intrigued me:
good and bad, intentional and unintentional, explicit and implicit.
The time spent at Gale was a mother-lode of gold nuggets for the would-be
but I do have a life-long interest
in group dynamics and anthropology... and unlike most software
developers, I also like to write.
Thus, this article.
There were many implicit (unexplained, undiscussed) behaviours
of the team that surprised and sometimes concerned me.
Some of these may be worth notice and possible discussion
To summarize, my purposes in writing this article include:
Just to emphasize... this is not a complaint or a diatribe.
Where I name names, it is only to point out what I consider
exceptionally positive behaviour.
I mean for this to be feedback in it's truest, engineering, sense.
- Point out implicit "fish-in-water" team behaviours that concerned me,
on the chance that they might be something that Pillar could improve on.
- Point out the same to Pillar folks who might be interested
in discussing them... because I might learn more that way.
- Scratch my itch, in good Agile/open-source style.
I can't believe <grin/> that no-one has written about
these things: so it might as well be me.
Just a little bit about me and where I come from, to provide
context to my observations.
III. Field Notes
- Wrote my first Fortran program in 1970 at the age of 14;
computers hooked me and I've never looked back.
- Graduated B.S. Comp Sci from
Rensselaer Polytechnic Institute
- Taught computer science 1980-1982 at Wayne State University.
Loved teaching, but the pay sucks.
- Lived through, embraced, and helped teach many paradigm changes:
structured programming, C, OO, the Web,
and now Agile/XP.
Confident enough to believe I can help others embrace new(er)
practices, humble enough to remember that my thinking will get up-ended
at each new shift.
- I am new, and not new, to Agile.
I spent three weeks at
Menlo Innovations in
Much of what I learned about XP there had the "aha" shock of
"this is what I've been fumbling towards for years!".
But I found Menlo too rigid, too cult-like, to stay.
I then tried to drag a small U.S. SaaS company
(with mostly remote Chinese developers)
into Agile practices, teaching myself simultaneously.
I made a lot of useful changes, but it wasn't pretty, and they
never really embraced TDD.
- So I explicitly chose to move on to some place that really
believed in Agile, but was not rigid.
These are literally field notes about the behaviours I saw that
were interesting, or that concerned me.
I have organized them into categories that (roughly!)
parallel published Agile methodologies.
I have tried to clearly separate out my recommendations
(in separate blocks following each section),
although I confess to
some (gasp!) interpretation of each behaviour.
IV. Closing Observations
- Pairing newbies.
During my first week at Gale, I was paired with another
new person, Laurence Rama.
This seems counter to normal pairing practice... but oddly enough,
it worked very well.
We were given cards about new features that were reasonably
transparent to anyone familiar with any java/web platform,
so we were able to add some velocity immediately.
Because we were both new, neither of us deferred to the other,
and we were able to make good use of our complementary skill-sets.
This was either brilliant planning on Matt Van Vleet's part,
or damned good luck,
as there were other possible first pairings that would not have worked
nearly as well.
When this happens, assign a specific person -- not the team manager,
not the tech lead -- to be their immediate go-to person for questions.
It's good when a newbie pair can work things out together, but some
may hesitate to ask questions when they're stuck.
(Fortunately, I have no fear of looking stupid, so I asked most
of the questions for our pair.)
- Vague pairing rules.
During my initial interview, I was told that pairing was frequent
but often optional.
During my first few weeks at Gale, almost all work was paired.
But as time went on, and as some people were slowly moved to other
projects, pairing behaviour changed a lot.
It was frequently unclear whether we should pair, with whom,
At one point, my then-pair "P" and I were working on three cards
A, B, and C: P on A, me on B, and "we" on C.
All three cards had significant built-in delays in the
test/develop cycle, which is what led to this curious arrangement.
We did examine each other's work on A and B before check-in,
to at least preserve the spirit and some of the benefits of pairing.
It was efficient, it worked, but it was weird.
Even later, when the number of developers grew very small,
I usually worked solo, but grabbed another person any time I
had a doubt, felt like I needed a sanity check, or before
This worked well (as far as I can tell!), but it
was never really discussed explicitly.
I did at times return to traditional pairing when there
was a need for deep knowledge sharing, or for me
to pick up another's work that was in-progress while
the first person went on vacation.
Be flexible about pairing, but be explicit about the flexibility.
Don't force pairing for everything.
Don't force pairing unless you really mean it.
Otherwise it lowers morale and encourages cynicism.
- Pairing difficulties.
The most fascinating thing about Agile, from an anthropologist's
point of view, is pairing behaviour.
With essentially no instruction or explicit norms, most of the
developers seemed to pair readily and effectively... even as the
styles of pairing varied widely.
There were a few difficulties that I saw on repeated occasions
that are worth noting:
- Large disparity in project-specific knowledge.
I've been on both sides of this pair.
In one case, I was new to the team, and simply overwhelmed with
the level of class dependency knowledge needed in order to work
with my (extremely thorough and knowledgeable) pair partner.
After several days, I recommended that I was not ready for
this pairing, that it would take too long for me to come
up to speed, and that I was slowing my pair down.
He agreed and was very gracious, and I moved to a different
pairing that had worked on cards with smaller scope, which
gave me a chance to absorb more about the class structure
without being overwhelmed.
In a different case, near the end of my ten weeks, I paired
with a young newcomer, recently out of college.
I specifically picked part of a card that I knew she could
handle, tried to give her a "10,000 feet down to this
point" overview, and handed her the keyboard.
Success was mixed, but she was able to be productive.
- "Introvert thinking".
There is a tendency for introverts to think silently,
to invisibly go from A to M, and then surface -- which
can leave everyone around them wondering how the heck
we just got to N?
This can be quite confusing in a pair, where frequent
conversation seems to "grease the wheels".
This is especially true if the introvert is a senior
team member, and the partner is new and more hesitant
to insist on step-by-step communication.
- High bandwidth, low communication.
The reverse of the introvert case is the pair partner who
talks almost constantly.
This "high bandwidth" can reduce actual communication,
and stir uncomfortable feelings in the other partner...
who may end up feeling responsible for managing
both sides of the conversation.
The high bandwidth partner is frequently very skilled,
knowledgeable, and productive... but the situation
can require great strength and delicacy from the other partner.
- Ambiguous speech: demonstrative pronouns and adjectives.
Words like "it", "that", and "this" in pair conversation
about classes and objects are prone to ambiguity,
especially in rapid or high-bandwidth monologues.
- Shoulds vs. Wants.
Different members of a pair naturally approach any given problem
(e.g. defining the responsibilities of a new class, or what a
reasonable set of tests would be) with different assumptions.
On multiple occasions I observed the beginnings of
conflicts about just "how far"
the responsibility of a class should go.
What I found helpful in these cases was to be explicit about
what we each thought a class should do at a minimum,
versus what we each wanted the class to do.
(In one case I actually drew out a 2x2 matrix of "should do/not-do"
vs. "want to do/not-do" to make it clear.)
The danger here is when pair partner A interprets B's "wants" as
"shoulds", and begins an argument about it.
When the distinction is made clear, the "virtual argument" disappears
and forward velocity resumes.
- Dunning-Kruger effect.
A fancy name for a common phenomenon.
In rare cases, a pair member may be so utterly confident -- and wrong --
about something, that the other partner may hesitate to question
In one (and only one) case in a triple, I felt that I had to be
rude in order to demonstrate that we had gone down a very wrong track.
- TDD. More to follow in a later draft.
- Story Cards.
- The quality and legibility of story cards varied enormously.
Most of the time it was necessary to find the person who wrote the card,
and get them to explain what they meant.
- Cards were taped to the (former) cube-walls.
They frequently fell off, and were retaped, and fell off...
- Mark Price repeatedly had to ask people to write in their
actual times on the cards, once the card was complete.
There were only vague (and frequently differing) understandings
of how to record time on a card.
- Provide a few made-up sample cards,
that show "best practices" for writing cards.
- Require card writers to put their name and a title in big letters
at the top of a card.
Cards without those get moved to a visible "not accepted" board.
- Allow developers to take a really ambiguous or unclear card
back to the card-writer, and require the writer to rewrite the card.
- Write short examples of how to record and fill-in times for
cards. Put them in the wiki.
- Find a better way to make cards stick.
I cut a 4x4 foot piece of
screwed some L-brackets onto it, and hung it over the cube walls.
People could tack or tape cards to it with having them fall off.
I'm sure there are even better/easier ways.
(Maybe duct-tape a big piece of stiff laminate to the wall, and tape cards
- Information Sharing
- Word of mouth.
Works great most of the time.
Sometimes it's hard to know when to interrupt others; sometimes
we get layers on layers of interrupts happening.
This belongs in a retrospective.
Each group is different.
How should we interrupt each other?
5 minutes of explicit conversation about it in the group
might save several person-hours of hesitation time.
- Written information.
This group seemed positively hostile to writing anything down.
Some key pieces of information got passed around via email, but we
never even had a consistent/updated email list(!!).
There is an understandable Agile reluctance to commit anything to
writing that isn't code.
That makes sense, far too much documentation gets out of synch with
live code 2 minutes after it is written.
But I suggest that this can go far beyond the natural Agile mindset;
it can instead become a rationalization for avoidance.
When you have to tell more than 3 people something, it's time to
write it down -- even if you know it will become out of date.
Especially if you know new people will be added to the team
- Build a team email list -- a real list with an alias, not an email
with a bunch of cc's that gets passed around.
The latter is a guarantee for failure.
If you could write a test for it, you know it would fail.
- Build a team wiki.
Commit the little nuggets of key information to it.
(A handful of real examples:
"How do I turn on/off eclipse/liferay debugging".
Or "Where did we put the modified Liferay files we have to plug-in
Or "How does
LinkedIn integration work?"
Or "What's the URL to access the integration server
report from my desktop?" Or... you get the idea.)
If you're tempted to send an email, don't -- put it in the wiki,
then send the email with a pointer to the relevant wiki page.
- The "tech lead as benevolent despot" role works well, to cut
So appoint a benevolent despot to rule the list and the wiki.
(It should not be the tech lead.)
Give him/her the power to insist that people use the wiki;
if developers try to use email, the despot gets to bounce it back
and tell them to immediately put it in the wiki.
Make sure the despot gets a little time now and then to keep
the list and the wiki clean.
- Don't depend on "someone" at Pillar to maintain blogs or wikis.
Make it local, control it locally.
(OK, maybe it can live on a Pillar server.
But it's got to feel local.)
One senior Pillar staff member told me "The Pillar Wiki is where
information goes to die."
- Classes, objects, naming.
The book "Clean Code" talks about readable code as a
That means that class names have to make sense
in the context of a larger language.
Granted, both of these are edgey -- they slide towards waterfall,
towards documentation rot.
But you cannot have a language without a basic vocabulary,
and you cannot have readable code in the style of
"Clean Code" if it doesn't at least hint
at a domain language.
Otherwise your "WTF?" count increases, as you
end up with not-very-meaningful class names like UrlTarget and
method calls like BigBlob.getTextSize().
- Define some domain-specific key-words at the outset,
or along the way as concepts evolve.
Write them on a whiteboard, a wiki, a shared file, anything.
- Write 2-3 sentences at the top of each class,
describing what the heck this thing is.
Not what it does, not how it works, but what it is!
Or even why it is!
- Integration Server.
- The integration server was mysterious.
There was a fair amount of knowledge about its operation,
and the norms for using, that just didn't live anywhere,
and couldn't easily be described by word-of-mouth.
A very few people worked hard to keep it clean, and it
was an honorable but unenviable task.
- The integration server builds broke. A lot.
If anything, the longer the project went on, the more often the
There was much good-natured pleading (esp by the tech lead) for
people to keep the builds from breaking, or to clean up after
a build broke... but there was no obvious pattern that anyone
was doing anything in particular wrong.
It was also hard to tell who was really responsible for any given build
Possible reasons for the frequent build breaks included:
And sometimes we just didn't know "why the damned thing broke" and just
kicked it again and waited for (or even forced) the next commit.
- Multi-phase commits.
Builds commenced on a commit, but since the production code and
the integration tests were in separate projects, a commit to one could
trigger a breaking build, because the integration test project commit
had not yet happened.
"False positives" like this encouraged a "just wait 'till the next
- EasyB test failures.
These were often the "shoemaker's children" of the project, possibly
because there was confusion about who was writing them from the beginning.
- Database changes.
Early on, all database changes were basically emailed around,
which caused confusion and sometimes broke a build.
Eventually a subversion-based tool was added that automatically ran
database updates, which was a vast improvement and deeply appreciated by all.
- Reduce false positives.
Arrange projects so that developers can easily choose to run
unit-tests separate from integration tests, yet both are handled
by a single commit.
Or make a build commence on a commit, but only after a certain
Or... I don't know, but this problem must have been solved somewhere
- Make it easier for anyone to take responsibility for integration
A one-or-two page guide might suffice.
- When frequent pleas don't get results, then lean on it in
These are good people, if something's not happening, then the
situation itself may be at fault.
- Information flow / teaching / learning.
Overall, these were great opportunities.
The publicity for them was mixed and often unclear.
If in doubt, I went anyway, and then left if it didn't seem useful.
The best was Dan Wiebe's talk on EasyMock -- not just because I
personally found it useful, but because he was very clear about
what this was, why he thought it useful, and only then
delved into details.
The worst was a departing Gale employee talking about "ATDD".
I stayed for 45 minutes, and never actually found out what it was
or why it might be useful.
(And as a former teacher, I was not, shall we say, a passive listener.)
As silly as it may seem, I think it would be possible to
do a lunch-and-learn on how to do lunch-and-learns.
Subject matter experts do not automatically know how to teach,
but there are some basic principles that can help.
The simplest is: start with
who I am,
what I'm talking about,
and why that might be useful to you.
Don't dive into details right away, start at 10,000 feet
and work down.
Do give short, specific examples of how something was useful.
I found it highly ironic that, at Gale of all places, nobody seemed
to use books.
There seemed to be a belief that, if you needed to learn something,
you should find it on the 'net.
A few people did pass around one or two books: if it looked useful,
I immediately bought my own copy.
(jQuery in Action
was particularly useful.)
I brought in a load of ~15 books that I thought helpful;
a few people used them on occasion.
There may be some tension here between "travel light"
and "gain mastery" that I do not understand.
It troubles me, but I do not (yet) have recommendations.
- Technology choices.
Obviously, the choice of technologies affects how
well a project progresses.
But more subtly, the choice of technologies affects how
well specific Agile practices work... or don't.
- The choice of
Liferay as the base
platform is a good example.
On the surface, it seemed like a reasonable choice.
But because of the way it "takes over" certain Tomcat functions,
it had the affect of really slowing down the
develop / test / deploy / test cycle.
Unlike "vanilla" Tomcat or JBoss deployments, with Liferay in the middle,
it was impossible to just change a JSP and immediately see the results.
This wasted a lot of time, and caused developers to "batch" many changes
Liferay seemed to offer some features that would be useful:
built-in user creation/authentication, and the handling and
arrangement of portlets.
But in practice, neither of these appeared to help very much,
and in fact required some serious "working around" their features
In 20-20 hindsight, this may be a case of
"use the simplest possible tool for the problem".
I believe the overall velocity would have been significantly
higher if we had built our own pieces just on top of Tomcat.
- The use of EasyB
was a well-intentioned attempt to have business stake-holders
write application/behaviour test skeletons, which would then be
filled-in with Groovy code by developers.
Unfortunately, it never happened that way, so the responsibility
for the application tests was always unclear, and thus
they always lagged behind "real" code.
And many UI problems had to be discovered by live human testing.
In this particular instance, we might have been much faster,
and the responsibility much clearer, if the developers were told
to just write the application tests directly in Selenium (no EasyB,
no Groovy), and run them straight from JUnit.
In general, the responsibility for the application tests should
be made explicitly clear at the outset, and then let that drive
the choice of technology.
Compared to my experiences in other similarly-sized environments,
at Pillar/Gale I observed very robust code, moderately high velocity,
and high developer enthusiasm.
I attribute this to these practices:
But you knew that already, right?
What I found particularly interesting, however, was a group
sense that somehow these practices could (or should)
solve all the problems that the developers faced.
When these practices did not lead to solutions,
I observed frequent pleas and exhortations to "do it better",
"take more responsibility", "remember to fill in times on the cards",
and so on.
The pleading was generally good-natured, but had a background feeling
of helplessness, of puzzled frustration that "damn it, we're good Agilists,
why aren't we solving these problems?".
Now, in many cases, some of these problems were solved,
often by someone "scratching their itch" after an issue was
repeatedly brought up during the retrospectives.
(The management of incremental database changes is a good example
of a problem that was -- eventually -- solved.)
But my main point here is that there can be a disparity
between what the team believes that "Agile can do", and what
the situation actually allows.
And like any group with a strong shared belief system,
when the beliefs fail to change reality, sometimes the frustration
turns inwards to the team, that somehow "we were not good enough,
did not practice our beliefs strongly enough".
(I am reminded -- distantly -- of many "millennialist" and Utopian
movements in the U.S. in the 1800's.
When the promised Second Coming or social revolution
did not occur, these groups felt
puzzled and defeated, and lost a great deal of their energy and
This "turning inwards" is actually much more positive than the
historical response of frustrated developers -- to blame the management,
choice of technology tools, lousy specifications, etc.
An inwards look can encourage developers to reflect on what they
could do differently, what itch could be scratched earlier,
and what different technology choices might help.
The retrospectives are a powerful tool to use this "turning inwards"
and redirect the energy towards identifying the real problems.
Curiously, I observed, at best, "mixed" energy levels during
I'm not sure why this was, but it was decidedly lower than
the energy during stand-up meetings, or during daily development.
- Buy-in of TDD across all levels.
- Story cards (mostly) ensured good-sized tasks.
- Developers could write story cards, especially for bugs, problems,
- The "shared honesty" enforced by pair-programming.
- Frequent rotation of pairing.
- Shared group norms of "Travel light", "don't speculate",
and "simplest solution that works".
- Strong encouragement of immediate refactoring.
- Interviewing process that (mostly) ensures basic competence with
all of the above.
- Many developers just wanted to get back to programming.
- Pointing out the same problems repeatedly was discouraging.
- Problems would be assigned to some people (esp managers),
but progress was unclear (not testable!)
I break with anthropological tradition here to
(briefly) sermonize, and end with a question.
Agile/TDD turns the most common programmer vice
-- "I just want to write code" -- into a virtue.
In doing so, it also releases a huge amount of energy --
the unspoken, often repressed anxiety that
"my code will break the next time it is changed".
(Which can lead to "nobody but me should ever change it.")
Agile solves huge problems.
But having done that, there is a temptation to believe it
solves all problems.
We must not, in our enthusiasm, fall into that
"Always test it!" is a powerful message and tool.
How can we extend it to problems that we have not been