Roland Eschenburg

Roland Eschenburg

An Infrastructure & Network Engineer who thought that Docker and Kubernetes looked way more fun. Building Jenkins jobs to fund his container habit and helping teams to automatically deliver software ever since.

Learning deliberately

9 minute read

A tale of continuous improvement via experimentation & fast feedback.

About “us”

A little bit about us and our background first.

We are the Motley Crew. A fairly young (10 months old) enablement & platform team. Our mission is to Help deliver value to customers faster, with more confidence. In practice, we look after our CI/CD & microservice infrastructure.

As a new team, we needed to overcome a few challenges:

  • We were still in a pandemic with most of us working mainly or entirely from home;
  • The team started as a collection of single points of failure - a few of us were “the only ones who knew a particular area/technology”; and
  • We assumed responsibility of a somewhat neglected and aging technology stack that fell through the cracks of previous re-orgs
    • which led to constant disruption by operational issues (aka “Jenkins is down again”).

To cope, we knew we had to try something different and experiment with ways of working that none of us were very familiar with. We started rummaging in the Agile toolbox to find ways to facilitate

  • Team (and trust) building;
  • Rapid knowledge sharing; and
  • Flexibility to cope with unplanned work & changing requirements - on a daily basis as opposed to a two weekly cadence.

To start with, we settled on a Scrumban approach with very tight WIP (work in process) limits & co-creation as our main way of working. Specifically, we would do mob-programming via video meetings every day. Since we were talking as a team all day every day, we had no need for dedicated daily standups - those would happen naturally first thing in our mobbing sessions (or not). We did have a need for planning and regular reflection, however. To keep the feedback cycle “short”, we settled on a twice-a-week catch-up, which was run like a “classic” retro (Even Better If, What Went Well, etc).

Learning how to work with each other

The above way of working served us well - we quickly got on top of some of the more pressing infrastructure issues, upgraded and migrated things to stabilise the platform and reduce the amount of unplanned work and interruptions. We learned a bunch along the way as well - it’s almost impossible not to, by meeting twice a week to talk about “What Went Well”.

We found that mobbing worked well for us on large problems that were mostly unknowns for the entire team, but not so well when we had a couple of experts on a given technology/problem space. So we started splitting the team into a couple of smaller mobs or pairs, trying to pair up experts with novices to still facilitate knowledge sharing. This also gives people a break as mob programming is very tiring and can get a bit much. Sometimes this created a swarm (i.e. the sub-teams worked on a shared, bigger problem),and sometimes it didn’t as different sub-teams dealt with competing priorities. We also learnt that fast feedback was still very necessary and (at a minimum) daily check-ins with the rest of the team are vital. Sometimes, tightening the feedback loop lead back to mobbing, where feedback was instant. Mobbing also led to us introducing a “Friday Morning Maintenance” slot - a dedicated time to work on regular maintenance tasks, small general improvements and all the stuff that fell by the wayside during the week.

Note how I’m saying “feedback” a lot. This is mostly team-internal feedback to make sure we are all still pulling in the same direction on a problem, and that the problem hasn’t changed. For customer feedback, we are lucky enough to be able to invite our (mostly internal) customers into our mobbing calls as necessary or reach them quickly via Slack. This, we learnt, is also incredibly important.

A lot of this learning was the result of an experiment and double-loop learning mindset (i.e. as opposed to trying to adjust our mobbing techniques only (single loop), we looked at whether mobbing was the best and only solution (double loop)). We tried different approaches and figured out what did and didn’t work for us. But if you look closely, all these lessons were on “how to work with each other” (which makes sense - as a new team we were focused on how to get on with each other).

At a lower level however, learning was - at most - accidental. While we thought things were going very well, we were still struggling with a bunch of basic things which are probably very familiar to anybody working in software development and are, I believe, fundamental to “how to do the work”.

Not all sunshine and roses

All of this sounds great - so what gives?

Things we struggled with regularly:

  • WIP;
  • Cycle time;
    • Predictability/estimateability of work
    • Tasks being way too large
  • Keeping tickets up to date and the (very short) backlog groomed; and
  • Boring retros that didn’t lead to much or any change.

Incidentally, those are all somewhat connected. And while we did note some of them occasionally in our retros (“Even better if.. we kept our tickets up to date”), the action would be “update tickets more” - if we were lucky, sometimes we would even check on the previous retro’s actions (although more often than not we forgot that, too). Writing it down like this makes it painfully obvious how unhelpful an “update tickets more” action is. It doesn’t address the root cause, nor does it offer anything on how to achieve it. And without any meaningful reflection we saw very little change in our behaviour.

Enter: Troubleshooting Agile

I’m a big fan of the Troubleshooting Agile Podcast - and a lot of our ways of working have been inspired by Squirrel and Jeffrey.

Over the summer they published a series of very short episodes on learning loops, three of which in particular stood out to me and caused a bit of an “Aha!” moment.

The 3 topics where:

We’re not a “traditional” scrum team, but the principals still apply, as it turns out. So a few days of pondering these episodes led to what I call

Deliberate Learning

Our new experiment works like this:

Every catch-up (still twice a week), we look at our backlog, groom/re-prioritise as necessary and then commit to the work we think we can achieve between now and our next catch up (two or three working days respectively). At our next catch up, we “walk the (very small) board” (another Summer Shorts learning loop episode!) and discuss every piece of work we had committed to:

  • Did we deliver it?
  • If not, why not?
    • Forensically, but blameless, of course - a micro RCA almost. At least an informal five whys
  • If yes, how did it go?

Essentially, we are deliberately looking for any tiny opportunity to learn and/or improve and try to distill this down to short lessons (small enough for a post-it).

We then plan the next two-three days’ worth of work, keeping our lessons in mind.

We’ve also started to consolidate and carry our lessons with us and deliberately review them until they turn to habit. (Visually - in Miro we drag the post-its with the lessons around with us to wherever we are doing the planning, in Jira, as an attempt to keep planning in a single tool, we keep a Confluence page open with the lessons prominently displayed).

We’ve also agreed on a daily demo/check-in at the end of the day to keep the feedback cycle short (turns out, having something like a daily standup is useful when done correctly… and not boring or a waste of time).

How is it going?

Suffice to say, our first few retros after starting the experiment were rather longer than usual. However, we started to see patterns and started to address some of our worst habits, making meaningful changes.

Some of the lessons learned include (taken straight from Miro):

  • Break down tasks more (we learned this one about four times, before we saw an improvement).
    • Discussing elephant carpaccio several times.
  • If you are making assumptions about a tasks, speak them out loud to verify them
    • Also a real eye opener, this one. Quietly making assumptions about stuff can be a killer.
  • Large-ish non-coding tasks need to still go into Jira to make work visible and keep track of WIP.
  • Review scope of stories/tasks frequently and adjust, don’t over-engineer or allow scope creep!
  • Where cross-team collaboration is required, check availability and setup meetings before committing to the work.
  • Talk about the positives more.
    • Initially we only discussed the “what could we have done better” when we didn’t deliver the planned tasks, without reflecting on the “what worked well for us”.
  • Occasionally check on alignment by discussing what certain words mean to each of us (“customer”, “completed”, “delivered” etc).
  • It is ok to have small tasks that take a long time - we’ve come across them when it turned out that nobody in the team had any current/deep expertise on a technology/topic, so we had to learn on the job. Initially we were confused and were looking for other lessons (could we have made the task smaller, would a spike have helped?), but came to the conclusion that a) we really enjoyed those tasks because we were all digging in as a team to figure stuff out and b) it was time well spent - no further lessons needed.

Results

We’ve only been doing this for just over eight weeks at the time of writing, but the changes are quite remarkable.

The biggest game-changer has been elephant carpaccio. What previously was a single story on the board, is now an Epic with lots of small tasks or stories (each still delivering fully shaped elephants!) As a result:

  • Our cycle time is down to around a day;
    • delivering three-four tasks in three days is now the new normal
  • Our WIP has come right down (one-three tasks in progress at any given time, depending on how many sub-teams we’ve split into, no more than one task per sub-team)
    • Since tasks are small, it’s much easier to finish them one after the other
    • Changing priorities/urgent requests don’t bother us as much anymore, as we can easily finish of a small task and then switch focus to something else
  • Tickets are being updated;
    • A small task only needs a small update, which is quickly done when moving a task across the board to ‘Done’
  • Retros are more interesting and meaningful, the team is more engaged and we keep learning more every day; and
  • Friday Morning Maintenance can probably be retired soon, as our new cycle time allows us enough flexibility to do those tasks ad-hoc.

Conclusion

Every day is a learning day - sometimes you just have to go out and deliberately try to make it so.

While our ‘rate of learning’ has slowed down, we are still learning new things every day and at every level. As our team grows and the problems change, so do the lessons.