Team First

It can be disheartening when I see posts online from Scrum teams that absolutely hated the experience and want to go back to Waterfall.  How can that be?  Agreed, Scrum can be a difficult pill to swallow at first when one is very used to the ebb and flow of the waterfall, but I feel the core principles of the Agile Manifesto and the core principles of Scrum should make sense and eventually impassion the team. So why does this not happen?

I believe one of the main reasons this might be happening is because some of us are not doing a good job as Scrum Masters.  We need to do a better job selling agile to the teams (and implementing, but that is for another post).  Selling does not mean shoving down their throats, blinding following the Scrum rules.  It means put yourself in their shoes and see the change from their eyes.  More meetings?  Possibly.  More transparency?  Of course.  More productivity?  Let’s hope so.  Less risk? Definitely.  These are not bad things for the team, but they may not see it that way initially.  It could take months, or longer, for them to “get it”.  Unfortunately, some of them may never “get it”, or may never want to.

Once we understand the changes that are occurring, not just in process, but also in culture, we can do a better job communicating to the team.  Each team member may see things differently, and we should acknowledge that fact and see if we can find a common avenue to help address issues or concerns across the team.

These types of discussions and transition plans could be well beyond the skill level and experience of a new Scrum Master, so in that case a professional agile or Scrum coach may be necessary.

We must not forget that a large part of Scrum and agile is team empowerment, and valuing people over process.  If the team is resisting, then the process probably isn’t working, and may need adjustment.  Use common sense.  I strongly feel that if the Scrum Master protects and empowers the team to the best of their ability, success in a better process in the end should be inevitable.  It may not be Scrum by-the-book, but I think that is perfectly fine.

Selling Leadership

If you are doing Scrum, or another agile process/methodology, hopefully leadership has bought in and is fully committed to going agile.  Perhaps they have not.  Even if they have stated that they are committed, they may not fully understand what it means to go agile.  Helping to form cross-functional empowered teams, providing dedicated Product Owners, and embracing and leading the agile cultural change, are all things that leadership should be doing.

If leadership does not appear to be fully on board, perhaps the benefits of agile are not clear?  Perhaps we need to do a better job selling agile to leadership.  Agile best practices are in their best interest as well.  Motivated, well-equipped, cross-functional teams working on the highest value items to the business at all times?  Reduce waste in process; increase productivity; increase transparency and thus reduce delivery risk.  Who would not want that?  Continuous Integration is an easy sale, in my opinion.  There are no real negatives that could possibly come close to outweighing the automation and reduced waste and risk that would ensue.  Test Driven Development is possibly more difficult to sell, but I am convinced it can be done with supporting arguments and information.

Everything can, and should, be started in moderation.  That also helps reduce the risk of spending time on processes or procedures that may ultimately fail.  It also reduces the risk that team members will be caught up focusing on areas that are not providing the highest value to the business.

With leadership support, agile teams can do so much more.  It is our job to help leadership see the benefits of going agile, and to try to obtain that support.

Selling the Client

I think much of the same arguments above apply to communicating the benefits of agile to clients or our users.  Shorter feedback loops, more transparency, adaptation to change; I see all of these as positives from a client perspective.  There is definitely a lot of adjustment that will probably have to be made (doing small iterations means delivering smaller chucks of features to the client at a time, which is possibly quite different than what they are used to), but there is a reason for this (see adaption to change and shorter feedback loops).

So if a Scrum or agile transition is not going as well as planned, or preparations are being made for a future transition, keep in mind that we must all be good stewards of agile.  We must be able to sell agile, to a certain extent.

After that, it should continue to sell itself.

I am very interested to see how they manage adopting and scaling the Agile process across such a large organization.  It is good to see that their leadership is committed to seeing it through, as I envision they have a difficult, but positive, journey in front of them.

This post does not aim to re-define CI, as there are plenty of other sources online that can give you the run-down, but it does aim to share my thoughts as to why it is so important.  Let me summarize CI at a high-level, and then break down the positive impact as I see it.

What Is Continuous Integration?

Good question, and an excellent place to start.  Continuous Integration, or “CI” for short, in the simplest form is a process which fully automates the compilation of your projects latest source code from source control.  There are some very important points in that sentence, so let me highlight them:

• fully automated
• compilation of latest source code
• from source control

These points are very important, but often seem to be missed at some point or another.  If developers are running build scripts manually (either from their machines, or on a server somewhere), that doesn’t qualify as CI.  If they are still doing builds manually (not even running scripts, ouch) from their machines, then keep reading….

The “fully automated” and “from source control” parts actually abstract out a few important concepts that I don’t want to skip over.  The concept is that developers should all be checking in their code as frequently as possible and practical for the project.  It is important that their code compiles against the latest source, but their entire feature or problem set they are working on may not have to be fully baked out.  The act of checking-in the code, will trigger a CI build.  This is where the “fully automated” part comes into play.  A proper CI server or tool will automatically check source control for any new changes.  When it detects that somebody has checked-in a change, it will get the latest code and force a build (compile) of all the latest source.  If the compile works, that’s great!  If the compile breaks for whatever reason, it should notify the developer who broke the build with their change, and optionally, other team members as well.

That’s the bare-bones CI process in a nutshell.  However, even that basic process can save a team from some major headaches down the line, especially for a geographically dispersed or outsourced team.

Improve Quality

A bare-bones CI process that simply compiles the latest source code may not seem to directly impact the quality of your software, but it can have some nice benefits that will indirectly improve your quality.  Since CI is fully automated, it will lay the foundation for a repeatable and consistent build process which will immediately notify the team where there are build issues, and what those issues are.  Having this type of stability in your build process will help improve the quality of your software indirectly.

In terms of having a direct impact on your software quality, a good CI process will lay the foundation for rolling in additional automated tasks into a daily or nightly build, such as running unit tests, static code analysis, security testing, building installers, deployment tasks and more.  Basically, any task that can be automated, should.  This will give your entire build process a much needed overhaul in terms of automation and consistency.  Having testing and analysis built into the build process is a really good way to start improving on the software quality.  These tools will not improve the quality themselves, but they will increase the transparency of potential issues to the project team, and help them locate and focus in on potential issues more proactively.

Improve Productivity

Using an automated build process should have a rather obvious direct correlation to team productivity.  If it is fully automated, then that means they are spending less time doing builds.  Correct!

Let me also highlight some other potential issues that can have dramatic impacts on team productivity.  Let me start with some questions.  Do you have a geographically disperse team?  How often do your developers check in their code?  Do they get the latest code and ensure their proposed changes will compile before they check those changes in?  Do they double-check that any new dependencies they are adding are also properly checked-in and working?

Any time a developer checks in their code without properly checking if their changes will negatively impact others on the team (i.e. they “broke” the build), there is the potential for some major downtime or troubleshooting while team members figure out why their build broke and how to best fix the problem.

When teams are spread out across the globe, or at least different timezones, this can become a bigger problem.  I have seen cases where a team member in the US checked in their latest code on a Friday night, shortly before leaving work.  On Sunday, team members in India needed the latest code to continue working, so they checked out the code and it didn’t compile!  The team member who “broke” the build had forgotten to ensure that all their files, including new dependencies, were checked-in and working before leaving the office.  Had this team had a proper CI build in place at that time, the developer (and team in our case) would have been notified of the broken build within minutes of having checked-in the code.  With any luck, the notification would have arrived before the team member left the office.

Now keep in mind that CI does not promote blindly checking in your code without performing other best practices (like ensuring the proposed code changes compile against the latest code, unit tests pass, etc).  However, a good CI process will help cover the gap for when these types of issues, or mistakes, do happen.

In the cases I mention above, a proper CI process will help improve productivity through a few means.  Since CI is automated, it only needs to be written once (and probably tweaked over time), which helps reduce the time team members are performing low-value tasks, such as building, running unit tests and analyzing the code.  In a few cases, CI can improve team productivity in the, hopefully rare, cases where a broken build would have cost the team hours of troubleshooting and fixing just to get back up and running.

Reduce Risk

Hopefully a few trends have become apparent above.  An automated and easily reproducible build process can help reduce risk, just in the fact that it can help eliminate potential human errors during a build process.  Having the team notified immediately as soon as a broken build is detected can also reduce delivery risk by helping to keep the team productive and the project on-track.  Building in additional steps into a nightly build, like automated unit testing, security testing, code analysis, documentation generation, installer generation, even deployment tasks (the list can go on), can help further reduce risk of potential undetected issues or human errors in each of those steps.

Summary

The more time that the team can focus on producing good, quality code, and spend less time manually running low-value tasks such as some of the above, means the team is able to produce value to the business.  And that is very important.

Best Practices

Here are some of the best practice that I encourage others to follow based on my experiences:

• Use source control and ensure the team is proactively checking-in code as frequently as possible
• Find a build server product that you like, and most importantly integrates with your source control product.  It should be easy to use and to configure/maintain.  There are quite a few CI servers out there, many of which are open-source and free – but just ensure you will be able to understand and easily configure the tool, as it will quickly become your best friend or worst enemy depending on how easy it is to use and the features it supports.  I do not claim to have a broad exposure to many CI servers, but I will post a future review on some build server tools I have used in the hopes that it will help somebody in their review.
• Keep different build configurations for each project:
  • CI Build Configuration (mandatory) – Keep it lean and mean.  Configure the build server get the latest code and compile full project on every check-in to source control.  Possibly run critical unit tests as long as they are fast.  Remember, the goal with this configuration is to notify the team of a broken build as soon as possible.  If possible, have the build server clean out the local source directory and dependencies before getting the latest code from source control.  This should help ensure a clean build every time.
  • Nightly Build Configuration (recommended) – Have this build on a regular schedule for the project.  This should perform in much the same manner as the CI Build Configuration (clean build), but you have the luxury of time now.  The primary objective is not to notify the team of a broken build as soon as possible, but to perform a full build and additional quality checks along the way.  Even deploying to a development or test environment may make sense depending on the project and the team’s needs.  Build servers will often let a team member manually trigger a build by clicking on a button or a link.  This means even if this build configuration is scheduled to occur in the middle of the night, the team can still use it to do a full build, run tests, scan the code and deploy, during the middle of the day if needed.
  • Weekly Build Configuration (optional) – This configuration was based on our needs around full static code analysis.  Full static code analysis can take a really long time, depending on the produce used, what it is scanning for, and the volume of source code it needs to analyze.  It can also put a heavy load on the build server, effectively locking it up for a lengthy period of time.  For us, it simply did not make sense to scan the code everyday, as it didn’t offer that much value to the team at this point.  Once a week seemed to give us the results we wanted, and did not task the build servers during the busy week days (when the build servers should be focused more on CI builds).
• Try to keep all project dependencies within a relative path under the project in source control.  In other words, try to refrain from referencing dependencies in the GAC (.Net) or on the local machine that the build server may not be able to easily retrieve from source control.  Build servers will typically support build agents, and may federate the build on any agent that is available.  Having dependencies required to be pre-installed, or pre-setup on each build agent in a real pain to maintain, and also complicates project deployment.  It is my opinion that having all dependencies stored in source control with the project is the best way to go.  This way the build server doesn’t require any special setup or steps to compile.  It simply gets the latest code (and dependencies) from source control and triggers the build.  This also allows the build agent to potentially purge the build directory and all dependencies before it gets the latest code – which can help ensure a clean build each and every time.  This also limits the potential impact of one project’s dependency from impacting another project, assuming they are sharing the same build agent (common to share build agents in our organization).
• Look for ways to further improve your build and deployment process.  Automate everything over time.
• Suggestion:  If cost is an issue, and virtual machines (VMs) are available in your organization, then use hosted VMs for the build server and build agents.  I find that in almost all cases, a properly configured VM will have no difficulty running CI builds quickly.  Full static code analysis can take much longer if the VM does not have sufficient resources.  VMs can be easier to setup, and/or clone, and can reduce cost across a medium or large deployment of build agents.  If you are using VMs, ensure you have enough disk space allocated for frequently building your selected projects.  In my case, having a dedicated logical drive for builds, at least 20-30 GB may be sufficient for a number of projects (again, it depends on the size of the projects, the logs and artifacts that get generated for each build, how often the CI builds will get triggered, how long you want to keep the build logs).  In our case, static code analysis consumes much more RAM, CPU and disk on the VMs, than just a regular CI build.  This is part of the reason for us doing it on the weekends during idle time.

How can a team accomplish so much (i.e. requirement docs, design, development, qa testing, etc) in 30 day iterations?   It can take weeks to get a large feature just developed, never mind the extra days for testing, extra days for bugs fixes and more time for regression testing.   I thought it would likely take a few sprints together, 60 or more likely 90 days, to fit all that in.

Those were my initial thoughts, both as a senior dev, architect and development manager.  My experience in those roles were telling me, “wow, this is kinda crazy. Kinda cool, but kinda crazy.”  It must work for somebody, otherwise Scrum wouldn’t be so popular, right?  I think so.

It is becoming more obvious over time where we need to make more cultural changes, process changes and further embrace the idea of team empowerment. What would you do if you had to cram all that effort and tasks into 30 days, and still have a production quality product ready at the end of each Sprint?

We are still moving in that direction, and I think we have a clear path so far.   We are automating a lot;  CI builds, soon Unit Tests, and whatever else we can.   Making the code self-testing, self-validating (read TDD for test driven development) appears to be the only feasible way of getting all the development done (and the testing), while maximizing the teams use of the full Sprint (i.e. no “code freeze” to fit in QA testing at the end of the sprint).

The rough part?  How to quickly move toward TDD on a very large, complex application suite that has been under heavy development for many years.  I will follow up shortly with some ideas on how we might address this issue.

Thankfully, our team has transitioned pretty well.  While we are not following Daily Scrum’s to the letter (i.e. we don’t always stand up; always go in the correct order; or require team members to stick money in the pot if they are a few seconds late), we do keep our discussions short (most cases shorter than 10 or 15 minutes.

Early on in the Scrum transition, one of the team members brought in a soft stress ball to the Daily Scrum.  They ended up volunteering to go first and then tossed the ball to somebody else (who inadvertently all eyes were on so they went next).  Since one of those first meetings, that is what the team has done.  They don’t go in order, or sit in the same place, but when they get the ball, they quickly go over the important information:

• What did I do yesterday
• What do I plan on doing today
• What is getting in the way of me getting my work done (Impediments)

Sometimes the team will go into more discussion right then and there when they think it is warranted, but they will keep it short (if it requires more time, they will parking lot the discussion and continue with it after the meeting concludes).  This is the format the team prefers, and it seems to be working really well – so no reason to change it.

One other transitional issue we had early on, was that most team members would look directly at and provide an update only to me (Scrum Master), rather than addressing the entire team.  First Tip:  I listened and gazed around the room, but avoided long-term eye contact with the person speaking.  They tended to start looking at others for eye contact, which is what I wanted.  Second Tip:  Due to other meetings and conflicts, on some days I was unable to make the Daily Scrum (but I was lucky enough to have another trained Scrum Master on the team to help run things).  I think my “not going” helped the team figure out how to update themselves and not worry about just updating me.  I would then always follow up with a few team members to see if anything critical or Impediments were discussed that I needed to be aware of.  Obviously, I still try to make every Daily Scrum that I can – but if I cannot, I feel it is critical that they still meet and talk to each other (as I am just there to facilitate and provide guidance when needed).

I am curious what other Scrum Masters do to encourage their teams to transition to a Daily Scrum rather than a status meeting?

We have not been using Scrum and Agile for a long time.  We are a little more than a year into our transition from waterfall to agile.

The transitional road has not been without its bumps, twists & bends, and occasional detours, but I think we are on the right track and working our way towards a more agile and productive team.

I plan to write a series of short posts to help myself and hopefully others reflect on our mis-steps and successes.  What worked for us may not necessarily work for you;  Nor to I want to give you the false impression that we are completely agile and following Scrum to the letter, because we have some gaps here and there.  Some gaps we are trying to close, and some gaps I am not sure we will even bother to address.  Our ultimate goal is to become more agile with each iteration, and continually find ways to improve our process, improve our team and the product at the same time.

This is tiny snapshot of where the project was more than a year ago:

• Requirements (what requirements?) were typically conversations or just an email with some description of what somebody wanted.  This was not necessarily a horrible thing, as it allowed the developers, business analyst and project manager/business owner to be creative and discuss the requirements as we went.  However the lack of documentation did cause problems along the way.
• Our release schedule was our iteration schedule (a release roughly every 3-4 months or so).  A rough plan was devised at the beginning of a release cycle, most of the time.
• Status meetings.  The team would meet roughly once a week or so, sometimes less frequent, with the project manager to go over each task status.  A lot of unknowns and large tasks resulted in a lot of “I am still working on it” from week to week.

Currently:

• We are still on a 3 month release cycle (as required by the business), but we work in 30 day iterations (or Sprints).
• Status meetings went away and replaced with a more natural quick “team discussion” called a Daily Scrum.  Here is where my team deviates a bit from the Scrum rules:  we answer the required 3 statements/questions 1) What did I do yesterday? (or before the meeting) 2) What do I plan on working on today? (or after the meeting) and 3) What impediments are preventing me from getting my work done?  Instead, each team member still goes through all three items, but more freestyle – just normal talking to the team.  Sometimes they take more time and get deeper in the details (rather than parking lot the discussion), but that is a conscious choice the team made, and it does not seem to be disrupting the flow or speed of the Daily Scrum too much.  As the team does not sit in a single room together, this is one of the way they mediate that and have a quick open discussion.
• Detailed requirements are being drafted, but as Use Cases.  We will likely start moving towards User Stories and using Story Points for estimating the product backlog.  Our BA attempts to focus on Use Cases a sprint ahead of the team, but this is not always the case, and the team will still work together to understand the requirements as they go.
• Review meetings:  The team demonstrates functionality and what they worked on during the sprint to our Product Owner.  This is a good place to get additional feedback on the overall direction, vision and change requests of the product from the Product Owner – which would make their way into the Product Backlog.
• Retrospective meetings: Arguably the most important meeting for the team from a cohesive, self-managing, happy and productive team standpoint.  This meeting has gotten noticeably more important, more detailed and more productive as we go.  The team is starting to notice more items that get in the way of them being productive – and are doing a better job in bringing these items up and discussing them with the team.  We try to tackle as many items, at least one, each sprint so we can always feel like we are changing something for the better – and so that the team feels that they are in control.  These discussions also occur at times during the Daily Scrum or anytime during the sprint.  We bring them back up during the retrospective so we can discuss and document it.  Depending on the item, we may try to address it during our sprint (if it is small enough, and makes sense to address it immediately rather than wait, and does not have a negative impact on what the team already committed to).

Both lists a actually much longer, but it should serve as an idea of the progress we are making.  The short of it is I think the team, and the Product Owner, is starting to see a lot of positives from the transition to be more agile.  The team is much more motivated and happy, and the productivity and product quality is continuously improving over time.  There is a lot of ground left to cover, and it may take us a while to get there, but this was definitely the correct direction to take from the fork in the road.

I will follow up with more stories, and links to useful articles and sites that I found to be helpful.