Building software is like building a house in many ways.  When building a house, we have to have a good foundation, strong pillars, proper plumbing and wiring, and so on and so forth.  When building software, we need to have a good design, clear and concise code, proper wiring of components, etc.

There is one subtle difference between building software and building a house though.  Usually when we build a house, we rarely make large renovations such as adding a story, moving around walls and windows, putting in a new electrical system, or modifying the foundation. Software, on the other hand, needs to be modified and updated quite often due to changes in business caused by new opportunities, changes triggered by the economy, mergers and acquisitions, and so on.  Life expectancy of a software product is generally between 3 to 5 years before major updates are required.  A great number of software products require such updates much earlier in their lifecycle.

In the world of software something that works is not necessarily something that can be easily modified, extended, or maintained.  General observation is that the chunk of cost of software is not what’s spent on building it in the first place, but what it takes to maintain and extend it over the years.

There are several attributes of quality software.  I will list the most important ones from business standpoint below:

Reliability

Software functions each and every time in a predictable manner.  You can’t afford to have your “Save” button save your documents sometimes but not others.

Maintainability

It’s easy to maintain software.  Fixing a bug, for example, does not spawn another bug.  Or moving your database to another server does not require code changes.

Scalability

Software can easily keep up with business growth and does not require code modifications as the user base or usage grows.  This also goes hands in hands with efficiency, which is another attribute of quality software.

Extendibility

It is relatively painless to add new features.  Also the addition of new features does not break old features.

Reusability

The entire software itself or certain parts of it can be reused to create new software to address other business uses.

There are many other attributes of quality software, for example, usability, security, fault tolerance, etc., hence a lot of thought goes into building a quality product.

The Challenge

When building software, quality is something that is baked in from the beginning, and can’t be sprinkled on top after the fact.  Also creating quality software requires additional time, effort, and possibly investment into proper tools.  This creates an interesting problem for business users and the technical leadership alike: how to balance quality of software with investment that may or may not bring any positive ROI.

There are software development methodologies such as Agile that promise quality software with minimum time-to-market.  The problem with such methodologies is that if not understood correctly and followed properly, they do not yield promised results.  I have come across many companies claiming to use to use Agile, for example, that are only disguising their chaotic software development under Agile’s name.

I evaluate each situation and devise a plan accordingly.  Quality of software is definitely important to me, but having an entrepreneurial mindset, I understand and appreciate business challenges as well.  For experienced technical teams creating quality software even with time and budget constraints is relatively easy.  For teams that are not as experienced, this remains to be a challenge, that often times result in major disasters.

According to the 2009 CHAOS report published by the market research firm, the Standish Group, IT Project failure rate increased in the last 2 years of recession.  No surprise.

According to the report the rate of successful projects that were delivered on-time, within budget, and with the required features, dropped from 35% in 2006 to 32% in the last two years.  Also the rate of failed projects increased from 19% in 2006 to 24% recently, which included projects that were deemed complete failure and either never delivered, or delivered but never used.  The rest of the projects were considered challenged, those that were delivered late, with incomplete features, or went significantly over-budget.

As IT leaders do we see something wrong with this picture?  Keep in mind that out of projects that are usually successfully delivered, some are trivial to begin with, while some others start out with a bigger-than-required budget due to over-estimation.  This consideration drives the rate of success as reported by Standish even further down.

There have been numerous people who have written about how the lack of user involvement, unrealistic expectations, poor requirements, poor design, scope creep, etc. play a role in the failure of IT projects.  I do not disagree, but I think the problem is much more deep-rooted than what is generally pointed out.

I view these high failure rates as a direct result of:

• Incapable IT Leadership; and
• Incapable Technical Leadership

Keep in mind that IT leadership is not necessarily the same as the Technical Leadership.  I consider IT Leadership as high-level IT executives that are aligned with the business, for example, CIO’s, CTO’s. etc.  Technical Leadership, on the other hand, consists of Technical Architects, Solution Architects, and the like.

Lack of user involvement or unrealistic expectations, for example, point straight at the failure of the IT leadership.  IT leaders must make sure that they have proper processes as well as resources in place to involve and educate the users.  They are also responsible for convincing the business one way or the other, and analyzing and mitigating any risks before they become cause for failures.

Similarly it’s the responsibility of the technical leadership to ensure that proper requirements are in place before they embark on their journey of design and implementation, and that their designs actually do address all the business and other requirements.

Whatever the reasons, failure should not be an option.  If a project fails, I believe it’s because the leadership failed to do their job.

With budget cuts that come with the recession that’s still holding strong, as well as the ever increasing complexity of IT projects, I see the rate of success driving even further down the next time the Standish Group publishes it’s CHAOS report.  I really hope that I am wrong, but without a major paradigm shift in the IT and the technical leadership, I really do not see how that would be possible.

On a recent team project I was using continuous integration or CI (an Agile practice) via CruiseControl.NET. It was a breeze to automatically build projects and solutions using MSBuild on our build server (a run-of-the-mill Dell workstation running Windows XP without Visual Studio 2005 installed) whenever someone checked-in any files to source control. This allowed us to be confident at all times that various pieces our distributed project integrate well. Moreover, since MSBuild can build projects and solutions from command line, I had some batch files setup that would build release versions of our projects, create setup files (using NSIS), archive older versions of setup files, and move new setup files to a network share where MIS / Operations folks could install the new version on the production server from.

In Visual Studio 2005, project and solution files are nothing more than MSBuild XML build scripts. This allows us full control over the build process. For example, in a web project, Visual Studio 2005 does not provide any user interface to modify pre and post build actions. However, since solution file is simply an MSBuild script, we can modify such actions manually by opening up the solution file in any text/XML editor.

MSBuild can perform several key tasks out of the box. Each task is essentially a unit of work (UOW) that contributes to the entire build process, e.g. copy files and folders, compile files, etc. If we ever require tasks that do not ship with MSBuild, we can always create custom tasks in any .NET language by simply implementing the ITask interface or deriving from the Task class.

In conclusion, MSBuild allows developers full control of the build process so that builds can be fully customized. This may not be so crucial in smaller projects, but for enterprise solutions that typically comprise of several projects and many dependencies, MSBuild is definitely a God-send!

• A team of peers with clear accountability, shared responsibility and open communications. Each role is accountable for a specific share of the quality of the overall solution.
• Advocacy for all key constituencies that must be represented on a successful software project. Every perspective is represented to provide the checks and balances that prevent errors of omission and lopsided decisions.
• Stretch to fit to the scale necessary for the specific project. Constituencies may be combined in small teams or further refined as teams scale for larger projects.

There are 7 advocacies that are represented in the MSF Agile Team Model. In my opinion it’s imperitive that these advocacies have proper representation in order to ensure success of a given project. Since MSF offers a stretch-to-fit approach, it’s possible to map these 7 advocacies into fewer role clusters as determined by the phyical size of the team.

Following are the advocacies and their respective focus as presented by the MSF Team Model:

Product Management

Goals

• Understand, communicate, and ensure success from the standpoint of the economic customer requesting the solution.

Functional Areas

• Marketing
• Business Value
• Customer Advocacy
• Product Planning

Responsibilities

• Acts as customer advocate
• Drives shared project vision/scope
• Manages customer requirements definition
• Develops and maintains business case
• Manages customer expectations
• Drives features vs. schedule vs. resources tradeoff decisions
• Manages marketing, evangelizing and public relations
• Develops, maintains, and executes the communications plan

Program Management

Goals

• Right solution is delivered at the right time and all expectations are understood, managed and met.
• Deployed solution will meet qualities of service & business objectives, and be viable in the long term.

Functional Areas

• Project Management
• Solution Architecture
• Process Assurance
• Administrative Services

Responsibilities

• Drives development process to ship product on time
• Manages product design and specifications
• Facilitates communication and negotiation within the team
• Implements and ensures standards
• Maintains the project schedule and reports project status
• Drives implementation of critical trade-off decisions
• Develops, maintains, and executes the project master plan and schedule
• Drives and manages risk assessment and risk management

Development

Goals

• Implementation, estimates, high quality maintainable code and unit tests.

Functional Areas

• Technology Consulting
• Implementation of Architecture and Design
• Application Development

Responsibilities

• Specifies the features of physical design
• Estimates time and effort to complete each feature
• Builds and/or supervises building of features
• Prepares product for deployment
• Provides technology subject matter expertise to the team

Quality Assurance

Goals

• Ensure solution quality.

Functional Areas

• Test Planning
• Test Engineering
• Test Reporting

Responsibilities

• Ensures all issues are known
• Develops testing strategy and plans
• Conducts testing
• Reports test results

Release/Operations

Goals

• Timely readiness and compatibility of infrastructure.

Functional Areas

• Infrastructure
• Support
• Operations
• Commercial Release Management

Responsibilities

• Act as advocate for operations, support and delivery channels
• Manage procurement
• Manage product deployment
• Drive manageability and supportability trade-off decisions
• Manage operations, support, and delivery channel relationships
• Provide logistical support to the project team

User Experience

Goals

• Provide user documentation and training
• Understand and communicate users’ context, and ensure usability from user perspective

Functional Areas

• Technical Communications
• Training
• Usability
• Graphic Design
• Internationalization
• Accessibility

Responsibilities

• Acts as user advocate on team
• Manages user requirements definition
• Designs and develops performance support systems
• Drives usability and user performance enhancement trade-off decisions
• Provides specifications for help features and files
• Develops and provides user training

For some companies migrating to Agile from other traditional methodologies, this may not be the case. I have worked with companies that have well-defined and assigned roles for Business Analysts, Database Administrators, Front-end Developers, Back-end Developers, etc. When such companies move towards Agile, they face a problem of optimal resource allocation and utilization.

As an example, a developer may not be productive while a DBA is creating new tables in the database and writing stored procedures. Once the DBA is done with his tasks, he may not have enough to do until the start of the next iteration. Since Agile is iterative, and iterations can’t overlap, this may result in poor resource utilization and may adversely effect the overall efficiency.

Before starting to implement an Agile methodology within your organization, it is important to make sure that your IT human-resource infrastructure actually supports it. While Agile may work in companies with specialized IT roles, it is definitely not designed and optimized for those scenarios.

While I have been successful in implementing many of the Agile practices such as earlier stakeholder involvement, EDUF, iterative development, code refactoring, unit testing, continuous integration, etc. in a relatively short amount of time, I still have to deal with the fact on a daily basis that they have to have very good estimates too far in the future. They must have this information from a budgetary standpoint and in order to be able to make executive decisions in advance about whether or not to proceed with a given project.

I have tried my best to explain how the Agile estimation works, and how trying to estimate too far in the future could be a wasted effort, but after all the time and energy spent, I am beginning to think that perhaps the Agile philosophy is not for all mindsets after all!