Larger Architecture Equals Fewer Architects

Large projects requiring multiple software systems and multiple technologies require careful design and planning before one line of code is written or one subsystem is spec'd out. Senior IT thought leaders—you may call them system architects or computer scientists or whatever—need to whiteboard and design the systems and their interactions before the level of effort can even be scoped.

Business Analysts

A long time ago in a galaxy far away there was science called systems analysis. The premise of systems analysis was that a reasonably intelligent person who understood information systems could work side-by-side with a business team and gain an understanding of the required business process. Armed with that knowledge they could then map that process onto software systems.

Somewhere in the last 30 years that methodology has been replaced with BA's interviewing business owners and gathering requirements. Notice I said business owners—BA's tend to gather their requirements from managers, not the people who do the work. Then they interview the manager's managers and since that group is even further removed from the actual work they are more concerned about reporting on the amount of work accomplished than the work itself. And the madness continues.

BA's must have the ability to understand the difference between real business requirements and nice-to-haves and fantasy world nice-to-haves. BA's do not need to be current employees. In fact you will probably be better served by using contract BA's with the requisite skill sets but without pre-conceived ideas or prejudices about the business. And never use a BA who is already part of the business unit being analyzed. They already “know” what is needed, which pretty much precludes the possibility of innovative thinking and discovery.

Software Developers

Now here's a tough group to deal with. I was one myself and understand why they can be so difficult. A common practice in large projects is to bring the software developers into the process as soon as the first rounds of business requirements are complete. Big mistake. Software developers want to help people; they want to build the cool things that their customers want.

Suddenly the fantasy world requirement is in scope. The work of the smart guys who did the initial high-level design is not complete when the high-level design is. That group—or their equivalent at the system or subsystem level—should take the first pass at mapping the business requirements to the appropriate components.

Software developers also tend to overthink system design. Edge cases are important but there is probably no need to code for those once-in-a-decade edge cases. I recently observed a discussion about transactions that occur in that undefined period when the east coast office has already flipped to daylight savings time and the midwest office won't for another hour. It is a non-issue because the data centers use a common time and any client side differences can be compensated for. Events happen at a specific point in time and that is not affected by location of the user. That is why the military does things in Zulu (Greenwich) time on a 24-hour clock. Ambiguity is a bad thing for military (and data processing) operations.

System Architecture and Design

Back to system design. Once the high level design is solid and approved, the lead team needs to decide what system is going to lie behind each of the abstracted service layers. Build or buy is always a tough call. If you can find commercial software that meets 90 percent of your requirements then go with that and forget about that missing 10 percent.

Most vendors claim that their product is accessible through a full featured API and that it is infinitely configurable to meet any of your requirements. Don't believe it unless you see it. More often than not the API is limited to whatever they needed to make the product function in the first place.

The database schema is hidden and doesn't matter because your license forbids you to read or write to it anyway. You have no access to source code. You are provided packages that can be deployed to your servers—and if it doesn't work it is always your problem.

That may sound a little negative and it is. If commercial software fits the bill than go for it, but don't force the vendor to make a lot of substantial changes to their product. System down or degraded situations are when you discover the true worth of a vendor and their product.

When I am on a call at 4 a.m. with my CIO listening in I don't want the vendor to tell me their lead developer is on a six-month sabbatical in Uzbekistan. And I also don't want the vendor telling me that they can't really support their product with this or that particular release of whatever.

I like throats to grab and I prefer to grab throats that get their check from the same place I do. Black box software systems are not supportable. Kind of like “building” a computer when what really happened was simpler than assembling an Ikea bookcase. And which requires zero knowledge of how a computer functions. All that being said I have had many excellent experiences with software and system vendors. They are not all scary. But you need to do your due diligence before making a commitment to build enterprise systems on their product.

Rolling Out Your Own

Inevitably many of the functional components of a large software system are going to be home grown. If nothing else you need to own the service layers with all the business logic and probably the UX/UI. Let's assume after much iteration and discussion the business requirements are final and let's assume that includes all the business and system-use cases. You are now getting very close to getting some value out of those scrum masters, but you aren't quite there yet.

The single most critical documents you will create are your system design specifications. These start out as high level representations of the interactions between one system and other systems—passing through the virtual service layers.

The teams that create these documents are small. Bite off a small chunk of functionality and get the team leads for the various subsystems involved and create process flows through each system. Then take each group of endpoint in the process flow and get those two or three people to design that sub process.

Large projects are made simple through decomposition. It sounds simple, but very few organizations have the patience and the conviction to make the process work. Deadlines are more compelling than process. Once you subvert process you are guaranteeing a certain degree of failure. Properly built design specifications can be turned over to any set of competent developers with an expectation of a reasonable degree of success.

Now is the time to build out virtual services layers (I mean really virtual; not abstracted) that will return expected data when consumed by other components. We can get a little agile now. The various subsystems can be built in parallel with straw man “virtual” services standing in for the real services or API's while we wait for them to be built. Fire up those sprints and be agile.

Basics

Large scale software development is not difficult. It is in fact easier than managing smaller projects since everything can be so easily siloed. It does require strong leadership— you need an overall program manager and program managers for each sub system. You need strong well qualified technical leaders who are capable of designing enterprise software systems. You need commitment from senior management to support the program management and the approved methodologies.

You need to manage people's time well. Twenty people in a room designing system interactions is a waste of project resources. Small groups (five to six or even fewer) can make good decisions. Anything larger than that and you end up designing a camel and you will never get that beach condo with a camel.