The Software Development Crisis is a name for the readily observed fact that, historically, the larger a software development project is, the most chance it has of failing, to the extent that the largest projects have a high risk of failure, almost certainty.
This high risk is a big problem for middle management of large organizations, who can get the budget for giant visionary projects that senior management can brag to the board or shareholders about, but not the budget for regular maintanence and development. Without regular maintenance and development work, the code base becomes old or stale or ricketty or non-performant, and this creates the scenario where the lets replace all these little systems with one big system snake oil can be pedaled.
The crisis has long been seen, as far back as the Brook’s seminal book The Mythical Man Month which pointed out that adding extra people to a project too late actually delays it rather than improves it. Responses have been better languages, tooling, and, most significantly, the Agile approaches such as Extreme Programming (how to make individual developers effective) and Scrum (how to make development teams predictable). But Test Driven Development (TDD) has been one of the big successes, because it has enabled the DevOps organization, where the tests are a delivery artifact that the deployment and operations teams benefit from (which fits in with ITIL disciplines.)
With that background in mind, I was reading a quote from TDD guru Kent Beck (in the 2008 paper Realizing quality improvement through test driven development: results and experiences of four industrial team ) that “Test-first coding is not a testing technique”, saying that in fact it is an analysis and design technique. The paper notes that adopting TDD decreases bugs significantly, but adds about 35% to delivery time (i.e. absent of other measures such as Scrum): a tradeoff to consider, given that bugs are traditionally deemed to 4 to 10 times more expensive to catch in a released system than during development (TODO: did I just make that up? where did that number come from, or what is it?) And the paper notes that bug fixes are themselves 40% more likely to contain bugs than normal code, even if still! You do want to fix bugs, but you want to start testing and operations with as few bugs as possible.
Let me re-phrase this: a mantra of the Agile world is that Big Design Upfront is dangerous, counter-productive, ineffective, and so on. However, paradoxically, Little Design Upfront is deemed somehow safening, productive, effective and so on. But if Waterfall is bad, why are little cascades good? It seems to me the key difference is that the little cascades are actually directly testable: there is no interpretation gap, or handwaving gap.
The connection with Schematron should be clear. One of the advantages of Schematron is that the Pattern/Rule/Assertion approach is suited to incremental development: whereas changing a DTD or Schema is a Big Deal, merely adding an assertion as some particular need arises is often without anxiety as to unintended consequences. So DTDs and closed Schemas tend to belong or require to the Waterfall mentality, with all its attendant problems.
(I wonder if this is why so often companies with large document corpuses find that every 10 or 15 years or so they need to shuffle the deckchairs and move to a completely new schema: because the grammar DTD/Schema technology cannot cope with maintenance demands well, and so gets a backlog of feature requests until the dam breaks and a large analysis and conversion effort is required. On the other hand, often the movements to a new schema are the MacGuffin to force a change in platform or technology. And I cannot swear that Schematron would be better, for lack of concrete knowledge.)
So, apart from these well-trodden paths of using Schematron at implementation time, and for increasing quality-in-use from feedback, what about using Schematron upfront as part of analysis and design? (Of course, there are many standards efforts, such as HL7, that have made Schematron schema upfront, so I guess my case here is for non-public-standard systems.)
Schematron lends itself to two on particular approach. The core of a Schematron schema is that it is a list of natural language (i.e. human) statements about what is supposed to be in the document. I suggest that the functional specification for software that accepts or produces XML should be written (by the domain expert) in the form of single natural language statements: for example, for a transformation between formats:
- Every title element in the old will have a corresponding title element in the new.
- Every title element in the new will be in the same order as the title in the old.
- No repossession notes will be issued for December because doing so makes our company appear heartless.
In a Scrum kind of environment, these will part of the detailed backlog, and presented initially by the product owner (as the contact person with the wider organization who must produce the business rules) and then by the team (as the details emerge). The team takes each of these and puts them into one or more Schematron schema. Initially these can be null entries:
<sch:rule context="/">
<sch:assert id="t1.1" role="UNKNOWN" test="true()">
Every title element in the old will have a corresponding title element in the new.
</sch:assert>
<sch:assert id="t1.2" role="UNKNOWN" test="true()">
Every title element in the new will be in the same order as the title in the old.
</sch:assert>
<sch:assert id="t1.3" role="UNKNOWN" test="true()">
No repossession notes will be issued for December because doing so makes our company appear heartless.
</sch:assert>
</sch:rule>As the team implements the code, it first fills out the assertion tests. So that, for example (XPaths indicative and untested!):
<sch:rule context="/">
<sch:assert id="t1.1" role="ERROR" test="count($old//old:title) = count(//new:Title)">
Every title element in the old will have a corresponding title element in the new.
</sch:assert>
</sch:rule>
<sch:rule context="new:Title">
<sch:let name="titles-before-me" value="count(preceding::new:Title)"/>
<sch:let name="matching-title-in-old value="$old//old:title[.=current()/.]"/>
<sch:let name="titles-before-match-in-old" value="count($matching-title-in-old/preceding::old:title)"/>
<sch:assert id="t1.2" role="ERROR" test="$titles-before-me = $titles-before-match-in-old">
Every title element in the new will be in the same order as the title in the old.
</sch:assert>
</sch:rule>
<sch:rule context="new:repossession-note">
<sch:assert id="t1.3" role="WARN" test="@month != 12">
No repossession notes will be issued for December because doing so makes our company appear heartless.
</sch:assert>
</sch:rule>The same approach can be used in a more Waterfall model, where the analysts should be required to state the (functional) requirements in simple testable, objective statements. It needs to be part of tasking that these requirements can be reviewed for their suitability to be made into tests, and adjusted/split/removed if they cannot be made into executable tests using Schematron.