{"id":79,"date":"2010-03-17T08:07:31","date_gmt":"2010-03-17T08:07:31","guid":{"rendered":"http:\/\/www.eqsim.com\/blog\/?p=79"},"modified":"2010-03-17T08:12:45","modified_gmt":"2010-03-17T08:12:45","slug":"why-software-fails-is-a-lot-like-why-simulations-fail","status":"publish","type":"post","link":"http:\/\/www.eqsim.com\/blog\/why-software-fails-is-a-lot-like-why-simulations-fail\/","title":{"rendered":"‘Why Software Fails’ is a lot like why simulations fail"},"content":{"rendered":"

I came across an article in Dr. Dobbs Journal entitled “Why Software Really Fails And What to Do About It<\/a>“, by Chuck Connell (March 11, 2010).\u00a0 I think the most succinct answer to the provocative title question appears on page 3:<\/p>\n

We fool ourselves about how well we understand the complex new software machines we are trying to build.<\/p><\/blockquote>\n

Wow.\u00a0 It hit me that this is the same sentiment I run into when I discuss building simulations with certain types of programmers–those inexperienced with creating simulations, but with egos that believe they can do anything just because they have the ability to program.\u00a0 I will call these people ‘naive simulation programmers.’\u00a0 I don’t mean they are necessarily naive or ignorant about programming or deficient in knowledge of programming languages, rather, they are naive about the true nature of simulation.\u00a0 To be honest, I used to fit in the latter category until I had a revelation during my PhD work, thanks to my advisors.<\/p>\n

The bottom line is that simulations express a model of some phenomenon (usually from the real world, like a device or a process),\u00a0but ALL models\u00a0intrinsically\u00a0have assumptions.\u00a0 Imagine, if a simulation\/model were made to be 100% accurate with the real-world manifestation, it could only be possible if the simulation WAS the manifestation itself–otherwise there would be some difference, albeit possibly small, which made it different.\u00a0 This is not simply anal retentive arguing.\u00a0 Once you internalize this fact and step back a bit, you realize that our goal with a model is to decide which are the important details of the real manifestation we need to replicate, and which we don’t have to, or which details we can simplify without losing our expected validity of the details we deemed important.<\/p>\n

Getting back to the connection with the quotation–I believe that the naive simulation programmers\u00a0value the objective of reproducing\u00a0fidelity to the real system, almost\u00a0like a macho benchmark,\u00a0over reasoning about the objective of the overall project and designing\u00a0a model with the right level of detail to meet that objective.\u00a0 That machoism gets people in trouble–more time, more resources, more cost, etc. than necessary, and sometimes the extra burden put on designing to an arbitrary but unnecessary fidelity can even sink the whole project.<\/p>\n

For example, I was brought in to oversee a project to convert a physical\u00a0computer with an 8-bit microprocessor (8088)\u00a0into a simulated computer with a simulated 8-bit microprocessor (should we call this a ‘mocku-processor’?\u00a0 wow, it’s late), to\u00a0enable students to perform calculations and to troubleshoot the computer.\u00a0 Rather than examine the types of tasks the students were asked to perform, the lead naive simulation programmer started off saying “we’re going to need an 8088 emulator for this.”\u00a0 This led to heated discussions about whether such detail was going to be necessary, and we really ought to look at what tasks students need to perform because there’s a good chance we don’t have to simulate all the functionality (and absorb the cost and integration time) of some fanciful ’emulator’ that is supposed to exist somewhere (because of course, even that emulator had assumptions that might not hold true for our tasks!).\u00a0 I certainly understood what he was thinking–solving the problem ‘in general’ was going to give us coverage for the wide range of tasks needed.\u00a0 However, he was going to make this proclamation before bothering to understand the level of fidelity and modeling required for the tasks at hand.\u00a0 In the end, because he was the team leader and had the ultimate decision, he took his team through a completely wasted pursuit of the generality illusion before settling back to a more simplified solution, based on the specific needed tasks, that actually could solve the problem at hand.<\/p>\n

In brief, naive simulation programmers gravitate to the pursuit of grandiose models that are supposed to simulate behavior comprehensively, under the pretext that it is going to be able to address the current problem and beyond, without sufficiently understanding what assumptions are perfectly\u00a0reasonably to be made.\u00a0 In a form similar to the quote from Chuck, I would say:<\/p>\n

We fool ourselves about how complex we have to make simulations to be effective.<\/em><\/p>\n

For example,\u00a0our CommandSim platform for training emergency responder officers\u00a0uses photographs, videos, and animations to present realistic incidents to the trainees.\u00a0 ‘Serious games’ enthusiasts take a look and argue that a 3D immersive environment would\u00a0produce better training, by virtue of the fact that some of our competitors produce 3D environments for emergency responder training.\u00a0 However, we have demonstrated that the performance improvement with CommandSim, for its specific purposes–which happen to be the same purposes the competitors are aiming their products–has been seen to be comparable to live exercises.\u00a0 That begs the question, “what exactly is the additional contribution of the 3D immersiveness?”\u00a0 Certainly not in simplified technology, nor ease of distribution.\u00a0 By carefully understanding what problems you are seeking to solve, you can come up with solutions designed at the ‘right’ level of detail, not just going full bore with all the detail possible.<\/p>\n

Improving the Success Rate<\/h3>\n

At the end of the article, Chuck lays out good arguments for ways in which we can improve the success rate of software, which I think parallel the ways we can improve the success rate of simulations (for their specific purposes).\u00a0 Here are two of them:<\/p>\n