Lesson 4 - Building Real World Models: The Modeling Process

As discussed at the beginning of this Unit, the ability to evaluate a system and conceptualize it in a generalized (and often abstract) way is one of the most difficult skills required of a modeler. Moreover, it is something that for the most part cannot be easily learned from a book, and requires practice and experience, and perhaps most importantly, interaction with and feedback from experienced modelers. Having said this, however, starting with a good understanding of the most appropriate way to approach a complex modeling project can help you to gain these skills in an efficient manner.  Therefore, this Lesson briefly discusses the modeling process.

The Iterative Modeling Process

The steps required in simulating any system are as follows:

1. Define your objectives and measures of performance.  Before attempting to simulate any system, it is critical to first clearly identify what types of questions you are trying to answer with the model.  The objectives of the model define the performance measures for the system.  A performance measure is a model output by which you can judge the performance of the system (e.g., the concentration of a particular contaminant at a specific location, the expected profit, the most likely cost).  Clearly identifying the objectives of the model will also help you determine the appropriate level of detail in the model. This step may seem obvious, but failing to clearly define the specific questions you are trying to answer prior to attempting to build a model is one of the most common errors made by inexperienced modelers.
2. Develop the conceptual model.  The most important step in simulating any system is developing a conceptual model of the system.  A conceptual model is a representation of the significant features, events and processes controlling the behavior of the system.  It is essentially a body of ideas, based on available information, that summarizes the current understanding of the system.  Development of a conceptual model is typically the most time-consuming part of any simulation effort.
3. Create the mathematical model. Once a conceptual model of the system is developed, it is necessary to represent it quantitatively within a mathematical model.  A mathematical model consists of a set of input assumptions, parameters, equations and algorithms describing the system.
4. Quantify the input parameters.  The mathematical model identifies specific inputs (e.g., the flow rate in a river, the financial discount rate) which are required in order to simulate the system.  These must be quantified by specifying their values or probability distributions.
5. Implement and solve the mathematical model using a computational tool.  For simple systems, the computational tool might be your brain (if you can solve the equations in your head), a calculator, or perhaps a spreadsheet.  For more complex systems, a specialized simulation tool (such as GoldSim) is required.
6. Document the model and evaluate, explain and present the results.  The final step in the simulation process is to produce results, evaluate and draw conclusions from these results, and document the model and present the results to interested parties. Of course, as discussed in Unit 15, you should not wait until the end of a project to document your model, but should do so continuously as you build it.

The most important thing to understand about these steps is that they should be carried out iteratively, responding to new information and/or preliminary modeling results. That is, modeling any system should be an iterative process:

The basic concept is that as new data are obtained (e.g., through a data collection program or research) and/or as new insights to the behavior of the system are obtained (based on preliminary model results), you should reevaluate and refine the model. In some cases, you might “loop back upward” in the middle of the process.  For example, development of a conceptual and corresponding mathematical model are dependent on the type of data available. If you determine that your available data is highly uncertain, the conceptual and mathematical models may need to be adjusted accordingly (e.g., if data is sparse and uncertain, a highly detailed model would likely be inappropriate).

Simulation models which are constructed and continuously modified in this manner can then do more than only provide predictions of performance; they can provide a systematic framework for organizing and evaluating the available information about the system, and can act as a management tool to aid decision-making with regard to data collection and resource allocation (what should be studied, when, and in what detail?).

Because development of the conceptual model of a system is perhaps the most important step of any simulation effort, it is worthwhile to discuss this step in somewhat greater detail below.

Conceptual Model Development: Charts and Diagrams

The first step in creating a conceptual model is to carefully define the system boundaries. That is, the system you are attempting to model (“in”) is only a subunit of the rest of the world (“out”). The system may be influenced by things outside of the system limits (i.e., through boundary conditions), but does not model any impacts on things outside of the system limits. Hence, clearly defining the boundary between “in” and “out” is a critical first step in developing a conceptual model of a system.

Once you have defined your system boundaries, conceptual model development often centers on the creation of flow charts or influence diagrams.  These are simply graphical tools which provide a framework upon which a quantitative description of an environmental system can be constructed.  The charts and diagrams themselves can take many formats, depending on the personal preference of the individuals who develop and use them.  Regardless of their format, their purpose is to present in a graphical manner the features, processes and events which may influence the system. 

These charts or diagrams typically form part of the documentation for a conceptual model.  Often, however, the process of creating the diagrams is more useful than the end product itself.  This is because the act of constructing a flow chart or influence diagram forces the modeler or modeling team at the outset of the project to step back and look at the “big picture” and view the entire integrated system with all of its interconnections and couplings, as opposed to focusing on specific system details.

Conceptual Model Development: Top-Down Versus Bottom-Up

One of the most common errors made when building a simulation model is to represent the model with an inappropriately high level of detail from the outset (and this starts with the conceptual model). 

This is because engineers are often taught to tackle problems from the “bottom-up”, describing all of the processes in great detail.  Bottom-up modeling approaches attempt from the outset to acquire data and model the various controlling processes in great detail, and typically make use of complex process-level models for the various system components.  The emphasis is on understanding and explaining the processes in great detail in order to eventually describe the behavior of the entire system. 

While such an approach may seem at first glance to be "scientifically correct", for the following reasons it is generally not the best way to build real world simulation models:

1. The level of detail in a model developed from the bottom-up often becomes inconsistent relative to the amount of available information.  That is, a model is only as good as its inputs, and if you don't have much information, a detailed model is generally no better than a simple one. 
2. It is often difficult to appropriately integrate and capture interdependencies among the various model components in a bottom-up model, since it is often impossible (or computationally impractical) to dynamically couple the various detailed process-level models used for the components of the system  As a  result, important interactions in the system are often intentionally or unintentionally ignored in a bottom-up model. 
3. It is easy for a bottom-up modeling project to lose sight of the "big picture" (i.e., to "not see the forest for the trees").  As a result, such an approach can be very time-consuming and expensive, with much effort being spent on issues that prove to have little or no impact on the ultimate objective of the project. 
4. Finally, and perhaps most importantly, such models tend to be very difficult to understand and explain (and hence be used) outside of the group of people who create them.

"Top-down" modeling approaches, on the other hand, start from the top (i.e., the ultimate objective of the modeling exercise) and concentrate on the integration and coupling of all system components.  The controlling processes may initially be represented by approximate high-level (i.e., less detailed or “abstracted”) models and parameters.   The model can then evolve by adding detail (and reducing uncertainty) for specific components as more is learned about the system (consistent with the iterative process discussed above).  Such an approach can help to keep a project focused on the goals of the modelling exercise without getting lost in what may prove to be unnecessary details. Moreover, because a properly designed top-down model tends to be only as complex as necessary, is well-organized, and is hierarchical, it is generally easier to understand and explain to others.

There are two key points in the application of a top-down modeling approach:

1. Top-down models must incorporate an accurate representation of the model and parameter uncertainty resulting from the approximations.
2. As opposed to representing all processes with great detail from the outset, details are only added when justified (e.g., if additional data are available, and if simulation results indicate that performance is sensitive to a process that is currently represented in a simplified manner).  That is, details are only added to those processes that are identified as being important with respect to system performance and where additional detail will reduce the uncertainty due to model simplifications.

It is important to understand that a top-down model does not have to be “simple”.  Whereas a “simple” model might completely ignore a key process, a well-designed top-down model approximates the process at an appropriate level while explicitly incorporating the resulting uncertainty that is introduced.

This key idea that detailed models are not necessarily better models is captured well in this quote by George Box (a British statistician, described as "one of the great statistical minds of the 20th century"):

Since all models are wrong the scientist cannot obtain a "correct" one by excessive elaboration… Just as the ability to devise simple but evocative models is the signature of the great scientist, so overelaboration and overparameterization is often the mark of mediocrity.

Although detailed models may not be directly implemented when using a top-down approach, this is not to say that detailed modeling is not required.  Quite to the contrary, detailed modelling results of various sub-systems often form the foundation for a top-down model and are always required to some extent in order to generate the appropriate input parameters, and to assess the degree of approximation involved in simplified approaches.

Modeling Process Summary

Although the skills required to become a good modeler generally require practice and experience, this Lesson has attempted to highlight a few key points regarding the modeling process that will hopefully help you in your quest to become an expert modeler.  In particular,

• Before attempting to simulate any system, it is critical to first clearly identify what types of questions you are trying to answer with the model.
• Modeling should be an iterative process.
• Clearly defining the boundary between “in” and “out” is a critical first step in developing a conceptual model of a system.
• The process of creating diagrams and flow charts while developing a conceptual model is often more useful than the end product itself, as it forces the modeler or modeling team at the outset of the project to step back and look at the “big picture”.
• You should take a “top-down” approach to modeling, representing processes at an appropriate level of detail given the objectives of the exercise and the available information. That is, keep your model as simple as possible (but not too simple).