Difference between revisions of "Validation"
(New page: ==Validation== The goal of the validation project is to create an environment which can be used to answer the question, "how do you know that your model has any connection to reality"? F...) |
|||
| Line 1: | Line 1: | ||
| − | ==Validation== | + | == Validation == |
| − | + | Validation means the ability to determine that a given model has some connection to some reality -- that a model is an accurate representation of a real system. If you are going to use a simulation model to try and predict how some real system is going to behave, you must have some reason to believe your results -- i.e., can you trust that an inference made from the model translates into a correct prediction for the real system. Validation of a model provides such a basis for trust. | |
| − | + | Every model has a '''target system''' that it is attempting to simulate. The first step in writing a simulation is to identify this target system and the level of detail and accuracy that the simulation is desired to reproduce. Once this is done, one can develop an '''abstract model''' of the target system. This is typically an exercise in managing the tradeoffs between complexity, resource requiremens and accuracy. This step has been called '''model qualification''' in the literature. This abstract model is then developed into an '''ns-3 model''' that implements the abstract model as a computer program. The process of getting the implementation to agree with the abstract model is called '''model verification''' in the literature. This is what people usually think of when they imagine software testing. The process of getting the ns-3 model behavior to agree with the target system behavior is called '''model validation''' in the literature. Since validation may ultimately be based on some non-deterministic physical process, it will probably be a stochastic process. | |
| + | |||
| + | The process used to validate a model is conceptually quite simple. One compares the behavior of the ns-3 model to the behavior of the target system, and makes adjustments to the abstract model and/or the ns-3 model to improve the correlation. This pat of the process is sometimes called '''calibrating''' the abstract model. | ||
| + | |||
| + | At the end of the validation process, it is desirable to have a number of repeatable results that can demonstrate to other users that a given ns-3 model is faithful to a given abstract model and that this abstract model is, in turn, a faithful representation of the target system based on the initial qualification process. We call this collection of results the '''Validation Suite'''. You can think of this as the collection of experiments that have been used to validate a model, which can be run collectively as a suite of tests on the model for use by other users. | ||
| + | |||
| + | === The Basic Process === | ||
| + | |||
| + | The process of modeling some target system is then composed of three main parts: model qualification, model verification and model validation. As you might expect, we are only going to address model validation here. In the literature, Naylor and Finger call this piece of the puzzle, '''Validation of Input-Output Transformations'''. Since the target system is often ultimately based on physical processes which are governed statistically, statistical comparisons between experiments done on the target system and simulations of experiments done on the ns-3 model will be in order. The validation project in ns-3 aims to provide you with tools to make these comparisons easier and translate them into processes that can be used to test your model behavior. | ||
---- | ---- | ||
| − | [[User:Craigdo|Craigdo]] | + | [[User:Craigdo|Craigdo]] 19:24, 1 April 2009 (UTC) |
Revision as of 19:24, 1 April 2009
Validation
Validation means the ability to determine that a given model has some connection to some reality -- that a model is an accurate representation of a real system. If you are going to use a simulation model to try and predict how some real system is going to behave, you must have some reason to believe your results -- i.e., can you trust that an inference made from the model translates into a correct prediction for the real system. Validation of a model provides such a basis for trust.
Every model has a target system that it is attempting to simulate. The first step in writing a simulation is to identify this target system and the level of detail and accuracy that the simulation is desired to reproduce. Once this is done, one can develop an abstract model of the target system. This is typically an exercise in managing the tradeoffs between complexity, resource requiremens and accuracy. This step has been called model qualification in the literature. This abstract model is then developed into an ns-3 model that implements the abstract model as a computer program. The process of getting the implementation to agree with the abstract model is called model verification in the literature. This is what people usually think of when they imagine software testing. The process of getting the ns-3 model behavior to agree with the target system behavior is called model validation in the literature. Since validation may ultimately be based on some non-deterministic physical process, it will probably be a stochastic process.
The process used to validate a model is conceptually quite simple. One compares the behavior of the ns-3 model to the behavior of the target system, and makes adjustments to the abstract model and/or the ns-3 model to improve the correlation. This pat of the process is sometimes called calibrating the abstract model.
At the end of the validation process, it is desirable to have a number of repeatable results that can demonstrate to other users that a given ns-3 model is faithful to a given abstract model and that this abstract model is, in turn, a faithful representation of the target system based on the initial qualification process. We call this collection of results the Validation Suite. You can think of this as the collection of experiments that have been used to validate a model, which can be run collectively as a suite of tests on the model for use by other users.
The Basic Process
The process of modeling some target system is then composed of three main parts: model qualification, model verification and model validation. As you might expect, we are only going to address model validation here. In the literature, Naylor and Finger call this piece of the puzzle, Validation of Input-Output Transformations. Since the target system is often ultimately based on physical processes which are governed statistically, statistical comparisons between experiments done on the target system and simulations of experiments done on the ns-3 model will be in order. The validation project in ns-3 aims to provide you with tools to make these comparisons easier and translate them into processes that can be used to test your model behavior.
Craigdo 19:24, 1 April 2009 (UTC)
