Data Collection Framework - Revision history

Tomh: /* Prioritized list for ns-3.23 and future releases */

2015-04-24T17:59:35Z

Prioritized list for ns-3.23 and future releases

← Older revision		Revision as of 17:59, 24 April 2015
Line 240:		Line 240:
	4) add ability to GnuplotHelper to add data series to single plot		4) add ability to GnuplotHelper to add data series to single plot

	---------------- (likely cutoff for ns-3.23) ---------------		----------------
			(likely cutoff for ns-3.23)
			----------------

	5) review and try to incorporate DCF code from Magister Solutions		5) review and try to incorporate DCF code from Magister Solutions

Tomh: /* Prioritized list for ns-3.23 */

2015-04-24T17:58:48Z

Prioritized list for ns-3.23

← Older revision		Revision as of 17:58, 24 April 2015
Line 225:		Line 225:
	The ns-3.22 release added a TimeProbe class for traced values that emit values of type ns3::Time.		The ns-3.22 release added a TimeProbe class for traced values that emit values of type ns3::Time.

	=== Prioritized list for ns-3.23 ===		=== Prioritized list for ns-3.23 and future releases ===

	Li Li, Felipe Perrone, and Tom Henderson are working on these enhancements planned for ns-3.23 if possible:		Li Li, Felipe Perrone, and Tom Henderson are working on these enhancements planned for ns-3.23 if possible:
Line 239:		Line 239:

	4) add ability to GnuplotHelper to add data series to single plot		4) add ability to GnuplotHelper to add data series to single plot

			---------------- (likely cutoff for ns-3.23) ---------------

	5) review and try to incorporate DCF code from Magister Solutions		5) review and try to incorporate DCF code from Magister Solutions
Line 247:		Line 249:

	8) Several collectors are nearly done in the http://code.nsnam.org/safe/ns-3.21-collector/ repository, such as BiVariateCollector. Prepare code reviews as necessary.		8) Several collectors are nearly done in the http://code.nsnam.org/safe/ns-3.21-collector/ repository, such as BiVariateCollector. Prepare code reviews as necessary.

			9) Add capability to set time resolution on x-axis (defaults to Seconds)

	=== Longer-term items ===		=== Longer-term items ===

Tomh: update DCF

2015-02-28T00:50:02Z

update DCF

← Older revision		Revision as of 00:50, 28 February 2015
Line 1:		Line 1:
	== Scope ==		== Scope ==
	Data Collection Framework is responsible for simulation data monitoring, on-line reduction and storage		Data Collection Framework is responsible for simulation data monitoring, on-line reduction and storage. This project has been going on for several years so this wiki page collects information across a long timespan; some has been overcome by events.

	== Arch and API Proposal ==		== Arch and API Proposal ==

	Simulation data collection process is a three-stage pipeline ~~(''Felipe, I am sorry for badly misusing your terms'')~~:		This is the original design/API proposal discussed between Pavel Boyko, Felipe Perrone, and Mathieu Lacage.

			Simulation data collection process is a three-stage pipeline:

	1. Models report ''events'' using existing trace source mechanism.		1. Models report ''events'' using existing trace source mechanism.
Line 214:		Line 216:
	== Development road map ==		== Development road map ==

	~~''This section~~ was ~~last edited July 29, 2013''~~		The initial set of code was merged in the ns-3.18 release (easily enable the dumping of raw ns-3 trace source data into files and gnuplots):

	~~We are planning to merge~~ the ~~initial portion of data collection framework for~~ ns-3.18 (~~August release~~).
	* basic probes		* basic probes
	* file aggregators		* file aggregators
	* gnuplot aggregators		* gnuplot aggregators
			* test code coverage for helpers, file and gnuplot aggregators, and additional probes
			* add coverage in ns-3 tutorial

			The ns-3.22 release added a TimeProbe class for traced values that emit values of type ns3::Time.

			=== Prioritized list for ns-3.23 ===

			Li Li, Felipe Perrone, and Tom Henderson are working on these enhancements planned for ns-3.23 if possible:

			1) rename TimeSeriesAdaptor to EventDrivenCollector

			2) Add two new collector types:
			* TimeSeriesCollector (reports the most recently reported value of a traced value, at regular intervals)
			* TimeAverageCollector (produce time-averaged data across regular intervals, such as data rates (e.g. packets per second))
			By default, these will not output values when the probe hasn't yet fired during the enabled period, although it may be a mode to enable in the future.

			3) add ability to scale y-axis data of GnuplotHelper by a scale factor

	~~Goal: easily enable the dumping of raw ns-3 trace source~~ data ~~into files and gnuplots~~		4) add ability to GnuplotHelper to add data series to single plot

	~~=== Items~~ to ~~complete for ns-3.18 ===~~		5) review and try to incorporate DCF code from Magister Solutions

	* test code coverage for ~~helpers~~, file ~~and gnuplot aggregators, and additional probes~~		6) extend base class of DataCollectionObject to allow for setting the Enable/Disable times in a single statement
	* add coverage in ns-3 ~~tutorial~~
			7) allow users to learn all of the available traced values to be able to hook to, from the command line or from an output file

			8) Several collectors are nearly done in the http://code.nsnam.org/safe/ns-3.21-collector/ repository, such as BiVariateCollector. Prepare code reviews as necessary.

	=== ~~Prioritized list for ns~~-~~3.19~~ ===		=== Longer-term items ===

			* perform offline steady state/transient detection
			* perform online steady state/transient detection
			* can we enable/disable multiple DCF elements in a path, with a single statement, by virtue of their relationships?
	* Support more probes of all ns-3 trace sources		* Support more probes of all ns-3 trace sources
	* BasicStatsCollector, helpers, and examples: compute statistics to reduce probe'd data before writing to files, plots, and databases		* BasicStatsCollector, helpers, and examples: compute statistics to reduce probe'd data before writing to files, plots, and databases

Vedranm: Note about statistics

2013-08-05T11:50:11Z

Note about statistics

← Older revision		Revision as of 11:50, 5 August 2013
Line 210:		Line 210:
	Vedran asked whether the Object Start/Object Stop time tracker could be implemented with DCF. Basically, this is a variation on the requirement stated above for "Wifi Throughput"; what is desired is something like a "Duty Cycle collector" that will keep track of the proportion of time that an object was on or off, and report statistics at the end of the simulation.		Vedran asked whether the Object Start/Object Stop time tracker could be implemented with DCF. Basically, this is a variation on the requirement stated above for "Wifi Throughput"; what is desired is something like a "Duty Cycle collector" that will keep track of the proportion of time that an object was on or off, and report statistics at the end of the simulation.

	In discussing this use case, we observed that there is a need to actually stop the statistics framework at the end of the simulation, so that the time from the last event was recorded to the end of the simulation. That is, if the last state transition was at time t=1000 seconds, but the simulation ended at time t=2000 seconds, we need to ensure that this kind of report covers the time range of 1000-2000 seconds, so we need to stimulate somehow the report at time t=2000 seconds even if there is not an underlying state change to drive it.		In discussing this use case, we observed that there is a need to actually stop the statistics framework at the end of the simulation, so that the time from the last event was recorded to the end of the simulation. That is, if the last state transition was at time t=1000 seconds, but the simulation ended at time t=2000 seconds, we need to ensure that this kind of report covers the time range of 1000-2000 seconds, so we need to stimulate somehow the report at time t=2000 seconds even if there is not an underlying state change to drive it. (Actually, '''this should be discussed further''': it could cause problems with reliability analysis -- it actually did in my experiments -- as it can happen that a certain object whose state changes depend on another object remains in started or stopped state until the end of simulation because that another object ended its state changing time. Sorry if this explanation isn't very clear; I'm willing to elaborate further if necessary. -- [[User:Vedranm\|Vedran Miletić]])

	== Development road map ==		== Development road map ==

Tomh: /* Prioritized list for ns-3.19 */

2013-07-31T14:25:44Z

Prioritized list for ns-3.19

← Older revision		Revision as of 14:25, 31 July 2013
Line 236:		Line 236:
	* Migrate Joe Kopena's data collection code and example to the new framework		* Migrate Joe Kopena's data collection code and example to the new framework
	** this will enable some SQLite database support		** this will enable some SQLite database support
			* Add ability to periodically write out interim results for long simulations

	=== Some core ns-3 issues ===		=== Some core ns-3 issues ===

Tomh: /* Development road map */

2013-07-29T19:06:55Z

Development road map

@@ Line 214: / Line 214: @@
 == Development road map ==
-''This section was last edited July 27, 2013''
+''This section was last edited July 29, 2013''
-We are hoping to merge the initial portion of data collection framework for ns-3.18 (August release).
+We are planning to merge the initial portion of data collection framework for ns-3.18 (August release).
 == Background and related work ==

Tomh: /* Use cases */

2013-07-29T18:47:49Z

Use cases

← Older revision		Revision as of 18:47, 29 July 2013
Line 154:		Line 154:
	This section describes a few use cases that the framework is intended to be able to support (not all capabilities are implemented).		This section describes a few use cases that the framework is intended to be able to support (not all capabilities are implemented).

	This supplements the discussion [Data_Collection_Framework#Informal_requirements \| above].		This supplements the discussion on informal requirements [[Data_Collection_Framework#Informal_requirements \| above]].

	=== interface bandwidth statistics ===		=== interface bandwidth statistics ===

	A user previously asked on ns-3-users:		A user previously asked on ns-3-users:


	Basically, what is suggested to take a percentage of how much		Basically, what is suggested to take a percentage of how much
Line 208:		Line 207:

	=== object start/stop time tracker ===		=== object start/stop time tracker ===

			Vedran asked whether the Object Start/Object Stop time tracker could be implemented with DCF. Basically, this is a variation on the requirement stated above for "Wifi Throughput"; what is desired is something like a "Duty Cycle collector" that will keep track of the proportion of time that an object was on or off, and report statistics at the end of the simulation.

			In discussing this use case, we observed that there is a need to actually stop the statistics framework at the end of the simulation, so that the time from the last event was recorded to the end of the simulation. That is, if the last state transition was at time t=1000 seconds, but the simulation ended at time t=2000 seconds, we need to ensure that this kind of report covers the time range of 1000-2000 seconds, so we need to stimulate somehow the report at time t=2000 seconds even if there is not an underlying state change to drive it.

	== Development road map ==		== Development road map ==

Tomh: /* Use cases */

2013-07-29T18:38:26Z

Use cases

← Older revision		Revision as of 18:38, 29 July 2013
Line 152:		Line 152:
	== Use cases ==		== Use cases ==

	This section describes a few use cases that the framework is intended to be able to support (not all capabilities are		This section describes a few use cases that the framework is intended to be able to support (not all capabilities are implemented).

			This supplements the discussion [Data_Collection_Framework#Informal_requirements \| above].

	=== interface bandwidth statistics ===		=== interface bandwidth statistics ===

Tomh: /* athstats helper */

2013-07-27T14:34:48Z

athstats helper

← Older revision		Revision as of 14:34, 27 July 2013
Line 179:		Line 179:
	There is an athstats helper class that prints out formatted text statistics of the Wifi NetDevice corresponding to what the athstats tool might print out:		There is an athstats helper class that prints out formatted text statistics of the Wifi NetDevice corresponding to what the athstats tool might print out:

	examples/wireless/wifi-ap.cc: AthstatsHelper athstats; examples/wireless/wifi-ap.cc: athstats.EnableAthstats ("athstats-sta", stas); examples/wireless/wifi-ap.cc: athstats.EnableAthstats ("athstats-ap", ap);		examples/wireless/wifi-ap.cc:
			AthstatsHelper athstats;
			examples/wireless/wifi-ap.cc: athstats.EnableAthstats ("athstats-sta", stas);
			examples/wireless/wifi-ap.cc: athstats.EnableAthstats ("athstats-ap", ap);

	This was written before the data collection framework existed. It consists of a helper class that mainly hooks a number of traces in the WifiNetDevice, collects statistics from them, and periodically writes the statistics out to file, and resets:		This was written before the data collection framework existed. It consists of a helper class that mainly hooks a number of traces in the WifiNetDevice, collects statistics from them, and periodically writes the statistics out to file, and resets:
Line 200:		Line 203:
	In the context of the data collection framework, this helper is analogous to a custom Collector object, that hooks directly to trace sources (without probes), and also contains FileAggregator support (that is, it is a combined Collector+Aggregator).		In the context of the data collection framework, this helper is analogous to a custom Collector object, that hooks directly to trace sources (without probes), and also contains FileAggregator support (that is, it is a combined Collector+Aggregator).

	The DCF way to write this would be as follows. First, if there were probes available for these trace signatures, they could be added, but this is not strictly necessary. The Athstats helper could be written still largely as a custom collector, but the file handling aspects could be handled perhaps by a stock file aggregator object, to which the specially formatted printf string format could be provided.		The DCF way to write this would be as follows. First, if there were probes available for these trace signatures, they could be added, but this is not strictly necessary. The Athstats helper could be written still largely as a custom collector, but the file handling aspects could be handled perhaps by a stock file aggregator object, to which the specially formatted printf string format could be provided.

	=== object start/stop time tracker ===		=== object start/stop time tracker ===

Tomh: dcf use cases

2013-07-27T14:33:57Z

dcf use cases

@@ Line 168: / Line 168: @@
 One could envision some kind of PointToPointHelper methods to print this information out, using BasicStatsCollector.
-   <nowiki>/* Plot sending and receiving throughput, averaged at 1 second
       intervals, in a Gnuplot */
-   PointToPointHelper::PlotInterfaceThroughput (Ptr<NetDevice> nd)  PointToPointHelper::PlotInterfaceThroughputAll (const NetDeviceContainer &ndc)  PointToPointHelper::WriteInterfaceThroughtput ...  /* File variant*/
+   PointToPointHelper::PlotInterfaceThroughput (Ptr<NetDevice> nd)
 === athstats helper ===
 === object start/stop time tracker ===