This repository has been archived by the owner on Jun 22, 2018. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathvxperf2.html
227 lines (218 loc) · 11.9 KB
/
vxperf2.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
<?xml version="1.0" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>vxperf2.pm</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<link rev="made" href="mailto:rurban@x-ray.at" />
</head>
<body style="background-color: white">
<!-- INDEX BEGIN -->
<div name="index">
<p><a name="__index__"></a></p>
<ul>
<li><a href="#name">NAME</a></li>
<li><a href="#synopsis">SYNOPSIS</a></li>
<li><a href="#description">DESCRIPTION</a></li>
<li><a href="#rules_file">RULES FILE</a></li>
<li><a href="#keywords">KEYWORDS</a></li>
<li><a href="#subroutines">SUBROUTINES</a></li>
<li><a href="#data_structures">DATA STRUCTURES</a></li>
<li><a href="#dependencies">DEPENDENCIES</a></li>
<li><a href="#limitations">LIMITATIONS</a></li>
<li><a href="#wishlist">WISHLIST</a></li>
<li><a href="#author">AUTHOR</a></li>
<li><a href="#credits">CREDITS</a></li>
<li><a href="#license_and_copyright">LICENSE AND COPYRIGHT</a></li>
<li><a href="#last_updated">LAST UPDATED</a></li>
</ul>
<hr name="index" />
</div>
<!-- INDEX END -->
<p>
</p>
<h1><a name="name">NAME</a></h1>
<p><strong>vxperf2</strong></p>
<p>
</p>
<hr />
<h1><a name="synopsis">SYNOPSIS</a></h1>
<pre>
use vxperf2;</pre>
<pre>
@database = &parseLogFile($logPath);
&applyRules($rulesFile, $resultsDir, \@database);
&applyRules($rulesFile2, $resultsDir2, \@database);</pre>
<pre>
@database1 = &parseLogFile($logPath1);
@database2 = &parseLogFile($logPath2);
@database3 = &parseLogFile($logPath3);
.
.
.
&applyRules($rulesFile, $resultsDir, \@database1, \@database2, \@database3, ...);</pre>
<p>
</p>
<hr />
<h1><a name="description">DESCRIPTION</a></h1>
<p><strong>vxperf2</strong> is a generic parser to analyze logs that are in a text database format. (e.g. Output generated by various OS monitoring utilities.) Using a rules file with six different keywords, the parser can perform several useful operations on one or more logs. It can summarize (basic functions count, max, min, sum, avg), visualize (plot), navigate (skip data points), zoom in (consider a subset of data points) and compare (within and across logs) any specified subset of fields of interest.</p>
<p>
</p>
<hr />
<h1><a name="rules_file">RULES FILE</a></h1>
<p>Below is an example rules file using all valid keywords. The first word and only the first word of any rule is treated as a keyword.</p>
<pre>
# This is a comment and the beginning of the rules file.
y-axis: columnA columnB columnC
y-axis: columnD columnE columnF columnG
z-axis: columnZ
plot: columnA columnB
plot: columnC columnE columnG
only: value1 value2
only: value3 value4
offset: n
points: m
# End of rules file.</pre>
<p>
</p>
<hr />
<h1><a name="keywords">KEYWORDS</a></h1>
<p><strong>vxperf2</strong> accepts six keywords: <strong>y-axis</strong>, <strong>z-axis</strong>, <strong>plot</strong>, <strong>only</strong>, <strong>offset</strong>, <strong>points</strong>. All lines which don't start with a keyword are ignored (comments).</p>
<p>There is no keyword "x-axis" because it is always fixed to the number of rows.</p>
<ul>
<li>
<p><strong>y-axis</strong>: A user might be interested in the summaries (count, sum, max, min, avg) related to specific fields, not all of them. Rules starting with <strong>y-axis</strong> list the fields to be considered for summarizing logs. Using the above example rules file <strong>vxperf2</strong> will summarize fields columnA, columnB, columnC, columnD, columnE, columnF and columnG.</p>
</li>
<li>
<p><strong>z-axis</strong>: At times a user is more interested in filtered logs than the whole. e.g. While collecting the output of Unix top (http://www.unixtop.org/), a user might actually be interested only in a few specific processes (specific PIDs or specific process names). The rule starting with <strong>z-axis</strong> contains the field whose values are treated as another dimension of the fields. Note that there can be only one <strong>z-axis</strong>. Only the first field in the first rule containing this keyword will be used. If valuez1, valuez2, valuez3 are distinct values of field columnZ then along with columnA, <strong>vxperf2</strong> will also summarize columnA-valuez1, columnA-valuez2, and columnA-valuez3. And so on for other fields.</p>
</li>
<li>
<p><strong>plot</strong>: A graph can at times convey more information than a few functions. Rules starting with <strong>plot</strong> list the fields which are to be plotted on the Y-axis (against the number of rows on the X-axis). All fields listed in one <strong>plot</strong> rule will be plotted in a single PNG image. A different PNG image will be created for each line starting with <strong>plot</strong>. <strong>vxperf2</strong> will plot columnA and columnB in one PNG image titled "columnA-columnB.png", and plot columnC, columnE and columnG in another PNG image titled "columnC-columnE-columnG.png".</p>
</li>
<li>
<p><strong>only</strong>: Plots often become cluttered with too many different curves, which is even more likely when a <strong>z-axis</strong> is specified. Rules starting with <strong>only</strong> list the values of <strong>z-axis</strong> which are to be plotted, and the remaining values will be discarded. <strong>vxperf2</strong> will plot columnA-value1, columnA-value2, columnB-value1, columnB-value2 in one PNG image titled "columnA-columnB.png". And so on for other plots.</p>
</li>
<li>
<p><strong>offset</strong>: The rule starting with <strong>offset</strong> contains the number of rows of each field from the beginning of a log to be ignored. Note that there can be only one <strong>offset</strong>. Only the first string in the first rule containing this keyword will be used. If one considers the plots, <strong>offset</strong> is a way to navigate along them.</p>
</li>
<li>
<p><strong>points</strong>: The rule starting with <strong>points</strong> contains the number of data points of each field of a log to be considered. Note that there can be only one <strong>points</strong>. Only the first string in the first rule containing this keyword will be used. If one considers the plots, <strong>points</strong> is a way to zoom into them.</p>
</li>
</ul>
<p>
</p>
<hr />
<h1><a name="subroutines">SUBROUTINES</a></h1>
<p>The <strong>vxperf2</strong> module contains three subroutines: <strong>equals</strong>, <strong>parseLogFile</strong> and <strong>applyRules</strong>.</p>
<ul>
<li>
<p><strong>equals</strong> takes the references to two arrays and returns 1 if the arrays are equal (length and values at each index) and 0 if they are not. This subroutine was needed internally for the module and thus implemented and thus exposed. Usage:</p>
<pre>
$boolean = &equals(\@array1, \@array2);</pre>
</li>
<li>
<p><strong>parseLogFile</strong> takes the path to a text file, parses it, creates and returns a database (an array of hashes) out of it. Each hash of the array corresponds to a line of the text file. Usage:</p>
<pre>
$database = &parseLogFile($logPath);</pre>
</li>
<li>
<p><strong>applyRules</strong> takes the path to a rules file, the path to a results directory, and references to one or more databases (arrays of hashes) created using the afore-mentioned <strong>parseLogFile</strong> subroutine. It parses the rules file, applies the extracted rules on the databases, writes the summaries into "$resultsDir/summary.txt" and the plots appropriately as PNG images in $resultsDir. Usage:</p>
<pre>
&applyRules($rulesFile, $resultsDir, \@database1, \@database2, \@database3, ...);</pre>
</li>
</ul>
<p>
</p>
<hr />
<h1><a name="data_structures">DATA STRUCTURES</a></h1>
<p>Apart from the basic Perl data types like integers, strings, arrays and hashes, <strong>vxperf2</strong> turned out to be a useful exercise in getting acquainted with some basic two-dimensional Perl data structures.</p>
<ul>
<li>
<p><strong>@tables</strong> is an array of arrays. Each element is a table name (sequence of fields) which corresponds to a unique header of a text file. Each table name is stored in the form of an array (list of fields).</p>
</li>
<li>
<p><strong>@database</strong> is an array of hashes. Each element is a row (hash) in one of the tables (fields as hash keys) which corresponds to a line in the text file (hash values).</p>
</li>
<li>
<p><strong>%ydata</strong> is a hash of arrays. Each element is a list of values corresponding to a field, and is used for plotting the graphs (against the number of rows).</p>
</li>
<li>
<p><strong>%functions</strong> is a hash of hashes. Each element is a hash (with functions as keys) which corresponds to a field.</p>
</li>
</ul>
<p>
</p>
<hr />
<h1><a name="dependencies">DEPENDENCIES</a></h1>
<p><strong>vxperf2</strong> uses the CPAN module <strong>Graphics::GnuplotIF</strong> (http://search.cpan.org/perldoc?Graphics::GnuplotIF) to plot various graphs, which provides a Perl API to <strong>gnuplot</strong> (http://sourceforge.net/projects/gnuplot).</p>
<p><strong>vxperf2.pl</strong> is a Perl wrapper around <strong>vxperf2</strong> which can be run from the command-line or called from non-Perl scripts.</p>
<p><strong>log2db</strong> which is also shipped with the tarball is not a dependency but a module to modify various standard monitoring logs into text databases, which can then be passed to <strong>vxperf2</strong>. <strong>log2db.pl</strong> is a Perl wrapper around <strong>log2db</strong> which can be run from the command-line or called from non-Perl scripts.</p>
<p>
</p>
<hr />
<h1><a name="limitations">LIMITATIONS</a></h1>
<p>Data integrity is assumed and never verified.</p>
<p><strong>vxperf2</strong> can reliably parse only files in a text database format. The format has a simple syntax: any line following one or more blank lines is considered a header (sequence of fields) of a table. All subsequent lines are considered rows of that table. Each field or value in any line is a "word" separated from other field(s) with whitespace.</p>
<p>Plotting is made assuming uniform intervals between any two rows of a certain field.</p>
<p>
</p>
<hr />
<h1><a name="wishlist">WISHLIST</a></h1>
<ul>
<li>
<p>HTML or some other ouptput type instead of a directory of files.</p>
</li>
<li>
<p>Single step, combining log2db and vxperf2, for a set of specific logs, using the logtype as input.</p>
</li>
<li>
<p>Better filenames and identifiers/labels in summary and plots.</p>
</li>
<li>
<p>Option to identify all tables and their fields in logs, and possible summarizing all of them.</p>
</li>
<li>
<p>Multiple rules with the keyword <strong>only</strong>.</p>
</li>
<li>
<p>More functions to summarize data.</p>
</li>
<li>
<p>Arithmetic expressions on y-axes, on z-values.</p>
</li>
<li>
<p>More options for plotting.</p>
</li>
<li>
<p>Detect correlation patterns between fields (y-axes).</p>
</li>
<li>
<p>Plotting against custom x-axes, and showing absolute time where possible.</p>
</li>
<li>
<p>BUG: Keyword <strong>offset</strong> doesn't consider rules defined with keyword <strong>only</strong>.</p>
</li>
</ul>
<p>
</p>
<hr />
<h1><a name="author">AUTHOR</a></h1>
<p><strong>Sravan Bhamidipati</strong> <a href="mailto:(Sravan_Bhamidipati@symantec.com,">(Sravan_Bhamidipati@symantec.com,</a> <a href="mailto:bsravanin@gmail.com)">bsravanin@gmail.com)</a></p>
<p>
</p>
<hr />
<h1><a name="credits">CREDITS</a></h1>
<p>Dhrubojyoti Biswas contributed to the design discussions. Shrinivas Chandukar contributed various ideas through the original vxperf (written mostly using Awk).</p>
<p>
</p>
<hr />
<h1><a name="license_and_copyright">LICENSE AND COPYRIGHT</a></h1>
<p>MIT License: <a href="http://www.opensource.org/licenses/mit-license.php">http://www.opensource.org/licenses/mit-license.php</a>
Symantec Corporation: <a href="http://www.symantec.com">http://www.symantec.com</a></p>
<p>
</p>
<hr />
<h1><a name="last_updated">LAST UPDATED</a></h1>
<p>4th January, 2012</p>
</body>
</html>