Bill Barth (mailto:bbarth@tacc.utexas.edu)
R. Todd Evans (mailto:rtevans@tacc.utexas.edu)
John Hammond (mailto:jhammond@tacc.utexas.edu)
Andy R. Terrel (mailto:aterrel@tacc.utexas.edu)
The tacc_stats repository consists of two complemetary components:
-
tacc_statsis a C based job-oriented and logically structured version of the conventional sysstat system monitor. -
job_pickles.pyis a Python based code that collects the raw stats data in a specified time range into a job-based pickled Python dictionary.
To get access to the tacc_stats source code
git clone https://github.com/billbarth/tacc_stats.git
Type these commands from the top of the tacc_stats
source directory to quickly build and install. The
executables will be placed in bin/.
$ mkdir build
$ cd build
$ ../do_configure.sh
$ make install
-
Introduction: The build system is based on CMake. It configures the C and Python routines for a particular computing platform. The configure process specifies both desired directories to store and read tacc_stats generated data from, and a list of device types from which to monitor.
-
Configure: All configuring should be specified in the
do_configure.shscript. The meaning of every field is specified in this script. The first part of the script specifies paths to locations where tacc_stats data is read and stored from. These will be system specific. The location of the python version to use, host name, and batch system should also be specified here. The second part of the script specifies which devices to monitor by building up a list labeledTYPES.TYPESthat are commented out will be ignored. If the system is missing anyTYPESunder the Chip types section, that type will automatically be skipped during the monitoring. All paths and types are set in the do_configure.sh. Once this script is set, make a directory to build the code.
This directory should be made in the top-level source directory, e.g.tacc_stats/. From this directorydo_configure.shshould then be called:$ mkdir build $ cd build $ ../do_configure.sh At this point the code will be configured for the system.
-
Build: From with the build directory type
make install. This will compile tacc_stats, then place the executabletacc_statsand all useful scripts into thebin/directory in the top level source directory. The python modules which support the executable scripts will be placed in theinclude/directory in the top level source directory. The executables typically used aretacc_stats,job_pickles.py, anddo_job_pickles_cron.sh.tacc_statsis typically invoked in the prologue of the batch system andcron_tabfile on each node.do_job_pickles.shis typically invoked from acron_tabfile and pickles jobs over the previous 24 hr period, storing the pickled data into thepickles_dirspecified indo_configure.sh.
tacc_stats has two complemetary components. The first component is a light-weight C
code called tacc_stats, initially called to configure Performance Monitoring Counter registers
for specific events before a job is begun. As the job is running the code is
repeatedly called to collect the counter registers values at regular time
intervals (specified for example in a cron_tab file). This counter data is
stored in "raw stats" files.
The second component is based the Python code job_pickles.py,
performed off line. The Python codes processes the raw stats files into
python dictionaries meant to ease analysis of the stats data.
tacc_stats can be run manually by:
$ tacc_stats begin jobid
$ tacc_stats collect
However, it is typically invoked by setting up cron scripts and prolog/epilog files as described in the example below, which corresponds to its usage on Stampede.
-
Invocation:
tacc_statsruns every 10-minutes (through cron), and at the beginning and end of every job (through SLURM prolog/epilog). In addition,tacc_statsmay be directly invoked by the user (or application) although we have not advertised this. -
Data Handling: On each invocation,
tacc_statscollects and records system statistics to a structured text file on ram backed storage local to the node. Stats files are rotated at every night at 23:55 localtime, and archived at sometime between 02:00--04:00 localtime to Stampede's/scratchfilesystem. A stats file created at epoch timeEPOCH, on nodeHOSTNAME, will be stored locally as/var/log/tacc_stats/EPOCH, and archived at/scratch/projects/tacc_stats/archive/HOSTNAME/EPOCH.gz. For example stats collected on Jun 14 2011 on c101-101, might correspond to files/var/log/tacc_stats/1308027601and/scratch/projects/tacc_stats/archive/c101-101.stampede.tacc.utexas.edu/1308027601.gz.
\warning Do not expect all stats files to be created at midnight exactly, or even approximately. As nodes are rebooted, new stats_files will be created as soon as a job begins or the cron task runs.
\warning Stats from a given job on a give host may span multiple files.
\warning Expect stats files to be missing occasionally, as nodes may crash before they can be archived. Since we use ram backed storage these files do not survive a reboot.
job_pickles.py can be run manually by:
$ ./job_pickles.py path_to_pickles/ date_start date_end
where the 3 required arguments have the following meaning
path_to_pickles/: the directory to store pickled dictionariesdate_start: the start of the date range, e.g."2013-09-25 04:00:00"date_end: the end of the date range, e.g."2013-09-26 05:00:00"
One could also run
$ ./do_job_pickles_cron.sh
to pickle all raw stats data in the 24 hour period yesterday to today. On Stampede
this script is invoked every 24 hours using a crontab file.
For pickling data with Intel Sandy Bridge core and uncore counters it is useful to
modify the event_map dictionaries in intel_snb.py to include whatever events you are counting. The dictionaries map a control register value to a Schema name.
You can have events in the event_map dictionaries that you are not counting,
but if missing an event it will be labeled in the Schema with it's control register
value.
A raw stats file consists of a multiline header, followed my one or more record groups. The first few lines of the header identify the version of tacc_stats, the FQDN of the host, it's uname, it's uptime in seconds, and other properties to be specified.
$tacc_stats 1.0.2
$hostname i101-101.ranger.tacc.utexas.edu
$uname Linux x86_64 2.6.18-194.32.1.el5_TACC #18 SMP Mon Mar 14 22:24:19 CDT 2011
$uptime 4753669
These are followed by schema descriptors for each of the types collected:
!amd64_pmc CTL0,C CTL1,C CTL2,C CTL3,C CTR0,E,W=48 CTR1,E,W=48 CTR2,E,W=48 CTR3,E,W=48
!cpu user,E,U=cs nice,E,U=cs system,E,U=cs idle,E,U=cs iowait,E,U=cs irq,E,U=cs softirq,E,U=cs
!lnet tx_msgs,E rx_msgs,E rx_msgs_dropped,E tx_bytes,E,U=B rx_bytes,E,U=B rx_bytes_dropped,E
!ps ctxt,E processes,E load_1 load_5 load_15 nr_running nr_threads
...
A schema descriptor consists of the character '!' followed by the type, followed by a space separated list of elements. Each element consists of a key name, followed by a comma-separated list of options; the options currently used are:
- E meaning that the counter is an event counter,
- W= meaning that the counter is wide (as opposed to 64),
- C meaning that the value is a control register, not a counter,
- U= meaning that the value is in units specified by .
Note especially the event and width options. Certain counters, such as the performance counters are subject to rollover, and as such their widths must be known for the values to be interpreted correctly.
\warning The archived stats files do not account for rollover. This task is left for postprocessing.
A record group consists of a blank line, a line containing the epoch time of the record and the current jobid, zero of more lines of marks (each starting with the % character), and several lines of statistics.
1307509201 1981063
%begin 1981063
amd64_pmc 11 4259958 4391234 4423427 4405240 235835341001110 187269740525248 62227761639015 177902917871843
amd64_pmc 10 4259958 4391234 4405239 4423427 221601328309784 187292967300939 47879507215852 174113618669738
amd64_pmc 13 4259958 4405238 4391234 4423427 211997466129346 215850892876689 2218837366391 233806061617899
amd64_pmc 12 4392928 4259958 4391234 4423427 6782043270201 102683296940807 2584394368284 174209034378272
...
cpu 11 429720418 0 1685980 43516346 447875 155 3443
cpu 10 429988676 0 1675476 43150935 559410 8 283
...
net ib0 0 0 55915434547 0 0 0 0 0 0 0 0 0 159301288 0 46963995550 0 0 97 0 0 0 31404022 0
...
ps - 4059349377 507410 1600 1600 1600 18 373
...
Each line of statistics contains the type (amd64_pmc, cpu, net, ps,...), the device (11,10,13,12,...,ib0,-...), followed by the counter values in the order given by the schema. Note that when we cannot meaningfully attach statistics to a device, we use '-' as the device name.
The TYPES that can be collected are:
amd64_pmc AMD Opteron performance counters (per core)
intel_nhm Intel Nehalem Processor (per core)
intel_uncore Westmere Uncore (per socket)
intel_snb Intel Sandy Brige Processor (per core)
intel_snb_cbo Caching Agent (CBo) (per socket)
intel_snb_pcu Power Control Unit (per socket)
intel_snb_imc Integrated Memory Controller (per socket)
intel_snb_qpi QPI Link Layer (per socket)
intel_snb_hau Home Agent Unit (per socket)
intel_snb_r2pci Ring to PCIe Agent (per socket)
ib Infiniband usage
ib_sw InfiniBand usage
ib_ext Infiniband usage
llite Lustre filesystem usage (per mount),
lnet Lustre network usage
mdc Lustre network usage
osc Lustre filesystem usage
block block device statistics (per device),
cpu scheduler accounting (per CPU),
mem memory usage (per socket)
net network device usage (per device)
nfs NFS system usage
numa weird NUMA statistics (per socket),
ps process statistics,
sysv_shm SysV shared memory segment usage,
tmpfs ram-backed filesystem usage (per mount),
vfs dentry/file/inode cache usage,
vm virtual memory statistics.
The TYPES to include in a build of tacc_stats are specified in
the do_configure.sh list TYPES. To add a new TYPE to tacc_stats,
write the appropriate TYPENAME.c file and place it in the monitor directory.
Then add the TYPENAME to the TYPES list.
For the keys associated with each TYPE, see the appropriate schema.
For the source and meanings of the counters, see the tacc_stats source
https://github.com/bbarth/tacc_stats, the CentOS 5.6 kernel source,
especially Documentation/*, and the manpages, especially proc(5).
I have not tracked down the meanings of all counters. However, if I
did (and it wasn't obvious from the counter name) then I put that
information in the source (see for example block.c).
All intel Sandy Bridge core and uncore counters are documented in detail
in their corresponding source code and via Doxygen, e.g. intel_snb.c.
Many processor-related performance counters are configurable using their
corresponding control registers. The use of these registers is described
in the source code and Doxygen.
\note All chip architecture related types are checked for existence at run time. Therefore, it is unnecessary for the user to filter for these types listed above - they will be filtered at run time. This should also work well for systems composed of multiple types of chip architectures.
\warning Due to a bug in Lustre, llite overreports read_bytes.
\warning Some event counters (from ib_sw, numa, and possibly others) suffer from occasional dips. This may be due to non-atomic accesses in the (kernel) code that presents the counter, a bug in tacc_stats, or some other condition. Spurious rollover is easy to detect, however, because a naive adjustment produced a riduculously large delta.
\warning We never reset counters, thus to determine the number of events that occurred during a job, you must subtract the value at begin from end.
\warning Due to a quirk in the Opteron performance counter
architecture, we do not assign the same set of events to each core,
see amd64_pmc.c in the tacc_stats source for details.
Pickled stats data will be placed in the directory specified by
pickles_dir. The pickled data is contained in a nested python
dictionary with the following key layers:
job : 1st key Job ID
host : 2nd key Host node used by Job ID
type : 3rd key TYPE specified in tacc_stats
device : 4th key device belonging to type
For example, to access Job ID 101's stats data on host c560-901 for
TYPE intel_snb for device cpu number 0 from within a python script:
pickle_file = open('101','r')
jobid = pickle.load(pickle_file)
pickle_file.close()
jobid['c560-901']['intel_snb']['0']
The value accessed by this key is a 2D array, with rows corresponding to record times and columns to specific counters for the device. To view the names for each counter add
jobid.get_schema('intel_snb')
or for a short version
jobid.get_schema('intel_snb').desc
(C) 2011 University of Texas at Austin
This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.
This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.
You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA