Intel® Trace Analyzer and Collector User and Reference Guide

Download PDF
ID 767272
Date 3/31/2023
Public
Document Table of Contents
Document Table of Contents x
Intel® Trace Analyzer and Collector User and Reference Guide
Intel® Trace Analyzer and Collector User and Reference Guide x
Introduction Install and Set Up Intel® Trace Analyzer and Collector Trace Your Applications Analyze Your Applications Intel® Trace Collector Reference Intel® Trace Analyzer Reference Notices and Disclaimers
Introduction x
Notational Conventions Get Help
Trace Your Applications x
Tracing Conventional MPI Applications Tracing Failing MPI Applications Tracing OpenSHMEM* Applications Tracing MPI File IO Handling of Communicator Names Tracing MPI Load Imbalance Tracing User Defined Events Configuring the Collector Filtering Trace Data Recording OpenMP* Regions Information Tracing System Calls (Linux* OS) Collecting Lightweight Statistics Recording Source Location Information Recording Hardware Performance Information (Linux* OS) Recording Operating System Counters Tracing Library Calls Correctness Checking Tracing Distributed Non-MPI Applications
Correctness Checking x
Correctness Checking of MPI Applications Running with Valgrind* (Linux* OS) Configuring Error Checks Analyzing the Results Debugger Integration
Debugger Integration x
TotalView* Debugger GNU* Symbolic Debugger Allinea* Distributed Debugging Tool* (DDT*)
Analyze Your Applications x
Starting Intel® Trace Analyzer Intel Trace Analyzer Graphical User Interface Navigating Timelines Concepts Viewing Correctness Checking Reports Comparing Two Trace Files Interoperability with Intel® VTune™ Profiler and Intel® Advisor OpenMP* Regions Display Support OTF2 Format Support
Navigating Timelines x
Zoom Stack
Concepts x
Level of Detail Aggregation Advanced Aggregation Tagging and Filtering
Viewing Correctness Checking Reports x
Event Timeline Correctness Checking Reports Qualitative Timeline Correctness Checking Reports Detailed Dialog
Comparing Two Trace Files x
Mappings in Comparison Views Comparison Charts
Mappings in Comparison Views x
Mapping of Processes Mapping of Functions
Comparison Charts x
Comparison Function Profile Comparison Message Profile Comparison Collective Operations Profile
Intel® Trace Collector Reference x
API Reference Configuration Reference Correctness Checking Errors Structured Tracefile Format stftool Utility Time Stamping Secure Loading of Dynamic Link Libraries* on Windows* OS
API Reference x
Initialization, Termination and Control Defining and Recording Source Locations Defining and Recording Functions or Regions Defining and Recording Scopes Defining Groups of Processes Defining and Recording Counters Recording Communication Events Additional API Calls in libVTcs C++ API
Initialization, Termination and Control x
VT_initialize VT_finalize VT_getrank VT_registerthread VT_registernamed VT_registerprefixed VT_getthrank VT_traceon VT_traceoff VT_tracestate VT_symstate VT_flush VT_timestamp VT_timestart VT_setfinalizecallback VT_getdescription VT_countsetcallback
Defining and Recording Functions or Regions x
New Interface Old Interface State Changes
C++ API x
VT_FuncDef Class Reference VT_SclDef Class Reference VT_Function Class Reference VT_Region Class Reference
Configuration Reference x
Configuration File Format Protocol File Configuration Options
Configuration Options x
ACTIVITY ALTSTACK AUTOFLUSH CHECK CHECK-LEAK-REPORT-SIZE CHECK-MAX-DATATYPES CHECK-MAX-ERRORS CHECK-MAX-PENDING CHECK-MAX-REPORTS CHECK-MAX-REQUESTS CHECK-SUPPRESSION-LIMIT CHECK-TIMEOUT CHECK-TRACING CLUSTER COMPRESS-RAW-DATA COUNTER CURRENT-DIR DEADLOCK-TIMEOUT DEADLOCK-WARNING DEMANGLE DETAILED-STATES ENTER-USERCODE ENVIRONMENT EXTENDED-VTF FLUSH-PID FLUSH-PREFIX GROUP HANDLE-SIGNALS INTERNAL-MPI KEEP-RAW-EVENTS LOGFILE-FORMAT LOGFILE-NAME LOGFILE-PREFIX LOGFILE-RANK MEM-BLOCKSIZE MEM-FLUSHBLOCKS MEM-INFO MEM-MAXBLOCKS MEM-MINBLOCKS MEM-OVERWRITE NMCMD OS-COUNTER-DELAY PCTRACE PCTRACE-CACHE PCTRACE-FAST PLUGIN PROCESS PROGNAME PROTOFILE-NAME STATISTICS STATE STF-PROCS-PER-FILE STF-USE-HW-STRUCTURE STOPFILE-NAME SYMBOL SYNC-MAX-DURATION SYNC-MAX-MESSAGES SYNC-PERIOD SYNCED-CLUSTER SYNCED-HOST TIME-WINDOWS (Experimental) TIMER TIMER-SKIP UNIFY-COUNTERS UNIFY-GROUPS UNIFY-SCLS UNIFY-SYMBOLS VERBOSE VT_START_PAUSED VT_COMPRESS_TRACE
Correctness Checking Errors x
Supported Errors How the Collection Works
How the Collection Works x
Parameter Checking Premature Exit Overlapping Memory Detecting Illegal Buffer Modifications Buffer Given to MPI Cannot Be Read or Written Distributed Memory Checking Illegal Memory Access Request Handling Datatype Handling Buffered Sends Deadlocks Checking Message Transmission Datatype Mismatches Data Modified during Transmission Checking Collective Operations Freeing Communicators
Structured Tracefile Format x
STF Components Single-File STF Configuring STF
stftool Utility x
stftool Utility Options Expanded ASCII output of STF Files
Time Stamping x
Clock Synchronization Choosing a Timer
Choosing a Timer x
gettimeofday/_ftime QueryPerformanceCounter CPU Cycle Counter Normalized CPU Cycle Counter MPI_Wtime() High Precision Event Timers POSIX* clock_gettime
Intel® Trace Analyzer Reference x
Graphical User Interface Reference Intel® Trace Analyzer Command Line Interface Reference Filter Expression Grammar Formal Description of Grammar Examples of Advanced Usage of Grammar otf2-to-stf Utility
Graphical User Interface Reference x
Welcome Page Summary Page Main Menu Bar View Menu Bar View Bars Charts Dialogs Settings
Main Menu Bar x
File Menu Options Menu Project Menu  Windows Menu Help Menu
View Menu Bar x
View Charts Navigate Advanced Layout Comparison Menu
Advanced x
Tagging Specific Events Filtering Events Simulating Ideal Communication Checking Application Imbalance Aggregating Results Aggregating Functions Creating Command Line for Intel® VTune™ Profiler and Intel® Advisor
View Bars x
Toolbar Trace Map Status Bar
Charts x
Event Timeline Qualitative Timeline Quantitative Timeline Counter Timeline Function Profile Message Profile Collective Operations Profile Performance Assistant Common Chart Features
Event Timeline x
Context Menu Filtering and Tagging
Qualitative Timeline x
Context Menu Filtering and Tagging
Quantitative Timeline x
Context Menu Filtering and Tagging
Counter Timeline x
Context Menu Filtering and Tagging
Function Profile x
Flat Profile Load Balance Call Tree Call Graph Context Menu Filtering and Tagging Function Profile Settings
Message Profile x
Context Menu Filtering and Tagging Aggregation Message Profile Settings
Collective Operations Profile x
Context Menu Filtering and Tagging Collective Operations Profile Settings
Dialogs x
Process Aggregation Function Aggregation Function Group Color Editor Filtering Dialog Box Tagging Dialog Box Idealization Dialog Box Imbalance Diagram Dialog Box Trace Merge Dialog Box  Details Dialog Box Source View Dialog Time Interval Selection Configuration Dialogs Find Dialog Box Command line for Intel® VTune™ Profiler and Intel® Advisor Dialog Box OTF2 to STF Conversion Dialog Box Configuration Assistant
Process Aggregation x
Comparison Mode
Function Aggregation x
Comparison Mode
Filtering Dialog Box x
Building Filter Expressions Using Graphical Interface Building Filter Expressions Manually Filter Expressions in Comparison Mode
Details Dialog Box x
Detailed Attributes of Function Events Detailed Attributes of Message Events Detailed Attributes of Collective Operation Events
Configuration Dialogs x
Load Configuration File Dialog Edit Configuration File Dialog
Settings x
Preferences Font Settings Number Formatting Settings
Preferences x
General Preferences Tracefile Preferences Event Timeline Settings Qualitative Timeline Settings Quantitative Timeline Settings Counter Timeline Settings
Notices and Disclaimers x
Appendix A Copyright and Licenses
Intel® Trace Analyzer and Collector User and Reference Guide

Filter Expression Grammar

The filter expression grammar creates a filter, which is a set of function (funcfilter), message (p2pfilter), and collective operation (collfilter) filters, each defining a filter for its respective kind of data. These sub-filter specifications are separated by the # sign and come in any order. Each filter class is specified either once, or more than once (in this case a Boolean AND is created from all subfilters for a given class), or not specified at all. An example where three classes of filters are specified is the expression generated in the graphical interface.

Each filter class specifier (funcfilter, p2pfilter, or collfilter) is followed by an expression put in parentheses. That expression can consist of any number of predicates that are different for each filter class and correspond to the entries described in the section Building Filter Expressions Using the Graphical Interface. These predicates are joined by using Boolean AND (&&) and OR (||) operators. Boolean expressions are parenthesized as needed. Also, a Boolean NOT (!) operator in front of any predicate or parenthesized expression negates the predicate/expression.

A filter class enables you to define a special expression to pass all or no data through the filter. For example, p2pfilter(NONE) filters out all messages, while collfilter(ALL) passes all collective operation. When the keywords ALL or NONE are used, ensure that it is the only argument to funcfilter, p2pfilter, or collfilter.

Keyword specification in the filter expression grammar is case-insensitive. Specifying names (for functions, processes and communicators, among others), however, is case-sensitive. Use double quotes for names that consist of several words or do not start with a letter or an underscore character (for example, "Major Function Groups"). Use double quotes for single word names (for example, "MPI") if necessary. White space (space and tab characters, as well as newlines) is ignored, unless it is part of a quoted name. If a process/group or function/group name is ambiguous, it is evaluated as if all matching groups were given.

Formal Description of Grammar

Here is a formal description of the filter expression grammar:

# The filter itself
FILTER ::= AFILTER
| FILTER # AFILTER
AFILTER ::= funcfilter ( FUNCFILTARG )
| collfilter ( COLLFILTARG )
| p2pfilter ( P2PFILTARG )
# Specifying functions
FUNCFILTARG ::= FUNCEXPR
| all
| none
FUNCEXPR ::= FUNCATOM
| FUNCEXPR && FUNCATOM
| FUNCEXPR || FUNCATOM
FUNCATOM ::= TG
| FG
| STARTTIME
| ( FUNCEXPR )
| ! FUNCATOM
# Specifying messages
P2PFILTARG ::= P2PEXPR
| all
| none
P2PEXPR ::= P2PATOM
| P2PEXPR && P2PATOM
| P2PEXPR || P2PATOM
P2PATOM ::= DURATION
| COMM
| TAG
| P2PVOLUME
| TGSENDER
| TGRECEIVER
| COMMSENDER
| COMMRECEIVER
| TGSRPAIR
| COMMSRPAIR
| TG
| COMMSR
| STARTTIME
| ENDTIME
| SENDER_FG
| RECEIVER_FG
| ( P2PEXPR )
| ! P2PATOM
# Specifying collective operations
COLLFILTARG ::= COLLEXPR
| all
| none
COLLEXPR ::= COLLATOM
| COLLEXPR && COLLATOM
| COLLEXPR || COLLATOM
COLLATOM ::= DURATION
| COMM
| COLLOPTYPE
| COLLVOLUME
| TGROOT
| COMMROOT
| TG
| STARTTIME
| ENDTIME
| ( COLLEXPR )
| ! COLLATOM
# Specifying times
STARTTIME ::= start ( TRIPLETS ; INTEGER )
| start ( TRIPLETS )
ENDTIME ::= end ( TRIPLETS ; INTEGER )
| end ( TRIPLETS )
DURATION ::= duration ( TRIPLETS )
# Specifying TGroups and FGroups
TG ::= tg ( NAMES )
FG ::= fg ( NAMES )
SENDER_FG ::= send_fg ( NAMES )
RECEIVER_FG ::= recv_fg ( NAMES )
# Specifying collective operation and message properties
COMM ::= comm ( TRIPLETS )
TAG ::= tag ( TRIPLETS )
COLLOPTYPE ::= type ( COLLNAMES )
COLLVOLUME ::= volume ( TRIPLETS )
P2PVOLUME ::= volume ( TRIPLETS )
# Specifying root, sender, or receiver, either by a TGroup name
# or by position in the communicator. If the operation has no
# root, then root() is always false.
TGROOT ::= root ( NAMES )
TGSENDER ::= sender ( NAMES )
TGRECEIVER ::= receiver ( NAMES )
# The predicate sr specifies both sender and receiver, separated
# by a semicolon.
TGSRPAIR ::= sr ( NAMES ; NAMES )
COMMROOT ::= root@ ( TRIPLETS )
COMMSENDER ::= sender@ ( TRIPLETS )
COMMRECEIVER ::= receiver@ ( TRIPLETS )
# The predicate sr@ specifies both sender and receiver ranks,
# separated by a semicolon.
COMMSRPAIR ::= sr@ ( TRIPLETS ; TRIPLETS )
COMMSR ::= tg@ ( TRIPLETS )
# Names containing fancy characters have to be double-quoted.
# Names map to TGroup and thread names, FGroup and function
# names, or collective operation types, depending on the context.
NAMES ::= NAME 
| TRIPLET
| NAMES , NAME
| NAMES , TRIPLET
COLLNAMES ::= COLLNAMELIST
| TRIPLETS
COLLNAMELIST ::= NAME
| COLLNAMELIST , NAME
NAME ::= [_a-zA-Z][_a-zA-Z0-9.]*
| \"[^\"]*\"
# Specifying triplets and numbers
TRIPLETS ::= TRIPLET
| TRIPLETS , TRIPLET
TRIPLET ::= INTEGER
| INTEGER :
| INTEGER : INTEGER
| INTEGER : INTEGER : INTEGER
INTEGER ::= [0-9]+

Examples of Advanced Usage of Grammar

This section includes several examples of the manual usage of the filter expression grammar and how the manual usage of the grammar provides advanced capabilities in filtering trace data and speeding up the process of selecting exactly what you want to analyze.

For the first example, consider a parenthesized structure that cannot be built in the graphical interface easily (messages sent by process 0 and starting or ending between 70000 and 80000 ticks):

p2pfilter( sender( 0 ) && ( start( 70000:80000 ) || end( 70000:80000 ) ) )

The following example uses the sr predicate, which is not available in the graphical interface. This predicate helps to efficiently filter out all messages between processes 0 and 1:

p2pfilter( ! sr( 0:1; 0:1 ) )

Finally, consider the following scenario. With the graphical interface, a complicated filter is specified for a certain filter class with a large number of predicates and Boolean operators (both AND and OR, the latter added by using the Add New Clause button). To negate everything that has been specified so far (that is, to get exactly the trace data that was previously being filtered out), use ! in front of the whole expression when in the manual mode. For example, the filter below specifies MPI_Barrier collective operations that last no longer than 2000 ticks, plus all collective operations with process 0 as the root:

collfilter( type( MPI_Barrier ) && duration( 0:2000 ) || root( 0 ) )

while the following filter specifies all the collective operations that do not match the description above:

collfilter( ! ( type( MPI_Barrier ) && duration( 0:2000 ) || root( 0 ) ) )
Parent topic: Intel® Trace Analyzer Reference
Level Two Title