Intel® Trace Analyzer and Collector User and Reference Guide

Download PDF
ID 767272
Date 3/31/2023
Public
Document Table of Contents
Document Table of Contents x
Intel® Trace Analyzer and Collector User and Reference Guide
Intel® Trace Analyzer and Collector User and Reference Guide x
Introduction Install and Set Up Intel® Trace Analyzer and Collector Trace Your Applications Analyze Your Applications Intel® Trace Collector Reference Intel® Trace Analyzer Reference Notices and Disclaimers
Introduction x
Notational Conventions Get Help
Trace Your Applications x
Tracing Conventional MPI Applications Tracing Failing MPI Applications Tracing OpenSHMEM* Applications Tracing MPI File IO Handling of Communicator Names Tracing MPI Load Imbalance Tracing User Defined Events Configuring the Collector Filtering Trace Data Recording OpenMP* Regions Information Tracing System Calls (Linux* OS) Collecting Lightweight Statistics Recording Source Location Information Recording Hardware Performance Information (Linux* OS) Recording Operating System Counters Tracing Library Calls Correctness Checking Tracing Distributed Non-MPI Applications
Correctness Checking x
Correctness Checking of MPI Applications Running with Valgrind* (Linux* OS) Configuring Error Checks Analyzing the Results Debugger Integration
Debugger Integration x
TotalView* Debugger GNU* Symbolic Debugger Allinea* Distributed Debugging Tool* (DDT*)
Analyze Your Applications x
Starting Intel® Trace Analyzer Intel Trace Analyzer Graphical User Interface Navigating Timelines Concepts Viewing Correctness Checking Reports Comparing Two Trace Files Interoperability with Intel® VTune™ Profiler and Intel® Advisor OpenMP* Regions Display Support OTF2 Format Support
Navigating Timelines x
Zoom Stack
Concepts x
Level of Detail Aggregation Advanced Aggregation Tagging and Filtering
Viewing Correctness Checking Reports x
Event Timeline Correctness Checking Reports Qualitative Timeline Correctness Checking Reports Detailed Dialog
Comparing Two Trace Files x
Mappings in Comparison Views Comparison Charts
Mappings in Comparison Views x
Mapping of Processes Mapping of Functions
Comparison Charts x
Comparison Function Profile Comparison Message Profile Comparison Collective Operations Profile
Intel® Trace Collector Reference x
API Reference Configuration Reference Correctness Checking Errors Structured Tracefile Format stftool Utility Time Stamping Secure Loading of Dynamic Link Libraries* on Windows* OS
API Reference x
Initialization, Termination and Control Defining and Recording Source Locations Defining and Recording Functions or Regions Defining and Recording Scopes Defining Groups of Processes Defining and Recording Counters Recording Communication Events Additional API Calls in libVTcs C++ API
Initialization, Termination and Control x
VT_initialize VT_finalize VT_getrank VT_registerthread VT_registernamed VT_registerprefixed VT_getthrank VT_traceon VT_traceoff VT_tracestate VT_symstate VT_flush VT_timestamp VT_timestart VT_setfinalizecallback VT_getdescription VT_countsetcallback
Defining and Recording Functions or Regions x
New Interface Old Interface State Changes
C++ API x
VT_FuncDef Class Reference VT_SclDef Class Reference VT_Function Class Reference VT_Region Class Reference
Configuration Reference x
Configuration File Format Protocol File Configuration Options
Configuration Options x
ACTIVITY ALTSTACK AUTOFLUSH CHECK CHECK-LEAK-REPORT-SIZE CHECK-MAX-DATATYPES CHECK-MAX-ERRORS CHECK-MAX-PENDING CHECK-MAX-REPORTS CHECK-MAX-REQUESTS CHECK-SUPPRESSION-LIMIT CHECK-TIMEOUT CHECK-TRACING CLUSTER COMPRESS-RAW-DATA COUNTER CURRENT-DIR DEADLOCK-TIMEOUT DEADLOCK-WARNING DEMANGLE DETAILED-STATES ENTER-USERCODE ENVIRONMENT EXTENDED-VTF FLUSH-PID FLUSH-PREFIX GROUP HANDLE-SIGNALS INTERNAL-MPI KEEP-RAW-EVENTS LOGFILE-FORMAT LOGFILE-NAME LOGFILE-PREFIX LOGFILE-RANK MEM-BLOCKSIZE MEM-FLUSHBLOCKS MEM-INFO MEM-MAXBLOCKS MEM-MINBLOCKS MEM-OVERWRITE NMCMD OS-COUNTER-DELAY PCTRACE PCTRACE-CACHE PCTRACE-FAST PLUGIN PROCESS PROGNAME PROTOFILE-NAME STATISTICS STATE STF-PROCS-PER-FILE STF-USE-HW-STRUCTURE STOPFILE-NAME SYMBOL SYNC-MAX-DURATION SYNC-MAX-MESSAGES SYNC-PERIOD SYNCED-CLUSTER SYNCED-HOST TIME-WINDOWS (Experimental) TIMER TIMER-SKIP UNIFY-COUNTERS UNIFY-GROUPS UNIFY-SCLS UNIFY-SYMBOLS VERBOSE VT_START_PAUSED VT_COMPRESS_TRACE
Correctness Checking Errors x
Supported Errors How the Collection Works
How the Collection Works x
Parameter Checking Premature Exit Overlapping Memory Detecting Illegal Buffer Modifications Buffer Given to MPI Cannot Be Read or Written Distributed Memory Checking Illegal Memory Access Request Handling Datatype Handling Buffered Sends Deadlocks Checking Message Transmission Datatype Mismatches Data Modified during Transmission Checking Collective Operations Freeing Communicators
Structured Tracefile Format x
STF Components Single-File STF Configuring STF
stftool Utility x
stftool Utility Options Expanded ASCII output of STF Files
Time Stamping x
Clock Synchronization Choosing a Timer
Choosing a Timer x
gettimeofday/_ftime QueryPerformanceCounter CPU Cycle Counter Normalized CPU Cycle Counter MPI_Wtime() High Precision Event Timers POSIX* clock_gettime
Intel® Trace Analyzer Reference x
Graphical User Interface Reference Intel® Trace Analyzer Command Line Interface Reference Filter Expression Grammar otf2-to-stf Utility
Graphical User Interface Reference x
Welcome Page Summary Page Main Menu Bar View Menu Bar View Bars Charts Dialogs Settings
Main Menu Bar x
File Menu Options Menu Project Menu  Windows Menu Help Menu
View Menu Bar x
View Charts Navigate Advanced Layout Comparison Menu
Advanced x
Tagging Specific Events Filtering Events Simulating Ideal Communication Checking Application Imbalance Aggregating Results Aggregating Functions Creating Command Line for Intel® VTune™ Profiler and Intel® Advisor
View Bars x
Toolbar Trace Map Status Bar
Charts x
Event Timeline Qualitative Timeline Quantitative Timeline Counter Timeline Function Profile Message Profile Collective Operations Profile Performance Assistant Common Chart Features
Event Timeline x
Context Menu Filtering and Tagging
Qualitative Timeline x
Context Menu Filtering and Tagging
Quantitative Timeline x
Context Menu Filtering and Tagging
Counter Timeline x
Context Menu Filtering and Tagging
Function Profile x
Flat Profile Load Balance Call Tree Call Graph Context Menu Filtering and Tagging Function Profile Settings
Message Profile x
Context Menu Filtering and Tagging Aggregation Message Profile Settings
Collective Operations Profile x
Context Menu Filtering and Tagging Collective Operations Profile Settings
Dialogs x
Process Aggregation Function Aggregation Function Group Color Editor Filtering Dialog Box Tagging Dialog Box Idealization Dialog Box Imbalance Diagram Dialog Box Trace Merge Dialog Box  Details Dialog Box Source View Dialog Time Interval Selection Configuration Dialogs Find Dialog Box Command line for Intel® VTune™ Profiler and Intel® Advisor Dialog Box OTF2 to STF Conversion Dialog Box Configuration Assistant
Process Aggregation x
Comparison Mode
Function Aggregation x
Comparison Mode
Filtering Dialog Box x
Building Filter Expressions Using Graphical Interface Processes Tab Functions Tab Messages Collective Operations Building Filter Expressions Manually Filter Expressions in Comparison Mode
Details Dialog Box x
Detailed Attributes of Function Events Detailed Attributes of Message Events Detailed Attributes of Collective Operation Events
Configuration Dialogs x
Load Configuration File Dialog Edit Configuration File Dialog
Settings x
Preferences Font Settings Number Formatting Settings
Preferences x
General Preferences Tracefile Preferences Event Timeline Settings Qualitative Timeline Settings Quantitative Timeline Settings Counter Timeline Settings
Notices and Disclaimers x
Appendix A Copyright and Licenses
Intel® Trace Analyzer and Collector User and Reference Guide

Building Filter Expressions Using Graphical Interface

Use the Filtering dialog box to specify filter expressions separately for function events, messages and collective operations under the respective tabs. You can also use the Processes tab to restrict the events to a subset of processes.

Some of the text fields in the filter dialog box can take triplets that are of the form start:stop:incr or the short form start:stop if the increment is one. Such a triplet describes all numbers greater than or equal to start and smaller than or equal to stop and that are of the form start+incr*n. The stop value is optional, too. When stop is omitted, all numbers, beginning from start, match.

Processes Tab

Use this tab to specify the processes to pass the filter. This tab has effects on all three sub-expressions: Functions, Messages and Collective Operations.

Use This: To Do This:
Show Processes

List the desired processes in the text field to see them in the resulting view and filter out all the other processes.

The text field can contain a comma-separated list of process ids, process group ids, triplets thereof, unquoted or double-quoted names of threads, processes or process groups. Names given match all equally named processes/groups.

Three-dot-labeled button Open the Process Group Selection dialog box and select one or more processes or process groups.
Invert

Swap the selection.

For example, if you want all processes except one to pass the filter, select the process to be filtered out and use the Invert check box. It is a logical NOT and it is marked with an exclamation mark before the predicate in the filter expression.

See the resulting filter expression below the filter clauses. To reuse the expression elsewhere, select and copy the expression in the manual mode of the filtering dialog box.

The context menu of the filter expression contains a Copy (Ctrl+C) entry and a Select All (Ctrl+A) entry.

Functions Tab

In the Functions tab, use the All, None and Custom radio buttons to select the mode of filtering.

To enable the filter clause tab, select the Custom radio button. The filter clause consists of the following entries:

Entry Description
Functions

Use this tab to add functions to the filter.

The text field can contain a comma-separated list of functions or function group ids, triplets thereof, unquoted or double-quoted names of functions or function groups that describe functions passing the filter. All given names match all equally named functions/groups.

Press the three-dot-labeled button to open the Function Group Selection sub-dialog and choose the function names from the list.

Check the Add Function Id check box to write the name and Id of a selected function into the text field. This is useful to resolve ambiguities of function/group names: if only the name is written into the text field, all functions with this name pass the filter; if the name is replaced by an id, only the function with this id passes the filter, not other functions with the same name.

Processes

Use this tab to add processes to the filter.

The text field can contain a comma-separated list of process ids, process group ids, triplets thereof, unquoted or double-quoted thread, process or process group names that describe processes and threads in the functions passing the filter. All given names match all equally named processes/groups.

Press the three-dot-labeled button to choose processes from a list.

In the Process Group Selection sub-dialog, check the Add Process Id check box to write the process name and Id into the text field.

To add another filter clause, press the Add New Clause button. To remove an existing filter clause tab, use the Remove Current Clause button. Clauses are connected by a logical OR, while attributes from the same tab are connected by a logical AND.

Messages

To enable the filter clause tab in the Messages tab, select the Custom radio button. The filter clause tab has the following entries:

Entry Description
Communicator

The text field can contain a comma-separated list of communicator ids, unquoted communicator names or communicator names in double quotes that pass the filter.

Press the three-dot-labeled button to choose from the list.

Tag The text field can contain a comma-separated list of non-negative integers and triplets that describe tag values that pass the filter.
Processes Makes a message pass the filter if either the sender or the receiver matches. Analogous to the logical OR of Sender and Receiver.
Sender

The text field can contain a comma-separated list of process ids, process group ids, triplets thereof, unquoted or double-quoted names of threads, processes or process groups that describe processes and threads in the message sender that make the message pass the filter. Given names match all equally named processes/groups.

Press the three-dot-labeled button to choose from the list.

Receiver Analogous to Sender.
Ranks Makes a message pass if either the sender or the receiver matches. Analogous to the logical OR of Rank of Sender and Rank of Receiver.
Ranks of Sender The text field can contain a comma-separated list of nonnegative integers and triplets that describe sender ranks (in the MPI communicator) that can pass the filter.
Ranks of Receiver Analogous to Rank of Sender.
Message Size The text field can contain a comma-separated list of non-negative integers and triplets that describe message sizes (in bytes) that pass the filter.
Start Time

The text field can contain a comma-separated list of non-negative integers and triplets that describe start time (in ticks) of the message that makes the operation pass the filter.

Press the three-dot-labeled button to enter/edit the time in ticks or seconds (default depending on the View current time unit).

End Time Analogous to Start Time.
Duration

The text field can contain a comma-separated list of non-negative integers and triplets that describe the duration (in ticks) of the message that makes the operation pass the filter.

Press the three-dot-labeled button to enter/edit the duration (shown as a time interval) in ticks or seconds (default depending on the View current time unit).

Collective Operations

To enable the filter clause tab in the Collective Operations tab, select the Custom radio button. The filter clause tab has the following entries:

Entry Description
Communicator

The text field can contain a comma-separated list of communicator ids, unquoted communicator names or communicator names in double quotes that pass the filter.

Press the three-dot-labeled button to choose from the list.

Collective Operation

The text field can contain a comma-separated list of unquoted or double-quoted names of collective operations that pass the filter.

Press the three-dot-labeled button to choose from the list.

Processes

The text field can contain a comma-separated list of process ids, process group ids, triplets thereof, unquoted or double-quoted names of threads, processes or process groups that pass the filter. Given names match all equally named processes/groups.

Press the three-dot-labeled button to choose from the list.

Root

The text field can contain a comma-separated list of process ids, process group ids, triplets thereof, unquoted or double-quoted names of threads, processes or process groups that describe processes and threads serving as Root in the operation that passes the filter. Given names match all equally named processes/groups.

Press the three-dot-labeled button to choose from a list.

Rank of Root The text field can contain a comma-separated list of non-negative integers and triplets that describe root ranks of the operation that passes the filter.
Transferred Volume The text field can contain a comma-separated list of nonnegative integers and triplets that describe all volumes (in total bytes per operation) that make a collective operation pass the filter.
Start Time

The text field can contain a comma-separated list of non-negative integers and triplets that describe start time (in ticks) of the operation that passes the filter.

Press the three-dot-labeled button to enter/edit the time in ticks or seconds (default depending on the View current time unit).

End Time Analogous to Start Time.
Duration

The text field can contain a comma-separated list of non-negative integers and triplets that describe the duration (in ticks) of the operation that passes the filter.

Press the three-dot-labeled button to enter/edit the duration (shown as a time interval) in ticks or seconds (default depending on the View current time unit).

Parent topic: Filtering Dialog Box
Level Two Title