Intel® Trace Analyzer and Collector User and Reference Guide

Download PDF
ID 767272
Date 3/31/2023
Public
Document Table of Contents
Document Table of Contents x
Intel® Trace Analyzer and Collector User and Reference Guide
Intel® Trace Analyzer and Collector User and Reference Guide x
Introduction Install and Set Up Intel® Trace Analyzer and Collector Trace Your Applications Analyze Your Applications Intel® Trace Collector Reference Intel® Trace Analyzer Reference Notices and Disclaimers
Introduction x
Notational Conventions Get Help
Trace Your Applications x
Tracing Conventional MPI Applications Tracing Failing MPI Applications Tracing OpenSHMEM* Applications Tracing MPI File IO Handling of Communicator Names Tracing MPI Load Imbalance Tracing User Defined Events Configuring the Collector Filtering Trace Data Recording OpenMP* Regions Information Tracing System Calls (Linux* OS) Collecting Lightweight Statistics Recording Source Location Information Recording Hardware Performance Information (Linux* OS) Recording Operating System Counters Tracing Library Calls Correctness Checking Tracing Distributed Non-MPI Applications
Correctness Checking x
Correctness Checking of MPI Applications Running with Valgrind* (Linux* OS) Configuring Error Checks Analyzing the Results Debugger Integration
Debugger Integration x
TotalView* Debugger GNU* Symbolic Debugger Allinea* Distributed Debugging Tool* (DDT*)
Analyze Your Applications x
Starting Intel® Trace Analyzer Intel Trace Analyzer Graphical User Interface Navigating Timelines Concepts Viewing Correctness Checking Reports Comparing Two Trace Files Interoperability with Intel® VTune™ Profiler and Intel® Advisor OpenMP* Regions Display Support OTF2 Format Support
Navigating Timelines x
Zoom Stack
Concepts x
Level of Detail Aggregation Advanced Aggregation Tagging and Filtering
Viewing Correctness Checking Reports x
Event Timeline Correctness Checking Reports Qualitative Timeline Correctness Checking Reports Detailed Dialog
Comparing Two Trace Files x
Mappings in Comparison Views Comparison Charts
Mappings in Comparison Views x
Mapping of Processes Mapping of Functions
Comparison Charts x
Comparison Function Profile Comparison Message Profile Comparison Collective Operations Profile
Intel® Trace Collector Reference x
API Reference Configuration Reference Correctness Checking Errors Structured Tracefile Format stftool Utility Time Stamping Secure Loading of Dynamic Link Libraries* on Windows* OS
API Reference x
Initialization, Termination and Control Defining and Recording Source Locations Defining and Recording Functions or Regions Defining and Recording Scopes Defining Groups of Processes Defining and Recording Counters Recording Communication Events Additional API Calls in libVTcs C++ API
Initialization, Termination and Control x
VT_initialize VT_finalize VT_getrank VT_registerthread VT_registernamed VT_registerprefixed VT_getthrank VT_traceon VT_traceoff VT_tracestate VT_symstate VT_flush VT_timestamp VT_timestart VT_setfinalizecallback VT_getdescription VT_countsetcallback
Defining and Recording Functions or Regions x
New Interface Old Interface State Changes
C++ API x
VT_FuncDef Class Reference VT_SclDef Class Reference VT_Function Class Reference VT_Region Class Reference
Configuration Reference x
Configuration File Format Protocol File Configuration Options
Configuration Options x
ACTIVITY ALTSTACK AUTOFLUSH CHECK CHECK-LEAK-REPORT-SIZE CHECK-MAX-DATATYPES CHECK-MAX-ERRORS CHECK-MAX-PENDING CHECK-MAX-REPORTS CHECK-MAX-REQUESTS CHECK-SUPPRESSION-LIMIT CHECK-TIMEOUT CHECK-TRACING CLUSTER COMPRESS-RAW-DATA COUNTER CURRENT-DIR DEADLOCK-TIMEOUT DEADLOCK-WARNING DEMANGLE DETAILED-STATES ENTER-USERCODE ENVIRONMENT EXTENDED-VTF FLUSH-PID FLUSH-PREFIX GROUP HANDLE-SIGNALS INTERNAL-MPI KEEP-RAW-EVENTS LOGFILE-FORMAT LOGFILE-NAME LOGFILE-PREFIX LOGFILE-RANK MEM-BLOCKSIZE MEM-FLUSHBLOCKS MEM-INFO MEM-MAXBLOCKS MEM-MINBLOCKS MEM-OVERWRITE NMCMD OS-COUNTER-DELAY PCTRACE PCTRACE-CACHE PCTRACE-FAST PLUGIN PROCESS PROGNAME PROTOFILE-NAME STATISTICS STATE STF-PROCS-PER-FILE STF-USE-HW-STRUCTURE STOPFILE-NAME SYMBOL SYNC-MAX-DURATION SYNC-MAX-MESSAGES SYNC-PERIOD SYNCED-CLUSTER SYNCED-HOST TIME-WINDOWS (Experimental) TIMER TIMER-SKIP UNIFY-COUNTERS UNIFY-GROUPS UNIFY-SCLS UNIFY-SYMBOLS VERBOSE VT_START_PAUSED VT_COMPRESS_TRACE
Correctness Checking Errors x
Supported Errors How the Collection Works
How the Collection Works x
Parameter Checking Premature Exit Overlapping Memory Detecting Illegal Buffer Modifications Buffer Given to MPI Cannot Be Read or Written Distributed Memory Checking Illegal Memory Access Request Handling Datatype Handling Buffered Sends Deadlocks Checking Message Transmission Datatype Mismatches Data Modified during Transmission Checking Collective Operations Freeing Communicators
Structured Tracefile Format x
STF Components Single-File STF Configuring STF
stftool Utility x
stftool Utility Options Expanded ASCII output of STF Files
Time Stamping x
Clock Synchronization Choosing a Timer
Choosing a Timer x
gettimeofday/_ftime QueryPerformanceCounter CPU Cycle Counter Normalized CPU Cycle Counter MPI_Wtime() High Precision Event Timers POSIX* clock_gettime
Intel® Trace Analyzer Reference x
Graphical User Interface Reference Intel® Trace Analyzer Command Line Interface Reference Filter Expression Grammar otf2-to-stf Utility
Graphical User Interface Reference x
Welcome Page Summary Page Main Menu Bar View Menu Bar View Bars Charts Dialogs Settings
Main Menu Bar x
File Menu Options Menu Project Menu  Windows Menu Help Menu
View Menu Bar x
View Charts Navigate Advanced Layout Comparison Menu
Advanced x
Tagging Specific Events Filtering Events Simulating Ideal Communication Checking Application Imbalance Aggregating Results Aggregating Functions Creating Command Line for Intel® VTune™ Profiler and Intel® Advisor
View Bars x
Toolbar Trace Map Status Bar
Charts x
Event Timeline Qualitative Timeline Quantitative Timeline Counter Timeline Function Profile Message Profile Collective Operations Profile Performance Assistant Common Chart Features
Event Timeline x
Context Menu Filtering and Tagging
Qualitative Timeline x
Context Menu Filtering and Tagging
Quantitative Timeline x
Context Menu Filtering and Tagging
Counter Timeline x
Context Menu Filtering and Tagging
Function Profile x
Flat Profile Load Balance Call Tree Call Graph Context Menu Filtering and Tagging Function Profile Settings
Message Profile x
Context Menu Filtering and Tagging Aggregation Message Profile Settings Preferences Colors Data See Also
Collective Operations Profile x
Context Menu Filtering and Tagging Collective Operations Profile Settings
Dialogs x
Process Aggregation Function Aggregation Function Group Color Editor Filtering Dialog Box Tagging Dialog Box Idealization Dialog Box Imbalance Diagram Dialog Box Trace Merge Dialog Box  Details Dialog Box Source View Dialog Time Interval Selection Configuration Dialogs Find Dialog Box Command line for Intel® VTune™ Profiler and Intel® Advisor Dialog Box OTF2 to STF Conversion Dialog Box Configuration Assistant
Process Aggregation x
Comparison Mode
Function Aggregation x
Comparison Mode
Filtering Dialog Box x
Building Filter Expressions Using Graphical Interface Building Filter Expressions Manually Filter Expressions in Comparison Mode
Details Dialog Box x
Detailed Attributes of Function Events Detailed Attributes of Message Events Detailed Attributes of Collective Operation Events
Configuration Dialogs x
Load Configuration File Dialog Edit Configuration File Dialog
Settings x
Preferences Font Settings Number Formatting Settings
Preferences x
General Preferences Tracefile Preferences Event Timeline Settings Qualitative Timeline Settings Quantitative Timeline Settings Counter Timeline Settings
Notices and Disclaimers x
Appendix A Copyright and Licenses
Intel® Trace Analyzer and Collector User and Reference Guide

Message Profile Settings

The Settings dialog box has three subsets: Preferences, Colors and Data.

Preferences

Use the Preferences to configure Message Profile display and layout.

Setting Description
Row Labels Display row headers
Column Labels Display column headers
Scale Display the colored scale next to the matrix
Grid Control display of black grid lines between cells
Keep Empty Rows/Columns when using Sender/Receiver Groupings

Control display of empty rows and columns.

This feature is only relevant for the groupings Sender and Receiver.

When enabled, all processes are shown. For example, empty rows and columns will be displayed. It keeps the form of the matrix constant making it easier to see patterns in the data.

When disabled, empty rows and columns are not shown in these groupings.

All other groupings suppress empty rows and columns to save screen space regardless of the state of this check box.

Communicator Names

Display helpful communicator names (if available in the trace file).

Displaying communicator names may take a lot of valuable screen space.

Communicator Ids Display only concise communicator ids
Automatic Cell Sizes Adjust cell sizes to make all text readable
Manual Cell Sizes

Specify the size of the cells either in the Cell Size group at the bottom of the tab.

In this mode, the alphanumerical data in the cells is displayed only if it fits or if it is switched off entirely by un-checking the check box Text in Cells.

Colors

In this tab you can change the default colors in which the messages in the Message Profile are shown.

The Message Profile Setting provide the following options:

Setting Description
Steps

Specify the number of color steps (1-255).

The chosen colors are considered as points in a color space. The scale colors are interpolated on a line through color space connecting these two points.

Maximum Color Choose the colors for the maximum attribute values
Minimum Color Choose the colors for the minimum attribute values
HSV/RG

Choose between HSV and RGB color space.

HSV is more fancy and colorful, but RGB is often more useful and readable.

For monochrome printing, you will get a better result if you choose a very light and a very dark color. For example, white for the minimum and black for the maximum.

Manual Scaling

When enabled you will need to specify the minimum and maximum values for the color scale in the two text input fields below.

This is very convenient when you compare two Message Profile Charts that may live in different Views.

Data

The Grouping of data is controlled by indicating what appears in rows and columns. Note that not all combinations are possible. For example, you cannot set the same grouping for both rows and columns and you cannot have Sender/Receiver at one axis and one of the Sender or Receiver on the other axis.

The available groupings are:

Grouping Description
Sender

Categorizes the messages by Sender.

The exact labels are defined by the current thread group that is given by the View (see the Views section).

Receiver

Categorizes the messages by Receiver.

The exact labels are defined by the current process group that is given by the View.

Sender/Receiver

Categorizes the messages by Sender/Receiver pairs.

The exact labels are defined by the current process group that is given by the View.

Tag Categorizes the messages by the MPI tag assigned to the message by the program at the sender side.
Communicator Categorizes the messages by the MPI communicator. The labels are either communicator ids or names. Names are displayed if they are available in the trace file and if they are chosen in the Preferences tab of the Message Profile Settings dialog box.
Volume

Categorizes the messages by their Volume; for example, size in bytes.

Grouping Volume by Receiver shows only messages with a volume of 2000 bytes.

Sending Function

Categorizes the messages by the function that sends them.

Labels are names of MPI functions such as MPI_Irsend. This categorization is not influenced by the current Function Aggregation.

This information is only available with traces created by the Intel® Trace Collector version 6 and higher.

Receiving Function

Categorize the messages by the function that receives them.

Labels are names of MPI functions like MPI_Waitany. This is not influenced by the current Function Aggregation.

This information is only available with traces created by the Intel® Trace Collector version 6 and higher.

Datum group

Allows choosing which attribute should be printed or painted in the cells.

See the description of the attributes.

Row Statistics group Allows switching the individual columns on or off. These columns hold the statistics for the rows.
Column Statistics group Allows switching the individual rows on or off. These hold the statistics for the columns.

Datum Attributes Description
Total Time

The total travel time of the messages, accumulated over all messages that fall into this cell.

The unit is either [s] or [tick] depending on the View setting.

MinimumTime

The minimum travel time of a message, minimized over all messages that fall into this cell.

The unit is either [s] or [tick] depending on the View setting.

Maximum Time

The maximum travel time of a message, maximized over all messages that fall into this cell.

The unit is either [s] or [tick] depending on the View setting.

Average Transfer Rate, [B/s]

The average transfer rate, averaged over the transfer rates of all messages that fall into this cell.

Messages are not weighted; for example, transfer rates of short messages have the same impact as transfer rates of long messages.

Minimum Transfer Rate, [B/s] The minimum transfer rate, minimized over all messages that fall into this cell.
Maximum Transfer Rate, [B/s] The maximum transfer rate, maximized over all messages that fall into this cell.
Total Data Volume, [B] The total data volume, accumulated over all messages that fall into this cell.
Minimum Data Volume, [B] The minimum data volume, minimized over all messages that fall into this cell.
Maximum Data Volume, [B] The maximum data volume, maximized over all messages that fall into this cell.
Count, [1] The number of messages that fall into this cell.

Example of grouping Volume by Receiver:

See Also

Views

Parent topic: Message Profile
Level Two Title