Intel® Trace Analyzer and Collector User and Reference Guide

Download PDF
ID 767272
Date 3/31/2023
Public
Document Table of Contents
Document Table of Contents x
Intel® Trace Analyzer and Collector User and Reference Guide
Intel® Trace Analyzer and Collector User and Reference Guide x
Introduction Install and Set Up Intel® Trace Analyzer and Collector Trace Your Applications Analyze Your Applications Intel® Trace Collector Reference Intel® Trace Analyzer Reference Notices and Disclaimers
Introduction x
Notational Conventions Get Help
Trace Your Applications x
Tracing Conventional MPI Applications Tracing Failing MPI Applications Tracing OpenSHMEM* Applications Tracing MPI File IO Handling of Communicator Names Tracing MPI Load Imbalance Tracing User Defined Events Configuring the Collector Filtering Trace Data Recording OpenMP* Regions Information Tracing System Calls (Linux* OS) Collecting Lightweight Statistics Recording Source Location Information Recording Hardware Performance Information (Linux* OS) Recording Operating System Counters Tracing Library Calls Correctness Checking Tracing Distributed Non-MPI Applications
Correctness Checking x
Correctness Checking of MPI Applications Running with Valgrind* (Linux* OS) Configuring Error Checks Analyzing the Results Debugger Integration
Debugger Integration x
TotalView* Debugger GNU* Symbolic Debugger Allinea* Distributed Debugging Tool* (DDT*)
Analyze Your Applications x
Starting Intel® Trace Analyzer Intel Trace Analyzer Graphical User Interface Navigating Timelines Concepts Viewing Correctness Checking Reports Comparing Two Trace Files Interoperability with Intel® VTune™ Profiler and Intel® Advisor OpenMP* Regions Display Support OTF2 Format Support
Navigating Timelines x
Zoom Stack
Concepts x
Level of Detail Aggregation Advanced Aggregation Tagging and Filtering
Viewing Correctness Checking Reports x
Event Timeline Correctness Checking Reports Qualitative Timeline Correctness Checking Reports Detailed Dialog
Comparing Two Trace Files x
Mappings in Comparison Views Comparison Charts
Mappings in Comparison Views x
Mapping of Processes Mapping of Functions
Comparison Charts x
Comparison Function Profile Comparison Message Profile Comparison Collective Operations Profile
Intel® Trace Collector Reference x
API Reference Configuration Reference Correctness Checking Errors Structured Tracefile Format stftool Utility Time Stamping Secure Loading of Dynamic Link Libraries* on Windows* OS
API Reference x
Initialization, Termination and Control Defining and Recording Source Locations Defining and Recording Functions or Regions Defining and Recording Scopes Defining Groups of Processes Defining and Recording Counters Recording Communication Events enum _VT_CommIDs VT_log_sendmsg VT_log_recvmsg VT_log_msgevent VT_log_op VT_log_opevent enum _VT_OpTypes VT_begin_unordered VT_end_unordered Additional API Calls in libVTcs C++ API
Initialization, Termination and Control x
VT_initialize VT_finalize VT_getrank VT_registerthread VT_registernamed VT_registerprefixed VT_getthrank VT_traceon VT_traceoff VT_tracestate VT_symstate VT_flush VT_timestamp VT_timestart VT_setfinalizecallback VT_getdescription VT_countsetcallback
Defining and Recording Functions or Regions x
New Interface Old Interface State Changes
C++ API x
VT_FuncDef Class Reference VT_SclDef Class Reference VT_Function Class Reference VT_Region Class Reference
Configuration Reference x
Configuration File Format Protocol File Configuration Options
Configuration Options x
ACTIVITY ALTSTACK AUTOFLUSH CHECK CHECK-LEAK-REPORT-SIZE CHECK-MAX-DATATYPES CHECK-MAX-ERRORS CHECK-MAX-PENDING CHECK-MAX-REPORTS CHECK-MAX-REQUESTS CHECK-SUPPRESSION-LIMIT CHECK-TIMEOUT CHECK-TRACING CLUSTER COMPRESS-RAW-DATA COUNTER CURRENT-DIR DEADLOCK-TIMEOUT DEADLOCK-WARNING DEMANGLE DETAILED-STATES ENTER-USERCODE ENVIRONMENT EXTENDED-VTF FLUSH-PID FLUSH-PREFIX GROUP HANDLE-SIGNALS INTERNAL-MPI KEEP-RAW-EVENTS LOGFILE-FORMAT LOGFILE-NAME LOGFILE-PREFIX LOGFILE-RANK MEM-BLOCKSIZE MEM-FLUSHBLOCKS MEM-INFO MEM-MAXBLOCKS MEM-MINBLOCKS MEM-OVERWRITE NMCMD OS-COUNTER-DELAY PCTRACE PCTRACE-CACHE PCTRACE-FAST PLUGIN PROCESS PROGNAME PROTOFILE-NAME STATISTICS STATE STF-PROCS-PER-FILE STF-USE-HW-STRUCTURE STOPFILE-NAME SYMBOL SYNC-MAX-DURATION SYNC-MAX-MESSAGES SYNC-PERIOD SYNCED-CLUSTER SYNCED-HOST TIME-WINDOWS (Experimental) TIMER TIMER-SKIP UNIFY-COUNTERS UNIFY-GROUPS UNIFY-SCLS UNIFY-SYMBOLS VERBOSE VT_START_PAUSED VT_COMPRESS_TRACE
Correctness Checking Errors x
Supported Errors How the Collection Works
How the Collection Works x
Parameter Checking Premature Exit Overlapping Memory Detecting Illegal Buffer Modifications Buffer Given to MPI Cannot Be Read or Written Distributed Memory Checking Illegal Memory Access Request Handling Datatype Handling Buffered Sends Deadlocks Checking Message Transmission Datatype Mismatches Data Modified during Transmission Checking Collective Operations Freeing Communicators
Structured Tracefile Format x
STF Components Single-File STF Configuring STF
stftool Utility x
stftool Utility Options Expanded ASCII output of STF Files
Time Stamping x
Clock Synchronization Choosing a Timer
Choosing a Timer x
gettimeofday/_ftime QueryPerformanceCounter CPU Cycle Counter Normalized CPU Cycle Counter MPI_Wtime() High Precision Event Timers POSIX* clock_gettime
Intel® Trace Analyzer Reference x
Graphical User Interface Reference Intel® Trace Analyzer Command Line Interface Reference Filter Expression Grammar otf2-to-stf Utility
Graphical User Interface Reference x
Welcome Page Summary Page Main Menu Bar View Menu Bar View Bars Charts Dialogs Settings
Main Menu Bar x
File Menu Options Menu Project Menu  Windows Menu Help Menu
View Menu Bar x
View Charts Navigate Advanced Layout Comparison Menu
Advanced x
Tagging Specific Events Filtering Events Simulating Ideal Communication Checking Application Imbalance Aggregating Results Aggregating Functions Creating Command Line for Intel® VTune™ Profiler and Intel® Advisor
View Bars x
Toolbar Trace Map Status Bar
Charts x
Event Timeline Qualitative Timeline Quantitative Timeline Counter Timeline Function Profile Message Profile Collective Operations Profile Performance Assistant Common Chart Features
Event Timeline x
Context Menu Filtering and Tagging
Qualitative Timeline x
Context Menu Filtering and Tagging
Quantitative Timeline x
Context Menu Filtering and Tagging
Counter Timeline x
Context Menu Filtering and Tagging
Function Profile x
Flat Profile Load Balance Call Tree Call Graph Context Menu Filtering and Tagging Function Profile Settings
Message Profile x
Context Menu Filtering and Tagging Aggregation Message Profile Settings
Collective Operations Profile x
Context Menu Filtering and Tagging Collective Operations Profile Settings
Dialogs x
Process Aggregation Function Aggregation Function Group Color Editor Filtering Dialog Box Tagging Dialog Box Idealization Dialog Box Imbalance Diagram Dialog Box Trace Merge Dialog Box  Details Dialog Box Source View Dialog Time Interval Selection Configuration Dialogs Find Dialog Box Command line for Intel® VTune™ Profiler and Intel® Advisor Dialog Box OTF2 to STF Conversion Dialog Box Configuration Assistant
Process Aggregation x
Comparison Mode
Function Aggregation x
Comparison Mode
Filtering Dialog Box x
Building Filter Expressions Using Graphical Interface Building Filter Expressions Manually Filter Expressions in Comparison Mode
Details Dialog Box x
Detailed Attributes of Function Events Detailed Attributes of Message Events Detailed Attributes of Collective Operation Events
Configuration Dialogs x
Load Configuration File Dialog Edit Configuration File Dialog
Settings x
Preferences Font Settings Number Formatting Settings
Preferences x
General Preferences Tracefile Preferences Event Timeline Settings Qualitative Timeline Settings Quantitative Timeline Settings Counter Timeline Settings
Notices and Disclaimers x
Appendix A Copyright and Licenses
Intel® Trace Analyzer and Collector User and Reference Guide

Recording Communication Events

These are API calls that allow logging of message send and receive and MPI-style collective operations. Because they are modeled after MPI operations, they use the same kind of communicator to define the context for the operation.

enum _VT_CommIDs

Logging send/receive events evaluates the rank local within the active communicator, and matches events only if they take place in the same communicator (in other words, it is the same behavior as in MPI).

Defining new communicators is not supported, but the predefined ones can be used.

Enumerator Description
VT_COMM_INVALID Invalid ID, do not pass to Intel® Trace Collector
VT_COMM_WORLD Global ranks are the same as local ones
VT_COMM_SELF Communicator that only contains the active process

VT_log_sendmsg

int VT_log_sendmsg(int other_rank, int count, 
	 int tag, int commid, int sclhandle)

Description

Logs sending of a message.

Fortran

VTLOGSENDMSG(other_rank, count, tag, commid, 
	 sclhandle, ierr)

Parameters

my_rank rank of the sending process
other_rank rank of the target process
count number of bytes sent
tag tag of the message
commid numeric ID for the communicator (VT_COMM_WORLD, VT_COMM_SELF, or see VT_commdef())
sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code

VT_log_recvmsg

int VT_log_recvmsg(int other_rank, int count, int tag, int commid, int sclhandle)

Description

Logs receiving of a message.

Fortran

VTLOGRECVMSG(other_rank, count, tag, commid, 
	 sclhandle, ierr)

Parameters

my_rank rank of the receiving process
other_rank rank of the source process
count number of bytes sent
tag tag of the message
commid numeric ID for the communicator (VT_COMM_WORLD, VT_COMM_SELF, or see VT_commdef())
sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code

The next three calls require a little extra care, because they generate events that not only have a time stamp, but also a duration. This means that you need to take a time stamp first, then do the operation and finally log the event.

VT_log_msgevent

int VT_log_msgevent(int sender, int receiver, int count, int tag, int commid, double sendts, int sendscl, int recvscl)

Description

Logs sending and receiving of a message.

Fortran

VTLOGMSGEVENT(sender, receiver, count, tag, commid, sendts, sendscl, recvscl, ierr)

Parameters

sender rank of the sending process
receiver rank of the target process
count number of bytes sent
tag tag of the message
commid numeric ID for the communicator (VT_COMM_WORLD, VT_COMM_SELF, or see VT_commdef())
sendts time stamp obtained with VT_timestamp()
sendscl handle as defined by VT_scldef() for the source code location where the message was sent, or VT_NOSCL
recvscl the same for the receive location

Return values

Returns error code

VT_log_op

int VT_log_op(int opid, int commid, int root, int bsend, int brecv, double startts, int sclhandle)

Description

Logs the duration and amount of transferred data of an operation for one process.

Fortran

VTLOGOP(opid, commid, root, bsend, brecv, startts, sclhandle, ierr)

Parameters

opid id of the operation; must be one of the predefined constants in enum _VT_OpTypes
commid numeric ID for the communicator; see VT_log_sendmsg() for valid numbers
root rank of the root process in the communicator (ignored for operations without root, must still be valid, though)
bsend bytes sent by process (ignored for operations that send no data)
brecv bytes received by process (ignored for operations that receive no data)
startts the start time of the operation (as returned by VT_timestamp())
sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code

VT_log_opevent

int VT_log_opevent(int opid, int commid, int root, int numprocs, int _ bsend, int _ brecv, double _ startts, int sclhandle)

Description

Logs the duration and amount of transferred data of an operation for all involved processes at once.

Intel® Trace Collector knows which processes send and receive data in each operation. Unused byte counts are ignored when writing the trace, so they can be left uninitialized, but NULL is not allowed as array address even if no entry is used at all.

Fortran

VTLOGOPEVENT(opid, commid, root, numprocs, bsend, brecv, startts, sclhandle, ierr)

Parameters

opid id of the operation; must be one of the predefined constants in enum _VT_OpTypes
commid numeric ID for the communicator; see VT_log_sendmsg() for valid numbers
root rank of the root process in the communicator (ignored for operations without root, must still be valid, though)
numprocs the number of processes in the communicator
bsend bytes sent by process
brecv bytes received by process
startts the start time of the operation (as returned by VT_timestamp())
sclhandle handle as defined by VT_scldef, or VT_NOSCL

Return values

Returns error code

enum _VT_OpTypes

These are operation IDs that can be passed to VT_log_op().

Their representation in the trace file matches that of the equivalent MPI operation.

User-defined operations are not supported.

Enumerator Description 
VT_OP_INVALID
 Undefined operation, should not be passed to Intel® Trace Collector 
VT_OP_COUNT
 Number of predefined operations 
VT_OP_BARRIER
VT_OP_IBARRIER
VT_OP_BCAST
VT_OP_IBCAST
VT_OP_GATHER
VT_OP_IGATHER
VT_OP_GATHERV
VT_OP_IGATHERV
VT_OP_SCATTER
VT_OP_ISCATTER
VT_OP_SCATTERV
VT_OP_ISCATTERV
VT_OP_ALLGATHER
VT_OP_IALLGATHER
VT_OP_ALLGATHERV
VT_OP_IALLGATHERV
VT_OP_ALLTOALL
VT_OP_IALLTOALL
VT_OP_ALLTOALLV
VT_OP_IALLTOALLV
VT_OP_ALLTOALLW
VT_OP_IALLTOALLW
VT_OP_NEIGHBOR_ALLGATHER
VT_OP_INEIGHBOR_ALLGATHER
VT_OP_NEIGHBOR_ALLGATHERV
VT_OP_INEIGHBOR_ALLGATHERV
VT_OP_NEIGHBOR_ALLTOALL
VT_OP_INEIGHBOR_ALLTOALL
VT_OP_NEIGHBOR_ALLTOALLV
VT_OP_INEIGHBOR_ALLTOALLV
VT_OP_NEIGHBOR_ALLTOALLW
VT_OP_INEIGHBOR_ALLTOALLW
VT_OP_REDUCE
VT_OP_REDUCE_LOCAL
VT_OP_IREDUCE
VT_OP_ALLREDUCE
VT_OP_IALLREDUCE
VT_OP_REDUCE_SCATTER
VT_OP_REDUCE_SCATTER_BLOCK
VT_OP_IREDUCE_SCATTER
VT_OP_IREDUCE_SCATTER_BLOCK
VT_OP_SCAN
VT_OP_ISCAN
VT_OP_EXSCAN
VT_OP_IEXSCAN
 MPI operation representations

Having a duration also may introduce the problem of having overlapping operations, which has to be taken care of with the following two calls.

VT_begin_unordered

int VT_begin_unordered(void)

Description

Starts a period with out-of-order events.

Most API functions log events with just one time stamp which is taken when the event is logged. That guarantees strict chronological order of the events.

VT_log_msgevent() and VT_log_opevent() are logged when the event has finished with a start time taken earlier with VT_timestamp(). This can break the chronological order, for example, like in the following two examples: 

t1: VT_timestamp() "start message"
t2: VT_end() "leave function"
t3: VT_log_msgevent( t1 ) "finish message"
t1: VT_timestamp() "start first message"
t2: VT_timestamp() "start second message"
t3: VT_log_msgevent( t1 ) "finish first message"
t4: VT_log_msgevent( t2 ) "finish second message"

In other words, it is okay to just log a complex event if and only if no other event is logged between its start and end in this thread. "logged" in this context includes other complex events that are logged later, but with a start time between the other events start and end time.

In all other cases you have to alert Intel® Trace Collector of the fact that out-of-order events will follow by calling VT_begin_unordered() before and VT_end_unordered() after these events. When writing the events into the trace file Intel® Trace Collector increases a counter per thread when it sees a VT_begin_unordered() and decrease it at a VT_end_unordered(). Events are remembered and sorted until the counter reaches zero, or till the end of the data.

This means that:

  • unordered periods can be nested

  • it is not necessary to close each unordered period at the end of the trace

  • but not closing them properly in the middle of a trace will force Intel® Trace Collector to use a lot more memory when writing the trace (proportional to the number of events till the end of the trace).

Fortran

VTBEGINUNORDERED(ierr)

VT_end_unordered

int VT_end_unordered (void)

Description

Close a period with out-of-order events that was started with VT_begin_unordered().

Fortran

VTENDNORDERED(ierr)
Parent topic: API Reference
Level Two Title