/*
* @file:   README
* CVS:     $Id$
* @author: Heike Jagode (jagode@icl.utk.edu)
* @author: Tony Castaldo (tonycastaldo@icl.utk.edu) & 
           Jiali Li (jli111@vols.utk.edu)
* @defgroup papi_components Components
* @brief Component Specific Readme file: pcp
*/


General Information
-------------------

+++ This component interfaces PAPI and the Performance Co-Pilot;
    for the Performance Metrics Domain agent: pmda=perfevent
...
+++ Purpose: Enable monitoring of P9 'nest' events via PCP without
             elevated privilege.
...
+++ P9 nest counters: any number of counters can be monitored
                      at any time. Meaning, there is no limitation
                      similar to P9 core counter monitoring, which
                      is limited to the number of counters (on P9
                      there are 6 counters: 4 programmable and
                      2 fixed (cycles, instructions completed).
...
+++ Repository: bitbucket/icl/papi/src/components/pcp


Code Specific Information
-------------------------

This code was tested on both Saturn and ORNL peak (Power9; P9).
------------------------------------------------------------------

The code handles an arbitrary number of PCP events; the necessary
tables are realloc()'ed as needed to accomodate any number of
events.   
------------------------------------------------------------------

The code is caching; because the communications with the PCP
daemon can be costly. This increases the complexity of the code
and memory footprint, but we strive to read event names, numerical
IDs, descriptors and domain data only once from the daemon, and
only as needed to support the PAPI user.
------------------------------------------------------------------

The P9 nest events contain both counters and "dutycycle" events.
The dutycycle events return instantaneous double precision
floating point values (between 0.0 and 1.0) that cannot be
'reset'. A PAPI_get_event_info() on these events will return, in
PAPI_event_info_t, a 'timescope' of PAPI_TIMESCOPE_POINT. Counters
will have a PAPI_TIMESCOPE_SINCE_START.  
------------------------------------------------------------------

The P9 PCP is installed without any "help text" descriptions of
the counters. The Saturn version of PCP does have help text.
------------------------------------------------------------------

This software translates all 32 bit counters (int, unsigned int,
float) into corresponding 64 bit counters (long long int, unsigned
long long, double) for return to PAPI. e.g. a float = -1.23 will
become a double = -1.23; an unsigned int of 0xFFFFFFFF will become
an unsigned long long of 0x00000000FFFFFFFF; a signed int of
0xFFFFFFFF (-1) becomes a long long of 0xFFFFFFFFFFFFFFFF (-1).
------------------------------------------------------------------

PAPI : PCP differences, and resolutions.

PCP STRING EVENTS AND OTHER EVENTS:-------------------------------
PCP has events that return arbitrarily length strings. There is no
char* data type in PAPI, so we exclude such events from the list
of events we support. The only one we excluded on P9 was a string
containing the version number of the PCP code in use.

We also exclude PCP aggregate types and some other types that
cannot be reasonably represented by PAPI's numerical types.

PCP ARRAYS OF VALUES: --------------------------------------------
A single PCP event can return an array of values; for example on
the Saturn test system nearly all events return an array of 64
values: 1 per Core. Others return 4 values; one per socket. This
includes a naming system for each of those values; e.g. cpu0 to
cpu63, or socket0 to socket63, etc. We call these multi-valued
events.

However, PAPI is designed to return exactly one value per event.
Our approach here is to 'explode' the multi-valued events, into
one PAPI event per value, and make the event names unique by
appending the PCP index name to the PCP event name with a ':'. 

e.g. PCP_EVENT_NAME -> PCP_EVENT_NAME:cpu0, PCP_EVENT_NAME:cpu1,
... PCP_EVENT_NAME:cpu63.

Internally we translate these names to a single event request from
PCP, and handle all the array indexing as needed; so to PAPI they
look like distinct events (including for zeroing or resets). 

This was an arbitrary coding choice that allows flexibility for
the user; an alternative would have been making these array
entries like '?' event mask parameters or options. 

PCP EVENT MASKS & OPTIONS:----------------------------------------
PCP doesn't have these. It returns a single value, or an array of
several named values.

PCP ROUND TRIP OPTIMIZATION:--------------------------------------
This interface communicates with a daemon; and on some systems
that can take significant time. On Saturn, every communication
costs 8.2 ms, so for ~10,000 events, a communication for each
could cost 82 seconds at initialization. For this reason the code
looks somewhat complex as we use batch queries that can move much
information at once; like pmTraversePMNS, pmLookup and pmFetch
that all take arrays of events at a time, and cost only one round
trip penalty. 

------------------------------------------------------------------
CODE OPTIONS for linux-pcp.

Near line 12: // #define DEBUG
To enable PAPI style debug messages; we don't use many, but there
are five SUBDBG (Substrate) messages in the _pcp_ctl() routine.

Near line 42: #define AGENT_NAME "perfevent"
PCP will only return events (during a pmTraversePMNS) that begin
with this string. At this writing 'nest' events are always within
'perfevents', but 'perfevents' is an umbrella that contains 
other events that are not 'nest' events. 

Near line 55: #define MYPCPLIB "libpcp.so" 
The name of the pcp library to be used. Note that it MUST be a
shared object library; a regular archive will not do.

Near line 194: #define COUNT_ROUTINES 1
Will enable a macro 'mRtnCnt' which will print on stderr the entry
of each PAPI interfacing routine the first time it is executed;
and when the component is shutdown a list of how often each
interface routine was executed. This is used to ensure the test
program exercised each interface routine at least once.

Near line 239: #define _prog_fprintf
change to 'if (1)' to enable printing (to stderr) of various
progress messages in the initialization stage.

Near line 240,241: #defines _time_fprintf, _time_gettimeofday.
change to 'if (1)' to enable printing (to stderr) of various
timings of the initilization operations, like reading names,
fetching values, etc. 


------------------------------------------------------------------

The testPCP.c code (bitbucket/icl/papi/src/components/pcp/tests)

This is straightforward application level code that reads through
every event, and tests every interface function of the component
at least once. First it finds the pcp component, then enumerates
all the events. It creates an event set, resets it, adds and
subtracts the event by name, gets its code, gets its info, and so
on. The final test is to add all the events to a single event set,
clean it up, destroy it, and execute a PAPI_shutdown.

It will print (to stdout) a CSV style text, with a heading,
detailing all the symbols, units, text description, time scope,
type and a sample value for for each event. 
------------------------------------------------------------------


How to install PAPI with the PCP component?
-------------------------------------------

This PAPI PCP component has been developed and tested using PCP
version 3.12.2.

This component uses the PCP location from the environment (or looks
for /usr by default).  Please set PCP_DIR during builds so that
the component can find the required header files and library.

Configure PAPI with PCP enabled.
    % cd src
    % ./configure --prefix=<prefix-location> --with-components="pcp"

Build with PCP_DIR specified
    % export PCP_DIR=/usr
    % make && make install

Testing the component requires that libraries for PAPI and PCP
can be found or are statically linked in to the executable.  You may
need to add the library directories as shown here.

    % export LD_LIBRARY_PATH=${PCP_DIR}/lib64:${LD_LIBRARY_PATH}

For general information on how to create and run components, the user
is referred to the INSTALL.txt section "CREATING AND RUNNING
COMPONENTS".


To check if PCP was successfully installed with PAPI.
    % <prefix-location>/bin/papi_component_avail

To find a list of PCP supported events.
    % <prefix-location>/bin/papi_native_avail | grep pcp

*/
