Activity/Event Statistics Logs - Explained

[#1]

This informational guide is aimed to aid you in understanding what you are looking for in the "-Statistics.log" and assist any troubleshooting.

Applicable Versions: 5.X, 6.0, 6.1

See Event Data Flow and Troubleshooting if needed

In general, you are looking for Event Data movement. High numbers or NO numbers in specific areas may be an indication of an issue.

BAM Statistics

Flow Path: BAM/Activity Monitor > Event Manager > Elasticsearch Search & SQLDB

These logs are typically located on the server containing the BAM (Activity Monitor) Service. See: Where Are The Logs article if needed.

For most targeted applications, the BAM (Activity Monitor) - Statistics.logs have the same information but can differ slightly. Below is an example of a Windows File Server Statistics log 

(click to enlarge image).

Log Line	Explanation	Indicators of Issues	Things to Try
Events received since the last statistics log.	New entries in the log appear every minute. Example: 168 events have shown up to the BAM (per the screenshot stats from the 1st screenshot).	Not up to date to the nearest minute. No events received since last statistics log.	Is this a highly active server? Does the service account have the proper permissions? If these logs are not up to date. Stop the Activity Monitor service. Backup and clear logs. Start BAM serviceCheck statistics logs for movement.
Since BAM Started	This is the total number of events that the BAM has detected	Zero	Is this a highly active server? Same steps as above Check exclusion rules on in the Activity Monitor Config. Check Permissions of the service account
Current OS events queue count or Events waiting in Queue	These are events that are waiting to be shipped to the Event Collector/Event Manager (See: Event Manager & Event Collector Explained for more details.)	Large Number	Stop the Activity Monitor service. Backup and clear logs. Start BAM serviceCheck connection to Event Manager server. Check Statistics of Event Manager
Events Processed since the last statistics log	These are events that have moved onto the Event Manager/Event Collector	Zero	Check connection to the Event Manager server

Event Collector Statistics

Flow Path: BAM/Activity Monitor > Event Manager > Elasticsearch Search & SQLDB

Note: The Event Manager is a two-component process. See Event Manager & Event Collector Explained for more details.

(Click to enlarge image)

Log Line	Explanation	Indicators of Issues	Things to Try If Issues Indicated
Time	This log is updated every minute	Log time not up to date to the nearest minute	If these logs are not up to date. Stop the Event Manager service. Backup and clear logs. Start Event Manager service (Remember Collector and Manager are 2 services in 1) Check statistics logs for movement
Total Events	Total Events the Event Collector has picked up from BAMs	Zero	Check BAM(s) connection to the Event Collector/Manager Server or check BAM config in Admin Console if there are multiple Event Managers in the environment. Check Disk space of the Event Collector/Manager server
Total requests in queue waiting for worker:	That indicates that Events are moving to the Event Manager. This number should be at zero or close to it.	High Numbers	Stop Event Manager Service Backup and clear logs Put in Debug for short time Check to see if there is data is flowing from the Event Manager to the Elasticsearch and MSSQL DB
New Events sent to MSMQ	Event moving to the Event Manager basically	Zero	Check BAM(s) connection to the Event Collector/Manager Server or check BAM config in Admin Console if there are multiple Event Managers in the environment. Check Disk space of the Event Collector/Manager server
New events sent to the MSMQ since event collector started	Total events sent	Zero	Check BAM(s) connection to the Event Collector/Manager Server or check BAM config in Admin Console if there are multiple Event Managers in the environment. Check Disk space of the Event Collector/Manager server

Event Manager Statistics & Event Manager.log

Flow Path: BAM/Activity Monitor > Event Manager > Elasticsearch Search & SQLDB

Note: The Event Manager is a two-component process. See Event Manager & Event Collector Explained for more details.

(Click to enlarge image)

Log Line	Description	Indicators of Issues	Things to Try if Issues Indicated
Time	This log is updated every minute	Log time not up to date to the nearest minute	Stop the Event Manager service. Backup and clear logs. Start Event Manager service (Remember Collector and Manager are 2 services in 1) Check statistics logs for movement
SQL sending Queue contains # bulks (# events). Resend cache stores # bulks (# events)  stored in memory and # buiks (# events) on Hard disk	Events being sent from the Event Manager to the SQL DB. Resend cache	High Numbers - Events that could not be sent for some reason and are waiting (being stored on the local storage)	Check the status of your SQLDB Is the server up? How is disk storage? Check the Activity Monitor's error logs
Policy Engine Events in queue	Events that have triggered a response, waiting to be processed	Varies	Check any discard policies that may be in place
New events saved since last statistics log (SQL: )	How many events have saved as activities in the SQL DB in the last minute	Varies - Meaning if this is a high traffic application and this number is low or zero. This may indicate issues	Check the status of your SQLDB Is the server up? How is disk storage? Check the Activity Monitor's error logs
New discarded events since last statistics log	Discard rules applied to the application	Varies - This is dependent on your configuration	If there are too many events coming in and taking up Elasticsearch and SQLDB space. Apply discard rules. This should start decreasing the number of events needing to be processed. Also, see this guide.
Event Collector's events in memory waiting to be proceed	Events in memory waiting to be proceed This should be zero or really close to it	High Number	Check the status of the Elasticsearch service in the Health Center
ElasticSearch Sending Queue contains # bulks (# events). Resend cache stores  #bulks (# events) stored in memory and # bulks (# events) on HardDisk	Events being sent from the Event Manager to the Elasticsearch DB. Resend cache	High Numbers - Events that could not be sent to Elasticsearch for some reason and are waiting (being stored on the local storage).	See: Elasticsearch DB Cache File Size Large for troubleshooting ( in progress).