Introduction
This informational guide is intended to provide a deep dive into the activity monitoring process that FAM employs when processing the events.
Below is a high level data flow diagram explaining how FAM Activity Monitor works.
Actors in the play
1. Activity Monitor (BAM)
2. Event Collector
3. Event Manager
4. Elasticsearch
5. SQL Server
Activity Monitoring Process
Below is the sequence of steps involved in the activity monitoring process.
1. Activity Monitor (BAM) receives/polls the events from the Endpoint.
Note: Activity Monitor uses the endpoint specific technology to poll/receive events depending on the endpoint type.
2. BAM processes the events as per the steps mentioned in the above table. After processing the events, the BAM sends the events to the Event Collector service (part of the Event Manager).
Note: Refer to the BAM logs to get more details on the # of events being processed. Refer to this article Activity/Event Statistics Logs - Explained to know how to read the BAM logs.
3. Event Collector (Queue) receives the events. These events are yet to be enriched using the Data Enrichment Connectors (DEC) associated to the endpoint from which the events are being processed.
Note: Refer to the Event Collector logs to get more details on the # of events received. Refer to this article Activity/Event Statistics Logs - Explained to know how to read the Event Collector logs.
4. Based on the events Backup configuration set in the Event Manager configuration file, events are stored in the "EventsBackUp" folder. These events are only reprocessed if need be. They get removed from the server following the backup events retention setting in the Event Manager configuration file.
Note: Refer to the "Events Backup" section in the EventManagerServiceHost.exe.config file for more details
5. Event Manager reads events from the Event Collector Queue and processes the events as per the steps mentioned in the table below.
6. Event Manager attempts to write the activity (enriched event) to Elasticsearch.
Note: Refer to the Event Manager logs to get more details on the # of events processed. Refer to this article Activity/Event Statistics Logs - Explained to know how to read the Event Manager logs.
7. In case the Event Manager fails to write to Elasticsearch, it holds the activity in a rewrite cache. This cache file is named "BulkWriter Elastic_ElasticData_Cache.db". These activities are reprocessed by the Event Manager when the underlying issue has been resolved.
Note: Ideal size of the Elasticsearch rewrite cache file should be less than 20KB indicating that the Event Manager is successfully processing incoming events.
6. Event Manager attempts to write the activity (enriched event) to SQL Server database.
Note: Refer to the Event Manager logs to get more details on the # of events processed. Refer to this article Activity/Event Statistics Logs - Explained to know how to read the Event Manager logs.
Activities in the database only serve as a backup and will be used by FAM incase you intend to re-index events back into Elasticsearch for any reason.
7. In case the Event Manager fails to write to SQL server database, it holds the activity in a rewrite cache. This cache file is named "BulkWriter SQL_SqlData_Cache.db". These activities are reprocessed by the Event Manager when the underlying issue has been resolved.
Note: Ideal size of the SQL Server rewrite cache file should be less than 20KB indicating that the Event Manager is successfully processing incoming events.
Activity Forensics
To run activity forensics on File Access Manager version 6.0 and above, login to the WebUI and navigate to Forensics --> Activities.
To run activity forensics on SecurityIQ version 6.0 and below, login to the admin console and navigate to Activities.
Note: FAM uses Elasticsearch to retrieve activity data while running forensics.
Re-Process Backup Events
Incase you intend to re-process backed up events due to any scenario, perform the following.
1. Open the Event Manager installation folder and confirm that you have a "EventsBackup" folder in it.
2. Open the EventsBackup folder and confirm that you see files representing events.
3. Open the Event Manager configuration file EventManagerServiceHost.exe.config and set the value for the key "RestoreBackedupEvents" to true.
4. Save the changes made to the configuration file and restart the Event Manager service.
Important Note:
A. As the events from the EventsBackup folder are reprocessed by the Event Manager, you might see duplicate events in Elasticsearch and/or SQLServer database as the Event Manager doesn't do any kind of pre-checks before re-processing the events.
B. Incoming new events will processed by the Event Manager after completing the processing of backup events. Until then the events are temporarily stored in the Event Collector queue.
