There are situations when you want to monitor specific workflows. For example you want to monitor a customer order as it proceeds through different steps or you want to monitor what a specifc, or a set of customers are doing on your applications. You may want to monitor all the requests that are coming in from a particular IP address in realtime for debugging purposes.
Most organizations use things like Trace Id, or Tracker Id, to trace all the logs that belong to a specific request but there are issues with that. These can be effective for requests that originated through an external interaction like a REST endpoint call or a Kafka message but you can not synchronize these ids with background activities that you system may be performing to complete your workflow.Since tools like Splunk go through a process of log creation, parsing and it is a batch process. It is really not very effective for real-time monitoring. I believe that we need a tool that can help us with the real-time monitoring in an effective way and not become an overhead for the application itself. I call these services contextual monitoring services.
The idea of this service is pretty simple, as developers are writing applications, they are adding logs. Depending on what is of interest, the developers could push some of these logs to contextual monitoring service. The only difference is that these logs are tied to a context which may be a User Id, Order Id, Partner Id, IP Address or any other identifier that you wish to use. We will not log all the requests through this mechanism but a small subset of request that you might be interested in.
The diagram below describes a mechanism through which we can implement a system for real-time monitoring of the applications.
|Basic Architecture of Real-time Monitoring|
|Real-time monitoring use case|
This service with help us build functionality that would help us take real-time actions on the behavior of the application.