Analyzing Application Performance
The analysis and monitoring of DataFlow applications can be approached in a number of ways. They include:
System level perspective
Includes monitoring CPU activity, disk activity, memory usage, and the usage of other system-level resources.
JVM perspective
Includes monitoring memory and GC activity, thread activity, and other JVM-level resources.
DataFlow application perspective
Includes understanding the logical graph layout, flow of data through the system, and CPU usage of operators.
The previous approaches are roughly ordered by increasing specificity and granularity of information. Generally, it is best to start looking at the system level, as constraints on system performance will necessarily be constraints for any application running on the system. This provides a general idea of the nature of the problem which can then guide investigation into JVM and application performance. In this way, you are not immediately overwhelmed by a wealth of information, struggling to find a clue to the solution in a large amount of data.
As discussed in
Debugging Distributed Applications, when working with a distributed application, it is important to remember that multiple machines are involved in execution. Thus, multiple machines may need to be monitored and the data combined. However, the point that local execution can sometime illustrate the same issues as distributed execution is just as valid with respect to analyzing performance as it is with debugging.