Monitoring DataFlow Jobs
In addition to managing configuration settings, Cluster Manager can display the following status information for DataFlow jobs in the cluster:
Cluster Summary
Provides basic cluster information such as Cluster Manager host and port. It also provides links to view the clustermgr.log and a Daemon Logs option that produces a .zip file of the clustermgr.log.
The following example shows the Cluster Summary view:
Pending Jobs
A job-centric view that shows jobs awaiting execution on the cluster. This includes information about the resource requirements. Administrators can cancel jobs from this page. Jobs are in the pending status until they receive resource allocation and container initialization.
The following example shows the Pending Jobs view:
Current Jobs
A job-centric view that provides the following:
Job Statistics summary
Lists the running DataFlow jobs and overall run-time data for those jobs. For each selected job, displays the per-process drill down statistics for its containers.
Per-container job logs
Lets you select a job to view the links to the job logs provided by YARN in the container list. Click any Logs link to open the YARN Node Manager log for the container.
A Kill button (available to administrators)
performs a "hard kill" of a job if this is the only way to end a running job. We do not recommend doing this because only the client can accomplish a normal, graceful shutdown. Killing a job terminates all of its executor processes and then deletes any temporary files in use by that job.
The following example shows the Current Jobs view:
Recent Jobs
Displays a job-centric view that shows the most recently executed jobs, overall run-time statistics for those jobs, and per-container drill-down statistics for those jobs.
The Job Statistics table summarizes the jobs currently running (one job per row). The job summary provides an aggregation of container statistics.
Click a job in the summary to display the details for the job at the container level, statistics of each container launched and running for the job. A link is provided for the container logs.
Click the link to view Node Manage logs for the container in a browser. For each job a Kill button is available for users with administrative rights. Click Kill to stop a job execution and free its resources.
Select an item in Job Statistics to view the links to Node Manager logs for each container.
Note: For more information about the recent job statistics, see Cluster Job Metrics.
Last modified date: 06/14/2024