Debugging YARN Applications

With YARN, executors and application masters run inside “containers”. YARN has two modes for handling container logs after an application has completed. If log aggregation is turned on (with the yarn.log-aggregation-enable config - this is the case when running Oozie or Spark jobs with the data platform), the container logs are copied to HDFS and deleted from the local machine. These logs can be viewed from anywhere on the cluster with the yarn logs command.

yarn logs -applicationId <app ID>

This prints out the contents of all log files from all containers from the given application.

For Spark, the logs are also available on the Spark Web UI under the Executors Tab. Both the Spark history server and the MapReduce history server have to be running.