org.apache.hadoop.mapred

Class TaskTracker

java.lang.Object

org.apache.hadoop.mapred.TaskTracker

All Implemented Interfaces:

Runnable, org.apache.hadoop.ipc.VersionedProtocol, TaskTrackerMXBean, TaskUmbilicalProtocol

public class TaskTracker
extends Object
implements TaskUmbilicalProtocol, Runnable, TaskTrackerMXBean

TaskTracker is a process that starts and tracks MR Tasks in a networked environment. It contacts the JobTracker for Task assignments and reporting results.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TaskTracker.DefaultShuffleProvider
static class	TaskTracker.MapOutputServlet This class is used in TaskTracker's Jetty to serve the map outputs to other nodes.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	ClientTraceLog
static long	COUNTER_UPDATE_INTERVAL
static long	DEFAULT_DISK_HEALTH_CHECK_INTERVAL How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
static int	FILE_NOT_FOUND
static String	FOR_REDUCE_TASK The reduce task number for which this map output is being transferred
static String	FROM_MAP_TASK The map task from which the map output data is being transferred
static int	HEARTBEAT_INTERVAL_MIN_DEFAULT
static org.apache.commons.logging.Log	LOG
static String	MAP_OUTPUT_LENGTH The custom http header used for the map output length.
static String	MR_CLIENTTRACE_FORMAT
static String	RAW_MAP_OUTPUT_LENGTH The custom http header used for the "raw" map output length.
static String	SHUFFLE_PROVIDER_PLUGIN_CLASSES
static String	SUBDIR
static int	SUCCESS
static String	TT_KEYTAB_FILE
static String	TT_LOG_TMP_DIR
static String	TT_RESOURCE_CALCULATOR_PLUGIN
static String	TT_USER_NAME
static String	WORKDIR

Fields inherited from interface org.apache.hadoop.mapred.TaskUmbilicalProtocol

versionID

Constructor Summary

Constructors

Constructor and Description
TaskTracker(JobConf conf) Start with the local machine name, and the default JobTracker

Method Summary

Methods

Modifier and Type	Method and Description
boolean	canCommit(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) Child checking whether it can commit
void	cleanupStorage() Deprecated.
void	close() Close down the TaskTracker and all its components.
void	commitPending(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext) Task is reporting that it is in commit_pending and it is waiting for the commit Response
void	done(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) The task is done.
void	fatalError(TaskAttemptID taskId, String msg, org.apache.hadoop.mapred.JvmContext jvmContext) A child task had a fatal error.
void	fsError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext) A child task had a local filesystem error.
String	getConfigVersion()
String	getHostname()
int	getHttpPort()
static Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation>	getInstrumentationClass(org.apache.hadoop.conf.Configuration conf)
static String	getIntermediateOutputDir(String user, String jobid, String taskid)
static String	getJobCacheSubdir(String user)
org.apache.hadoop.mapred.InterTrackerProtocol	getJobClient() The connection to the JobTracker, used by the TaskRunner for locating remote files.
JobConf	getJobConf() Get the default job conf for this tracker.
JobConf	getJobConf(JobID jobId) Get the specific job conf for a running job.
static String	getJobJarFile(String user, String jobid)
String	getJobTrackerUrl()
org.apache.hadoop.mapred.JvmManager	getJvmManagerInstance()
static String	getLocalJobDir(String user, String jobid)
static String	getLocalTaskDir(String user, String jobid, String taskid)
static String	getLocalTaskDir(String user, String jobid, String taskid, boolean isCleanupAttempt)
MapTaskCompletionEventsUpdate	getMapCompletionEvents(JobID jobId, int fromEventId, int maxLocs, TaskAttemptID id, org.apache.hadoop.mapred.JvmContext jvmContext) Called by a reduce task to get the map output locations for finished maps.
static String	getPrivateDistributedCacheDir(String user)
org.apache.hadoop.ipc.ProtocolSignature	getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash)
long	getProtocolVersion(String protocol, long clientVersion)
static String	getPublicDistributedCacheDir()
int	getRpcPort()
JvmTask	getTask(org.apache.hadoop.mapred.JvmContext context) Called upon startup by the child process, to fetch Task data.
TaskController	getTaskController()
org.apache.hadoop.mapred.TaskMemoryManagerThread	getTaskMemoryManager()
String	getTasksInfoJson()
org.apache.hadoop.mapred.TaskTrackerInstrumentation	getTaskTrackerInstrumentation()
InetSocketAddress	getTaskTrackerReportAddress() Return the port at which the tasktracker bound to
static String	getUserDir(String user)
String	getVersion()
boolean	isHealthy()
boolean	isIdle() Is this task tracker idle?
boolean	isTaskMemoryManagerEnabled() Is the TaskMemoryManager Enabled on this system?
static void	main(String[] argv) Start the TaskTracker, point toward the indicated JobTracker
void	mapOutputLost(TaskAttemptID taskid, String errorMsg) A completed map task's output has been lost.
boolean	ping(TaskAttemptID taskid, org.apache.hadoop.mapred.JvmContext jvmContext) Child checking to see if we're alive.
void	reportDiagnosticInfo(TaskAttemptID taskid, String info, org.apache.hadoop.mapred.JvmContext jvmContext) Called when the task dies before completion, and we want to report back diagnostic info
void	reportNextRecordRange(TaskAttemptID taskid, org.apache.hadoop.mapred.SortedRanges.Range range, org.apache.hadoop.mapred.JvmContext jvmContext) Report the record range which is going to process next by the Task.
void	run() The server retry loop.
static void	setInstrumentationClass(org.apache.hadoop.conf.Configuration conf, Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t)
void	shuffleError(TaskAttemptID taskId, String message, org.apache.hadoop.mapred.JvmContext jvmContext) A reduce-task failed to shuffle the map-outputs.
void	shutdown()
boolean	statusUpdate(TaskAttemptID taskid, TaskStatus taskStatus, org.apache.hadoop.mapred.JvmContext jvmContext) Called periodically to report Task progress, from 0.0 to 1.0.
void	updatePrivateDistributedCacheSizes(JobID jobId, long[] sizes) The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SHUFFLE_PROVIDER_PLUGIN_CLASSES

public static final String SHUFFLE_PROVIDER_PLUGIN_CLASSES

See Also:

Constant Field Values

LOG

public static final org.apache.commons.logging.Log LOG

MR_CLIENTTRACE_FORMAT

public static final String MR_CLIENTTRACE_FORMAT

See Also:

Constant Field Values

ClientTraceLog

public static final org.apache.commons.logging.Log ClientTraceLog

SUBDIR

public static final String SUBDIR

See Also:

Constant Field Values

TT_LOG_TMP_DIR

public static final String TT_LOG_TMP_DIR

See Also:

Constant Field Values

TT_RESOURCE_CALCULATOR_PLUGIN

public static final String TT_RESOURCE_CALCULATOR_PLUGIN

See Also:

Constant Field Values

TT_USER_NAME

public static final String TT_USER_NAME

See Also:

Constant Field Values

TT_KEYTAB_FILE

public static final String TT_KEYTAB_FILE

See Also:

Constant Field Values

HEARTBEAT_INTERVAL_MIN_DEFAULT

public static final int HEARTBEAT_INTERVAL_MIN_DEFAULT

See Also:

Constant Field Values

COUNTER_UPDATE_INTERVAL

public static final long COUNTER_UPDATE_INTERVAL

See Also:

Constant Field Values

DEFAULT_DISK_HEALTH_CHECK_INTERVAL

public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL

How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval

See Also:

Constant Field Values

SUCCESS

public static final int SUCCESS

See Also:

Constant Field Values

FILE_NOT_FOUND

public static final int FILE_NOT_FOUND

See Also:

Constant Field Values

MAP_OUTPUT_LENGTH

public static final String MAP_OUTPUT_LENGTH

The custom http header used for the map output length.

See Also:

Constant Field Values

RAW_MAP_OUTPUT_LENGTH

public static final String RAW_MAP_OUTPUT_LENGTH

The custom http header used for the "raw" map output length.

See Also:

Constant Field Values

FROM_MAP_TASK

public static final String FROM_MAP_TASK

The map task from which the map output data is being transferred

See Also:

Constant Field Values

FOR_REDUCE_TASK

public static final String FOR_REDUCE_TASK

The reduce task number for which this map output is being transferred

See Also:

Constant Field Values

WORKDIR

public static final String WORKDIR

See Also:

Constant Field Values

Constructor Detail

TaskTracker

public TaskTracker(JobConf conf)
            throws IOException,
                   InterruptedException

Start with the local machine name, and the default JobTracker

Throws:

IOException

InterruptedException

Method Detail

getTaskTrackerInstrumentation

public org.apache.hadoop.mapred.TaskTrackerInstrumentation getTaskTrackerInstrumentation()

getTaskController

public TaskController getTaskController()

getUserDir

public static String getUserDir(String user)

getPrivateDistributedCacheDir

public static String getPrivateDistributedCacheDir(String user)

getPublicDistributedCacheDir

public static String getPublicDistributedCacheDir()

getJobCacheSubdir

public static String getJobCacheSubdir(String user)

getLocalJobDir

public static String getLocalJobDir(String user,
                    String jobid)

getJobJarFile

public static String getJobJarFile(String user,
                   String jobid)

getIntermediateOutputDir

public static String getIntermediateOutputDir(String user,
                              String jobid,
                              String taskid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                     String jobid,
                     String taskid)

getLocalTaskDir

public static String getLocalTaskDir(String user,
                     String jobid,
                     String taskid,
                     boolean isCleanupAttempt)

getProtocolVersion

public long getProtocolVersion(String protocol,
                      long clientVersion)
                        throws IOException

Specified by:

getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

IOException

getProtocolSignature

public org.apache.hadoop.ipc.ProtocolSignature getProtocolSignature(String protocol,
                                                           long clientVersion,
                                                           int clientMethodsHash)
                                                             throws IOException

Specified by:

getProtocolSignature in interface org.apache.hadoop.ipc.VersionedProtocol

Throws:

IOException

getInstrumentationClass

public static Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> getInstrumentationClass(org.apache.hadoop.conf.Configuration conf)

setInstrumentationClass

public static void setInstrumentationClass(org.apache.hadoop.conf.Configuration conf,
                           Class<? extends org.apache.hadoop.mapred.TaskTrackerInstrumentation> t)

cleanupStorage

@Deprecated
public void cleanupStorage()
                    throws IOException

Deprecated.

Removes all contents of temporary storage. Called upon startup, to remove any leftovers from previous run. Use MRAsyncDiskService.moveAndDeleteAllVolumes instead.

Throws:

IOException

See Also:

org.apache.hadoop.mapreduce.util.MRAsyncDiskService#cleanupDirsInAllVolumes()

shutdown

public void shutdown()
              throws IOException,
                     InterruptedException

Throws:

IOException

InterruptedException

close

public void close()
           throws IOException,
                  InterruptedException

Close down the TaskTracker and all its components. We must also shutdown any running tasks or threads, and cleanup disk space. A new TaskTracker within the same process space might be restarted, so everything must be clean.

Throws:

InterruptedException

IOException

getJobClient

public org.apache.hadoop.mapred.InterTrackerProtocol getJobClient()

The connection to the JobTracker, used by the TaskRunner for locating remote files.

getTaskTrackerReportAddress

public InetSocketAddress getTaskTrackerReportAddress()

Return the port at which the tasktracker bound to

getJvmManagerInstance

public org.apache.hadoop.mapred.JvmManager getJvmManagerInstance()

run

public void run()

The server retry loop. This while-loop attempts to connect to the JobTracker. It only loops when the old TaskTracker has gone bad (its state is stale somehow) and we need to reinitialize everything.

Specified by:

run in interface Runnable

getTask

public JvmTask getTask(org.apache.hadoop.mapred.JvmContext context)
                throws IOException

Called upon startup by the child process, to fetch Task data.

Specified by:

getTask in interface TaskUmbilicalProtocol

Parameters:

context - the JvmContext of the JVM w.r.t the TaskTracker that launched it

Returns:

Task object

Throws:

IOException

statusUpdate

public boolean statusUpdate(TaskAttemptID taskid,
                   TaskStatus taskStatus,
                   org.apache.hadoop.mapred.JvmContext jvmContext)
                     throws IOException

Called periodically to report Task progress, from 0.0 to 1.0.

Specified by:

statusUpdate in interface TaskUmbilicalProtocol

Parameters:

taskid - task-id of the child

taskStatus - status of the child

jvmContext - context the jvmContext running the task.

Returns:

True if the task is known

Throws:

IOException

reportDiagnosticInfo

public void reportDiagnosticInfo(TaskAttemptID taskid,
                        String info,
                        org.apache.hadoop.mapred.JvmContext jvmContext)
                          throws IOException

Called when the task dies before completion, and we want to report back diagnostic info

Specified by:

reportDiagnosticInfo in interface TaskUmbilicalProtocol

Parameters:

taskid - the id of the task involved

info - the text to report

jvmContext - context the jvmContext running the task.

Throws:

IOException

reportNextRecordRange

public void reportNextRecordRange(TaskAttemptID taskid,
                         org.apache.hadoop.mapred.SortedRanges.Range range,
                         org.apache.hadoop.mapred.JvmContext jvmContext)
                           throws IOException

Description copied from interface: TaskUmbilicalProtocol

Report the record range which is going to process next by the Task.

Specified by:

reportNextRecordRange in interface TaskUmbilicalProtocol

Parameters:

taskid - the id of the task involved

range - the range of record sequence nos

jvmContext - context the jvmContext running the task.

Throws:

IOException

ping

public boolean ping(TaskAttemptID taskid,
           org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException

Child checking to see if we're alive. Normally does nothing.

Specified by:

ping in interface TaskUmbilicalProtocol

Parameters:

taskid - the id of the task involved

jvmContext - context the jvmContext running the task.

Returns:

True if the task is known

Throws:

IOException

commitPending

public void commitPending(TaskAttemptID taskid,
                 TaskStatus taskStatus,
                 org.apache.hadoop.mapred.JvmContext jvmContext)
                   throws IOException

Task is reporting that it is in commit_pending and it is waiting for the commit Response

Specified by:

commitPending in interface TaskUmbilicalProtocol

Parameters:

taskid - task's id

taskStatus - status of the child

jvmContext - context the jvmContext running the task.

Throws:

IOException

canCommit

public boolean canCommit(TaskAttemptID taskid,
                org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException

Child checking whether it can commit

Specified by:

canCommit in interface TaskUmbilicalProtocol

jvmContext - context the jvmContext running the task.

Returns:

true/false

Throws:

IOException

done

public void done(TaskAttemptID taskid,
        org.apache.hadoop.mapred.JvmContext jvmContext)
          throws IOException

The task is done.

Specified by:

done in interface TaskUmbilicalProtocol

Parameters:

taskid - task's id

jvmContext - context the jvmContext running the task.

Throws:

IOException

shuffleError

public void shuffleError(TaskAttemptID taskId,
                String message,
                org.apache.hadoop.mapred.JvmContext jvmContext)
                  throws IOException

A reduce-task failed to shuffle the map-outputs. Kill the task.

Specified by:

shuffleError in interface TaskUmbilicalProtocol

Throws:

IOException

fsError

public void fsError(TaskAttemptID taskId,
           String message,
           org.apache.hadoop.mapred.JvmContext jvmContext)
             throws IOException

A child task had a local filesystem error. Kill the task.

Specified by:

fsError in interface TaskUmbilicalProtocol

Throws:

IOException

fatalError

public void fatalError(TaskAttemptID taskId,
              String msg,
              org.apache.hadoop.mapred.JvmContext jvmContext)
                throws IOException

A child task had a fatal error. Kill the task.

Specified by:

fatalError in interface TaskUmbilicalProtocol

Throws:

IOException

getMapCompletionEvents

public MapTaskCompletionEventsUpdate getMapCompletionEvents(JobID jobId,
                                                   int fromEventId,
                                                   int maxLocs,
                                                   TaskAttemptID id,
                                                   org.apache.hadoop.mapred.JvmContext jvmContext)
                                                     throws IOException

Description copied from interface: TaskUmbilicalProtocol

Called by a reduce task to get the map output locations for finished maps. Returns an update centered around the map-task-completion-events. The update also piggybacks the information whether the events copy at the task-tracker has changed or not. This will trigger some action at the child-process.

Specified by:

getMapCompletionEvents in interface TaskUmbilicalProtocol

Parameters:

jobId - the reducer job id

fromEventId - the index starting from which the locations should be fetched

maxLocs - the max number of locations to fetch

id - The attempt id of the task that is trying to communicate

Returns:

A MapTaskCompletionEventsUpdate

Throws:

IOException

mapOutputLost

public void mapOutputLost(TaskAttemptID taskid,
                 String errorMsg)
                   throws IOException

A completed map task's output has been lost.

Throws:

IOException

getJobConf

public JobConf getJobConf(JobID jobId)
                   throws IOException

Get the specific job conf for a running job.

Throws:

IOException

getJobConf

public JobConf getJobConf()

Get the default job conf for this tracker.

isIdle

public boolean isIdle()

Is this task tracker idle?

Returns:

has this task tracker finished and cleaned up all of its tasks?

main

public static void main(String[] argv)
                 throws Exception

Start the TaskTracker, point toward the indicated JobTracker

Throws:

Exception

isTaskMemoryManagerEnabled

public boolean isTaskMemoryManagerEnabled()

Is the TaskMemoryManager Enabled on this system?

Returns:

true if enabled, false otherwise.

getTaskMemoryManager

public org.apache.hadoop.mapred.TaskMemoryManagerThread getTaskMemoryManager()

getHostname

public String getHostname()

Specified by:

getHostname in interface TaskTrackerMXBean

Returns:

the hostname of the tasktracker

getVersion

public String getVersion()

Specified by:

getVersion in interface TaskTrackerMXBean

Returns:

the version of the code base

getConfigVersion

public String getConfigVersion()

Specified by:

getConfigVersion in interface TaskTrackerMXBean

Returns:

the config version (from a config properties)

getJobTrackerUrl

public String getJobTrackerUrl()

Specified by:

getJobTrackerUrl in interface TaskTrackerMXBean

Returns:

the URL of the jobtracker

getRpcPort

public int getRpcPort()

Specified by:

getRpcPort in interface TaskTrackerMXBean

Returns:

the RPC port of the tasktracker

getHttpPort

public int getHttpPort()

Specified by:

getHttpPort in interface TaskTrackerMXBean

Returns:

the HTTP port of the tasktracker

isHealthy

public boolean isHealthy()

Specified by:

isHealthy in interface TaskTrackerMXBean

Returns:

the health status of the tasktracker

getTasksInfoJson

public String getTasksInfoJson()

Specified by:

getTasksInfoJson in interface TaskTrackerMXBean

Returns:

a json formatted info about tasks of the tasktracker

updatePrivateDistributedCacheSizes

public void updatePrivateDistributedCacheSizes(JobID jobId,
                                      long[] sizes)
                                        throws IOException

Description copied from interface: TaskUmbilicalProtocol

The job initializer needs to report the sizes of the archive objects and directories in the private distributed cache.

Specified by:

updatePrivateDistributedCacheSizes in interface TaskUmbilicalProtocol

Parameters:

jobId - the job to update

sizes - the array of sizes that were computed

Throws:

IOException