JobTracker (Hadoop 2.6.0-mr1-cdh5.7.1 API)

java.lang.Object
- org.apache.hadoop.mapred.JobTracker

All Implemented Interfaces:

org.apache.hadoop.ipc.VersionedProtocol, AdminOperationsProtocol, JobTrackerMXBean, JTProtocols, RefreshAuthorizationPolicyProtocol, RefreshUserMappingsProtocol, GetUserMappingsProtocol
```
public class JobTracker
extends Object
implements JTProtocols, JobTrackerMXBean
```
JobTracker is the central location for submitting and tracking MR jobs in a network environment.

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`JobTracker.IllegalStateException` A client tried to submit a job before the Job Tracker was ready.
`static class`	`JobTracker.State`

Field Summary

Fields
Modifier and Type	Field and Description
`static long`	`COUNTER_UPDATE_INTERVAL`
`static long`	`DEFAULT_DISK_HEALTH_CHECK_INTERVAL` How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
`static long`	`DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT`
`static String`	`DELEGATION_KEY_UPDATE_INTERVAL_KEY`
`static long`	`DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT`
`static String`	`DELEGATION_TOKEN_MAX_LIFETIME_KEY`
`static long`	`DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT`
`static String`	`DELEGATION_TOKEN_RENEW_INTERVAL_KEY`
`static int`	`FILE_NOT_FOUND`
`static String`	`FOR_REDUCE_TASK` The reduce task number for which this map output is being transferred
`static String`	`FROM_MAP_TASK` The map task from which the map output data is being transferred
`static int`	`HEARTBEAT_INTERVAL_MIN_DEFAULT`
`static String`	`JT_KEYTAB_FILE`
`static String`	`JT_USER_NAME`
`static org.apache.commons.logging.Log`	`LOG`
`static String`	`MAP_OUTPUT_LENGTH` The custom http header used for the map output length.
`static String`	`MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY`
`static String`	`MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY`
`static String`	`MAX_USER_JOBCONF_SIZE_KEY` the config key for max user jobconf size
`static String`	`RAW_MAP_OUTPUT_LENGTH` The custom http header used for the "raw" map output length.
`static String`	`SHUFFLE_SSL_ADDRESS_DEFAULT`
`static String`	`SHUFFLE_SSL_ADDRESS_KEY`
`static boolean`	`SHUFFLE_SSL_ENABLED_DEFAULT`
`static String`	`SHUFFLE_SSL_ENABLED_KEY`
`static int`	`SHUFFLE_SSL_PORT_DEFAULT`
`static String`	`SHUFFLE_SSL_PORT_KEY`
`static int`	`SUCCESS`
`static int`	`TRACKERS_OK`
`static int`	`UNKNOWN_TASKTRACKER`
`static long`	`versionID` version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with `heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short)` version 4 changed TaskReport for HADOOP-549.
`static long`	`versionID`
`static String`	`WORKDIR`

Fields inherited from interface org.apache.hadoop.mr1security.RefreshUserMappingsProtocol
versionID

Fields inherited from interface org.apache.hadoop.mr1tools.GetUserMappingsProtocol
versionID

Fields inherited from interface org.apache.hadoop.mr1security.authorize.RefreshAuthorizationPolicyProtocol
versionID

Fields inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol
versionID

Method Summary

Methods
Modifier and Type	Method and Description
`Collection<TaskTrackerStatus>`	`activeTaskTrackers()` Get the active task tracker statuses in the cluster
`void`	`addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)` Registers a `JobInProgressListener` for updates from this `TaskTrackerManager`.
`Collection<TaskTrackerStatus>`	`blacklistedTaskTrackers()` Get the blacklisted task tracker statuses in the cluster
`void`	`cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)` Discard a current delegation token.
`Vector<JobInProgress>`	`completedJobs()`
`Vector<JobInProgress>`	`failedJobs()`
`void`	`failJob(JobInProgress job)` Fail a job and inform the listeners.
`static InetSocketAddress`	`getAddress(org.apache.hadoop.conf.Configuration conf)`
`String`	`getAliveNodesInfoJson()`
`JobStatus[]`	`getAllJobs()` Get all the jobs submitted.
`String`	`getAssignedTracker(TaskAttemptID taskId)` Get tracker name for a given task id.
`String`	`getBlacklistedNodesInfoJson()`
`String`	`getBuildVersion()` Returns the VersionInfo build version of the JobTracker
`TaskReport[]`	`getCleanupTaskReports(JobID jobid)` Grab a bunch of info on the cleanup tasks that make up the job
`org.apache.hadoop.mapred.Clock`	`getClock()`
`ClusterMetrics`	`getClusterMetrics()`
`ClusterStatus`	`getClusterStatus()` Deprecated. use `getClusterStatus(boolean)`
`ClusterStatus`	`getClusterStatus(boolean detailed)` Get the current status of the cluster
`List<JobInProgress>`	`getCompletedJobs()`
`JobConf`	`getConf()` Returns a handle to the JobTracker's Configuration
`String`	`getConfigVersion()`
`org.apache.hadoop.security.token.Token<DelegationTokenIdentifier>`	`getDelegationToken(org.apache.hadoop.io.Text renewer)` Get a new delegation token.
`DelegationTokenSecretManager`	`getDelegationTokenSecretManager()`
`List<JobInProgress>`	`getFailedJobs()`
`String`	`getFilesystemName()` Grab the local fs name
`String[]`	`getGroupsForUser(String user)` Get the groups which are mapped to the given user.
`String`	`getHostname()`
`int`	`getInfoPort()`
`static Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation>`	`getInstrumentationClass(org.apache.hadoop.conf.Configuration conf)`
`JobInProgress`	`getJob(JobID jobid)` Obtain the job object identified by jobid
`Counters`	`getJobCounters(JobID jobid)` Grab the current job counters
`JobProfile`	`getJobProfile(JobID jobid)` Grab a handle to a job that is already known to the JobTracker.
`JobStatus[]`	`getJobsFromQueue(String queue)` Gets all the jobs submitted to the particular Queue
`JobStatus`	`getJobStatus(JobID jobid)` Grab a handle to a job that is already known to the JobTracker.
`String`	`getJobTrackerMachine()`
`static String`	`getLocalJobFilePath(JobID jobId)` Get the localized job file path on the job trackers local file system
`TaskReport[]`	`getMapTaskReports(JobID jobid)` Grab a bunch of info on the map tasks that make up the job
`JobID`	`getNewJobId()` Allocates a new JobId string.
`int`	`getNextHeartbeatInterval()` Calculates next heartbeat interval using cluster size.
`org.apache.hadoop.net.Node`	`getNode(String name)` Return the Node in the network topology that corresponds to the hostname
`Collection<org.apache.hadoop.net.Node>`	`getNodesAtMaxLevel()` Returns a collection of nodes at the max level
`int`	`getNumberOfUniqueHosts()`
`int`	`getNumResolvedTaskTrackers()`
`int`	`getNumTaskCacheLevels()`
`static org.apache.hadoop.net.Node`	`getParentNode(org.apache.hadoop.net.Node node, int level)`
`org.apache.hadoop.ipc.ProtocolSignature`	`getProtocolSignature(String protocol, long clientVersion, int clientMethodsHash)`
`long`	`getProtocolVersion(String protocol, long clientVersion)`
`org.apache.hadoop.mapred.QueueAclsInfo[]`	`getQueueAclsForCurrentUser()` Gets the Queue ACLs for current user
`org.apache.hadoop.security.authorize.AccessControlList`	`getQueueAdmins(String queueName)` Get the administrators of the given job-queue.
`JobQueueInfo`	`getQueueInfo(String queue)` Gets scheduling information associated with the particular Job queue
`String`	`getQueueInfoJson()`
`org.apache.hadoop.mapred.QueueManager`	`getQueueManager()` Return the `QueueManager` associated with the JobTracker.
`JobQueueInfo[]`	`getQueues()` Gets set of Job Queues associated with the Job Tracker
`long`	`getRecoveryDuration()` How long the jobtracker took to recover from restart.
`TaskReport[]`	`getReduceTaskReports(JobID jobid)` Grab a bunch of info on the reduce tasks that make up the job
`List<JobInProgress>`	`getRunningJobs()` Version that is called from a timer thread, and therefore needs to be careful to synchronize.
`TaskReport[]`	`getSetupTaskReports(JobID jobid)` Grab a bunch of info on the setup tasks that make up the job
`String`	`getStagingAreaDir()` Get a hint from the JobTracker where job-specific files are to be placed.
`long`	`getStartTime()`
`String`	`getSummaryJson()`
`String`	`getSystemDir()` Grab the jobtracker system directory path where job-specific files are to be placed.
`TaskCompletionEvent[]`	`getTaskCompletionEvents(JobID jobid, int fromEventId, int maxEvents)` Get task completion events for the jobid, starting from fromEventId.
`String[]`	`getTaskDiagnostics(TaskAttemptID taskId)` Get the diagnostics for a given task
`TaskTracker`	`getTaskTracker(String trackerID)`
`TaskTrackerStatus`	`getTaskTrackerStatus(String trackerID)`
`int`	`getThreadCount()`
`TaskInProgress`	`getTip(TaskID tipid)` Returns specified TaskInProgress, or null.
`int`	`getTotalSubmissions()`
`String`	`getTrackerIdentifier()` Get the unique identifier (ie.
`int`	`getTrackerPort()`
`String`	`getVersion()`
`String`	`getVIVersion()` Returns the VersionInfo version of the JobTracker
`boolean`	`hasRecovered()` Whether the JT has recovered upon restart
`boolean`	`hasRestarted()` Whether the JT has restarted
`org.apache.hadoop.mapred.HeartbeatResponse`	`heartbeat(TaskTrackerStatus status, boolean restarted, boolean initialContact, boolean acceptNewTasks, short responseId)` The periodic heartbeat mechanism between the `TaskTracker` and the `JobTracker`.
`void`	`initJob(JobInProgress job)` Initialize the Job
`boolean`	`isBlacklisted(String trackerID)` Whether the tracker is blacklisted or not
`JobStatus[]`	`jobsToComplete()` Get the jobs that are not completed and not failed
`void`	`killJob(JobID jobid)` Kill the indicated job
`boolean`	`killTask(TaskAttemptID taskid, boolean shouldFail)` Kill indicated task attempt.
`static void`	`main(String[] argv)` Start the JobTracker process.
`void`	`offerService()` Run forever
`void`	`refreshNodes()` Rereads the config to get hosts and exclude list file names.
`void`	`refreshQueues()` Refresh the queue acls in use currently.
`void`	`refreshServiceAcl()` Refresh the service-level authorization policy in-effect.
`void`	`refreshSuperUserGroupsConfiguration()` Refresh superuser proxy group list
`void`	`refreshUserToGroupsMappings()` Refresh user to group mappings.
`void`	`removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)` Unregisters a `JobInProgressListener` from this `TaskTrackerManager`.
`long`	`renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)` Renew a delegation token to extend its lifetime.
`void`	`reportTaskTrackerError(String taskTracker, String errorClass, String errorMessage)` Report a problem to the job tracker.
`org.apache.hadoop.net.Node`	`resolveAndAddToTopology(String name)`
`Vector<JobInProgress>`	`runningJobs()`
`static void`	`setInstrumentationClass(org.apache.hadoop.conf.Configuration conf, Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> t)`
`void`	`setJobPriority(JobID jobid, String priority)` Set the priority of the specified job
`static JobTracker`	`startTracker(JobConf conf)` Start the JobTracker with given configuration.
`static JobTracker`	`startTracker(JobConf conf, String identifier)`
`void`	`stopTracker()`
`JobStatus`	`submitJob(JobID jobId, String jobSubmitDir, org.apache.hadoop.security.Credentials ts)` JobTracker.submitJob() kicks off a new job.
`List<List<String>>`	`taskTrackerNames()` Get the active and blacklisted task tracker names in the cluster.
`Collection<TaskTrackerStatus>`	`taskTrackers()` Get all the task trackers in the cluster

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - MAX_USER_JOBCONF_SIZE_KEY
```
public static final String MAX_USER_JOBCONF_SIZE_KEY
```
    the config key for max user jobconf size
    
    See Also:
    Constant Field Values
  - DELEGATION_KEY_UPDATE_INTERVAL_KEY
```
public static final String DELEGATION_KEY_UPDATE_INTERVAL_KEY
```
    See Also:
    Constant Field Values
  - DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT
```
public static final long DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT
```
    See Also:
    Constant Field Values
  - DELEGATION_TOKEN_RENEW_INTERVAL_KEY
```
public static final String DELEGATION_TOKEN_RENEW_INTERVAL_KEY
```
    See Also:
    Constant Field Values
  - DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
```
public static final long DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
```
    See Also:
    Constant Field Values
  - DELEGATION_TOKEN_MAX_LIFETIME_KEY
```
public static final String DELEGATION_TOKEN_MAX_LIFETIME_KEY
```
    See Also:
    Constant Field Values
  - DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
```
public static final long DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
```
    See Also:
    Constant Field Values
  - SHUFFLE_SSL_ENABLED_KEY
```
public static final String SHUFFLE_SSL_ENABLED_KEY
```
    See Also:
    Constant Field Values
  - SHUFFLE_SSL_ENABLED_DEFAULT
```
public static final boolean SHUFFLE_SSL_ENABLED_DEFAULT
```
    See Also:
    Constant Field Values
  - SHUFFLE_SSL_ADDRESS_KEY
```
public static final String SHUFFLE_SSL_ADDRESS_KEY
```
    See Also:
    Constant Field Values
  - SHUFFLE_SSL_ADDRESS_DEFAULT
```
public static final String SHUFFLE_SSL_ADDRESS_DEFAULT
```
    See Also:
    Constant Field Values
  - SHUFFLE_SSL_PORT_KEY
```
public static final String SHUFFLE_SSL_PORT_KEY
```
    See Also:
    Constant Field Values
  - SHUFFLE_SSL_PORT_DEFAULT
```
public static final int SHUFFLE_SSL_PORT_DEFAULT
```
    See Also:
    Constant Field Values
  - LOG
```
public static final org.apache.commons.logging.Log LOG
```
  - JT_USER_NAME
```
public static final String JT_USER_NAME
```
    See Also:
    Constant Field Values
  - JT_KEYTAB_FILE
```
public static final String JT_KEYTAB_FILE
```
    See Also:
    Constant Field Values
  - MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY
```
public static final String MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY
```
    See Also:
    Constant Field Values
  - MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY
```
public static final String MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY
```
    See Also:
    Constant Field Values
  - HEARTBEAT_INTERVAL_MIN_DEFAULT
```
public static final int HEARTBEAT_INTERVAL_MIN_DEFAULT
```
    See Also:
    Constant Field Values
  - COUNTER_UPDATE_INTERVAL
```
public static final long COUNTER_UPDATE_INTERVAL
```
    See Also:
    Constant Field Values
  - DEFAULT_DISK_HEALTH_CHECK_INTERVAL
```
public static final long DEFAULT_DISK_HEALTH_CHECK_INTERVAL
```
    How often TaskTracker needs to check the health of its disks, if not configured using mapred.disk.healthChecker.interval
    
    See Also:
    Constant Field Values
  - SUCCESS
```
public static final int SUCCESS
```
    See Also:
    Constant Field Values
  - FILE_NOT_FOUND
```
public static final int FILE_NOT_FOUND
```
    See Also:
    Constant Field Values
  - MAP_OUTPUT_LENGTH
```
public static final String MAP_OUTPUT_LENGTH
```
    The custom http header used for the map output length.
    
    See Also:
    Constant Field Values
  - RAW_MAP_OUTPUT_LENGTH
```
public static final String RAW_MAP_OUTPUT_LENGTH
```
    The custom http header used for the "raw" map output length.
    
    See Also:
    Constant Field Values
  - FROM_MAP_TASK
```
public static final String FROM_MAP_TASK
```
    The map task from which the map output data is being transferred
    
    See Also:
    Constant Field Values
  - FOR_REDUCE_TASK
```
public static final String FOR_REDUCE_TASK
```
    The reduce task number for which this map output is being transferred
    
    See Also:
    Constant Field Values
  - WORKDIR
```
public static final String WORKDIR
```
    See Also:
    Constant Field Values
  - versionID
```
public static final long versionID
```
    version 3 introduced to replace emitHearbeat/pollForNewTask/pollForTaskWithClosedJob with heartbeat(TaskTrackerStatus, boolean, boolean, boolean, short) version 4 changed TaskReport for HADOOP-549. version 5 introduced that removes locateMapOutputs and instead uses getTaskCompletionEvents to figure finished maps and fetch the outputs version 6 adds maxTasks to TaskTrackerStatus for HADOOP-1245 version 7 replaces maxTasks by maxMapTasks and maxReduceTasks in TaskTrackerStatus for HADOOP-1274 Version 8: HeartbeatResponse is added with the next heartbeat interval. version 9 changes the counter representation for HADOOP-2248 version 10 changes the TaskStatus representation for HADOOP-2208 version 11 changes string to JobID in getTaskCompletionEvents(). version 12 changes the counters representation for HADOOP-1915 version 13 added call getBuildVersion() for HADOOP-236 Version 14: replaced getFilesystemName with getSystemDir for HADOOP-3135 Version 15: Changed format of Task and TaskStatus for HADOOP-153 Version 16: adds ResourceStatus to TaskTrackerStatus for HADOOP-3759 Version 17: Changed format of Task and TaskStatus for HADOOP-3150 Version 18: Changed status message due to changes in TaskStatus Version 19: Changed heartbeat to piggyback JobTracker restart information so that the TaskTracker can synchronize itself. Version 20: Changed status message due to changes in TaskStatus (HADOOP-4232) Version 21: Changed information reported in TaskTrackerStatus' ResourceStatus and the corresponding accessor methods (HADOOP-4035) Version 22: Replaced parameter 'initialContact' with 'restarted' in heartbeat method (HADOOP-4305) Version 23: Added parameter 'initialContact' again in heartbeat method (HADOOP-4869) Version 24: Changed format of Task and TaskStatus for HADOOP-4759 Version 25: JobIDs are passed in response to JobTracker restart Version 26: Added numRequiredSlots to TaskStatus for MAPREDUCE-516 Version 27: Adding node health status to TaskStatus for MAPREDUCE-211 Version 28: Adding user name to the serialized Task for use by TT. Version 29: Adding available memory and CPU usage information on TT to TaskTrackerStatus for MAPREDUCE-1218 Version 30: Adding disk failure to TaskTrackerStatus for MAPREDUCE-3015 Version 31: Efficient serialization format for Framework counters (MAPREDUCE-901) Version 32: Adding version methods for HADOOP-8209
    
    See Also:
    Constant Field Values
  - TRACKERS_OK
```
public static final int TRACKERS_OK
```
    See Also:
    Constant Field Values
  - UNKNOWN_TASKTRACKER
```
public static final int UNKNOWN_TASKTRACKER
```
    See Also:
    Constant Field Values
  - versionID
```
public static final long versionID
```
    See Also:
    Constant Field Values
- Method Detail
  - getClock
```
public org.apache.hadoop.mapred.Clock getClock()
```
  - startTracker
```
public static JobTracker startTracker(JobConf conf)
                               throws IOException,
                                      InterruptedException
```
    Start the JobTracker with given configuration. The conf will be modified to reflect the actual ports on which the JobTracker is up and running if the user passes the port as zero.
    
    Parameters:
    conf - configuration for the JobTracker.
    
    Throws:
    
    IOException
    
    InterruptedException
  - startTracker
```
public static JobTracker startTracker(JobConf conf,
                      String identifier)
                               throws IOException,
                                      InterruptedException
```
    Throws:
    
    IOException
    
    InterruptedException
  - stopTracker
```
public void stopTracker()
                 throws IOException
```
    Throws:
    
    IOException
  - getProtocolVersion
```
public long getProtocolVersion(String protocol,
                      long clientVersion)
                        throws IOException
```
    Specified by:
    
    getProtocolVersion in interface org.apache.hadoop.ipc.VersionedProtocol
    
    Throws:
    
    IOException
  - getProtocolSignature
```
public org.apache.hadoop.ipc.ProtocolSignature getProtocolSignature(String protocol,
                                                           long clientVersion,
                                                           int clientMethodsHash)
                                                             throws IOException
```
    Specified by:
    
    getProtocolSignature in interface org.apache.hadoop.ipc.VersionedProtocol
    
    Throws:
    
    IOException
  - getDelegationTokenSecretManager
```
public DelegationTokenSecretManager getDelegationTokenSecretManager()
```
  - hasRestarted
```
public boolean hasRestarted()
```
    Whether the JT has restarted
  - hasRecovered
```
public boolean hasRecovered()
```
    Whether the JT has recovered upon restart
  - getRecoveryDuration
```
public long getRecoveryDuration()
```
    How long the jobtracker took to recover from restart.
  - getInstrumentationClass
```
public static Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> getInstrumentationClass(org.apache.hadoop.conf.Configuration conf)
```
  - setInstrumentationClass
```
public static void setInstrumentationClass(org.apache.hadoop.conf.Configuration conf,
                           Class<? extends org.apache.hadoop.mapred.JobTrackerInstrumentation> t)
```
  - getAddress
```
public static InetSocketAddress getAddress(org.apache.hadoop.conf.Configuration conf)
```
  - offerService
```
public void offerService()
                  throws InterruptedException,
                         IOException
```
    Run forever
    
    Throws:
    
    InterruptedException
    
    IOException
  - getTotalSubmissions
```
public int getTotalSubmissions()
```
  - getJobTrackerMachine
```
public String getJobTrackerMachine()
```
  - getTrackerIdentifier
```
public String getTrackerIdentifier()
```
    Get the unique identifier (ie. timestamp) of this job tracker start.
    
    Returns:
    a string with a unique identifier
  - getTrackerPort
```
public int getTrackerPort()
```
  - getInfoPort
```
public int getInfoPort()
```
  - getStartTime
```
public long getStartTime()
```
  - runningJobs
```
public Vector<JobInProgress> runningJobs()
```
  - getRunningJobs
```
public List<JobInProgress> getRunningJobs()
```
    Version that is called from a timer thread, and therefore needs to be careful to synchronize.
  - failedJobs
```
public Vector<JobInProgress> failedJobs()
```
  - getFailedJobs
```
public List<JobInProgress> getFailedJobs()
```
  - completedJobs
```
public Vector<JobInProgress> completedJobs()
```
  - getCompletedJobs
```
public List<JobInProgress> getCompletedJobs()
```
  - taskTrackers
```
public Collection<TaskTrackerStatus> taskTrackers()
```
    Get all the task trackers in the cluster
    
    Returns:
    Collection of TaskTrackerStatus
  - activeTaskTrackers
```
public Collection<TaskTrackerStatus> activeTaskTrackers()
```
    Get the active task tracker statuses in the cluster
    
    Returns:
    Collection of active TaskTrackerStatus
  - taskTrackerNames
```
public List<List<String>> taskTrackerNames()
```
    Get the active and blacklisted task tracker names in the cluster. The first element in the returned list contains the list of active tracker names. The second element in the returned list contains the list of blacklisted tracker names.
  - blacklistedTaskTrackers
```
public Collection<TaskTrackerStatus> blacklistedTaskTrackers()
```
    Get the blacklisted task tracker statuses in the cluster
    
    Returns:
    Collection of blacklisted TaskTrackerStatus
  - isBlacklisted
```
public boolean isBlacklisted(String trackerID)
```
    Whether the tracker is blacklisted or not
    
    Parameters:
    trackerID -
    
    Returns:
    true if blacklisted, false otherwise
  - getTaskTrackerStatus
```
public TaskTrackerStatus getTaskTrackerStatus(String trackerID)
```
  - getTaskTracker
```
public TaskTracker getTaskTracker(String trackerID)
```
  - resolveAndAddToTopology
```
public org.apache.hadoop.net.Node resolveAndAddToTopology(String name)
```
  - getNodesAtMaxLevel
```
public Collection<org.apache.hadoop.net.Node> getNodesAtMaxLevel()
```
    Returns a collection of nodes at the max level
  - getParentNode
```
public static org.apache.hadoop.net.Node getParentNode(org.apache.hadoop.net.Node node,
                                       int level)
```
  - getNode
```
public org.apache.hadoop.net.Node getNode(String name)
```
    Return the Node in the network topology that corresponds to the hostname
  - getNumTaskCacheLevels
```
public int getNumTaskCacheLevels()
```
  - getNumResolvedTaskTrackers
```
public int getNumResolvedTaskTrackers()
```
  - getNumberOfUniqueHosts
```
public int getNumberOfUniqueHosts()
```
    Returns:
    The number of unique hosts running tasktrackers.
  - addJobInProgressListener
```
public void addJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)
```
    Registers a JobInProgressListener for updates from this TaskTrackerManager.
  - removeJobInProgressListener
```
public void removeJobInProgressListener(org.apache.hadoop.mapred.JobInProgressListener listener)
```
    Unregisters a JobInProgressListener from this TaskTrackerManager.
  - getQueueManager
```
public org.apache.hadoop.mapred.QueueManager getQueueManager()
```
    Return the QueueManager associated with the JobTracker.
    
    Returns:
    the QueueManager
  - getVIVersion
```
public String getVIVersion()
                    throws IOException
```
    Returns the VersionInfo version of the JobTracker
    
    Throws:
    
    IOException
  - getBuildVersion
```
public String getBuildVersion()
                       throws IOException
```
    Returns the VersionInfo build version of the JobTracker
    
    Throws:
    
    IOException
  - heartbeat
```
public org.apache.hadoop.mapred.HeartbeatResponse heartbeat(TaskTrackerStatus status,
                                                   boolean restarted,
                                                   boolean initialContact,
                                                   boolean acceptNewTasks,
                                                   short responseId)
                                                     throws IOException
```
    The periodic heartbeat mechanism between the TaskTracker and the JobTracker. The JobTracker processes the status information sent by the TaskTracker and responds with instructions to start/stop tasks or jobs, and also 'reset' instructions during contingencies.
    
    Parameters:
    status - the status update
    restarted - true if the process has just started or restarted, false otherwise
    initialContact - true if this is first interaction since 'refresh', false otherwise.
    acceptNewTasks - true if the TaskTracker is ready to accept new tasks to run.
    responseId - the last responseId successfully acted upon by the TaskTracker.
    
    Returns:
    a HeartbeatResponse with fresh instructions.
    
    Throws:
    
    IOException
  - getNextHeartbeatInterval
```
public int getNextHeartbeatInterval()
```
    Calculates next heartbeat interval using cluster size. Heartbeat interval is incremented by 1 second for every 100 nodes by default.
    
    Returns:
    next heartbeat interval.
  - getFilesystemName
```
public String getFilesystemName()
                         throws IOException
```
    Grab the local fs name
    
    Throws:
    
    IOException
  - getConf
```
public JobConf getConf()
```
    Returns a handle to the JobTracker's Configuration
  - reportTaskTrackerError
```
public void reportTaskTrackerError(String taskTracker,
                          String errorClass,
                          String errorMessage)
                            throws IOException
```
    Report a problem to the job tracker.
    
    Parameters:
    taskTracker - the name of the task tracker
    errorClass - the kind of error (eg. the class that was thrown)
    errorMessage - the human readable error message
    
    Throws:
    
    IOException - if there was a problem in communication or on the remote side
  - getNewJobId
```
public JobID getNewJobId()
                  throws IOException
```
    Allocates a new JobId string.
    
    Returns:
    a unique job name for submitting jobs.
    
    Throws:
    
    IOException
  - submitJob
```
public JobStatus submitJob(JobID jobId,
                  String jobSubmitDir,
                  org.apache.hadoop.security.Credentials ts)
                    throws IOException
```
    JobTracker.submitJob() kicks off a new job. Create a 'JobInProgress' object, which contains both JobProfile and JobStatus. Those two sub-objects are sometimes shipped outside of the JobTracker. But JobInProgress adds info that's useful for the JobTracker alone.
    
    Throws:
    
    IOException
  - getStagingAreaDir
```
public String getStagingAreaDir()
                         throws IOException
```
    Get a hint from the JobTracker where job-specific files are to be placed.
    
    Returns:
    the directory where job-specific files are to be placed.
    
    Throws:
    
    IOException
    See Also:
    JobSubmissionProtocol.getStagingAreaDir()
  - getClusterStatus
```
@Deprecated
public ClusterStatus getClusterStatus()
```
    Deprecated. use getClusterStatus(boolean)
    
    Returns:
    a summary of the cluster's status.
  - getClusterStatus
```
public ClusterStatus getClusterStatus(boolean detailed)
```
    Get the current status of the cluster
    
    Parameters:
    detailed - if true then report tracker names and memory usage
    
    Returns:
    summary of the state of the cluster
  - getClusterMetrics
```
public ClusterMetrics getClusterMetrics()
```
  - killJob
```
public void killJob(JobID jobid)
             throws IOException
```
    Kill the indicated job
    
    Throws:
    
    IOException
    See Also:
    JobSubmissionProtocol.killJob(org.apache.hadoop.mapred.JobID)
  - cancelDelegationToken
```
public void cancelDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                           throws IOException,
                                  InterruptedException
```
    Discard a current delegation token.
    
    Parameters:
    token - the token to cancel
    
    Throws:
    
    IOException
    
    InterruptedException
  - getDelegationToken
```
public org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> getDelegationToken(org.apache.hadoop.io.Text renewer)
                                                                                     throws IOException,
                                                                                            InterruptedException
```
    Get a new delegation token.
    
    Parameters:
    renewer - the user other than the creator (if any) that can renew the token
    
    Returns:
    the new delegation token
    
    Throws:
    
    IOException
    
    InterruptedException
  - renewDelegationToken
```
public long renewDelegationToken(org.apache.hadoop.security.token.Token<DelegationTokenIdentifier> token)
                          throws IOException,
                                 InterruptedException
```
    Renew a delegation token to extend its lifetime.
    
    Parameters:
    token - the token to renew
    
    Returns:
    the new expiration time
    
    Throws:
    
    IOException
    
    InterruptedException
  - initJob
```
public void initJob(JobInProgress job)
```
    Initialize the Job
    
    Parameters:
    job - JobInProgress object
  - failJob
```
public void failJob(JobInProgress job)
```
    Fail a job and inform the listeners. Other components in the framework should use this to fail a job.
    
    Parameters:
    job - JobInProgress object
  - setJobPriority
```
public void setJobPriority(JobID jobid,
                  String priority)
                    throws IOException
```
    Set the priority of the specified job
    
    Parameters:
    jobid - ID of the job
    priority - Priority to be set for the job
    
    Throws:
    
    IOException
  - getJobProfile
```
public JobProfile getJobProfile(JobID jobid)
```
    Grab a handle to a job that is already known to the JobTracker.
    
    Returns:
    Profile of the job, or null if not found.
  - getJobStatus
```
public JobStatus getJobStatus(JobID jobid)
```
    Grab a handle to a job that is already known to the JobTracker.
    
    Returns:
    Status of the job, or null if not found.
  - getJobCounters
```
public Counters getJobCounters(JobID jobid)
                        throws IOException
```
    Grab the current job counters
    
    Throws:
    
    IOException
  - getMapTaskReports
```
public TaskReport[] getMapTaskReports(JobID jobid)
                               throws IOException
```
    Grab a bunch of info on the map tasks that make up the job
    
    Throws:
    
    IOException
  - getReduceTaskReports
```
public TaskReport[] getReduceTaskReports(JobID jobid)
                                  throws IOException
```
    Grab a bunch of info on the reduce tasks that make up the job
    
    Throws:
    
    IOException
  - getCleanupTaskReports
```
public TaskReport[] getCleanupTaskReports(JobID jobid)
                                   throws IOException
```
    Grab a bunch of info on the cleanup tasks that make up the job
    
    Throws:
    
    IOException
  - getSetupTaskReports
```
public TaskReport[] getSetupTaskReports(JobID jobid)
                                 throws IOException
```
    Grab a bunch of info on the setup tasks that make up the job
    
    Throws:
    
    IOException
  - getTaskCompletionEvents
```
public TaskCompletionEvent[] getTaskCompletionEvents(JobID jobid,
                                            int fromEventId,
                                            int maxEvents)
                                              throws IOException
```
    Get task completion events for the jobid, starting from fromEventId. Returns empty aray if no events are available.
    
    Parameters:
    jobid - job id
    fromEventId - event id to start from.
    maxEvents - the max number of events we want to look at
    
    Returns:
    array of task completion events.
    
    Throws:
    
    IOException
  - getTaskDiagnostics
```
public String[] getTaskDiagnostics(TaskAttemptID taskId)
                            throws IOException
```
    Get the diagnostics for a given task
    
    Parameters:
    taskId - the id of the task
    
    Returns:
    an array of the diagnostic messages
    
    Throws:
    
    IOException
  - getTip
```
public TaskInProgress getTip(TaskID tipid)
```
    Returns specified TaskInProgress, or null.
  - killTask
```
public boolean killTask(TaskAttemptID taskid,
               boolean shouldFail)
                 throws IOException
```
    Kill indicated task attempt.
    
    Parameters:
    taskid - the id of the task to kill.
    shouldFail - if true the task is failed and added to failed tasks list, otherwise it is just killed, w/o affecting job failure status.
    
    Returns:
    true if the task was found and successfully marked to kill
    
    Throws:
    
    IOException
    See Also:
    JobSubmissionProtocol.killTask(TaskAttemptID, boolean)
  - getAssignedTracker
```
public String getAssignedTracker(TaskAttemptID taskId)
```
    Get tracker name for a given task id.
    
    Parameters:
    taskId - the name of the task
    
    Returns:
    The name of the task tracker
  - jobsToComplete
```
public JobStatus[] jobsToComplete()
```
    Get the jobs that are not completed and not failed
    
    Returns:
    array of JobStatus for the running/to-be-run jobs.
  - getAllJobs
```
public JobStatus[] getAllJobs()
```
    Get all the jobs submitted.
    
    Returns:
    array of JobStatus for the submitted jobs
    See Also:
    JobSubmissionProtocol.getAllJobs()
  - getSystemDir
```
public String getSystemDir()
```
    Grab the jobtracker system directory path where job-specific files are to be placed.
    
    Returns:
    the system directory where job-specific files are to be placed.
    See Also:
    JobSubmissionProtocol.getSystemDir()
  - getQueueAdmins
```
public org.apache.hadoop.security.authorize.AccessControlList getQueueAdmins(String queueName)
                                                                      throws IOException
```
    Get the administrators of the given job-queue. This method is for hadoop internal use only.
    
    Returns:
    Queue administrators ACL for the queue to which job is submitted to
    
    Throws:
    
    IOException
    See Also:
    JobSubmissionProtocol.getQueueAdmins(String)
  - getJob
```
public JobInProgress getJob(JobID jobid)
```
    Obtain the job object identified by jobid
    
    Returns:
    jobInProgress object
  - refreshNodes
```
public void refreshNodes()
                  throws IOException
```
    Rereads the config to get hosts and exclude list file names. Rereads the files to update the hosts and exclude lists.
    
    Specified by:
    
    refreshNodes in interface AdminOperationsProtocol
    
    Throws:
    
    IOException
  - getLocalJobFilePath
```
public static String getLocalJobFilePath(JobID jobId)
```
    Get the localized job file path on the job trackers local file system
    
    Parameters:
    jobId - id of the job
    
    Returns:
    the path of the job conf file on the local file system
  - main
```
public static void main(String[] argv)
                 throws IOException,
                        InterruptedException
```
    Start the JobTracker process. This is used only for debugging. As a rule, JobTracker should be run as part of the DFS Namenode process.
    
    Throws:
    
    IOException
    
    InterruptedException
  - getQueues
```
public JobQueueInfo[] getQueues()
                         throws IOException
```
    Gets set of Job Queues associated with the Job Tracker
    
    Returns:
    Array of the Job Queue Information Object
    
    Throws:
    
    IOException
  - getQueueInfo
```
public JobQueueInfo getQueueInfo(String queue)
                          throws IOException
```
    Gets scheduling information associated with the particular Job queue
    
    Parameters:
    queue - Queue Name
    
    Returns:
    Scheduling Information of the Queue
    
    Throws:
    
    IOException
  - getJobsFromQueue
```
public JobStatus[] getJobsFromQueue(String queue)
                             throws IOException
```
    Gets all the jobs submitted to the particular Queue
    
    Parameters:
    queue - Queue name
    
    Returns:
    array of JobStatus for the submitted jobs
    
    Throws:
    
    IOException
  - getQueueAclsForCurrentUser
```
public org.apache.hadoop.mapred.QueueAclsInfo[] getQueueAclsForCurrentUser()
                                                                    throws IOException
```
    Gets the Queue ACLs for current user
    
    Returns:
    array of QueueAclsInfo object for current user.
    
    Throws:
    
    IOException
  - refreshServiceAcl
```
public void refreshServiceAcl()
                       throws IOException
```
    Description copied from interface: RefreshAuthorizationPolicyProtocol
    
    Refresh the service-level authorization policy in-effect.
    
    Specified by:
    
    refreshServiceAcl in interface RefreshAuthorizationPolicyProtocol
    
    Throws:
    
    IOException
  - refreshSuperUserGroupsConfiguration
```
public void refreshSuperUserGroupsConfiguration()
```
    Description copied from interface: RefreshUserMappingsProtocol
    
    Refresh superuser proxy group list
    
    Specified by:
    
    refreshSuperUserGroupsConfiguration in interface RefreshUserMappingsProtocol
  - getGroupsForUser
```
public String[] getGroupsForUser(String user)
                          throws IOException
```
    Description copied from interface: GetUserMappingsProtocol
    
    Get the groups which are mapped to the given user.
    
    Specified by:
    
    getGroupsForUser in interface GetUserMappingsProtocol
    
    Parameters:
    user - The user to get the groups for.
    
    Returns:
    The set of groups the user belongs to.
    
    Throws:
    
    IOException
  - refreshUserToGroupsMappings
```
public void refreshUserToGroupsMappings()
                                 throws IOException
```
    Description copied from interface: RefreshUserMappingsProtocol
    
    Refresh user to group mappings.
    
    Specified by:
    
    refreshUserToGroupsMappings in interface RefreshUserMappingsProtocol
    
    Throws:
    
    IOException
  - refreshQueues
```
public void refreshQueues()
                   throws IOException
```
    Description copied from interface: AdminOperationsProtocol
    
    Refresh the queue acls in use currently. Refresh the queues used by the jobtracker and scheduler. Access control lists and queue states are refreshed.
    
    Specified by:
    
    refreshQueues in interface AdminOperationsProtocol
    
    Throws:
    
    IOException
  - getHostname
```
public String getHostname()
```
    Specified by:
    
    getHostname in interface JobTrackerMXBean
    
    Returns:
    hostname of the jobtracker
  - getVersion
```
public String getVersion()
```
    Specified by:
    
    getVersion in interface JobTrackerMXBean
    
    Returns:
    version of the code base
  - getConfigVersion
```
public String getConfigVersion()
```
    Specified by:
    
    getConfigVersion in interface JobTrackerMXBean
    
    Returns:
    the config version (from a config property)
  - getThreadCount
```
public int getThreadCount()
```
    Specified by:
    
    getThreadCount in interface JobTrackerMXBean
    
    Returns:
    number of threads of the jobtracker jvm
  - getSummaryJson
```
public String getSummaryJson()
```
    Specified by:
    
    getSummaryJson in interface JobTrackerMXBean
    
    Returns:
    the summary info in json
  - getAliveNodesInfoJson
```
public String getAliveNodesInfoJson()
```
    Specified by:
    
    getAliveNodesInfoJson in interface JobTrackerMXBean
    
    Returns:
    the alive nodes info in json
  - getBlacklistedNodesInfoJson
```
public String getBlacklistedNodesInfoJson()
```
    Specified by:
    
    getBlacklistedNodesInfoJson in interface JobTrackerMXBean
    
    Returns:
    the blacklisted nodes info in json
  - getQueueInfoJson
```
public String getQueueInfoJson()
```
    Specified by:
    
    getQueueInfoJson in interface JobTrackerMXBean
    
    Returns:
    the queue info json

Class JobTracker

Nested Class Summary

Field Summary

Fields inherited from interface org.apache.hadoop.mr1security.RefreshUserMappingsProtocol

Fields inherited from interface org.apache.hadoop.mr1tools.GetUserMappingsProtocol

Fields inherited from interface org.apache.hadoop.mr1security.authorize.RefreshAuthorizationPolicyProtocol

Fields inherited from interface org.apache.hadoop.mapred.AdminOperationsProtocol

Method Summary

Methods inherited from class java.lang.Object

Field Detail

MAX_USER_JOBCONF_SIZE_KEY

DELEGATION_KEY_UPDATE_INTERVAL_KEY

DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT

DELEGATION_TOKEN_RENEW_INTERVAL_KEY

DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT

DELEGATION_TOKEN_MAX_LIFETIME_KEY

DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT

SHUFFLE_SSL_ENABLED_KEY

SHUFFLE_SSL_ENABLED_DEFAULT

SHUFFLE_SSL_ADDRESS_KEY

SHUFFLE_SSL_ADDRESS_DEFAULT

SHUFFLE_SSL_PORT_KEY

SHUFFLE_SSL_PORT_DEFAULT

LOG

JT_USER_NAME

JT_KEYTAB_FILE

MAPRED_CLUSTER_MAP_MEMORY_MB_PROPERTY

MAPRED_CLUSTER_REDUCE_MEMORY_MB_PROPERTY

HEARTBEAT_INTERVAL_MIN_DEFAULT

COUNTER_UPDATE_INTERVAL

DEFAULT_DISK_HEALTH_CHECK_INTERVAL

SUCCESS

FILE_NOT_FOUND

MAP_OUTPUT_LENGTH

RAW_MAP_OUTPUT_LENGTH

FROM_MAP_TASK

FOR_REDUCE_TASK

WORKDIR

versionID

TRACKERS_OK

UNKNOWN_TASKTRACKER

versionID

Method Detail

getClock

startTracker

startTracker

stopTracker

getProtocolVersion

getProtocolSignature

getDelegationTokenSecretManager

hasRestarted

hasRecovered

getRecoveryDuration

getInstrumentationClass

setInstrumentationClass

getAddress

offerService

getTotalSubmissions

getJobTrackerMachine

getTrackerIdentifier

getTrackerPort

getInfoPort

getStartTime

runningJobs

getRunningJobs

failedJobs

getFailedJobs

completedJobs

getCompletedJobs

taskTrackers

activeTaskTrackers

taskTrackerNames

blacklistedTaskTrackers

isBlacklisted

getTaskTrackerStatus

getTaskTracker

resolveAndAddToTopology

getNodesAtMaxLevel

getParentNode

getNode