org.apache.hadoop.mapred

Class HAUtil

java.lang.Object

org.apache.hadoop.mapred.HAUtil

public class HAUtil
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static String	HA_DT_SERVICE_PREFIX A prefix put before the jobtracker address inside the "service" field of a delegation token, indicating that the address is a logical (HA) address.
static String[]	JOB_TRACKER_SPECIFIC_KEYS
static int	MR_CLIENT_FAILOVER_CONNECTION_RETRIES_DEFAULT
static String	MR_CLIENT_FAILOVER_CONNECTION_RETRIES_KEY
static int	MR_CLIENT_FAILOVER_CONNECTION_RETRIES_ON_SOCKET_TIMEOUTS_DEFAULT
static String	MR_CLIENT_FAILOVER_CONNECTION_RETRIES_ON_SOCKET_TIMEOUTS_KEY
static int	MR_CLIENT_FAILOVER_MAX_ATTEMPTS_DEFAULT
static String	MR_CLIENT_FAILOVER_MAX_ATTEMPTS_KEY
static String	MR_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX
static int	MR_CLIENT_FAILOVER_SLEEPTIME_BASE_DEFAULT
static String	MR_CLIENT_FAILOVER_SLEEPTIME_BASE_KEY
static int	MR_CLIENT_FAILOVER_SLEEPTIME_MAX_DEFAULT
static String	MR_CLIENT_FAILOVER_SLEEPTIME_MAX_KEY
static String	MR_HA_ACTIVE_CHECK_MILLIS
static int	MR_HA_ACTIVE_CHECK_MILLIS_DEFAULT
static boolean	MR_HA_AUTO_FAILOVER_ENABLED_DEFAULT
static String	MR_HA_AUTO_FAILOVER_ENABLED_KEY
static String	MR_HA_FENCING_METHODS_KEY
static String	MR_HA_JOBTRACKER_HTTP_REDIRECT_ADDRESS_KEY
static String	MR_HA_JOBTRACKER_ID_KEY
static String	MR_HA_JOBTRACKER_RPC_ADDRESS_KEY
static String	MR_HA_JOBTRACKERS_KEY_PREFIX
static int	MR_HA_ZKFC_PORT_DEFAULT
static String	MR_HA_ZKFC_PORT_KEY
static String	MR_JOBTRACKER_ADDRESS_KEY
static String	MR_JOBTRACKER_HTTP_ADDRESS_KEY
static String	MR_JOBTRACKER_OLD_HTTP_ADDRESS_KEY
static String	MR_JOBTRACKER_OLD_HTTP_PORT_KEY
static String	MR_JOBTRACKER_RPC_ADDRESS_KEY

Method Summary

Methods

Modifier and Type	Method and Description
static String	addKeySuffixes(String key, String... suffixes) Return configuration key of format key.suffix1.suffix2...suffixN
static org.apache.hadoop.io.Text	buildTokenServiceForLogicalAddress(String haJtAddress) Get the service name used in the delegation token for the given logical HA service.
static void	cloneDelegationTokenForLogicalAddress(org.apache.hadoop.security.UserGroupInformation ugi, String haJtAddress, Collection<InetSocketAddress> jtAddresses)
static Map<String,InetSocketAddress>	getHaJtRpcAddresses(org.apache.hadoop.conf.Configuration conf)
static String	getJobTrackerId(org.apache.hadoop.conf.Configuration conf) Get the jobtracker Id by matching the addressKey with the the address of the local node.
static String	getJobTrackerIdOfOtherNode(org.apache.hadoop.conf.Configuration conf) Get the jobtracker Id of the other node in an HA setup.
static String	getJtHaHttpRedirectAddress(org.apache.hadoop.conf.Configuration conf, String jtId)
static InetSocketAddress	getJtHaRpcAddress(org.apache.hadoop.conf.Configuration conf)
static InetSocketAddress	getJtHaRpcAddress(org.apache.hadoop.conf.Configuration conf, String jtId)
static String	getLogicalName(org.apache.hadoop.conf.Configuration conf)
static String	getLogicalName(String jtAddress)
static String	getServiceAddressFromToken(org.apache.hadoop.security.token.Token<?> token)
static boolean	isHAEnabled() Returns true if jobtracker HA is configured.
static boolean	isHAEnabled(org.apache.hadoop.conf.Configuration conf, String jtAddress) Returns true if jobtracker HA is configured.
static boolean	isTokenForLogicalAddress(org.apache.hadoop.security.token.Token<?> token)
static void	setGenericConf(org.apache.hadoop.conf.Configuration conf, String logicalName, String jtId, String... keys)
static void	setJtHaRpcAddress(org.apache.hadoop.conf.Configuration conf, String address)
static void	setJtRpcAddress(org.apache.hadoop.conf.Configuration conf) Set the JT address from the RPC address so that the wrapped JobTracker starts on the correct address.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MR_JOBTRACKER_ADDRESS_KEY

public static final String MR_JOBTRACKER_ADDRESS_KEY

See Also:

Constant Field Values

MR_JOBTRACKER_RPC_ADDRESS_KEY

public static final String MR_JOBTRACKER_RPC_ADDRESS_KEY

See Also:

Constant Field Values

MR_JOBTRACKER_HTTP_ADDRESS_KEY

public static final String MR_JOBTRACKER_HTTP_ADDRESS_KEY

See Also:

Constant Field Values

MR_JOBTRACKER_OLD_HTTP_ADDRESS_KEY

public static final String MR_JOBTRACKER_OLD_HTTP_ADDRESS_KEY

See Also:

Constant Field Values

MR_JOBTRACKER_OLD_HTTP_PORT_KEY

public static final String MR_JOBTRACKER_OLD_HTTP_PORT_KEY

See Also:

Constant Field Values

MR_HA_JOBTRACKERS_KEY_PREFIX

public static final String MR_HA_JOBTRACKERS_KEY_PREFIX

See Also:

Constant Field Values

MR_HA_JOBTRACKER_RPC_ADDRESS_KEY

public static final String MR_HA_JOBTRACKER_RPC_ADDRESS_KEY

See Also:

Constant Field Values

MR_HA_JOBTRACKER_HTTP_REDIRECT_ADDRESS_KEY

public static final String MR_HA_JOBTRACKER_HTTP_REDIRECT_ADDRESS_KEY

See Also:

Constant Field Values

MR_HA_JOBTRACKER_ID_KEY

public static final String MR_HA_JOBTRACKER_ID_KEY

See Also:

Constant Field Values

MR_HA_FENCING_METHODS_KEY

public static final String MR_HA_FENCING_METHODS_KEY

See Also:

Constant Field Values

MR_HA_AUTO_FAILOVER_ENABLED_KEY

public static final String MR_HA_AUTO_FAILOVER_ENABLED_KEY

See Also:

Constant Field Values

MR_HA_AUTO_FAILOVER_ENABLED_DEFAULT

public static final boolean MR_HA_AUTO_FAILOVER_ENABLED_DEFAULT

See Also:

Constant Field Values

MR_HA_ZKFC_PORT_KEY

public static final String MR_HA_ZKFC_PORT_KEY

See Also:

Constant Field Values

MR_HA_ZKFC_PORT_DEFAULT

public static final int MR_HA_ZKFC_PORT_DEFAULT

See Also:

Constant Field Values

MR_HA_ACTIVE_CHECK_MILLIS

public static final String MR_HA_ACTIVE_CHECK_MILLIS

See Also:

Constant Field Values

MR_HA_ACTIVE_CHECK_MILLIS_DEFAULT

public static final int MR_HA_ACTIVE_CHECK_MILLIS_DEFAULT

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX

public static final String MR_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_MAX_ATTEMPTS_KEY

public static final String MR_CLIENT_FAILOVER_MAX_ATTEMPTS_KEY

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_MAX_ATTEMPTS_DEFAULT

public static final int MR_CLIENT_FAILOVER_MAX_ATTEMPTS_DEFAULT

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_SLEEPTIME_BASE_KEY

public static final String MR_CLIENT_FAILOVER_SLEEPTIME_BASE_KEY

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_SLEEPTIME_BASE_DEFAULT

public static final int MR_CLIENT_FAILOVER_SLEEPTIME_BASE_DEFAULT

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_SLEEPTIME_MAX_KEY

public static final String MR_CLIENT_FAILOVER_SLEEPTIME_MAX_KEY

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_SLEEPTIME_MAX_DEFAULT

public static final int MR_CLIENT_FAILOVER_SLEEPTIME_MAX_DEFAULT

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_CONNECTION_RETRIES_KEY

public static final String MR_CLIENT_FAILOVER_CONNECTION_RETRIES_KEY

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_CONNECTION_RETRIES_DEFAULT

public static final int MR_CLIENT_FAILOVER_CONNECTION_RETRIES_DEFAULT

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_CONNECTION_RETRIES_ON_SOCKET_TIMEOUTS_KEY

public static final String MR_CLIENT_FAILOVER_CONNECTION_RETRIES_ON_SOCKET_TIMEOUTS_KEY

See Also:

Constant Field Values

MR_CLIENT_FAILOVER_CONNECTION_RETRIES_ON_SOCKET_TIMEOUTS_DEFAULT

public static final int MR_CLIENT_FAILOVER_CONNECTION_RETRIES_ON_SOCKET_TIMEOUTS_DEFAULT

See Also:

Constant Field Values

JOB_TRACKER_SPECIFIC_KEYS

public static final String[] JOB_TRACKER_SPECIFIC_KEYS

HA_DT_SERVICE_PREFIX

public static final String HA_DT_SERVICE_PREFIX

A prefix put before the jobtracker address inside the "service" field of a delegation token, indicating that the address is a logical (HA) address.

See Also:

Constant Field Values

Method Detail

isHAEnabled

public static boolean isHAEnabled()

Returns true if jobtracker HA is configured.

Returns:

true if HA is configured in the configuration; else false.

isHAEnabled

public static boolean isHAEnabled(org.apache.hadoop.conf.Configuration conf,
                  String jtAddress)

Returns true if jobtracker HA is configured.

Parameters:

conf - Configuration

jtAddress - the (logical) jobtracker address

Returns:

true if HA is configured in the configuration; else false.

getLogicalName

public static String getLogicalName(String jtAddress)

getLogicalName

public static String getLogicalName(org.apache.hadoop.conf.Configuration conf)

getJobTrackerId

public static String getJobTrackerId(org.apache.hadoop.conf.Configuration conf)

Get the jobtracker Id by matching the addressKey with the the address of the local node. If MR_HA_JOBTRACKER_ID_KEY is not specifically configured, this method determines the jobtracker Id by matching the local node's address with the configured addresses. When a match is found, it returns the JT Id from the corresponding configuration key.

Parameters:

conf - Configuration

Returns:

jobtracker Id on success, null on failure.

Throws:

org.apache.hadoop.HadoopIllegalArgumentException - on error

getJobTrackerIdOfOtherNode

public static String getJobTrackerIdOfOtherNode(org.apache.hadoop.conf.Configuration conf)

Get the jobtracker Id of the other node in an HA setup. If MR_HA_JOBTRACKER_ID_KEY is not specifically configured, this method determines the jobtracker Id by matching the local node's address with the configured addresses. When a match is found, it returns the JT Id from the corresponding configuration key.

Parameters:

conf - Configuration

Returns:

jobtracker Id on success, null on failure.

Throws:

org.apache.hadoop.HadoopIllegalArgumentException - on error

getHaJtRpcAddresses

public static Map<String,InetSocketAddress> getHaJtRpcAddresses(org.apache.hadoop.conf.Configuration conf)

getJtHaRpcAddress

public static InetSocketAddress getJtHaRpcAddress(org.apache.hadoop.conf.Configuration conf)

setJtHaRpcAddress

public static void setJtHaRpcAddress(org.apache.hadoop.conf.Configuration conf,
                     String address)

getJtHaRpcAddress

public static InetSocketAddress getJtHaRpcAddress(org.apache.hadoop.conf.Configuration conf,
                                  String jtId)

setJtRpcAddress

public static void setJtRpcAddress(org.apache.hadoop.conf.Configuration conf)

Set the JT address from the RPC address so that the wrapped JobTracker starts on the correct address.

getJtHaHttpRedirectAddress

public static String getJtHaHttpRedirectAddress(org.apache.hadoop.conf.Configuration conf,
                                String jtId)

addKeySuffixes

public static String addKeySuffixes(String key,
                    String... suffixes)

Return configuration key of format key.suffix1.suffix2...suffixN

setGenericConf

public static void setGenericConf(org.apache.hadoop.conf.Configuration conf,
                  String logicalName,
                  String jtId,
                  String... keys)

buildTokenServiceForLogicalAddress

public static org.apache.hadoop.io.Text buildTokenServiceForLogicalAddress(String haJtAddress)

Get the service name used in the delegation token for the given logical HA service.

Parameters:

haJtAddress - the logical URI of the cluster

Returns:

the service name

isTokenForLogicalAddress

public static boolean isTokenForLogicalAddress(org.apache.hadoop.security.token.Token<?> token)

Returns:

true if this token corresponds to a logical address rather than a specific jobtracker.

getServiceAddressFromToken

public static String getServiceAddressFromToken(org.apache.hadoop.security.token.Token<?> token)

cloneDelegationTokenForLogicalAddress

public static void cloneDelegationTokenForLogicalAddress(org.apache.hadoop.security.UserGroupInformation ugi,
                                         String haJtAddress,
                                         Collection<InetSocketAddress> jtAddresses)