org.apache.hadoop.mapred.nativetask

Class Platform

java.lang.Object

org.apache.hadoop.mapred.nativetask.Platform

@InterfaceAudience.Public
@InterfaceStability.Evolving
public abstract class Platform
extends Object

Base class for platforms. A platform is a framework running on top of MapReduce, like Hadoop, Hive, Pig, Mahout. Each framework defines its own key type and value type across a MapReduce job. For each platform, we should implement serializers such that we could communicate data with native side and native comparators so our native output collectors could sort them and write out. We've already provided the HadoopPlatform that supports all key types of Hadoop and users could implement their custom platform.

Field Summary

Fields

Modifier and Type	Field and Description
protected Set<String>	keyClassNames

Constructor Summary

Constructors

Constructor and Description
Platform()

Method Summary

Methods

Modifier and Type	Method and Description
protected abstract boolean	define(Class<?> keyComparator) whether it's the platform that has defined a custom Java comparator NativeTask doesn't support custom Java comparators (set with mapreduce.job.output.key.comparator.class) but a platform (e.g Pig) could also set that conf and implement native comparators so we shouldn't bail out.
abstract void	init() initialize a platform, where we should call registerKey
abstract String	name()
protected void	registerKey(String keyClassName, Class<?> key) associate a key class with its serializer and platform
protected abstract boolean	support(String keyClassName, INativeSerializer<?> serializer, JobConf job) whether a platform supports a specific key should at least satisfy two conditions 1.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

keyClassNames

protected Set<String> keyClassNames

Constructor Detail

Platform

public Platform()

Method Detail

init

public abstract void init()
                   throws IOException

initialize a platform, where we should call registerKey

Throws:

IOException

name

public abstract String name()

Returns:

name of a Platform, useful for logs and debug

registerKey

protected void registerKey(String keyClassName,
               Class<?> key)
                    throws IOException

associate a key class with its serializer and platform

Parameters:

keyClassName - map out key class name

key - key serializer class

Throws:

IOException

support

protected abstract boolean support(String keyClassName,
              INativeSerializer<?> serializer,
              JobConf job)

whether a platform supports a specific key should at least satisfy two conditions 1. the key belongs to the platform 2. the associated serializer must implement INativeComparable interface

Parameters:

keyClassName - map out put key class name

serializer - serializer associated with key via registerKey

job - job configuration

Returns:

true if the platform has implemented native comparators of the key and false otherwise

define

protected abstract boolean define(Class<?> keyComparator)

whether it's the platform that has defined a custom Java comparator NativeTask doesn't support custom Java comparators (set with mapreduce.job.output.key.comparator.class) but a platform (e.g Pig) could also set that conf and implement native comparators so we shouldn't bail out.

Parameters:

keyComparator - comparator set with mapreduce.job.output.key.comparator.class