org.apache.pig.backend.hadoop.executionengine.mapReduceLayer

Class PigOutputFormat

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat

public class PigOutputFormat
extends org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

The better half of PigInputFormat which is responsible for the Store functionality. It is the exact mirror image of PigInputFormat having RecordWriter instead of a RecordReader.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	PigOutputFormat.PigRecordWriter Wrapper class which will delegate calls to the actual RecordWriter - this should only get called in the single store case.

Field Summary

Fields

Modifier and Type	Field and Description
static String	MAPRED_OUTPUT_DIR hadoop job output directory
static String	MAPRED_TASK_PARTITION hadoop partition number
static String	PIG_MAPRED_OUTPUT_DIR the temporary directory for the multi store
static String	PIG_TMP_PATH the relative path that can be used to build a temporary place to store the output from a number of map-reduce tasks

Constructor Summary

Constructors

Constructor and Description
PigOutputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobcontext)
org.apache.hadoop.mapreduce.OutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.WritableComparable,Tuple>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
static void	setLocation(org.apache.hadoop.mapreduce.JobContext jobContext, POStore store) Before delegating calls to underlying OutputFormat or OutputCommitter Pig needs to ensure the Configuration in the JobContext contains the output location and StoreFunc for the specific store - so set these up in the context for this specific store

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAPRED_OUTPUT_DIR

public static final String MAPRED_OUTPUT_DIR

hadoop job output directory

See Also:

Constant Field Values

MAPRED_TASK_PARTITION

public static final String MAPRED_TASK_PARTITION

hadoop partition number

See Also:

Constant Field Values

PIG_MAPRED_OUTPUT_DIR

public static final String PIG_MAPRED_OUTPUT_DIR

the temporary directory for the multi store

See Also:

Constant Field Values

PIG_TMP_PATH

public static final String PIG_TMP_PATH

the relative path that can be used to build a temporary place to store the output from a number of map-reduce tasks

See Also:

Constant Field Values

Constructor Detail

PigOutputFormat

public PigOutputFormat()

Method Detail

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<org.apache.hadoop.io.WritableComparable,Tuple> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
                                                                                                        throws IOException,
                                                                                                               InterruptedException

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

Throws:

IOException

InterruptedException

setLocation

public static void setLocation(org.apache.hadoop.mapreduce.JobContext jobContext,
               POStore store)
                        throws IOException

Before delegating calls to underlying OutputFormat or OutputCommitter Pig needs to ensure the Configuration in the JobContext contains the output location and StoreFunc for the specific store - so set these up in the context for this specific store

Parameters:

jobContext - the JobContext

store - the POStore

Throws:

IOException - on failure

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext jobcontext)
                      throws IOException,
                             InterruptedException

Specified by:

checkOutputSpecs in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

Throws:

IOException

InterruptedException

getOutputCommitter

public org.apache.hadoop.mapreduce.OutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext taskattemptcontext)
                                                               throws IOException,
                                                                      InterruptedException

Specified by:

getOutputCommitter in class org.apache.hadoop.mapreduce.OutputFormat<org.apache.hadoop.io.WritableComparable,Tuple>

Throws:

IOException

InterruptedException