Package | Description |
---|---|
org.apache.hadoop.io |
Generic i/o code for use when reading and writing data to the network,
to databases, and to files.
|
org.apache.hadoop.io.compress | |
org.apache.hadoop.mapred | |
org.apache.hadoop.mapreduce.lib.output | |
org.apache.hadoop.mapreduce.task.reduce |
Modifier and Type | Method and Description |
---|---|
CompressionCodec |
SequenceFile.Writer.getCompressionCodec()
Returns the compression codec of data in this file.
|
CompressionCodec |
SequenceFile.Reader.getCompressionCodec()
Returns the compression codec of data in this file.
|
Constructor and Description |
---|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
Modifier and Type | Interface and Description |
---|---|
interface |
SplittableCompressionCodec
This interface is meant to be implemented by those compression codecs
which are capable to compress / de-compress a stream starting at any
arbitrary position.
|
Modifier and Type | Class and Description |
---|---|
class |
BZip2Codec
This class provides output and input streams for bzip2 compression
and decompression.
|
class |
DefaultCodec |
class |
DeflateCodec
Alias class for DefaultCodec to enable codec discovery by 'deflate' name.
|
class |
GzipCodec
This class creates gzip compressors/decompressors.
|
class |
Lz4Codec
This class creates lz4 compressors/decompressors.
|
class |
SnappyCodec
This class creates snappy compressors/decompressors.
|
Modifier and Type | Method and Description |
---|---|
CompressionCodec |
CompressionCodecFactory.getCodec(Path file)
Find the relevant compression codec for the given file based on its
filename suffix.
|
CompressionCodec |
CompressionCodecFactory.getCodecByClassName(String classname)
Find the relevant compression codec for the codec's canonical class name.
|
CompressionCodec |
CompressionCodecFactory.getCodecByName(String codecName)
Find the relevant compression codec for the codec's canonical class name
or by codec alias.
|
Modifier and Type | Method and Description |
---|---|
Class<? extends CompressionCodec> |
CompressionCodecFactory.getCodecClassByName(String codecName)
Find the relevant compression codec for the codec's canonical class name
or by codec alias and returns its implemetation class.
|
static List<Class<? extends CompressionCodec>> |
CompressionCodecFactory.getCodecClasses(Configuration conf)
Get the list of codecs discovered via a Java ServiceLoader, or
listed in the configuration.
|
Modifier and Type | Method and Description |
---|---|
static Compressor |
CodecPool.getCompressor(CompressionCodec codec) |
static Compressor |
CodecPool.getCompressor(CompressionCodec codec,
Configuration conf)
Get a
Compressor for the given CompressionCodec from the
pool or a new one. |
static Decompressor |
CodecPool.getDecompressor(CompressionCodec codec)
Get a
Decompressor for the given CompressionCodec from the
pool or a new one. |
static int |
CodecPool.getLeasedCompressorsCount(CompressionCodec codec)
Return the number of leased
Compressor s for this
CompressionCodec |
static int |
CodecPool.getLeasedDecompressorsCount(CompressionCodec codec)
Return the number of leased
Decompressor s for this
CompressionCodec |
Modifier and Type | Method and Description |
---|---|
CompressionCodec |
ShuffleConsumerPlugin.Context.getCodec() |
Modifier and Type | Method and Description |
---|---|
Class<? extends CompressionCodec> |
JobConf.getMapOutputCompressorClass(Class<? extends CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the map outputs. |
static Class<? extends CompressionCodec> |
FileOutputFormat.getOutputCompressorClass(JobConf conf,
Class<? extends CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the job outputs. |
Modifier and Type | Method and Description |
---|---|
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
int inMemSegments,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
boolean sortSegments,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
boolean sortSegments,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Path[] inputs,
boolean deleteInputs,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Counters.Counter mergedMapOutputsCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Path[] inputs,
boolean deleteInputs,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
Modifier and Type | Method and Description |
---|---|
Class<? extends CompressionCodec> |
JobConf.getMapOutputCompressorClass(Class<? extends CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the map outputs. |
static Class<? extends CompressionCodec> |
FileOutputFormat.getOutputCompressorClass(JobConf conf,
Class<? extends CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the job outputs. |
void |
JobConf.setMapOutputCompressorClass(Class<? extends CompressionCodec> codecClass)
Set the given class as the
CompressionCodec for the map outputs. |
static void |
FileOutputFormat.setOutputCompressorClass(JobConf conf,
Class<? extends CompressionCodec> codecClass)
Set the
CompressionCodec to be used to compress job outputs. |
Constructor and Description |
---|
IFile.Reader(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
Counters.Counter readsCounter)
Construct an IFile Reader.
|
IFile.Reader(Configuration conf,
FSDataInputStream in,
long length,
CompressionCodec codec,
Counters.Counter readsCounter)
Construct an IFile Reader.
|
IFile.Writer(Configuration conf,
FileSystem fs,
Path file,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Counters.Counter writesCounter) |
IFile.Writer(Configuration conf,
FSDataOutputStream out,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Counters.Counter writesCounter) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
boolean preserve) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
boolean preserve,
Counters.Counter mergedMapOutputsCounter) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
boolean preserve,
Counters.Counter mergedMapOutputsCounter,
long rawDataLength) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
long segmentOffset,
long segmentLength,
CompressionCodec codec,
boolean preserve) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
long segmentOffset,
long segmentLength,
CompressionCodec codec,
boolean preserve,
Counters.Counter mergedMapOutputsCounter) |
ShuffleConsumerPlugin.Context(TaskAttemptID reduceId,
JobConf jobConf,
FileSystem localFS,
TaskUmbilicalProtocol umbilical,
LocalDirAllocator localDirAllocator,
Reporter reporter,
CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter shuffledMapsCounter,
Counters.Counter reduceShuffleBytes,
Counters.Counter failedShuffleCounter,
Counters.Counter mergedMapOutputsCounter,
TaskStatus status,
Progress copyPhase,
Progress mergePhase,
Task reduceTask,
MapOutputFile mapOutputFile) |
Modifier and Type | Method and Description |
---|---|
static Class<? extends CompressionCodec> |
FileOutputFormat.getOutputCompressorClass(JobContext job,
Class<? extends CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the job outputs. |
Modifier and Type | Method and Description |
---|---|
static Class<? extends CompressionCodec> |
FileOutputFormat.getOutputCompressorClass(JobContext job,
Class<? extends CompressionCodec> defaultValue)
Get the
CompressionCodec for compressing the job outputs. |
static void |
FileOutputFormat.setOutputCompressorClass(Job job,
Class<? extends CompressionCodec> codecClass)
Set the
CompressionCodec to be used to compress job outputs. |
Constructor and Description |
---|
MergeManagerImpl(TaskAttemptID reduceId,
JobConf jobConf,
FileSystem localFS,
LocalDirAllocator localDirAllocator,
Reporter reporter,
CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter mergedMapOutputsCounter,
ExceptionReporter exceptionReporter,
Progress mergePhase,
MapOutputFile mapOutputFile) |
Copyright © 2013 Apache Software Foundation. All rights reserved.