Modifier and Type | Class and Description |
---|---|
static class |
DelegationTokenRenewer.RenewAction<T extends FileSystem & DelegationTokenRenewer.Renewable>
An action that will renew and replace the file system's delegation
tokens automatically.
|
Modifier and Type | Class and Description |
---|---|
class |
ChecksumFileSystem
Abstract Checksumed FileSystem.
|
class |
FilterFileSystem
A
FilterFileSystem contains
some other file system, which it uses as
its basic file system, possibly transforming
the data along the way or providing additional
functionality. |
class |
HarFileSystem
This is an implementation of the Hadoop Archive
Filesystem.
|
class |
LocalFileSystem
Implement the FileSystem API for the checksumed local filesystem.
|
class |
RawLocalFileSystem
Implement the FileSystem API for the raw local filesystem.
|
Modifier and Type | Field and Description |
---|---|
protected FileSystem |
FilterFileSystem.fs |
protected FileSystem |
TrashPolicy.fs |
protected FileSystem |
DelegateToFileSystem.fsImpl |
Modifier and Type | Method and Description |
---|---|
<T extends FileSystem & DelegationTokenRenewer.Renewable> |
DelegationTokenRenewer.addRenewAction(T fs)
Add a renew action to the queue.
|
<T extends FileSystem & DelegationTokenRenewer.Renewable> |
DelegationTokenRenewer.removeRenewAction(T fs)
Remove the associated renew action from the queue
|
Modifier and Type | Method and Description |
---|---|
static FileSystem |
FileSystem.get(Configuration conf)
Returns the configured filesystem implementation.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.get(URI uri,
Configuration conf,
String user)
Get a filesystem instance based on the uri, the passed
configuration and the user
|
FileSystem[] |
FilterFileSystem.getChildFileSystems() |
FileSystem[] |
FileSystem.getChildFileSystems()
Get all the immediate child FileSystems embedded in this FileSystem.
|
FileSystem |
Path.getFileSystem(Configuration conf)
Return the FileSystem that owns this Path.
|
protected FileSystem |
FsShell.getFS() |
protected static FileSystem |
FileSystem.getFSofPath(Path absOrFqPath,
Configuration conf) |
static FileSystem |
FileSystem.getNamed(String name,
Configuration conf)
Deprecated.
call #get(URI,Configuration) instead.
|
FileSystem |
LocalFileSystem.getRaw() |
FileSystem |
FilterFileSystem.getRawFileSystem()
Get the raw file system
|
FileSystem |
ChecksumFileSystem.getRawFileSystem()
get the raw file system
|
static FileSystem |
FileSystem.newInstance(Configuration conf)
Returns a unique configured filesystem implementation.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority.
|
static FileSystem |
FileSystem.newInstance(URI uri,
Configuration conf,
String user)
Returns the FileSystem for this URI's scheme and authority and the
passed user.
|
Modifier and Type | Method and Description |
---|---|
static Class<? extends FileSystem> |
FileSystem.getFileSystemClass(String scheme,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
static boolean |
FileUtil.copy(File src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy local files to a FileSystem.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path[] srcs,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf) |
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
File dst,
boolean deleteSource,
Configuration conf)
Copy FileSystem files to local files.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
boolean overwrite,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copy(FileSystem srcFS,
Path src,
FileSystem dstFS,
Path dst,
boolean deleteSource,
Configuration conf)
Copy files between FileSystems.
|
static boolean |
FileUtil.copyMerge(FileSystem srcFS,
Path srcDir,
FileSystem dstFS,
Path dstFile,
boolean deleteSource,
Configuration conf,
String addString)
Copy all files in a directory to one output file (merge).
|
static FSDataOutputStream |
FileSystem.create(FileSystem fs,
Path file,
FsPermission permission)
create a file with the provided permission
The permission of the file is set to be the provided permission as in
setPermission, not permission&~umask
It is implemented using two RPCs.
|
static void |
FileUtil.fullyDelete(FileSystem fs,
Path dir)
Deprecated.
|
static TrashPolicy |
TrashPolicy.getInstance(Configuration conf,
FileSystem fs,
Path home)
Get an instance of the configured TrashPolicy based on the value
of the configuration parameter fs.trash.classname.
|
void |
TrashPolicyDefault.initialize(Configuration conf,
FileSystem fs,
Path home) |
abstract void |
TrashPolicy.initialize(Configuration conf,
FileSystem fs,
Path home)
Used to setup the trash policy.
|
Path |
Path.makeQualified(FileSystem fs)
Deprecated.
|
static boolean |
FileSystem.mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
create a directory with the provided permission
The permission of the directory is set to be the provided permission as in
setPermission, not permission&~umask
|
static boolean |
Trash.moveToAppropriateTrash(FileSystem fs,
Path p,
Configuration conf)
In case of the symlinks or mount points, one has to move the appropriate
trashbin in the actual volume of the path p being deleted.
|
abstract T |
FileSystemLinkResolver.next(FileSystem fs,
Path p)
Calls the abstract FileSystem call equivalent to the specialized subclass
implementation in
FileSystemLinkResolver.doCall(Path) . |
T |
FileSystemLinkResolver.resolve(FileSystem filesys,
Path path)
Attempt calling overridden
FileSystemLinkResolver.doCall(Path) method with
specified FileSystem and Path . |
Modifier and Type | Method and Description |
---|---|
static FileSystem.Statistics |
FileSystem.getStatistics(String scheme,
Class<? extends FileSystem> cls)
Get the statistics for a particular file system
|
Constructor and Description |
---|
ChecksumFileSystem(FileSystem fs) |
DelegateToFileSystem(URI theUri,
FileSystem theFsImpl,
Configuration conf,
String supportedScheme,
boolean authorityRequired) |
FilterFileSystem(FileSystem fs) |
HarFileSystem(FileSystem fs)
Constructor to create a HarFileSystem with an
underlying filesystem.
|
LocalFileSystem(FileSystem rawLocalFileSystem) |
Trash(FileSystem fs,
Configuration conf)
Construct a trash can accessor for the FileSystem provided.
|
Modifier and Type | Class and Description |
---|---|
class |
FTPFileSystem
A
FileSystem backed by an FTP client provided by Apache Commons Net. |
Modifier and Type | Class and Description |
---|---|
class |
HttpFSFileSystem
HttpFSServer implementation of the FileSystemAccess FileSystem.
|
Modifier and Type | Method and Description |
---|---|
Void |
FSOperations.FSAppend.execute(FileSystem fs)
Executes the filesystem operation.
|
Void |
FSOperations.FSConcat.execute(FileSystem fs)
Executes the filesystem operation.
|
Map |
FSOperations.FSContentSummary.execute(FileSystem fs)
Executes the filesystem operation.
|
Void |
FSOperations.FSCreate.execute(FileSystem fs)
Executes the filesystem operation.
|
org.json.simple.JSONObject |
FSOperations.FSDelete.execute(FileSystem fs)
Executes the filesystem operation.
|
Map |
FSOperations.FSFileChecksum.execute(FileSystem fs)
Executes the filesystem operation.
|
Map |
FSOperations.FSFileStatus.execute(FileSystem fs)
Executes the filesystem operation.
|
org.json.simple.JSONObject |
FSOperations.FSHomeDir.execute(FileSystem fs)
Executes the filesystem operation.
|
Map |
FSOperations.FSListStatus.execute(FileSystem fs)
Executes the filesystem operation.
|
org.json.simple.JSONObject |
FSOperations.FSMkdirs.execute(FileSystem fs)
Executes the filesystem operation.
|
InputStream |
FSOperations.FSOpen.execute(FileSystem fs)
Executes the filesystem operation.
|
org.json.simple.JSONObject |
FSOperations.FSRename.execute(FileSystem fs)
Executes the filesystem operation.
|
Void |
FSOperations.FSSetOwner.execute(FileSystem fs)
Executes the filesystem operation.
|
Void |
FSOperations.FSSetPermission.execute(FileSystem fs)
Executes the filesystem operation.
|
org.json.simple.JSONObject |
FSOperations.FSSetReplication.execute(FileSystem fs)
Executes the filesystem operation.
|
Void |
FSOperations.FSSetTimes.execute(FileSystem fs)
Executes the filesystem operation.
|
Modifier and Type | Class and Description |
---|---|
class |
S3FileSystem
A block-based
FileSystem backed by
Amazon S3. |
Modifier and Type | Class and Description |
---|---|
class |
NativeS3FileSystem
A
FileSystem for reading and writing files stored on
Amazon S3. |
Modifier and Type | Field and Description |
---|---|
FileSystem |
PathData.fs |
Modifier and Type | Class and Description |
---|---|
class |
ViewFileSystem
ViewFileSystem (extends the FileSystem interface) implements a client-side
mount table.
|
Modifier and Type | Method and Description |
---|---|
FileSystem[] |
ViewFileSystem.getChildFileSystems() |
Modifier and Type | Class and Description |
---|---|
class |
DistributedFileSystem
Implementation of the abstract FileSystem for the DFS system.
|
class |
HftpFileSystem
An implementation of a protocol for accessing filesystems over HTTP.
|
class |
HsftpFileSystem
An implementation of a protocol for accessing filesystems over HTTPS.
|
Modifier and Type | Method and Description |
---|---|
static InetSocketAddress |
HAUtil.getAddressOfActive(FileSystem fs)
Get the internet address of the currently-active NN.
|
Modifier and Type | Class and Description |
---|---|
class |
WebHdfsFileSystem
A FileSystem for HDFS over the web.
|
Modifier and Type | Method and Description |
---|---|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Deprecated.
|
static SequenceFile.Writer |
SequenceFile.createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
Deprecated.
|
static void |
BloomMapFile.delete(FileSystem fs,
String name) |
static void |
MapFile.delete(FileSystem fs,
String name)
Deletes the named map file.
|
static long |
MapFile.fix(FileSystem fs,
Path dir,
Class<? extends Writable> keyClass,
Class<? extends Writable> valueClass,
boolean dryrun,
Configuration conf)
This method attempts to fix a corrupt MapFile by re-creating its index.
|
protected FSDataInputStream |
SequenceFile.Reader.openFile(FileSystem fs,
Path file,
int bufferSize,
long length)
Override this method to specialize the type of
FSDataInputStream returned. |
static void |
MapFile.rename(FileSystem fs,
String oldName,
String newName)
Renames an existing map directory.
|
Constructor and Description |
---|
ArrayFile.Reader(FileSystem fs,
String file,
Configuration conf)
Construct an array reader for the named file.
|
ArrayFile.Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass)
Create the named file for values of the named class.
|
ArrayFile.Writer(Configuration conf,
FileSystem fs,
String file,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Create the named file for values of the named class.
|
BloomMapFile.Reader(FileSystem fs,
String dirName,
Configuration conf)
Deprecated.
|
BloomMapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Deprecated.
|
BloomMapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf,
boolean open)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class<? extends Writable> valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
|
BloomMapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Deprecated.
|
MapFile.Reader(FileSystem fs,
String dirName,
Configuration conf)
Deprecated.
|
MapFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Deprecated.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
CompressionCodec codec,
Progressable progress)
Deprecated.
Use Writer(Configuration, Path, Option...) instead.
|
MapFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
Class valClass,
SequenceFile.CompressionType compress,
Progressable progress)
Deprecated.
Use Writer(Configuration, Path, Option...)} instead.
|
SequenceFile.Reader(FileSystem fs,
Path file,
Configuration conf)
Deprecated.
Use Reader(Configuration, Option...) instead.
|
SequenceFile.Sorter(FileSystem fs,
Class<? extends WritableComparable> keyClass,
Class valClass,
Configuration conf)
Sort and merge files containing the named classes.
|
SequenceFile.Sorter(FileSystem fs,
RawComparator comparator,
Class keyClass,
Class valClass,
Configuration conf)
Sort and merge using an arbitrary
RawComparator . |
SequenceFile.Sorter(FileSystem fs,
RawComparator comparator,
Class keyClass,
Class valClass,
Configuration conf,
SequenceFile.Metadata metadata)
Sort and merge using an arbitrary
RawComparator . |
SequenceFile.Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Deprecated.
|
SequenceFile.Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
Progressable progress,
SequenceFile.Metadata metadata)
Deprecated.
|
SequenceFile.Writer(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
Progressable progress,
SequenceFile.Metadata metadata)
Deprecated.
|
SetFile.Reader(FileSystem fs,
String dirName,
Configuration conf)
Construct a set reader for the named set.
|
SetFile.Reader(FileSystem fs,
String dirName,
WritableComparator comparator,
Configuration conf)
Construct a set reader for the named set using the named comparator.
|
SetFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass,
SequenceFile.CompressionType compress)
Create a set naming the element class and compression type.
|
SetFile.Writer(Configuration conf,
FileSystem fs,
String dirName,
WritableComparator comparator,
SequenceFile.CompressionType compress)
Create a set naming the element comparator and compression type.
|
SetFile.Writer(FileSystem fs,
String dirName,
Class<? extends WritableComparable> keyClass)
Deprecated.
pass a Configuration too
|
Modifier and Type | Method and Description |
---|---|
FileSystem |
FileSystemAccess.createFileSystem(String user,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
T |
FileSystemAccess.FileSystemExecutor.execute(FileSystem fs) |
void |
FileSystemAccess.releaseFileSystem(FileSystem fs) |
Modifier and Type | Method and Description |
---|---|
protected FileSystem |
FileSystemAccessService.createFileSystem(Configuration namenodeConf) |
FileSystem |
FileSystemAccessService.createFileSystem(String user,
Configuration conf) |
FileSystem |
FileSystemAccessService.createFileSystemInternal(String user,
Configuration conf) |
Modifier and Type | Method and Description |
---|---|
protected void |
FileSystemAccessService.checkNameNodeHealth(FileSystem fileSystem) |
protected void |
FileSystemAccessService.closeFileSystem(FileSystem fs) |
void |
FileSystemAccessService.releaseFileSystem(FileSystem fs) |
Modifier and Type | Method and Description |
---|---|
static void |
FileSystemReleaseFilter.setFileSystem(FileSystem fs)
Static method that sets the
FileSystem to release back to
the FileSystemAccess service on servlet request completion. |
Modifier and Type | Method and Description |
---|---|
FileSystem |
JobClient.getFs()
Get a filesystem handle.
|
FileSystem |
ShuffleConsumerPlugin.Context.getLocalFS() |
Modifier and Type | Method and Description |
---|---|
protected void |
FileInputFormat.addInputPathRecursively(List<FileStatus> result,
FileSystem fs,
Path path,
PathFilter inputFilter)
Add files in the input path recursively into the results.
|
void |
OutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job)
Check for validity of the output-specification for the job.
|
void |
SequenceFileAsBinaryOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
FileOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
static MapFile.Reader[] |
MapFileOutputFormat.getReaders(FileSystem ignored,
Path dir,
Configuration conf)
Open the output generated by this format.
|
RecordWriter<K,V> |
TextOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
SequenceFileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<WritableComparable,Writable> |
MapFileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
OutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress)
Get the
RecordWriter for the given job. |
RecordWriter<BytesWritable,BytesWritable> |
SequenceFileAsBinaryOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
abstract RecordWriter<K,V> |
FileOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
static boolean |
JobClient.isJobDirValid(Path jobDirPath,
FileSystem fs)
Checks if the job directory is clean and has all the required components
for (re) starting the job
|
protected boolean |
TextInputFormat.isSplitable(FileSystem fs,
Path file) |
protected boolean |
FileInputFormat.isSplitable(FileSystem fs,
Path filename)
Is the given filename splitable? Usually, true, but if the file is
stream compressed, it will not be.
|
protected boolean |
KeyValueTextInputFormat.isSplitable(FileSystem fs,
Path file) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
int inMemSegments,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
boolean sortSegments,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
boolean sortSegments,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Path[] inputs,
boolean deleteInputs,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Counters.Counter mergedMapOutputsCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Path[] inputs,
boolean deleteInputs,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
int inMemSegments,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
boolean sortSegments,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
boolean sortSegments,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
static <K,V> RawKeyValueIterator |
Merger.merge(Configuration conf,
FileSystem fs,
Class<K> keyClass,
Class<V> valueClass,
List<Merger.Segment<K,V>> segments,
int mergeFactor,
Path tmpDir,
RawComparator<K> comparator,
Progressable reporter,
Counters.Counter readsCounter,
Counters.Counter writesCounter,
Progress mergePhase) |
Constructor and Description |
---|
IFile.Reader(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
Counters.Counter readsCounter)
Construct an IFile Reader.
|
IFile.Writer(Configuration conf,
FileSystem fs,
Path file) |
IFile.Writer(Configuration conf,
FileSystem fs,
Path file,
Class<K> keyClass,
Class<V> valueClass,
CompressionCodec codec,
Counters.Counter writesCounter) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
boolean preserve) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
boolean preserve,
Counters.Counter mergedMapOutputsCounter) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
CompressionCodec codec,
boolean preserve,
Counters.Counter mergedMapOutputsCounter,
long rawDataLength) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
long segmentOffset,
long segmentLength,
CompressionCodec codec,
boolean preserve) |
Merger.Segment(Configuration conf,
FileSystem fs,
Path file,
long segmentOffset,
long segmentLength,
CompressionCodec codec,
boolean preserve,
Counters.Counter mergedMapOutputsCounter) |
ShuffleConsumerPlugin.Context(TaskAttemptID reduceId,
JobConf jobConf,
FileSystem localFS,
TaskUmbilicalProtocol umbilical,
LocalDirAllocator localDirAllocator,
Reporter reporter,
CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter shuffledMapsCounter,
Counters.Counter reduceShuffleBytes,
Counters.Counter failedShuffleCounter,
Counters.Counter mergedMapOutputsCounter,
TaskStatus status,
Progress copyPhase,
Progress mergePhase,
Task reduceTask,
MapOutputFile mapOutputFile) |
Modifier and Type | Field and Description |
---|---|
protected FileSystem |
CombineFileRecordReader.fs |
Modifier and Type | Method and Description |
---|---|
void |
NullOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
FilterOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
void |
LazyOutputFormat.checkOutputSpecs(FileSystem ignored,
JobConf job) |
protected abstract RecordWriter<K,V> |
MultipleOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected RecordWriter<K,V> |
MultipleTextOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected RecordWriter<K,V> |
MultipleSequenceFileOutputFormat.getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
RecordWriter<K,V> |
MultipleOutputFormat.getRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3)
Create a composite record writer that can write key/value data to different
output files
|
RecordWriter<K,V> |
NullOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
FilterOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
RecordWriter<K,V> |
LazyOutputFormat.getRecordWriter(FileSystem ignored,
JobConf job,
String name,
Progressable progress) |
protected boolean |
CombineFileInputFormat.isSplitable(FileSystem fs,
Path file) |
Modifier and Type | Method and Description |
---|---|
void |
DBOutputFormat.checkOutputSpecs(FileSystem filesystem,
JobConf job)
Check for validity of the output-specification for the job.
|
RecordWriter<K,V> |
DBOutputFormat.getRecordWriter(FileSystem filesystem,
JobConf job,
String name,
Progressable progress)
Get the
RecordWriter for the given job. |
Modifier and Type | Method and Description |
---|---|
FileSystem |
Cluster.getFileSystem()
Get the file system where job-specific files are stored
|
Modifier and Type | Method and Description |
---|---|
org.apache.hadoop.mapreduce.JobSubmitter |
Job.getJobSubmitter(FileSystem fs,
ClientProtocol submitClient)
Only for mocking via unit tests.
|
Modifier and Type | Method and Description |
---|---|
static void |
DistributedCache.addArchiveToClassPath(Path archive,
Configuration conf,
FileSystem fs)
Deprecated.
Add an archive path to the current set of classpath entries.
|
static void |
DistributedCache.addFileToClassPath(Path file,
Configuration conf,
FileSystem fs)
Deprecated.
Add a file path to the current set of classpath entries.
|
Constructor and Description |
---|
EventReader(FileSystem fs,
Path name)
Create a new Event Reader
|
JobHistoryParser(FileSystem fs,
Path historyFile)
Create the job history parser for the given history file using the
given file system
|
JobHistoryParser(FileSystem fs,
String file)
Create a job history parser for the given history file using the
given file system
|
Modifier and Type | Field and Description |
---|---|
protected FileSystem |
CombineFileRecordReader.fs |
Modifier and Type | Method and Description |
---|---|
protected void |
FileInputFormat.addInputPathRecursively(List<FileStatus> result,
FileSystem fs,
Path path,
PathFilter inputFilter)
Add files in the input path recursively into the results.
|
protected BlockLocation[] |
CombineFileInputFormat.getFileBlockLocations(FileSystem fs,
FileStatus stat) |
Modifier and Type | Method and Description |
---|---|
static void |
JobSplitWriter.createSplitFiles(Path jobSubmitDir,
Configuration conf,
FileSystem fs,
InputSplit[] splits) |
static <T extends InputSplit> |
JobSplitWriter.createSplitFiles(Path jobSubmitDir,
Configuration conf,
FileSystem fs,
List<InputSplit> splits) |
static <T extends InputSplit> |
JobSplitWriter.createSplitFiles(Path jobSubmitDir,
Configuration conf,
FileSystem fs,
T[] splits) |
static JobSplit.TaskSplitMetaInfo[] |
SplitMetaInfoReader.readSplitMetaInfo(JobID jobId,
FileSystem fs,
Configuration conf,
Path jobSubmitDir) |
Constructor and Description |
---|
MergeManagerImpl(TaskAttemptID reduceId,
JobConf jobConf,
FileSystem localFS,
LocalDirAllocator localDirAllocator,
Reporter reporter,
CompressionCodec codec,
Class<? extends Reducer> combinerClass,
Task.CombineOutputCollector<K,V> combineCollector,
Counters.Counter spilledRecordsCounter,
Counters.Counter reduceCombineInputCounter,
Counters.Counter mergedMapOutputsCounter,
ExceptionReporter exceptionReporter,
Progress mergePhase,
MapOutputFile mapOutputFile) |
Modifier and Type | Method and Description |
---|---|
protected FileSystem |
MRAppMaster.getFileSystem(Configuration conf)
Create the default file System for this job.
|
Modifier and Type | Method and Description |
---|---|
protected FileSystem |
JobImpl.getFileSystem(Configuration conf)
Create the default file System for this job.
|
Constructor and Description |
---|
StreamBaseRecordReader(FSDataInputStream in,
FileSplit split,
Reporter reporter,
JobConf job,
FileSystem fs) |
StreamXmlRecordReader(FSDataInputStream in,
FileSplit split,
Reporter reporter,
JobConf job,
FileSystem fs) |
Modifier and Type | Method and Description |
---|---|
static boolean |
DistCpUtils.checksumsAreEqual(FileSystem sourceFS,
Path source,
FileSystem targetFS,
Path target)
Utility to compare checksums for the paths specified.
|
static boolean |
DistCpUtils.compareFs(FileSystem srcFs,
FileSystem destFs) |
static void |
DistCpUtils.preserve(FileSystem targetFS,
Path path,
FileStatus srcFileStatus,
EnumSet<DistCpOptions.FileAttribute> attributes)
Preserve attribute on file matching that of the file status being sent
as argument.
|
static Path |
DistCpUtils.sortListing(FileSystem fs,
Configuration conf,
Path sourceListing)
Sort sequence file containing FileStatus and Text as key and value respecitvely
|
Modifier and Type | Method and Description |
---|---|
protected FileSystem |
LogAggregationService.getFileSystem(Configuration conf) |
Modifier and Type | Field and Description |
---|---|
protected FileSystem |
FileSystemRMStateStore.fs |
Copyright © 2013 Apache Software Foundation. All rights reserved.