public abstract class CopyListing extends Configured
Modifier | Constructor and Description |
---|---|
protected |
CopyListing(Configuration configuration,
Credentials credentials)
Protected constructor, to initialize configuration.
|
Modifier and Type | Method and Description |
---|---|
void |
buildListing(Path pathToListFile,
DistCpOptions options)
Build listing function creates the input listing that distcp uses to
perform the copy.
|
protected abstract void |
doBuildListing(Path pathToListFile,
DistCpOptions options)
The interface to be implemented by sub-classes, to create the source/target file listing.
|
protected abstract long |
getBytesToCopy()
Return the total bytes that distCp should copy for the source paths
This doesn't consider whether file is same should be skipped during copy
|
static CopyListing |
getCopyListing(Configuration configuration,
Credentials credentials,
DistCpOptions options)
Public Factory method with which the appropriate CopyListing implementation may be retrieved.
|
protected Credentials |
getCredentials()
get credentials to update the delegation tokens for accessed FS objects
|
protected abstract long |
getNumberOfPaths()
Return the total number of paths to distcp, includes directories as well
This doesn't consider whether file/dir is already present and should be skipped during copy
|
protected void |
setCredentials(Credentials credentials)
set Credentials store, on which FS delegatin token will be cached
|
protected abstract void |
validatePaths(DistCpOptions options)
Validate input and output paths
|
getConf, setConf
protected CopyListing(Configuration configuration, Credentials credentials)
configuration
- The input configuration,
with which the source/target FileSystems may be accessed.credentials
- - Credentials object on which the FS delegation tokens are cached.If null
delegation token caching is skippedpublic final void buildListing(Path pathToListFile, DistCpOptions options) throws IOException
pathToListFile
- - Output file where the listing would be storedoptions
- - Input options to distcpIOException
- - Exception if anyprotected abstract void validatePaths(DistCpOptions options) throws IOException, org.apache.hadoop.tools.CopyListing.InvalidInputException
options
- - Input optionsInvalidInputException:
- If inputs are invalidIOException:
- any Exception with FSIOException
org.apache.hadoop.tools.CopyListing.InvalidInputException
protected abstract void doBuildListing(Path pathToListFile, DistCpOptions options) throws IOException
pathToListFile
- Path on HDFS where the listing file is written.options
- Input Options for DistCp (indicating source/target paths.)IOException:
- Thrown on failure to create the listing file.IOException
protected abstract long getBytesToCopy()
protected abstract long getNumberOfPaths()
protected void setCredentials(Credentials credentials)
credentials
- - Credentials objectprotected Credentials getCredentials()
public static CopyListing getCopyListing(Configuration configuration, Credentials credentials, DistCpOptions options)
configuration
- The input configuration.credentials
- Credentials object on which the FS delegation tokens are cachedoptions
- The input Options, to help choose the appropriate CopyListing Implementation.Copyright © 2013 Apache Software Foundation. All rights reserved.