@InterfaceAudience.Private public class DFSUtil extends Object
Modifier and Type | Class and Description |
---|---|
static class |
DFSUtil.ConfiguredNNAddress
Represent one of the NameNodes configured in the cluster.
|
static class |
DFSUtil.DecomStaleComparator
Comparator for sorting DataNodeInfo[] based on decommissioned/stale states.
|
Modifier and Type | Field and Description |
---|---|
static Comparator<DatanodeInfo> |
DECOM_COMPARATOR
Compartor for sorting DataNodeInfo[] based on decommissioned states.
|
static byte[] |
EMPTY_BYTES |
static org.apache.commons.cli.Option |
helpOpt |
static org.apache.commons.cli.Options |
helpOptions |
static org.apache.commons.logging.Log |
LOG |
Modifier and Type | Method and Description |
---|---|
static String |
addKeySuffixes(String key,
String... suffixes)
Return configuration key of format key.suffix1.suffix2...suffixN
|
static void |
addPBProtocol(Configuration conf,
Class<?> protocol,
com.google.protobuf.BlockingService service,
RPC.Server server)
Add protobuf based protocol to the
RPC.Server |
static String |
addressMapToString(Map<String,Map<String,InetSocketAddress>> map)
Format the given map, as returned by other functions in this class,
into a string suitable for debugging display.
|
static byte[] |
byteArray2bytes(byte[][] pathComponents)
Given a list of path components returns a byte array
|
static String |
byteArray2PathString(byte[][] pathComponents)
Given a list of path components returns a path as a UTF8 String
|
static byte[][] |
bytes2byteArray(byte[] bytes,
byte separator)
Splits the array of bytes into array of arrays of bytes
on byte separator
|
static byte[][] |
bytes2byteArray(byte[] bytes,
int len,
byte separator)
Splits first len bytes in bytes to array of arrays of bytes
on byte separator
|
static String |
bytes2String(byte[] bytes)
Converts a byte array to a string using UTF8 encoding.
|
static String |
bytes2String(byte[] bytes,
int offset,
int length)
Decode a specific range of bytes of the given byte array to a string
using UTF8.
|
static int |
compareBytes(byte[] left,
byte[] right)
Compare two byte arrays by lexicographical order.
|
static ClientDatanodeProtocol |
createClientDatanodeProtocolProxy(DatanodeID datanodeid,
Configuration conf,
int socketTimeout,
boolean connectToDnViaHostname,
LocatedBlock locatedBlock)
Create a
ClientDatanodeProtocol proxy |
static ClientDatanodeProtocol |
createClientDatanodeProtocolProxy(InetSocketAddress addr,
UserGroupInformation ticket,
Configuration conf,
SocketFactory factory)
Create a
ClientDatanodeProtocol proxy |
static URI |
createUri(String scheme,
InetSocketAddress address)
Create a URI from the scheme and address
|
static List<DFSUtil.ConfiguredNNAddress> |
flattenAddressMap(Map<String,Map<String,InetSocketAddress>> map)
Flatten the given map, as returned by other functions in this class,
into a flat list of
DFSUtil.ConfiguredNNAddress instances. |
static Set<String> |
getAllNnPrincipals(Configuration conf) |
static String |
getBackupNameServiceId(Configuration conf)
Get nameservice Id for the BackupNode based on backup node RPC address
matching the local node address.
|
static Map<String,Map<String,InetSocketAddress>> |
getBackupNodeAddresses(Configuration conf)
Returns list of InetSocketAddress corresponding to backup node rpc
addresses from the configuration.
|
static Map<String,Map<String,InetSocketAddress>> |
getHaNnRpcAddresses(Configuration conf)
Returns list of InetSocketAddress corresponding to HA NN RPC addresses from
the configuration.
|
static String |
getInfoServer(InetSocketAddress namenodeAddr,
Configuration conf,
boolean httpsAddress)
return server http or https address from the configuration for a
given namenode rpc address.
|
static float |
getInvalidateWorkPctPerIteration(Configuration conf)
Get DFS_NAMENODE_INVALIDATE_WORK_PCT_PER_ITERATION from configuration.
|
static Collection<String> |
getNameNodeIds(Configuration conf,
String nsId)
Namenode HighAvailability related configuration.
|
static String |
getNamenodeNameServiceId(Configuration conf)
Get nameservice Id for the
NameNode based on namenode RPC address
matching the local node address. |
static String |
getNamenodeServiceAddr(Configuration conf,
String nsId,
String nnId)
Map a logical namenode ID to its service address.
|
static String |
getNameServiceIdFromAddress(Configuration conf,
InetSocketAddress address,
String... keys)
Given the InetSocketAddress this method returns the nameservice Id
corresponding to the key with matching address, by doing a reverse
lookup on the list of nameservices until it finds a match.
|
static Collection<String> |
getNameServiceIds(Configuration conf)
Returns collection of nameservice Ids from the configuration.
|
static Collection<URI> |
getNameServiceUris(Configuration conf,
String... keys)
Get a URI for each configured nameservice.
|
static Map<String,Map<String,InetSocketAddress>> |
getNNServiceRpcAddresses(Configuration conf)
Returns list of InetSocketAddresses corresponding to namenodes from the
configuration.
|
static Collection<URI> |
getNsServiceRpcUris(Configuration conf)
Get a URI for each configured nameservice.
|
static String |
getOnlyNameServiceIdOrNull(Configuration conf)
If the configuration refers to only a single nameservice, return the
name of that nameservice.
|
static float |
getPercentRemaining(long remaining,
long capacity)
Return remaining as percentage of capacity
|
static float |
getPercentUsed(long used,
long capacity)
Return used as percentage of capacity
|
static Random |
getRandom() |
static int |
getReplWorkMultiplier(Configuration conf)
Get DFS_NAMENODE_REPLICATION_WORK_MULTIPLIER_PER_ITERATION from
configuration.
|
static Map<String,Map<String,InetSocketAddress>> |
getSecondaryNameNodeAddresses(Configuration conf)
Returns list of InetSocketAddresses of corresponding to secondary namenode
http addresses from the configuration.
|
static String |
getSecondaryNameServiceId(Configuration conf)
Get nameservice Id for the secondary node based on secondary http address
matching the local node address.
|
static SecureRandom |
getSecureRandom() |
static String |
getSpnegoKeytabKey(Configuration conf,
String defaultKey)
Get SPNEGO keytab Key from configuration
|
static boolean |
isValidName(String src)
Whether the pathname is valid.
|
static BlockLocation[] |
locatedBlocks2Locations(List<LocatedBlock> blocks)
Convert a List
|
static BlockLocation[] |
locatedBlocks2Locations(LocatedBlocks blocks)
Convert a LocatedBlocks to BlockLocations[]
|
static String |
nnAddressesAsString(Configuration conf) |
static boolean |
parseHelpArgument(String[] args,
String helpDescription,
PrintStream out,
boolean printGenericCommandUsage)
Parse the arguments for commands
|
static String |
path2String(Object path)
Convert an object representing a path to a string.
|
static String |
percent2String(double percentage)
Convert percentage to a string.
|
static int |
roundBytesToGB(long bytes)
Round bytes to GiB (gibibyte)
|
static void |
setGenericConf(Configuration conf,
String nameserviceId,
String nnId,
String... keys)
Sets the node specific setting into generic configuration key.
|
static byte[] |
string2Bytes(String str)
Converts a string to a byte array using UTF8 encoding.
|
static String |
substituteForWildcardAddress(String configuredAddress,
String defaultHost)
Substitute a default host in the case that an address has been configured
with a wildcard.
|
public static final org.apache.commons.logging.Log LOG
public static final byte[] EMPTY_BYTES
public static final Comparator<DatanodeInfo> DECOM_COMPARATOR
public static org.apache.commons.cli.Options helpOptions
public static org.apache.commons.cli.Option helpOpt
public static int compareBytes(byte[] left, byte[] right)
public static Random getRandom()
public static SecureRandom getSecureRandom()
public static boolean isValidName(String src)
public static String bytes2String(byte[] bytes)
public static String bytes2String(byte[] bytes, int offset, int length)
bytes
- The bytes to be decoded into charactersoffset
- The index of the first byte to decodelength
- The number of bytes to decodepublic static byte[] string2Bytes(String str)
public static String byteArray2PathString(byte[][] pathComponents)
public static byte[] byteArray2bytes(byte[][] pathComponents)
public static String path2String(Object path)
public static byte[][] bytes2byteArray(byte[] bytes, byte separator)
bytes
- the array of bytes to splitseparator
- the delimiting bytepublic static byte[][] bytes2byteArray(byte[] bytes, int len, byte separator)
bytes
- the byte array to splitlen
- the number of bytes to splitseparator
- the delimiting bytepublic static BlockLocation[] locatedBlocks2Locations(LocatedBlocks blocks)
blocks
- a LocatedBlockspublic static BlockLocation[] locatedBlocks2Locations(List<LocatedBlock> blocks)
blocks
- A Listpublic static Collection<String> getNameServiceIds(Configuration conf)
conf
- configurationpublic static Collection<String> getNameNodeIds(Configuration conf, String nsId)
conf
- configurationnsId
- the nameservice ID to look at, or null for non-federatedpublic static String addKeySuffixes(String key, String... suffixes)
public static Set<String> getAllNnPrincipals(Configuration conf) throws IOException
IOException
public static Map<String,Map<String,InetSocketAddress>> getHaNnRpcAddresses(Configuration conf)
conf
- configurationpublic static Map<String,Map<String,InetSocketAddress>> getBackupNodeAddresses(Configuration conf) throws IOException
conf
- configurationIOException
- on errorpublic static Map<String,Map<String,InetSocketAddress>> getSecondaryNameNodeAddresses(Configuration conf) throws IOException
conf
- configurationIOException
- on errorpublic static Map<String,Map<String,InetSocketAddress>> getNNServiceRpcAddresses(Configuration conf) throws IOException
conf
- configurationIOException
- on errorpublic static List<DFSUtil.ConfiguredNNAddress> flattenAddressMap(Map<String,Map<String,InetSocketAddress>> map)
DFSUtil.ConfiguredNNAddress
instances.public static String addressMapToString(Map<String,Map<String,InetSocketAddress>> map)
public static String nnAddressesAsString(Configuration conf)
public static Collection<URI> getNsServiceRpcUris(Configuration conf)
conf
- configurationpublic static Collection<URI> getNameServiceUris(Configuration conf, String... keys)
conf
- configurationkeys
- configuration keys to try in order to get the URI for non-HA
nameservicespublic static String getNameServiceIdFromAddress(Configuration conf, InetSocketAddress address, String... keys)
conf
- - configurationaddress
- - InetSocketAddress for configured communication with NN.
Configured addresses are typically given as URIs, but we may have to
compare against a URI typed in by a human, or the server name may be
aliased, so we compare unambiguous InetSocketAddresses instead of just
comparing URI substrings.keys
- - list of configured communication parameters that should
be checked for matches. For example, to compare against RPC addresses,
provide the list DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY,
DFS_NAMENODE_RPC_ADDRESS_KEY. Use the generic parameter keys,
not the NameServiceId-suffixed keys.public static String getInfoServer(InetSocketAddress namenodeAddr, Configuration conf, boolean httpsAddress) throws IOException
conf
- namenodeAddr
- - namenode RPC addresshttpsAddress
- -If true, and if security is enabled, returns server
https address. If false, returns server http address.IOException
public static String substituteForWildcardAddress(String configuredAddress, String defaultHost) throws IOException
configuredAddress
- the address found in the configurationdefaultHost
- the host to substitute with, if configuredAddress
is a local/wildcard address.IOException
- if it is a wildcard address and security is enabledpublic static void setGenericConf(Configuration conf, String nameserviceId, String nnId, String... keys)
conf
- Configuration object to lookup specific key and to set the value
to the key passed. Note the conf object is modified.nameserviceId
- nameservice Id to construct the node specific key. Pass null if
federation is not configuration.nnId
- namenode Id to construct the node specific key. Pass null if
HA is not configured.keys
- The key for which node specific value is looked uppublic static float getPercentUsed(long used, long capacity)
public static float getPercentRemaining(long remaining, long capacity)
public static String percent2String(double percentage)
public static int roundBytesToGB(long bytes)
bytes
- number of bytespublic static ClientDatanodeProtocol createClientDatanodeProtocolProxy(DatanodeID datanodeid, Configuration conf, int socketTimeout, boolean connectToDnViaHostname, LocatedBlock locatedBlock) throws IOException
ClientDatanodeProtocol
proxyIOException
public static ClientDatanodeProtocol createClientDatanodeProtocolProxy(InetSocketAddress addr, UserGroupInformation ticket, Configuration conf, SocketFactory factory) throws IOException
ClientDatanodeProtocol
proxyIOException
public static String getNamenodeNameServiceId(Configuration conf)
NameNode
based on namenode RPC address
matching the local node address.public static String getBackupNameServiceId(Configuration conf)
public static String getSecondaryNameServiceId(Configuration conf)
public static URI createUri(String scheme, InetSocketAddress address)
public static void addPBProtocol(Configuration conf, Class<?> protocol, com.google.protobuf.BlockingService service, RPC.Server server) throws IOException
RPC.Server
conf
- configurationprotocol
- Protocol interfaceservice
- service that implements the protocolserver
- RPC server to which the protocol & implementation is added toIOException
public static String getNamenodeServiceAddr(Configuration conf, String nsId, String nnId)
conf
- ConfigurationnsId
- which nameservice nnId is a part of, optionalnnId
- the namenode ID to get the service addr forpublic static String getOnlyNameServiceIdOrNull(Configuration conf)
public static boolean parseHelpArgument(String[] args, String helpDescription, PrintStream out, boolean printGenericCommandUsage)
args
- the argument to be parsedhelpDescription
- help information to be printed outout
- PrinterprintGenericCommandUsage
- whether to print the
generic command usage defined in ToolRunnerpublic static float getInvalidateWorkPctPerIteration(Configuration conf)
conf
- Configurationpublic static int getReplWorkMultiplier(Configuration conf)
conf
- Configurationpublic static String getSpnegoKeytabKey(Configuration conf, String defaultKey)
conf
- ConfigurationdefaultKey
- Copyright © 2013 Apache Software Foundation. All rights reserved.