@InterfaceAudience.Private public abstract class Storage extends StorageInfo
Local storage information is stored in a separate file VERSION. It contains type of the node, the storage layout version, the namespace id, and the fs state creation time.
Local storage can reside in multiple directories. Each directory should contain the same VERSION file as the others. During startup Hadoop servers (name-node and data-nodes) read their local storage information from them.
The servers hold a lock for each storage directory while they run so that other nodes were not able to startup sharing the same storage. The locks are released when the servers stop (normally or abnormally).
Modifier and Type | Class and Description |
---|---|
static interface |
Storage.FormatConfirmable
Interface for classes which need to have the user confirm their
formatting during NameNode -format and other similar operations.
|
static class |
Storage.StorageDirectory
One of the storage directories.
|
static interface |
Storage.StorageDirType
An interface to denote storage directory type
Implementations can define a type for storage directory by implementing
this interface.
|
static class |
Storage.StorageState |
Modifier and Type | Field and Description |
---|---|
static int |
LAST_PRE_UPGRADE_LAYOUT_VERSION |
protected static String |
LAST_UPGRADABLE_HADOOP_VERSION |
static int |
LAST_UPGRADABLE_LAYOUT_VERSION |
static int[] |
LAYOUT_VERSIONS_203
Layout versions of 0.20.203 release
|
static org.apache.commons.logging.Log |
LOG |
static String |
STORAGE_1_BBW
The blocksBeingWritten directory which was used in some 1.x and earlier
releases.
|
static String |
STORAGE_DIR_CURRENT |
static String |
STORAGE_DIR_PREVIOUS |
static String |
STORAGE_FILE_LOCK |
protected static String |
STORAGE_FILE_VERSION |
static String |
STORAGE_PREVIOUS_CKPT |
static String |
STORAGE_TMP_FINALIZED |
static String |
STORAGE_TMP_LAST_CKPT |
static String |
STORAGE_TMP_PREVIOUS |
static String |
STORAGE_TMP_REMOVED |
protected List<Storage.StorageDirectory> |
storageDirs |
protected HdfsServerConstants.NodeType |
storageType |
clusterID, cTime, layoutVersion, namespaceID
Modifier | Constructor and Description |
---|---|
protected |
Storage(HdfsServerConstants.NodeType type)
Create empty storage info of the specified type
|
protected |
Storage(HdfsServerConstants.NodeType type,
StorageInfo storageInfo) |
Modifier and Type | Method and Description |
---|---|
protected void |
addStorageDir(Storage.StorageDirectory sd) |
static void |
checkVersionUpgradable(int oldVersion)
Checks if the upgrade from the given old version is supported.
|
static boolean |
confirmFormat(Iterable<? extends Storage.FormatConfirmable> items,
boolean force,
boolean interactive)
Iterate over each of the
Storage.FormatConfirmable objects,
potentially checking with the user whether it should be formatted. |
static void |
deleteDir(File dir)
Recursively delete all the content of the directory first and then
the directory itself from the local filesystem.
|
Iterable<Storage.StorageDirectory> |
dirIterable(Storage.StorageDirType dirType) |
Iterator<Storage.StorageDirectory> |
dirIterator()
Return default iterator
This iterator returns all entries in storageDirs
|
Iterator<Storage.StorageDirectory> |
dirIterator(Storage.StorageDirType dirType)
Return iterator based on Storage Directory Type
This iterator selects entries in storageDirs of type dirType and returns
them via the Iterator
|
static String |
getBuildVersion() |
List<File> |
getFiles(Storage.StorageDirType dirType,
String fileName) |
int |
getNumStorageDirs() |
static String |
getRegistrationID(StorageInfo storage) |
Storage.StorageDirectory |
getSingularStorageDir() |
Storage.StorageDirectory |
getStorageDir(int idx) |
static boolean |
is203LayoutVersion(int layoutVersion) |
abstract boolean |
isPreUpgradableLayout(Storage.StorageDirectory sd)
Return true if the layout of the given storage directory is from a version
of Hadoop prior to the introduction of the "current" and "previous"
directories which allow upgrade and rollback.
|
String |
listStorageDirectories()
generate storage list (debug line)
|
void |
readPreviousVersionProperties(Storage.StorageDirectory sd)
Read properties from the the previous/VERSION file in the given storage directory.
|
void |
readProperties(Storage.StorageDirectory sd)
Read properties from the VERSION file in the given storage directory.
|
static Properties |
readPropertiesFile(File from) |
static void |
rename(File from,
File to) |
protected void |
setClusterId(Properties props,
int layoutVersion,
Storage.StorageDirectory sd)
Validate and set clusterId from
Properties |
protected void |
setcTime(Properties props,
Storage.StorageDirectory sd)
Validate and set ctime from
Properties |
protected void |
setFieldsFromProperties(Properties props,
Storage.StorageDirectory sd)
Get common storage fields.
|
protected void |
setLayoutVersion(Properties props,
Storage.StorageDirectory sd)
Validate and set layout version from
Properties |
protected void |
setNamespaceID(Properties props,
Storage.StorageDirectory sd)
Validate and set namespaceID version from
Properties |
protected void |
setPropertiesFromFields(Properties props,
Storage.StorageDirectory sd)
Set common storage fields into the given properties object.
|
protected void |
setStorageType(Properties props,
Storage.StorageDirectory sd)
Validate and set storage type from
Properties |
void |
unlockAll()
Unlock all storage directories.
|
void |
writeAll()
Write all data storage files.
|
void |
writeProperties(File to,
Storage.StorageDirectory sd) |
void |
writeProperties(Storage.StorageDirectory sd)
Write properties to the VERSION file in the given storage directory.
|
getClusterID, getCTime, getLayoutVersion, getNamespaceID, setStorageInfo, toColonSeparatedString, toString, versionSupportsFederation
public static final org.apache.commons.logging.Log LOG
public static final int LAST_PRE_UPGRADE_LAYOUT_VERSION
public static final int LAST_UPGRADABLE_LAYOUT_VERSION
protected static final String LAST_UPGRADABLE_HADOOP_VERSION
public static final int[] LAYOUT_VERSIONS_203
public static final String STORAGE_FILE_LOCK
protected static final String STORAGE_FILE_VERSION
public static final String STORAGE_DIR_CURRENT
public static final String STORAGE_DIR_PREVIOUS
public static final String STORAGE_TMP_REMOVED
public static final String STORAGE_TMP_PREVIOUS
public static final String STORAGE_TMP_FINALIZED
public static final String STORAGE_TMP_LAST_CKPT
public static final String STORAGE_PREVIOUS_CKPT
public static final String STORAGE_1_BBW
protected HdfsServerConstants.NodeType storageType
protected List<Storage.StorageDirectory> storageDirs
protected Storage(HdfsServerConstants.NodeType type)
protected Storage(HdfsServerConstants.NodeType type, StorageInfo storageInfo)
public List<File> getFiles(Storage.StorageDirType dirType, String fileName)
public Iterator<Storage.StorageDirectory> dirIterator()
public Iterator<Storage.StorageDirectory> dirIterator(Storage.StorageDirType dirType)
public Iterable<Storage.StorageDirectory> dirIterable(Storage.StorageDirType dirType)
public String listStorageDirectories()
public int getNumStorageDirs()
public Storage.StorageDirectory getStorageDir(int idx)
public Storage.StorageDirectory getSingularStorageDir()
protected void addStorageDir(Storage.StorageDirectory sd)
public abstract boolean isPreUpgradableLayout(Storage.StorageDirectory sd) throws IOException
IOException
public static void checkVersionUpgradable(int oldVersion) throws IOException
oldVersion
- IOException
public static boolean confirmFormat(Iterable<? extends Storage.FormatConfirmable> items, boolean force, boolean interactive) throws IOException
Storage.FormatConfirmable
objects,
potentially checking with the user whether it should be formatted.
If running in interactive mode, will prompt the user for each
directory to allow them to format anyway. Otherwise, returns
false, unless 'force' is specified.force
- format regardless of whether dirs existinteractive
- prompt the user when a dir existsIOException
- if some storage cannot be accessedprotected void setFieldsFromProperties(Properties props, Storage.StorageDirectory sd) throws IOException
props
- IOException
protected void setPropertiesFromFields(Properties props, Storage.StorageDirectory sd) throws IOException
props
- the Properties object to write intoIOException
public void readProperties(Storage.StorageDirectory sd) throws IOException
IOException
public void readPreviousVersionProperties(Storage.StorageDirectory sd) throws IOException
IOException
public void writeProperties(Storage.StorageDirectory sd) throws IOException
IOException
public void writeProperties(File to, Storage.StorageDirectory sd) throws IOException
IOException
public static Properties readPropertiesFile(File from) throws IOException
IOException
public static void rename(File from, File to) throws IOException
IOException
public static void deleteDir(File dir) throws IOException
dir
- The directory to deleteIOException
public void writeAll() throws IOException
IOException
public void unlockAll() throws IOException
IOException
public static String getBuildVersion()
public static String getRegistrationID(StorageInfo storage)
protected void setStorageType(Properties props, Storage.StorageDirectory sd) throws InconsistentFSStateException
Properties
InconsistentFSStateException
protected void setcTime(Properties props, Storage.StorageDirectory sd) throws InconsistentFSStateException
Properties
InconsistentFSStateException
protected void setClusterId(Properties props, int layoutVersion, Storage.StorageDirectory sd) throws InconsistentFSStateException
Properties
InconsistentFSStateException
protected void setLayoutVersion(Properties props, Storage.StorageDirectory sd) throws IncorrectVersionException, InconsistentFSStateException
Properties
protected void setNamespaceID(Properties props, Storage.StorageDirectory sd) throws InconsistentFSStateException
Properties
InconsistentFSStateException
public static boolean is203LayoutVersion(int layoutVersion)
Copyright © 2013 Apache Software Foundation. All rights reserved.