public class ShuffleHandler extends AuxiliaryService
Service.STATE
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_MAX_SHUFFLE_CONNECTIONS |
static boolean |
DEFAULT_SHUFFLE_MANAGE_OS_CACHE |
static int |
DEFAULT_SHUFFLE_PORT |
static int |
DEFAULT_SHUFFLE_READAHEAD_BYTES |
static int |
DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE |
static String |
MAPREDUCE_SHUFFLE_SERVICEID |
static String |
MAX_SHUFFLE_CONNECTIONS |
protected org.apache.hadoop.mapred.ShuffleHandler.HttpPipelineFactory |
pipelineFact |
static String |
SHUFFLE_MANAGE_OS_CACHE |
static String |
SHUFFLE_PORT_CONFIG_KEY |
static String |
SHUFFLE_READAHEAD_BYTES |
static String |
SUFFLE_SSL_FILE_BUFFER_SIZE_KEY |
Constructor and Description |
---|
ShuffleHandler() |
Modifier and Type | Method and Description |
---|---|
static int |
deserializeMetaData(ByteBuffer meta)
A helper function to deserialize the metadata returned by ShuffleHandler.
|
ByteBuffer |
getMetaData()
Retrieve meta-data for this
AuxiliaryService . |
protected org.apache.hadoop.mapred.ShuffleHandler.Shuffle |
getShuffle(Configuration conf) |
void |
initializeApplication(ApplicationInitializationContext context)
A new application is started on this NodeManager.
|
static ByteBuffer |
serializeMetaData(int port)
Serialize the shuffle port into a ByteBuffer for use later on.
|
static ByteBuffer |
serializeServiceData(Token<JobTokenIdentifier> jobToken)
A helper function to serialize the JobTokenIdentifier to be sent to the
ShuffleHandler as ServiceData.
|
protected void |
serviceInit(Configuration conf)
All initialization code needed by a service.
|
protected void |
serviceStart()
Actions called during the INITED to STARTED transition.
|
protected void |
serviceStop()
Actions called during the transition to the STOPPED state.
|
void |
stopApplication(ApplicationTerminationContext context)
An application is finishing on this NodeManager.
|
initializeContainer, stopContainer
close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop
public static final String SHUFFLE_MANAGE_OS_CACHE
public static final boolean DEFAULT_SHUFFLE_MANAGE_OS_CACHE
public static final String SHUFFLE_READAHEAD_BYTES
public static final int DEFAULT_SHUFFLE_READAHEAD_BYTES
protected org.apache.hadoop.mapred.ShuffleHandler.HttpPipelineFactory pipelineFact
public static final String MAPREDUCE_SHUFFLE_SERVICEID
public static final String SHUFFLE_PORT_CONFIG_KEY
public static final int DEFAULT_SHUFFLE_PORT
public static final String SUFFLE_SSL_FILE_BUFFER_SIZE_KEY
public static final int DEFAULT_SUFFLE_SSL_FILE_BUFFER_SIZE
public static final String MAX_SHUFFLE_CONNECTIONS
public static final int DEFAULT_MAX_SHUFFLE_CONNECTIONS
public static ByteBuffer serializeMetaData(int port) throws IOException
port
- the port to be sent to the ApplciationMasterIOException
public static int deserializeMetaData(ByteBuffer meta) throws IOException
meta
- the metadata returned by the ShuffleHandlerIOException
public static ByteBuffer serializeServiceData(Token<JobTokenIdentifier> jobToken) throws IOException
jobToken
- the job token to be used for authentication of
shuffle data requests.IOException
public void initializeApplication(ApplicationInitializationContext context)
AuxiliaryService
AuxiliaryService
about the application initialization.initializeApplication
in class AuxiliaryService
context
- context for the application's initializationpublic void stopApplication(ApplicationTerminationContext context)
AuxiliaryService
AuxiliaryService
about the same.stopApplication
in class AuxiliaryService
context
- context for the application terminationprotected void serviceInit(Configuration conf) throws Exception
AbstractService
AbstractService.init(Configuration)
prevents re-entrancy.
The base implementation checks to see if the subclass has created
a new configuration instance, and if so, updates the base class valueserviceInit
in class AbstractService
conf
- configurationException
- on a failure -these will be caught,
possibly wrapped, and wil; trigger a service stopprotected void serviceStart() throws Exception
AbstractService
AbstractService.start()
prevents re-entrancy.serviceStart
in class AbstractService
Exception
- if needed -these will be caught,
wrapped, and trigger a service stopprotected void serviceStop() throws Exception
AbstractService
AbstractService.stop()
prevents re-entrancy.
Implementations MUST write this to be robust against failures, including
checks for null references -and for the first failure to not stop other
attempts to shut down parts of the service.serviceStop
in class AbstractService
Exception
- if needed -these will be caught and logged.public ByteBuffer getMetaData()
AuxiliaryService
AuxiliaryService
. Applications using
this AuxiliaryService
SHOULD know the format of the meta-data -
ideally each service should provide a method to parse out the information
to the applications. One example of meta-data is contact information so
that applications can access the service remotely. This will only be called
after the service's AbstractService.start()
method has finished. the result may be
cached.
The information is passed along to applications via
StartContainersResponse.getAllServicesMetaData()
that is returned by
ContainerManagementProtocol.startContainers(StartContainersRequest)
getMetaData
in class AuxiliaryService
protected org.apache.hadoop.mapred.ShuffleHandler.Shuffle getShuffle(Configuration conf)
Copyright © 2013 Apache Software Foundation. All rights reserved.