public class CustomOutputCommitter extends OutputCommitter
Modifier and Type | Field and Description |
---|---|
static String |
JOB_ABORT_FILE_NAME |
static String |
JOB_COMMIT_FILE_NAME |
static String |
JOB_SETUP_FILE_NAME |
static String |
TASK_ABORT_FILE_NAME |
static String |
TASK_COMMIT_FILE_NAME |
static String |
TASK_SETUP_FILE_NAME |
Constructor and Description |
---|
CustomOutputCommitter() |
Modifier and Type | Method and Description |
---|---|
void |
abortJob(JobContext jobContext,
int status)
For aborting an unsuccessful job's output.
|
void |
abortTask(TaskAttemptContext taskContext)
Discard the task output.
|
void |
commitJob(JobContext jobContext)
For committing job's output after successful job completion.
|
void |
commitTask(TaskAttemptContext taskContext)
To promote the task's temporary output to final output location.
|
boolean |
needsTaskCommit(TaskAttemptContext taskContext)
Check whether task needs a commit.
|
void |
setupJob(JobContext jobContext)
For the framework to setup the job output during initialization.
|
void |
setupTask(TaskAttemptContext taskContext)
Sets up output for the task.
|
abortJob, abortTask, cleanupJob, cleanupJob, commitJob, commitTask, isRecoverySupported, needsTaskCommit, recoverTask, recoverTask, setupJob, setupTask
public static final String JOB_SETUP_FILE_NAME
public static final String JOB_COMMIT_FILE_NAME
public static final String JOB_ABORT_FILE_NAME
public static final String TASK_SETUP_FILE_NAME
public static final String TASK_ABORT_FILE_NAME
public static final String TASK_COMMIT_FILE_NAME
public void setupJob(JobContext jobContext) throws IOException
OutputCommitter
setupJob
in class OutputCommitter
jobContext
- Context of the job whose output is being written.IOException
- if temporary output could not be createdpublic void commitJob(JobContext jobContext) throws IOException
OutputCommitter
commitJob
in class OutputCommitter
jobContext
- Context of the job whose output is being written.IOException
public void abortJob(JobContext jobContext, int status) throws IOException
OutputCommitter
JobStatus.FAILED
or
JobStatus.KILLED
. This is called from the application
master process for the entire job. This may be called multiple times.abortJob
in class OutputCommitter
jobContext
- Context of the job whose output is being written.status
- final runstate of the jobIOException
public void setupTask(TaskAttemptContext taskContext) throws IOException
OutputCommitter
setupTask
in class OutputCommitter
taskContext
- Context of the task whose output is being written.IOException
public boolean needsTaskCommit(TaskAttemptContext taskContext) throws IOException
OutputCommitter
needsTaskCommit
in class OutputCommitter
IOException
public void commitTask(TaskAttemptContext taskContext) throws IOException
OutputCommitter
OutputCommitter.needsTaskCommit(TaskAttemptContext)
returns true and this
task is the task that the AM determines finished first, this method
is called to commit an individual task's output. This is to mark
that tasks output as complete, as OutputCommitter.commitJob(JobContext)
will
also be called later on if the entire job finished successfully. This
is called from a task's process. This may be called multiple times for the
same task, but different task attempts. It should be very rare for this to
be called multiple times and requires odd networking failures to make this
happen. In the future the Hadoop framework may eliminate this race.commitTask
in class OutputCommitter
taskContext
- Context of the task whose output is being written.IOException
- if commit is notpublic void abortTask(TaskAttemptContext taskContext) throws IOException
OutputCommitter
abortTask
in class OutputCommitter
IOException
Copyright © 2013 Apache Software Foundation. All rights reserved.