public class OfflineSorter
extends java.lang.Object
sort(String)
Modifier and Type | Class and Description |
---|---|
static class |
OfflineSorter.BufferSize
A bit more descriptive unit for constructors.
|
static class |
OfflineSorter.ByteSequencesReader
Utility class to read length-prefixed byte[] entries from an input.
|
static class |
OfflineSorter.ByteSequencesWriter
Utility class to emit length-prefixed byte[] entries to an output stream for sorting.
|
(package private) static class |
OfflineSorter.FileAndTop |
private class |
OfflineSorter.MergePartitionsTask
Merges multiple file-based partitions to a single on-disk partition.
|
private static class |
OfflineSorter.Partition
Holds one partition of items, either loaded into memory or based on a file.
|
class |
OfflineSorter.SortInfo
Sort info (debugging mostly).
|
private class |
OfflineSorter.SortPartitionTask
Sorts one in-memory partition, writes it to disk, and returns the resulting file-based partition.
|
Modifier and Type | Field and Description |
---|---|
static long |
ABSOLUTE_MIN_SORT_BUFFER_SIZE
Absolute minimum required buffer size for sorting.
|
private java.util.Comparator<BytesRef> |
comparator |
static java.util.Comparator<BytesRef> |
DEFAULT_COMPARATOR
Default comparator: sorts in binary (codepoint) order
|
private Directory |
dir |
private java.util.concurrent.ExecutorService |
exec |
static long |
GB
Convenience constant for gigabytes
|
static int |
MAX_TEMPFILES
Maximum number of temporary files before doing an intermediate merge.
|
private int |
maxTempFiles |
static long |
MB
Convenience constant for megabytes
|
static long |
MIN_BUFFER_SIZE_MB
Minimum recommended buffer size for sorting.
|
private static java.lang.String |
MIN_BUFFER_SIZE_MSG |
private java.util.concurrent.Semaphore |
partitionsInRAM |
private OfflineSorter.BufferSize |
ramBufferSize |
(package private) OfflineSorter.SortInfo |
sortInfo |
private java.lang.String |
tempFileNamePrefix |
private int |
valueLength |
Constructor and Description |
---|
OfflineSorter(Directory dir,
java.lang.String tempFileNamePrefix)
Defaults constructor.
|
OfflineSorter(Directory dir,
java.lang.String tempFileNamePrefix,
java.util.Comparator<BytesRef> comparator)
Defaults constructor with a custom comparator.
|
OfflineSorter(Directory dir,
java.lang.String tempFileNamePrefix,
java.util.Comparator<BytesRef> comparator,
OfflineSorter.BufferSize ramBufferSize,
int maxTempfiles,
int valueLength,
java.util.concurrent.ExecutorService exec,
int maxPartitionsInRAM)
All-details constructor.
|
Modifier and Type | Method and Description |
---|---|
java.util.Comparator<BytesRef> |
getComparator()
Returns the comparator in use to sort entries
|
Directory |
getDirectory()
Returns the
Directory we use to create temp files. |
private OfflineSorter.Partition |
getPartition(java.util.concurrent.Future<OfflineSorter.Partition> future) |
protected OfflineSorter.ByteSequencesReader |
getReader(ChecksumIndexInput in,
java.lang.String name)
Subclasses can override to change how byte sequences are read from disk.
|
java.lang.String |
getTempFileNamePrefix()
Returns the temp file name prefix passed to
Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext) to generate temporary files. |
protected OfflineSorter.ByteSequencesWriter |
getWriter(IndexOutput out,
long itemCount)
Subclasses can override to change how byte sequences are written to disk.
|
(package private) void |
mergePartitions(Directory trackingDir,
java.util.List<java.util.concurrent.Future<OfflineSorter.Partition>> segments)
Merge the most recent
maxTempFile partitions into a new partition. |
(package private) OfflineSorter.Partition |
readPartition(OfflineSorter.ByteSequencesReader reader)
Read in a single partition of data, setting isExhausted[0] to true if there are no more items.
|
java.lang.String |
sort(java.lang.String inputFileName)
Sort input to a new temp file, returning its name.
|
private void |
verifyChecksum(java.lang.Throwable priorException,
OfflineSorter.ByteSequencesReader reader)
Called on exception, to check whether the checksum is also corrupt in this source, and add that
information (checksum matched or didn't) as a suppressed exception.
|
public static final long MB
public static final long GB
public static final long MIN_BUFFER_SIZE_MB
public static final long ABSOLUTE_MIN_SORT_BUFFER_SIZE
private static final java.lang.String MIN_BUFFER_SIZE_MSG
public static final int MAX_TEMPFILES
private final Directory dir
private final int valueLength
private final java.lang.String tempFileNamePrefix
private final java.util.concurrent.ExecutorService exec
private final java.util.concurrent.Semaphore partitionsInRAM
private final OfflineSorter.BufferSize ramBufferSize
OfflineSorter.SortInfo sortInfo
private int maxTempFiles
private final java.util.Comparator<BytesRef> comparator
public static final java.util.Comparator<BytesRef> DEFAULT_COMPARATOR
public OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix) throws java.io.IOException
java.io.IOException
OfflineSorter.BufferSize.automatic()
public OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix, java.util.Comparator<BytesRef> comparator) throws java.io.IOException
java.io.IOException
OfflineSorter.BufferSize.automatic()
public OfflineSorter(Directory dir, java.lang.String tempFileNamePrefix, java.util.Comparator<BytesRef> comparator, OfflineSorter.BufferSize ramBufferSize, int maxTempfiles, int valueLength, java.util.concurrent.ExecutorService exec, int maxPartitionsInRAM)
valueLength
is -1 (the default), the length of each value differs; otherwise,
all values have the specified length. If you pass a non-null ExecutorService
then it will be
used to run sorting operations that can be run concurrently, and maxPartitionsInRAM is the maximum
concurrent in-memory partitions. Thus the maximum possible RAM used by this class while sorting is
maxPartitionsInRAM * ramBufferSize
.public java.lang.String getTempFileNamePrefix()
Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext)
to generate temporary files.public java.lang.String sort(java.lang.String inputFileName) throws java.io.IOException
java.io.IOException
private void verifyChecksum(java.lang.Throwable priorException, OfflineSorter.ByteSequencesReader reader) throws java.io.IOException
java.io.IOException
void mergePartitions(Directory trackingDir, java.util.List<java.util.concurrent.Future<OfflineSorter.Partition>> segments) throws java.io.IOException
maxTempFile
partitions into a new partition.java.io.IOException
OfflineSorter.Partition readPartition(OfflineSorter.ByteSequencesReader reader) throws java.io.IOException, java.lang.InterruptedException
java.io.IOException
java.lang.InterruptedException
protected OfflineSorter.ByteSequencesWriter getWriter(IndexOutput out, long itemCount) throws java.io.IOException
java.io.IOException
protected OfflineSorter.ByteSequencesReader getReader(ChecksumIndexInput in, java.lang.String name) throws java.io.IOException
java.io.IOException
public java.util.Comparator<BytesRef> getComparator()
private OfflineSorter.Partition getPartition(java.util.concurrent.Future<OfflineSorter.Partition> future) throws java.io.IOException
java.io.IOException