bes  Updated for version 3.20.6
dmrpp::Chunk Class Reference

#include <Chunk.h>

Collaboration diagram for dmrpp::Chunk:
Collaboration graph

Public Member Functions

virtual void add_tracking_query_param ()
 Modify this chunk's data URL so that it includes tracking info. More...
 
 Chunk ()
 Get an empty chunk. More...
 
 Chunk (const Chunk &h4bs)
 
 Chunk (const std::string &data_url, unsigned long long size, unsigned long long offset, const std::vector< unsigned int > &pia_vec)
 Get a chunk initialized with values. More...
 
 Chunk (const std::string &data_url, unsigned long long size, unsigned long long offset, std::string pia_str="")
 Get a chunk initialized with values. More...
 
virtual void dump (std::ostream &strm) const
 
virtual unsigned long long get_bytes_read () const
 Get the number of bytes read so far for this Chunk. More...
 
virtual std::string get_curl_range_arg_string ()
 Returns a curl range argument. The libcurl requires a string argument for range-ge activitys, this method constructs one in the required syntax from the offset and size information for this byteStream. More...
 
virtual std::string get_data_url () const
 Get the data url string for this Chunk's data block. More...
 
virtual bool get_is_inflated () const
 
virtual unsigned long long get_offset () const
 Get the offset to this Chunk's data block. More...
 
virtual const std::vector< unsigned int > & get_position_in_array () const
 
virtual char * get_rbuf ()
 
virtual unsigned long long get_rbuf_size () const
 
virtual unsigned long long get_size () const
 Get the size of this Chunk's data block on disk. More...
 
virtual void inflate_chunk (bool deflate, bool shuffle, unsigned int chunk_size, unsigned int elem_width)
 Decompress data in the chunk, managing the Chunk's data buffers. More...
 
Chunkoperator= (const Chunk &rhs)
 
virtual void read_chunk ()
 
virtual void set_bytes_read (unsigned long long bytes_read)
 Set the size of this Chunk's data block. More...
 
virtual void set_data_url (const std::string &data_url)
 Get the data url string for this Chunk's data block. More...
 
virtual void set_is_inflated (bool state)
 
virtual void set_is_read (bool state)
 
virtual void set_position_in_array (const std::string &pia)
 parse the chunk position string More...
 
virtual void set_position_in_array (const std::vector< unsigned int > &pia)
 Set the chunk's position in the Array. More...
 
virtual void set_rbuf (char *buf, unsigned int size)
 Set the read buffer. More...
 
virtual void set_rbuf_to_size ()
 Allocates the internal read buffer to be d_size bytes. More...
 
virtual std::string to_string () const
 

Protected Member Functions

void _duplicate (const Chunk &bs)
 

Friends

class ChunkTest
 
class DmrppCommonTest
 

Detailed Description

This class is used to encapsulate the state and behavior needed for reading chunked data associated with a DAP variable. In particular it is based on the semantics of an hdf4:Chunk object, which is used to represent a chunk of data in a (potentially complex) HDF4/HDF5 file.

Definition at line 43 of file Chunk.h.

Constructor & Destructor Documentation

◆ Chunk() [1/3]

dmrpp::Chunk::Chunk ( )
inline

Get an empty chunk.

Note
This constructor does not read the Query String marker from the BES context system. You must call Chunk::add_tracking_query_param() if you want that information added with Chunks created using this constructor.
See also
Chunk::add_tracking_query_param()

Definition at line 95 of file Chunk.h.

◆ Chunk() [2/3]

dmrpp::Chunk::Chunk ( const std::string &  data_url,
unsigned long long  size,
unsigned long long  offset,
std::string  pia_str = "" 
)
inline

Get a chunk initialized with values.

Parameters
data_urlWhere to read this chunk's data
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_strA string that provides the logical position of this chunk in an Array. Has the syntax '[1,2,3,4]'.

Definition at line 110 of file Chunk.h.

◆ Chunk() [3/3]

dmrpp::Chunk::Chunk ( const std::string &  data_url,
unsigned long long  size,
unsigned long long  offset,
const std::vector< unsigned int > &  pia_vec 
)
inline

Get a chunk initialized with values.

Parameters
data_urlWhere to read this chunk's data
sizeThe number of bytes to read
offsetRead
  • size bytes starting from this offset
pia_vecThe logical position of this chunk in an Array; a std::vector of unsigned ints.

Definition at line 127 of file Chunk.h.

Member Function Documentation

◆ add_tracking_query_param()

void dmrpp::Chunk::add_tracking_query_param ( )
virtual

Modify this chunk's data URL so that it includes tracking info.

Add information to the Query string of a URL, intended primarily to aid in tracking the origin of requests when reading data from S3. The information added to the query string comes from a BES Context command sent to the BES by a client (e.g., the OLFS). The addition takes the form "?tracking_context=<context value>".

Note
This is only added to data URLs that reference S3.

Cloudydap test hack where we tag the S3 URLs with a query string for the S3 log in order to track S3 requests. The tag is submitted as a BESContext with the request. Here we check to see if the request is for an AWS S3 object, if it is AND we have the magic BESContext "cloudydap" then we add a query parameter to the S3 URL for tracking purposes.

Should this be a function? FFS why? This is the ONLY place where this needs happen, as close to the curl call as possible and we can just turn it off down the road. - ndp 1/20/17 (EOD)

Well, it's a function now... ;-) jhrg 8/6/18

Definition at line 401 of file Chunk.cc.

◆ dump()

void dmrpp::Chunk::dump ( std::ostream &  strm) const
virtual

unsigned long long d_size; unsigned long long d_offset; std::string d_md5; std::string d_uuid; std::string d_data_url; std::vector<unsigned int> d_chunk_position_in_array;

Definition at line 578 of file Chunk.cc.

◆ get_bytes_read()

virtual unsigned long long dmrpp::Chunk::get_bytes_read ( ) const
inlinevirtual

Get the number of bytes read so far for this Chunk.

Definition at line 200 of file Chunk.h.

◆ get_curl_range_arg_string()

string dmrpp::Chunk::get_curl_range_arg_string ( )
virtual

Returns a curl range argument. The libcurl requires a string argument for range-ge activitys, this method constructs one in the required syntax from the offset and size information for this byteStream.

Definition at line 383 of file Chunk.cc.

◆ get_data_url()

virtual std::string dmrpp::Chunk::get_data_url ( ) const
inlinevirtual

Get the data url string for this Chunk's data block.

Definition at line 177 of file Chunk.h.

◆ get_offset()

virtual unsigned long long dmrpp::Chunk::get_offset ( ) const
inlinevirtual

Get the offset to this Chunk's data block.

Definition at line 169 of file Chunk.h.

◆ get_position_in_array()

virtual const std::vector<unsigned int>& dmrpp::Chunk::get_position_in_array ( ) const
inlinevirtual
Returns
The chunk's position in the array, as a vector of ints.

Definition at line 272 of file Chunk.h.

◆ get_rbuf()

virtual char* dmrpp::Chunk::get_rbuf ( )
inlinevirtual

Returns a pointer to the memory buffer for this Chunk. The return value is NULL if no memory has been allocated.

Definition at line 235 of file Chunk.h.

◆ get_rbuf_size()

virtual unsigned long long dmrpp::Chunk::get_rbuf_size ( ) const
inlinevirtual

Returns the size, in bytes, of the read buffer for this Chunk.

Definition at line 264 of file Chunk.h.

◆ get_size()

virtual unsigned long long dmrpp::Chunk::get_size ( ) const
inlinevirtual

Get the size of this Chunk's data block on disk.

Definition at line 161 of file Chunk.h.

◆ inflate_chunk()

void dmrpp::Chunk::inflate_chunk ( bool  deflate,
bool  shuffle,
unsigned int  chunk_size,
unsigned int  elem_width 
)
virtual

Decompress data in the chunk, managing the Chunk's data buffers.

This method tracks if a chunk has already been decompressed, so, like read_chunk() it can be called for a chunk that has already been decompressed without error.

Parameters
deflateTrue if the chunk should be 'inflated'
shuffleTrue if the chunk should be 'unshuffled'
chunk_sizeThe expected chunk size, in elements; used to allocate storage
elem_widthThe number of bytes per element

Definition at line 472 of file Chunk.cc.

◆ operator=()

Chunk& dmrpp::Chunk::operator= ( const Chunk rhs)
inline

I think this is broken. vector<Chunk> assignment fails in the read_atomic() method but 'assignment' using a reference works. This bug shows up in DmrppCommnon::read_atomic(). jhrg 4/10/18

Definition at line 149 of file Chunk.h.

◆ read_chunk()

void dmrpp::Chunk::read_chunk ( )
virtual

This method is for reading one chunk after the other, using a CURL handle from the CurlHandlePool.

Parameters
deflate
shuffle
chunk_size
elem_width

Definition at line 541 of file Chunk.cc.

◆ set_bytes_read()

virtual void dmrpp::Chunk::set_bytes_read ( unsigned long long  bytes_read)
inlinevirtual

Set the size of this Chunk's data block.

Parameters
sizeSize of the data in bytes

Definition at line 209 of file Chunk.h.

◆ set_data_url()

virtual void dmrpp::Chunk::set_data_url ( const std::string &  data_url)
inlinevirtual

Get the data url string for this Chunk's data block.

Definition at line 192 of file Chunk.h.

◆ set_position_in_array() [1/2]

void dmrpp::Chunk::set_position_in_array ( const std::string &  pia)
virtual

parse the chunk position string

Extract information from the chunk position string and store as a vector of integers in the instance

Note
If we can change the DMR++ syntax to be less verbose and use a list of ints with whitespace as a separator, then the parsing code will be much simpler since istringstream is designed to deal with exactly that form of input.
Parameters
piaThe chunk position string. Syntax parsed: "[1,2,3,4]"

Definition at line 331 of file Chunk.cc.

◆ set_position_in_array() [2/2]

void dmrpp::Chunk::set_position_in_array ( const std::vector< unsigned int > &  pia)
virtual

Set the chunk's position in the Array.

Use this method when the vector<unsigned int> is known.

See also
Chunk::set_position_in_array(const string &pia)
Parameters
piaA vector of unsigned ints.

Definition at line 367 of file Chunk.cc.

◆ set_rbuf()

virtual void dmrpp::Chunk::set_rbuf ( char *  buf,
unsigned int  size 
)
inlinevirtual

Set the read buffer.

Transfer control of the buffer to this object. The buffer must have been allocated using 'new char[size]'. This object will delete any previously allocated buffer and take control of the one passed in with this method. The size and number of bytes read are set to the value of 'size.'

Parameters
bufThe new buffer to be used by this instance.
sizeThe size of the new buffer.

Definition at line 251 of file Chunk.h.

◆ set_rbuf_to_size()

virtual void dmrpp::Chunk::set_rbuf_to_size ( )
inlinevirtual

Allocates the internal read buffer to be d_size bytes.

The memory of the read buffer is managed internally by this method. Calling this method will release any previously allocated read buffer memory and then allocate a new memory block. The bytes_read counter is reset to zero.

Definition at line 222 of file Chunk.h.


The documentation for this class was generated from the following files: