Working with Streaming Buffer

Cluster

Cluster is a continuous memory region (a sequence of bytes), which is described by the class Cluster in the file io_buffer.h.

The class contains the cluster start (shift - for disk clusters, address - for memory) and the cluster length in bytes.

The class implements the methods for accessing the cluster data:

• uint16 _len() - Get the cluster length.

• size_t _off() - Get the cluster start field.

• const uint8 * _ptr() - Get the cluster start as an address.

• uint8 * _getMemPointer() - Get the cluster start as an address.

• void clear() - Clear the cluster.

• void AllocFile( size_t beg, uint16 len ) - Memorize the shift in file and the bytes count.

• void SetClusterOffset( size_t off ) - Set the cluster start.

• void SetClusterLength( uint16 len ) - Set the cluster length.

• static size_t SizeOf( VERSION version ) - Size of the cluster data in the stream for a given version.

File Space

File space is a set of clusters. File space is described by the class FileSpace in the file io_buffer.h.

The class contains the array of cluster indexes and the number of used bytes in the last cluster.

The class implements the methods for accessing the array of cluster indexes:

• size_t Count() - Get the number of elements in the array of cluster indexes.

• void Flush() - Set the number of elements in the array to zero.

• void RemoveInd( size_t ) - Delete an element from the array.

• size_t * Add() - Add an element to the end of the array.

• size_t * Add( const size_t & ) - Add a given element to the end of the array.

• void SetSize( size_t, bool ) - Set the new array size.

• void Reserve( size_t ) - Reserve space for a given number of elements.

• bool IsExist( size_t & ) - Return true, if the element found.

• size_t FindIt( size_t & el ) - Find an index of the element (if not found, return -1).

• size_t * InsertInd( size_t index, const size_t & el ) - Insert an empty element before the given one.

• const size_t * GetAddr() - Get the address of the beginning of the array.

• size_t * AddItems( size_t n ) - Add n elements to the end of the array.

• size_t & operator []( size_t loc ) - Access operator by index.

• uint16 & rest() - Get the number of used bytes in the last cluster.

Read/write Position

The class ClusterReference describes the position in the cluster for reading/writing. It is defined in the file io_buffer.h.

The class contains the cluster index in the array of cluster indexes of the buffer iobuf_Seq and the offset in this cluster.

Streaming Sequential Buffer

The class iobuf_Seq describes streaming buffer, which provides only sequential writing/reading (the base class). The class iobuf_Seq and its inheritors serve for reading/writing operations in the interest of stream (the class tape). The class is defined in the file io_buffer.h.

The following terms are used:

• Storage - a disk file or a memory region.

• File - a file space inside a storage.

The main class data:

• FileSpace sys - «system» file space. Opens in the constructor of the class tape.

• PArray<FileSpace> files - an array of file spaces, which are contained in iobuf_Seq. The first element of the array is always an address of sys.

• uint32 stateFlag - the buffer state.

• VERSION storageVers - the storage version.

• VERSION curFileVers - the version of the current open file space (stream). In general case the storage version and a version of any file space in it may be different.

• uint8 bufferMode - mode in which the buffer can operate.

• uint8 curFileMode - the opening mode of the current file. In general case the storage mode and the mode of open file in this storage can be different. Constraint - if the buffer mode is io::in, then an attempt to open the file for writing (io::out) doesn’t result in opening.

• uint8 * base - a pointer to the beginning of the buffer in memory.

• uint8 * ptr - a pointer to the next symbol in memory.

• uint8 * end - a pointer to the end of buffer in memory. When working with the disk, pointers are set to a fixed memory block the sections of file are loaded to while reading. When working with the memory, membuf sets them to the memory allocated for the cluster.

The main methods of the class:

• Access to the array of the clusters:

  • void Reserve( size_t n, bool addAdditionalSpace ) - Reserve space for a given number of elements.

  • void Flush() - Set the number of elements to null.

  • void HardFlush() - Free the whole memory.

  • void Adjust() - Free the unnecessary memory.

  • Cluster * Add() - Add an element to the end of the array.

  • Cluster * Add( const Cluster & ) - Add a given element to the end of the array.

  • size_t Count() - Get the number of elements in array.

  • Cluster & operator []( size_t ) - access by index operator.

• Access to file spaces:

  • void ReserveFiles( size_t n, bool addAdditionalSpace ) - Reserve space for a given number of FileSpace.

  • ClusterReference getCurrentClusterPos() - Get current position in the buffer.

  • FileSpace & sysFile() - Get access to the system file.

  • FileSpace * openedFile() - Get access to the open file.

• Access to versions:

  • void SetVersionsByStorage() - Set the current version to be equal to the storage version.

  • VERSION MathVersion() - Return the main version (version of the mathematical kernel).

  • VERSION AppVersion( size_t ) - Return the additional version (version of the target application).

  • const VersionContainer & GetVersionsContainer() - Get the buffer versions.

  • VERSION GetStorageVersion() - Get the storage version.

  • VERSION GetFormatVersion() - Get the format version.

  • void SetFormatVersion( VERSION ) - Set the format version.

Streaming Buffer with Arbitrary Access

The class VERSION describes a streaming buffer with arbitrary access (the base class). It inherits from the class iobuf_Seq and extends its functionality with the ability to delete and overwrite file spaces. It is defined in the file io_buffer.h.

The class contains the file space with the list of freed clusters. When a file space is deleted, all its clusters are moved here. While writing, when it is necessary to allocate the new cluster, this array is firstly checked for availability of free clusters. Clusters in «freed» are always ordered by shift from the beginning of the file space.

The class methods:

• bool open( FileSpace &, uint8, const VersionContainer &, bool fullCheck = true ) - Open the file space if it is one’s own file. The flag fullCheck = false switches off excessive checks (for the sake of performance).

• bool del( FileSpace & ) - Free space allocated for the file space.

• bool truncate( FileSpace &, size_t ) - Truncate the file space.

• bool detach( FileSpace & ) - Detach the file space from the buffer.

• bool speedOnClose() - Whether should be ordered while closing.

• void speedOnClose( bool ) - Set the flag of ordering while closing.

• void free( size_t ) - Free the cluster.

Memory Streaming Buffer

The class membuf implements memory stream buffer. It inherits from the class iobuf. The class is defined in the file io_memory_buffer.h.

Memory stream buffer is intended for using in read and write streams. Besides the instruments needed for the buffer, it has functions for packing to a contiguous memory block, that enables transferring data to another application via the memory.

The class contains the array of file spaces, which stores all used file spaces except «sys» defined in iobuf_Seq. When using the extended format, objects of each level are saved to a separate file space. Index of a file space in the array corresponds to the level of the object in the model.

The main class methods:

• bool isEmpty() - Whether the buffer is empty.

• size_t toMemory( const char *& m, size_t addSize = 0 ) - Write to contiguous memory. If m != 0 (memory is already allocated), then addSize is equal to memory size, if m = 0, then addSize defines a number of bytes to be added at the beginning and zeroed.

• bool fromMemory( const char * ) - Read from the contiguous memory.

• size_t getMemLen() - Compute necessary length of the contiguous memory block for a buffer.

• void clean() - Clear the buffer.

• void closeBuff() - Close the buffer.

Reading and Writing Memory Buffer

The functions for writing a contiguous memory buffer to a disk file and reading to contiguous memory buffer from a disk file are defined in the file io_memory_buffer.h.

Reading to a contiguous memory buffer from a disk file:

• iobuf & createiobuf( const TCHAR * fileName ).

Writing a contiguous memory buffer to a disk file:

• bool writeiobuftodisk( const TCHAR * fileName, membuf & buf ).