Table of Contents

Class ColumnWriter

Namespace
ParquetSharp
Assembly
ParquetSharp.dll

Writer of physical Parquet values to a single column.

public abstract class ColumnWriter : IDisposable
Inheritance
ColumnWriter
Implements
Derived
Inherited Members

Properties

ColumnDescriptor

Get the ColumnDescriptor for the column.

public ColumnDescriptor ColumnDescriptor { get; }

Property Value

ColumnDescriptor

ColumnIndex

Get the index of the column within the row group.

public int ColumnIndex { get; }

Property Value

int

ElementType

Get the element Type of the data being written.

public abstract Type ElementType { get; }

Property Value

Type

EstimatedBufferedStats

Estimated sizes of data buffered by this column writer (levels, values and dictionary) that has not yet been written to pages.

public BufferedWriterStats EstimatedBufferedStats { get; }

Property Value

BufferedWriterStats

Remarks

This does not include completed data pages held in memory while the column is dictionary encoded, which are reported by TotalCompressedBytes.

When the column is dictionary encoded, ValueBytes is a conservative estimate of the encoded dictionary indices size and may be non-zero when no values are buffered. After the column is closed, DictBytes continues to report the size of the dictionary, although the dictionary page has been written.

Handle

protected nint Handle { get; }

Property Value

nint

LogicalTypeFactory

Get the LogicalTypeFactory for the Parquet file writer.

public LogicalTypeFactory LogicalTypeFactory { get; }

Property Value

LogicalTypeFactory

LogicalWriteConverterFactory

Get the LogicalWriteConverterFactory for the Parquet file writer.

public LogicalWriteConverterFactory LogicalWriteConverterFactory { get; }

Property Value

LogicalWriteConverterFactory

RowWritten

Get the number of rows written to the column so far.

public long RowWritten { get; }

Property Value

long

TotalBytesWritten

Total uncompressed bytes of data and dictionary pages written by the page writer for this column so far.

public long TotalBytesWritten { get; }

Property Value

long

Remarks

This does not include completed data pages held in memory while the column is dictionary encoded, which are reported by TotalCompressedBytes.

TotalCompressedBytes

Total compressed bytes of data pages for this column that have been completed but not yet written by the page writer.

public long TotalCompressedBytes { get; }

Property Value

long

Remarks

Data pages are held in memory rather than written while the column is dictionary encoded, until the dictionary page is written when the column is closed. This is an estimate, as the size of each page header is over-estimated.

TotalCompressedBytesWritten

Total compressed bytes of data and dictionary pages written by the page writer for this column so far. If the column is uncompressed, this is equal to TotalBytesWritten.

public long TotalCompressedBytesWritten { get; }

Property Value

long

Remarks

This does not include completed data pages held in memory while the column is dictionary encoded, which are reported by TotalCompressedBytes.

Type

Get the physical type of the column.

public PhysicalType Type { get; }

Property Value

PhysicalType

WriterProperties

Get the WriterProperties for the column.

public WriterProperties WriterProperties { get; }

Property Value

WriterProperties

Methods

Apply<TReturn>(IColumnWriterVisitor<TReturn>)

Apply a visitor to the column writer.

public abstract TReturn Apply<TReturn>(IColumnWriterVisitor<TReturn> visitor)

Parameters

visitor IColumnWriterVisitor<TReturn>

The visitor instance.

Returns

TReturn

The result of the visitor operation.

Type Parameters

TReturn

The return type of the visitor.

Close()

public long Close()

Returns

long

Dispose()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.

public void Dispose()

LogicalWriter(int)

public LogicalColumnWriter LogicalWriter(int bufferLength = 4096)

Parameters

bufferLength int

The buffer length in bytes. Default is 4KB.

Returns

LogicalColumnWriter

A LogicalColumnWriter instance.

LogicalWriterOverride<TElement>(int)

Create a strongly-typed LogicalColumnWriter with an explicit element type override.

public LogicalColumnWriter<TElement> LogicalWriterOverride<TElement>(int bufferLength = 4096)

Parameters

bufferLength int

The buffer length in bytes. Default is 4KB.

Returns

LogicalColumnWriter<TElement>

A LogicalColumnWriter instance.

Type Parameters

TElement

The type of the data to write.

LogicalWriter<TElement>(int)

Create a strongly-typed LogicalColumnWriter without an explicit element type override.

public LogicalColumnWriter<TElement> LogicalWriter<TElement>(int bufferLength = 4096)

Parameters

bufferLength int

The buffer length in bytes. Default is 4KB.

Returns

LogicalColumnWriter<TElement>

A LogicalColumnWriter instance.

Type Parameters

TElement

The type of the data to write.