Data Compression

So as to be able to keep as much data in memory as possible, unique values within each column are stored in a lookup array. Another array is used as an index that corresponds with the actual value in the lookup array.

This means each value is stored once only instead of multiple times. In the compressed physical file, unique values are saved first, followed by the required number of bits used to represent it.

When loaded into memory 8 bits (char) are used if the number of unique values is 255 or less, 16 bits (ushort) if its 65535 or less, otherwise 32 bits (int).