Canon's CRW Raw File Format Specification
Although not publicly documented by Canon, their CRW RAW file format specifiction is scattered in bits and pieces across the web. The CRW format is embedded in Canon's CIFF structure, a copy of the believed to be most recently publically published is CIFFspecV1R04.pdf. Compiled here is believed to be a complete specification for a CRW coder/decoder and a zip file of its C-code implementation. The companion Canon CR2 Specification may also be of interest.
The specifiction is listed below and is included in the crw.zip (revised 11/10/08) file as comments of the primary code file CrwCodec.cpp. The zip file contains the following:
CrwCodec.cpp - Primary C-code for CRW coder/decoder
CrwCodec.exe - its sample excutable
CrwCodec.h - its header file
CiffTags.h - CIFF format definitions
Prop.h - Properties type definitions
RawUtl.cpp - Various raw utilities
RawUtl.h - and its header file
getput.cpp - Low level input/output
getput.h - and its header file
jpeg.h - Various JPEG definitions
The executable CrwCodec.exe must be run in a Windows command shell. Executing the command without parameters yields its syntax as follows:
Usage: CrwCodec.exe [options] file.crw outfile
Decompress raw image data of file.crw
-b: decode then save binary file (default)
-c: decode then recode file
-h: decode then create (text) histogram
-t: decode then save as 8-bit TIFF file
-T: decode then save as 16-bit TIFF file
Version 1.0
Wildtramper.com, Copyright (c), All rights reserved
Although the -t or -T options create a TIFF file from RAW, expect crude colorization.
/*****************************************************************************
* CRW Compression Description
******************************************************************************
* The intent of this paper is to describe Canon's Raw (CRW) photo image
* compression algorithm. Related topics such as the file structure that
* contains the image, called CIFF, are discussed only as needed to understand
* the compression algorithm. For more information on CIFF, refer to
* http://xyrion.org/ciff, although version 1.0, revision 4 is incomplete and
* sometimes in error.
*
* An image file that uses the CRW compression algorithm is composed of a CIFF
* file structure which identifies various segments. The structure of a CIFF is
* shown below:
*
* +=====================================+ Start of CIFF/CRW file
* | CIFF Header |
* | Size = 26 |
* +=====================================+ Start of HEAP
* | Block of NULL data | RAW data Segment
* | Size = 514 bytes |
* +-- -- -- -- -- -- -- -- -- -+
* | Optional uncompressed low bits |
* | Size = ImageWidth * ImageHeight / 4 |
* +-- -- -- -- -- -- -- -- -- -+
* | Compressed RAW data |
* | incorporates JPG markers |
* | ends with JPG end-of-image marker |
* | Size = variable length, but even |
* +=====================================+
* | JPG of RAW | JPG segment
* | Size = variable length |
* +=====================================+
* | Thumbnail | Thumbnail segment
* | Size = variable length |
* +=====================================+
* | Image Properties | Image properties segment
* | SensorInfo(ImageWidth,ImageHeight)|
* | DecoderTableNumber |
* | Size = variable length |
* +=====================================+
* | Root Directory | Root directory segment
* | identifies location of RAW, JPG, |
* | Thumbnail, and Image Prop segments. |
* | Size = 42 bytes typical |
* +=====================================+
* | Pointer to Root Directory relative | Root directory offset
* | to start of HEAP |
* | Size = 4 bytes |
* +=====================================+ End of CIFF/CRW file
*
* What is relevant in the CIFF to understanding the CRW compression algorithm
* is the RAW data segment, a few parameters in the Image properties segment
* (i.e. ImageWidth, ImageHeight, and DecoderTableNumber), and the means to
* identify these segments through the Root directory segment and the Root
* directory offset.
*
* Before proceeding further, it is worthwhile to note that the CRW compression
* algorithm is very loosely based on the lossless JPEG algorithm, see CCITT
* T.81 (www.w3.org/Graphics/JPEG/itu-t81.pdf).
*
* The basic compression algorithm supports a coding depth of 10 bits. While
* some CRW files limit pixel depth to 10 bits others use an additional step to
* obtain a depth of 12 bits. This step saves the two least significant bits of
* each pixel into an independent concatenated 'uncompressed low bits' data
* segment. The choice of either 10 or 12 bit pixel depth is camera model
* dependent, e.g. Canon's Powershot G2 supports 10-bit pixel depth while the
* Powershot S70 supports 12-bit pixel depth.
*
* When uncompressed low bits are used, this data stream is placed in the RAW
* segment and it precedes the compressed data. The bit ordering of bytes of
* the four pixel duo-bits for the uncompressed low bits is:
*
* 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0
* +---+---+---+---+---+---+---+---+ +---+---+---+---+---+---+---+---+
* ...| p[n+3]| p[n+2]| p[n+1]| p[n] | | p[n+7]| p[n+6]| p[n+5]| p[n+4]|...
* +---+---+---+---+---+---+---+---+ +---+---+---+---+---+---+---+---+
* msb lsb msb lsb
*
* The CRW compressed image data is a continuous bit stream organized in 8-bit
* bytes to facilitate easy file transport. The stream consists of multiple
* concatenated couplets consisting of a variable length Huffmann instruction
* (i.e. HufCode or codeword) followed by a variable length difference code
* (i.e. Diff) bit stream as shown below:
*
* +--------------------+-------------+
* ... | HufCode[Rpt,nBits] | Diff[nBits] | ...
* +--------------------+-------------+
*
* As inferred by the couplet fields, compression of image data is done on a
* pixel by pixel basis with the difference between like color pixels being
* encoded. When an overall 10-bit color depth is encoded, then this difference
* is between two 10-bit quantities. When an overall 12-bit color depth is
* encoded, then the 2 LSBs are packed into an uncompressed low bits segment,
* and the remaining 10 bits are then used for the difference.
*
* As used in most digital camera, the compressed file encodes the raw sensor
* data, typically as a Bayer grid, with single color per pixel. When
* decompressed, the Bayer grid is usually extrapolated into three colors (RGB)
* per pixel. Bayer grids used by most manufacturers are:
*
* 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5
* 0 R G R G R G 0 G R G R G R 0 B G B G B G
* 1 G B G B G B 1 B G B G B G 1 G R G R G R
* 2 R G R G R G 2 G R G R G R 2 B G B G B G
* 3 G B G B G B 3 B G B G B G 3 G R G R G R
*
* Loosely speaking within the compressed data space, there is one couplet for
* each compressed pixel. Couplets represent differences between like colors.
* But when the difference bits represent no (i.e. 0) difference between the
* current pixel and the preceding one of the same color then fewer couplets are
* required. Here, the repeat count in the HufCode codeword is used to convey
* zero difference information.
*
* Each Huffmann code maps exactly into a codeword which conveys two pieces of
* information: A repeat count (Rpt) and another count (nBits) of how many bits
* the difference field requires. The repeat count is a number from 0 to 15,
* and the number of bits for the difference field is also a number from 0 to 15
* although in practical applications its upper limit is 10 or in rare
* circumstances is 11.
*
* The actual difference value contained in the Diff[nBits] is organized loosely
* as a signed magnitude number, but has its own specific rules. First, it
* should be noted that the value of nBits determines the range of the number.
* As an example if nBits = 3, then the magnitude (of a DiffVal that can be +/-
* this magnitude) is 4 <= magnitude <= 7, or more specifically:
*
* (1 << (nBits - 1)) <= DiffVal <= ((1 << nBits) - 1) // + DiffVal
* -(1 << (nBits - 1)) >= DiffVal >= -((1 << nBits) - 1) // - DiffVal
*
* If the most significant bit (MSB) of Diff[nBits] is one, then DiffVal is
* positive and has a magnitude directly represented by binary value of these
* bits. If however, the MSB is zero, then DiffVal is negative and has a
* magnitude that is the compliment of these bits. As an example if
* Diff[nBits=3] = 101, then DiffVal is 5, whereas if Diff[nBits=3] = 001, then
* DiffVal is -6.
*
* The HufCode/Diff couplet stream is further segmented into blocks of 64, that
* is 64 pixels, where additional rules apply. These 64 pixel blocks are
* asynchronous in alignment with both rows and columns of an image.
*
* Two unique Huffmann code tables are used within a 64 pixel block. When the
* 64 pixel block count (call it i64) is 0, then Huffmann table 1 is used,
* otherwise table 2 is used. When a new block is encountered, much of the
* history of the previous block is ignored, but not all. Within the block more
* information is retained. That is, Huffman table 1 always assumes the repeat
* count is 0, while table 2 uses a dynamic repeat count. There are also
* various sets of these two tables, although only three are known to exist.
* One of the ImageProperties is DecoderTableNumber.
*
* Of these three known table sets, they allow compression of 10-bit samples,
* mostly. Each of the table 1s can accommodate an 11-bit sample whereas each
* of the table 2s can accommodate a 10-bit sample, the reason for this
* ambiguity is not known. This can be seen by noting that the nBits field of
* table 1s ranges from 0 to 11, while for table 2s it ranges from 0 to 10.
* Repeat counts (Rpt) of table 1s are always 0, while for table 2s ranges from
* 0 to 15. In table set 2, certain codewords do not make sense which are
* excluded, while one special codeword is included. The codewords in table set
* 2 which are excluded are those where the number of Diff bits (nBits) is 0
* when the repeat count is from 1 to 14. These are removed because including
* them would, for all intense and purpose, have the meaning of repeat by Rpt+1
* since an additional repeat would be derived from a Diff that is 0 bits in
* length or a value of 0. But the codeword Rpt=15 with nBits=0 is included
* since there is no repeat by 16 codeword. The special codeword with Rpt=0 and
* nBits=0 when encountered means to continue repeating for the duration of the
* 64 pixel (i64) block, but it should be noted that this only applies when
* table set 2 is active.
*
* A typical table set with associated Huffman code lengths is:
*
* Table 1, set 0:
* { 0,1,4,2,3,1,1,0,0,0,0,0,0,0,0,0 }, // Huffmann code lengths
* { 0x04,0x03,0x05,0x06,0x02,0x07,0x01,0x08, // Rpt nibble + nBits nibble
* 0x09,0x00,0x0a,0x0b }
*
* Table 2, set 0:
* { 0,2,2,2,1,4,2,1,2,5,1,1,0, 0,0,139 }, // Huffmann code lengths
* { 0x03,0x04,0x02,0x05,0x01,0x06,0x07,0x08, // Rpt nibble + nBits nibble
* 0x12,0x13,0x11,0x14,0x09,0x15,0x22,0x00,0x21,0x16,0x0a,0xf0,
* 0x23,0x17,0x24,0x31,0x32,0x18,0x19,0x33,0x25,0x41,0x34,0x42,
* 0x35,0x51,0x36,0x37,0x38,0x29,0x79,0x26,0x1a,0x39,0x56,0x57,
* 0x28,0x27,0x52,0x55,0x58,0x43,0x76,0x59,0x77,0x54,0x61,0xf9,
* 0x71,0x78,0x75,0x96,0x97,0x49,0xb7,0x53,0xd7,0x74,0xb6,0x98,
* 0x47,0x48,0x95,0x69,0x99,0x91,0xfa,0xb8,0x68,0xb5,0xb9,0xd6,
* 0xf7,0xd8,0x67,0x46,0x45,0x94,0x89,0xf8,0x81,0xd5,0xf6,0xb4,
* 0x88,0xb1,0x2a,0x44,0x72,0xd9,0x87,0x66,0xd4,0xf5,0x3a,0xa7,
* 0x73,0xa9,0xa8,0x86,0x62,0xc7,0x65,0xc8,0xc9,0xa1,0xf4,0xd1,
* 0xe9,0x5a,0x92,0x85,0xa6,0xe7,0x93,0xe8,0xc1,0xc6,0x7a,0x64,
* 0xe1,0x4a,0x6a,0xe6,0xb3,0xf1,0xd3,0xa5,0x8a,0xb2,0x9a,0xba,
* 0x84,0xa4,0x63,0xe5,0xc5,0xf3,0xd2,0xc4,0x82,0xaa,0xda,0xe4,
* 0xf2,0xca,0x83,0xa3,0xa2,0xc3,0xea,0xc2,0xe2,0xe3 }
*
* Note in these tables, there are exactly 12 Huffmann codes required for table
* 1s and 162 for table 2s - these counts are the sum of the Huffmann code
* lengths in a table. The specific Huffman codes follow the normal Huffmann
* generation rules, such as described in CCITT T.81 when the Huffmann code
* lengths array is used with the codeword meanings. Thus, expanding table 1,
* set 0 produces:
*
* HuffmannCode HuffmannLength RepeatCount nBits
* 00 2 0 4
* 010 3 0 3
* 011 3 0 5
* 100 3 0 6
* 101 3 0 2
* 1100 4 0 7
* 1101 4 0 1
* 11100 5 0 8
* 11101 5 0 9
* 11110 5 0 0
* 111110 6 0 10
* 1111110 7 0 11
*
* The arrangement of the Huffmann codewords with their implied meaning is a
* statistical endeavor which assigns more likely events to shorter codewords
* with the intend of ultimately creating the smallest compressed file bit
* stream. Unlike JPG where Huffmann tables are included in the JPG file, only
* a table set number is conveyed in CRW files. This certainly simplifies
* encoding of the image, but it also likely creates compressed files that might
* have been compressed more densely. So simplicity trumps efficiency.
*
* It also should be noted that the assignment of Huffmann code lengths from the
* 3 known table sets do not generate an optimized Huffmann code set. This is
* realized when the last Huffmann code is not all ones, as is the case for all
* these tables. There may be a reason for this, but it remains unknown. The
* amount of inefficiency is probably small, certainly not much different from
* the fact that fixed (i.e. 1 of 3) table sets are used rather than dynamic
* tables.
*
* Before continuing further, it should be noted that the RAW compressed image
* data follows JPG marker rules (refer to CCITT T.81). A JPG marker is a file
* oriented, byte aligned, two byte sequence where the first byte is 0xFF and
* the second byte describes the action required. Only two markers are known in
* the CRW compressed image data segment. The first is the sequence 0xFF:0x00,
* and the second sequence is 0xFF:0xD9. The 0xFF:0x00 is a marker that means
* the marker is image data with a value 0xFF. The 0xFF:0xD9 is a marker that
* means end-of-image (EOI), and this is expected at the end of the image data
* block. Because all CIFF segments must be of even word length, the EOI marker
* may be followed by an additional NULL byte which should be ignored.
*
* When the EOI marker is encountered before the image is fully decompressed,
* then the decoder should assume that all subsequent pixel values repeat. It
* is not known why the compressed image is truncated in many CRW files, but it
* has been observed as such. The truncation has only been observed in the
* guard band area of an image that must be trimmed anyway. This guard band
* area is a few rows at the top and bottom of the image that are black, and a
* few columns at the right and left of the image that are also black. One
* observation is the EOI truncation occurred exactly one 64-bit block before
* the true end of the image.
*
* Since the EOI marker is byte aligned and the RAW data is bit aligned, it is
* likely that from 1 to 7 unused bits will follow the last RAW data before the
* EOI. These bits should be all ones, as observed.
*
* So at this point one can partially decompress a CRW file. What is known is
* (a) how to create two decoder tables of Huffmann codewords, (b) how to handle
* the JPEG markers in the compressed data stream, (c) how to divide the data
* stream into 64 pixel blocks, (d) how to use the two decoder tables based on
* the position in a 64 pixel block, (e) and what to do when an end of file JPG
* marker is encountered. What is not known are the rules associated when new
* rows are encountered, and how to get a start value at the beginning of a 64
* pixel block. This will now be discussed followed by a few words on image
* parameters extracted from the CIFF directory.
*
* Once a set of 64 differences is collected, it is processed to generate pixel
* values. There are a few rules to follow. Given image buffer IB[i] is the
* current pixel in the image buffer and DifBuf[i64] is the current difference
* value, then:
*
* IB[i] = IB[i-2] + DifBuf[i64]; // normal
* or
* IB[i] = 512 + DifBuf[i64]; // when i is column 0 or 1 of row
*
* But, another variable is required. This is a compensation of the first value
* in the DifBuf[] buffer that must be done after the Diffs are collected but
* before they are applied to the pixel image buffer. Essentially, the
* DifBuf[0] value is relative to the previous DifBuf[0]'s value one 64-bit
* block earlier. Thus:
*
* DifBuf[0] += Diff64Init; // reference Diff[0] to previous block
* Diff64Init = DifBuf[0]; // save for next block
* and
* Diff64Init = 0; // at start of process
*
* At this stage, a CRW file can almost be correctly decompressed. A few
* additional tokens of information are needed. First are several parameters
* embedded in the CIFF directory. These include: (a) Image Width, (b) Image
* Height, (c) Decoder Table Number, and (d) Absolute address and size of RAW
* data in the CIFF file.
*
* The RAW data segment has a leader of zeros before the actual data, and it is
* not known why this is the case. This leader is exactly 514 bytes in size.
*
* So there you have it. Good luck.
*****************************************************************************/