Reading Binary Files

To handle DNA data coming out of old and new generation sequencing machines, I needed to learn how to read the binary SCF files that are typically output by the software these machines use. To do so, I needed to know the byte format of these files, and I had to translate from C definitions of the data types to IDL data types.

The byte format page describes in detail how the bytes are ordered. There's a header, some comments, base calls, and traces. The base calls include probabilities and a cross-reference to the trace position where the base was called. The traces themselves are stored as consecutive differences as opposed to absolute values, so translating the SCF traces to something you're used to is non-trivial.

The biggest trick in this was figuring out how to use IDL's READU function, the OPENU function, the latter with the SWAP_ENDIAN keyword set. The byte-ordering (at least on the 3.00 version SCF files I used) is different from what IDL expects. Also note that READU shifts the file pointer according to the data type of the variable you'll be reading in, something I found out the hard way and later read in the documentation.

I found the POINT_LUN,unit,pos function was most useful for shifting and tracking the file pointer. Defining a variable = -1 * unit (i.e. the negative of the logical unit) makes POINT_LUN return the current position.

apply pixel mask

This is going to be an on-going post about making this piece of code run, . I'm just going to add to it bit-by-bit until I have something complete. Usually the first thing to check is:

setenv LD_LIBRARY_PATH /usr/local/lib/

in your .cshrc. There are other steps required, but on my current libraries, installed on a new machine, this is the thing I forget to add most often. /usr/local/lib is where I usually keep all the needed C/C++ libraries, but it certainly change if you move to other machines.