Peter Siebel - Practical Common Lisp

(defconstant +null+ (code-char 0))

(defun read-null-terminated-ascii (in)

(with-output-to-string (s)

(loop for char = (code-char (read-byte in))

until (char= char +null+) do (write-char char s))))

The WITH-OUTPUT-TO-STRING macro, which I mentioned in Chapter 14, is an easy way to build up a string when you don't know how long it'll be. It creates a STRING-STREAM and binds it to the variable name specified, sin this case. All characters written to the stream are collected into a string, which is then returned as the value of the WITH-OUTPUT-TO-STRING form.

To write a string back out, you just need to translate the characters back to numeric values that can be written with WRITE-BYTE and then write the null terminator after the string contents.

(defun write-null-terminated-ascii (string out)

(loop for char across string

do (write-byte (char-code char) out))

(write-byte (char-code +null+) out))

As these examples show, the main intellectual challenge—such as it is—of reading and writing primitive elements of binary files is understanding how exactly to interpret the bytes that appear in a file and to map them to Lisp data types. If a binary file format is well specified, this should be a straightforward proposition. Actually writing functions to read and write a particular encoding is, as they say, a simple matter of programming.

Now you can turn to the issue of reading and writing more complex on-disk structures and how to map them to Lisp objects.

Since binary formats are usually used to represent data in a way that makes it easy to map to in-memory data structures, it should come as no surprise that composite on-disk structures are usually defined in ways similar to the way programming languages define in-memory structures. Usually a composite on-disk structure will consist of a number of named parts, each of which is itself either a primitive type such as a number or a string, another composite structure, or possibly a collection of such values.

For instance, an ID3 tag defined in the 2.2 version of the specification consists of a header made up of a three-character ISO-8859-1 string, which is always "ID3"; two one-byte unsigned integers that specify the major version and revision of the specification; eight bits worth of boolean flags; and four bytes that encode the size of the tag in an encoding particular to the ID3 specification. Following the header is a list of frames , each of which has its own internal structure. After the frames are as many null bytes as are necessary to pad the tag out to the size specified in the header.

If you look at the world through the lens of object orientation, composite structures look a lot like classes. For instance, you could write a class to represent an ID3 tag.

(defclass id3-tag ()

((identifier :initarg :identifier :accessor identifier)

(major-version :initarg :major-version :accessor major-version)

(revision :initarg :revision :accessor revision)

(flags :initarg :flags :accessor flags)

(size :initarg :size :accessor size)

(frames :initarg :frames :accessor frames)))

An instance of this class would make a perfect repository to hold the data needed to represent an ID3 tag. You could then write functions to read and write instances of this class. For example, assuming the existence of certain other functions for reading the appropriate primitive data types, a read-id3-tagfunction might look like this:

(defun read-id3-tag (in)

(let ((tag (make-instance 'id3-tag)))

(with-slots (identifier major-version revision flags size frames) tag

(setf identifier (read-iso-8859-1-string in :length 3))

(setf major-version (read-u1 in))

(setf revision (read-u1 in))

(setf flags (read-u1 in))

(setf size (read-id3-encoded-size in))

(setf frames (read-id3-frames in :tag-size size)))

tag))

The write-id3-tagfunction would be structured similarly—you'd use the appropriate write-*functions to write out the values stored in the slots of the id3-tagobject.

It's not hard to see how you could write the appropriate classes to represent all the composite data structures in a specification along with read-fooand write-foofunctions for each class and for necessary primitive types. But it's also easy to tell that all the reading and writing functions are going to be pretty similar, differing only in the specifics of what types they read and the names of the slots they store them in. It's particularly irksome when you consider that in the ID3 specification it takes about four lines of text to specify the structure of an ID3 tag, while you've already written eighteen lines of code and haven't even written write-id3-tagyet.

What you'd really like is a way to describe the structure of something like an ID3 tag in a form that's as compressed as the specification's pseudocode yet that can also be expanded into code that defines the id3-tagclass and the functions that translate between bytes on disk and instances of the class. Sounds like a job for a macro.

Since you already have a rough idea what code your macros will need to generate, the next step, according to the process for writing a macro I outlined in Chapter 8, is to switch perspectives and think about what a call to the macro should look like. Since the goal is to be able to write something as compressed as the pseudocode in the ID3 specification, you can start there. The header of an ID3 tag is specified like this:

ID3/file identifier "ID3"

ID3 version $02 00

ID3 flags %xx000000

ID3 size 4 * %0xxxxxxx

In the notation of the specification, this means the "file identifier" slot of an ID3 tag is the string "ID3" in ISO-8859-1 encoding. The version consists of two bytes, the first of which—for this version of the specification—has the value 2 and the second of which—again for this version of the specification—is 0. The flags slot is eight bits, of which all but the first two are 0, and the size consists of four bytes, each of which has a 0 in the most significant bit.

Some information isn't captured by this pseudocode. For instance, exactly how the four bytes that encode the size are to be interpreted is described in a few lines of prose. Likewise, the spec describes in prose how the frame and subsequent padding is stored after this header. But most of what you need to know to be able to write code to read and write an ID3 tag is specified by this pseudocode. Thus, you ought to be able to write an s-expression version of this pseudocode and have it expanded into the class and function definitions you'd otherwise have to write by hand—something, perhaps, like this:

(define-binary-class id3-tag

((file-identifier (iso-8859-1-string :length 3))

(major-version u1)

(revision u1)

(flags u1)

(size id3-tag-size)

(frames (id3-frames :tag-size size))))

The basic idea is that this form defines a class id3-tagsimilar to the way you could with DEFCLASS , but instead of specifying things such as :initargand :accessors, each slot specification consists of the name of the slot— file-identifier, major-version, and so on—and information about how that slot is represented on disk. Since this is just a bit of fantasizing, you don't have to worry about exactly how the macro define-binary-classwill know what to do with expressions such as (iso-8859-1-string :length 3), u1, id3-tag-size, and (id3-frames :tag-size size); as long as each expression contains the information necessary to know how to read and write a particular data encoding, you should be okay.