Sample Code
You can download my sample project here:
I'll discuss the major issues and points of interest below.
Kinds of Buttons
Although push buttons, check boxes, and radio buttons are all implemented by the NSButton control class and NSButtonCell cell class, I will handle checkboxes and radio buttons separately from push buttons. Push buttons usually have text or an image surrounded by a border, while checkboxes and radio buttons have no border, just an image and text. In the case of checkboxes, it may also be necessary to display a third state, NSMixedState.
States and Highlights
A button may display 2 or 3 possible states: NSOffState, NSOnState, and possibly NSMixedState. Independently, it may display no highlight, or a "pressed" highlight when the mouse is down on it, or perhaps a "mouse-over" highlight when the mouse cursor is over the button without the mouse button being pressed. With a maximum of 3 states and 3 highlights, there are up to 9 combined conditions, which may all have distinct appearances.
Quick and Dirty Customization
You can get a simple customized checkbox or radio button without any programming, just by setting the image and alternate image. However, this approach doesn't handle the mixed state, doesn't fully handle the "pressed" highlight, and does not provide any mouse-over effect.
Images
You will specify the appearance for each combination of state and highlight by providing a set of image resources. To avoid hard-coding the image names, I used an indirect approach. The cell class has a string property serving as a name for the whole image collection, and you can specify this property in the nib as a user-defined runtime attribute. With the aid of a helper class, this name will be looked up in a helper class, resulting in a dictionary that maps names of state/highlight combinations to image resource names.
Mouse-over Detection
To keep track of whether the mouse cursor is over the control, you might think you would need to subclass the control and add a tracking area. But in fact, the cell subclass can override showsBorderOnlyWhileMouseInside to return YES, and then the cell will receive mouseEntered: and mouseExited: messages.
Drawing
The key part of the cell subclass is the override of the NSButtonCell method drawImage:withFrame:inView:. This method finds the appropriate image for the current state and highlight, and draws it.
Nib Setup
You will need to created images for each desired combination of state and highlight, and add them to the project. You also need to add a property list named UISkin.plist that maps some name of your image collection to a dictionary, which in turn maps state names to image names. See the sample project.
To add a skinned checkbox or radio button to a nib, start by dropping in a standard checkbox or matrix of radio buttons. Select a cell to be customized in the hierarchical list of objects in the nib. Use the Identity Inspector to change the class from NSButtonCell to SkinCheckboxCell. Under "User Defined Runtime Attributes", add an attribute named "images", with type String, and value being the image collection name used in UISkin.plist. FInally, in the Attributes Inspector, set the image to one of the images in your image collection. This lets the button know the size of your images.