The nut plays a very important role in the tone and feel of the banjo. It should be made of bone or some material that is at least as hard. Plastic is too soft, wears out too quickly, and dampens the volume.
The spacing between the strings can affect playability. Too narrow, difficult to finger chords; too wide, might push the 1st and 4th strings off the side of the neck. The width of your neck will determine how wide you can space your strings. As with the bridge, the slots should be evenly spaced between the strings. The strings have to be high enough to clear the first fret or severe buzzing will occur. If the strings are too high, the action will feel stiff and tuning distortion with fretting and capoing at the lower frets will be a problem. You may want to take this to someone with experience in setup to get this critical adjustment made. The cut of the slot is also important. The slot should angle back towards the peg head at the same angle as the string to the tuner. Otherwise, a dull, fuzzy sound can occur from the nut. This works for the bridge slots as well. The slots should be about the same size as the string or slightly wider. This will help eliminate string binding in the slot. Also, the slots should not be too deep. Just a slight bit of nut or bridge material above the level of the string will be plenty.
5th String Nut
A very important and often overlooked item is the 5th string nut. This nut should also be bone. Bone blanks (1/8 diameter) are available from First Quality Music. The location is critical. It should be fit snug against the 5th fret, with the slot cut high enough to clear the fret. Some folks put a spike here and run the string off the fret, but, unlike capoing spikes, this holds the string against the fret constantly. Eventually, it wears a groove in the fret and buzzes. The fret can be dressed, but all that can be avoided with a bone nut. The distance from the 4th string should be checked. On some banjos, the 5th nut is too close to the 4th string. This should be checked and relocated by an experienced repairman.