You need three race portraits in your race directory, for any race, custom or otherwise, to function as intended.
You should replace the word racename, with your races name, in all instances below.
This needs to be named Racename_Race_LargePortrait.jpg, in a jpg format and be 512x512 pixels in size.
This needs to be named Racename_race_portrait.bmp and needs to be 128 x 128 pixels in size, using a 256 color bmp format.
This needs to be named Racename_Race_SmallPortraits.bmp, 82x44 pixels in size and in a 256 color bmp format.
On the original races portraits, they seem to be composed of a 36x36 pixel mini square portrait and a 42x42 pixel small portrait.
A good way to create these is to copy an existing races small portrait file, then copy and paste your own mini portraits over the top of theirs. In Photoshop or Paintshop Pro its easier to line up if you use a selection square.
(Photoshop users, to avoid errors)
To create the racename_race_portrait.bmp and the racename_Race_SmallPortraits.bmp You can just use the basic Microsoft paint program and convert the images to 256 colors but you lose a lot of detail, at first glance photoshop doesn't do this directly. As far as I can see the best way to keep high quality images, at a decent level, is to go into photoshop and convert to a gif file, using the 'save for the web' option; then go into paint, load up the file and save as a gif again. (Don't ask me why you have to save it again but I had to for it to work) Load the file again in paint and the detail is okay at this point, so you can convert to a 256 color bmp without too much bother.