Well the primary purpose of the redline isn't to just put an RPM limit after the point at which the engine makes its max. power... it's so you don't literally blow up the engine. Push the engine beyond a certain speed and a piston will bang into a valve, resulting in lots of little pieces everywhere in what's left of the motor. So, you want to establish a redline RPM at least slightly under that. As for how to calculate it, you got me.
.. Like Scott said, talk to reputable engine builders and see if they have a common way of figuring it out (that they'd be willing to share).