There is a direct correlation between the size of a turbo and the amount of time it takes to spool. So, for your purposes, more power = more time waiting for boost to kick in. This is why a larger than stock turbo is not usually a benefit for autoX or drivability.
Then you can look at cost of build, dependability after build, noise, etc, etc...just be careful what you wish for.
