Finding the direction of gravity may not be required if you can say that the camera does not rotate or you know its extrinsics related to gravity. This may shorten or remove the "initialization phase" entirly.