Optimal rotation can be considered from different perspectives. For example
- as little support as possible. This way app tries to orient object so as much surface as possible doesn't overhang and doesn't need support
- bed only support. App tries to orient object so all overhanging parts can be supported directly from the heatbed (not from the object itself)
But have a look on Meshmixer as Ryan Carlyle suggested. In this app you can see 3 kinds of auto orientation procedures. Additionally Meshmixer can create angle support and tree-like support.
In terms of mathematics.
I cannot suggest any sources but you can imagine the way it calculates such optimal rotation. Let's say it will be dead simple explanation of the second approach:
Let's get all faces (facets) and let's project their center points on the "floor" (heatbed) or "ceiling". If imagined projection line slices any other faces then rotation is not optimal. App can calculate "a weight" of such orientation and then try to find the smallest weight of all possible orientations.
Of course it's not that simple. App has to manage concavity, which face requires support at all, and many other aspects but in general you can imagine the algorithm.
edit
"as little support as possible" could be considered as easier to imagine "as little shaddow as possible" and the problem can be simplified to projection and ray tracing
similarly "bed only support" could be simplified to "cast shaddows but not on itself"