Sim-compile performance

[timsherwood]

I was playing around with different optimizations of sim-compile... we know that compile time is a big problem as the design scales, and so I was looking into auto-extraction of procedures but a VERY funny thing is happening. In the past I guess I just assumed that -O0 would be the fasted compile time (I mean it make sense right? no optimizations?) but here is what I foud.

At -O0, function extraction helps compile time (482s → 178-190s) but bloats .so and hurts sim time due to unoptimized function call overhead (boo!) so then I tried it with -O1.

Just switching to -O1 takes vanilla baseline compile time from 482s to 4.3s — a 112x speedup in compile time! The .so drops from 807K to 145K and sim is also 12x faster. GCC's own optimizer at -O1 does everything our extraction does and more.

At -O1, function extraction seems to actively hurt — compile goes from 4.3s to 20-23s (the extra function definitions give GCC more to analyze), while .so and sim are essentially unchanged.

I have not tested this rigorously, but for 100x compile time speed up for a 1 byte change to the source code seems like a tradeoff worth investigating :)

[Jbalkind]

Have you tried -Os? It helped a lot for Metro-MPI

[timsherwood]

  Flag     Compile    .so KB    Sim                                                                                                          
  ----     -------    ------    -------                                                                                                      
  -O0      481.7s     807K      0.0084s                                                                                                      
  -Og        4.8s     145K      0.0007s                                                                                                      
  -O1        4.3s     145K      0.0007s
  -O3        8.7s     145K      0.0007s                                                                                                      
  -O2        9.6s     145K      0.0007s                                                                                                      
  -Os       10.0s     145K      0.0007s

[Jbalkind]

What we saw was that once the design got large enough, -Os would win because it would reduce the size of the graph thus the amount of frontend stalls due to icache and itlb misses. Could be interesting if there's a heuristic tradeoff in terms of design size at some point