Problem statement

MXNet CI is running OOM [1] while building MXNet binaries for unix-cpu and unix-gpu stages. This is an intermittent failure and the work around is to re-trigger CI a few times. The issue is caused due to some of the numpy .cc files being too large causing gcc to use too much memory. The issue was not pronounced with gcc7, but with the recent update to use gcc8 [2] for CI builds, we have started to see this OOM error.

The fix is to refactor the numpy .cc files into smaller files so that the objects created during compilation don't use much memory. Here is the list of the largest objects (>10MB in size) generated currently on Mac CPU build:

 11M	./operator/numpy/linalg/np_norm_backward.cc.o
 11M	./operator/numpy/np_kron.cc.o
 11M	./operator/numpy/random/np_location_scale_op.cc.o
 12M	./operator/numpy/np_insert_op_slice.cc.o
 12M	./operator/numpy/np_insert_op_tensor.cc.o
 13M	./operator/numpy/np_elemwise_broadcast_op_extended_sec.cc.o
 13M	./operator/numpy/np_elemwise_unary_op_basic.cc.o
 13M	./operator/numpy/np_percentile_op.cc.o
 14M	./operator/numpy/np_matrix_op.cc.o
 14M	./operator/numpy/np_moments_op.cc.o
 14M	./operator/numpy/np_where_op.cc.o
 15M	./operator/numpy/np_einsum_op.cc.o
 16M	./operator/numpy/np_elemwise_broadcast_op_extended.cc.o
 21M	./operator/numpy/np_broadcast_reduce_op_value.cc.o
 22M	./operator/numpy/linalg/np_norm_forward.cc.o
 24M	./operator/numpy/np_elemwise_broadcast_op.cc.o
 34M	./operator/numpy/np_elemwise_broadcast_logic_op.cc.o

The corresponding cc files to above objects contains more than 210 operator registrations, and to refactor those into smaller files will need a considerable time and effort from the community. With 5 operators per day, that's more than 40 days of developers effort.

Proposed solutions

Option 1: We keep using gcc8 for CI builds and start working on refactoring these numpy .cc files. This would mean the community will have to face the CI failures for 40 days (could be less if more community members contribute).

Option 2: We go back to using gcc7 for CI builds, potentially solving the CI problem immediately, while we work on refactoring the numpy files. Reverting to gcc7 would take 2 days and then refactoring would take another 40 days.

I personally would prefer Option 2 for the reason that it saves contributors time in getting their PRs merged quickly, as well as saves on the CI resources. Would like to request community feedback on the same.

Going forward we also need to add a check to MXNet CI for build time memory usage. Any ideas for the same would be highly appreciated.

References

• [1] CI failure: https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-19588/12/pipeline/
• [2] CI builds gcc update commit: afc76b0
• Related issues out of memory during compilation on CI #19623 gcc8+ memory usage regression for compiling indexing_op.o #18501