Starting from ZNC 0.062, you can use the --disable-optimization to configure. This will disable "-O2" and instead use "-O0". GCC should need less memory now.
If you still can't compile ZNC, try ./configure --disable-optimization CXXFLAGS="--param ggc-min-expand=2 --param ggc-min-heapsize=20000" but be prepared for looong wait.
documented at http://wiki.znc.in/FAQ#The_compiler_dies_saying_it_has_not_enough_memory.21