16 bit FP operations seem to be more than enough for most networks - in fact for inference, google claims in their TPU paper that most network parameters can be quantized down to 8 integer bits!

There is also some work on pure binary networks, xnornet and binarynet. Low precision doesn't seem to effect these networks as long as they are trained with binary weights in the first place.