Hi, glad to see you here. We were using spark for logistic regression training but switch to MPI now, because of the 2GB problem. I think LR was spark's killer feature, would you make it better? Thanks.
You'll hit this bug when your model size is larger than 2GB.
BTW, Recomputation of RDDs may result in duplicated accumulator updates. So please do not use accumulator in your trainer for gradient summation. They know that, but they said they will not fix that.
Would you mind shooting me an email rxin at databricks.com so I can understand more the issues you run into?