My personal opinion is to leave things as they are until you hear from Google. Simply be aware of the possible interpretations of the ToS. I do believe Argorak has the correct sentiment: that it's ultimately up to the individual font designer/cretor.
Your temporary caching, to me, seems fine. I read the ToS as saying that permitting your end user to obtain permanent copies might be disallowed ... at the discretion of the font designers. To be absolutely certain, you should check each font's license. Considering the fonts aren't Google's to license, I doubt you'll hear from them.
Interestingly, Google's own Fonts API which I'm using with my service, doesn't provide any information about the underlying license of each font (I previously wanted to include that info in the UI as well).
Your temporary caching, to me, seems fine. I read the ToS as saying that permitting your end user to obtain permanent copies might be disallowed ... at the discretion of the font designers. To be absolutely certain, you should check each font's license. Considering the fonts aren't Google's to license, I doubt you'll hear from them.