Hi folks, I’m Damien, a scientist at Tumult Labs =)
We’re a company focused on differential privacy, a technology that allows organizations like the U.S. Census Bureau, the IRS, or the Wikimedia Foundation to safely publish statistics from very sensitive data.
To support these use cases, we’ve built Tumult Analytics, the best-in-class framework for differential privacy. And today, we’re releasing it as open-source software
You can find more information about differential privacy in the linked blog post, and you can also directly dive into the software documentation: https://docs.tmlt.dev/analytics/latest/ =)
If you have any questions, don't hesitate to post them below! We'll be monitoring this thread today and we're happy to get feedback & answer questions.
This looks exciting, thank you for sharing! Implementing differential privacy correctly is famously difficult and easy to get wrong. Will you be making the privacy-critical part of your code available publicly for people to audit?
Hi, I'm one of the authors of the scientific paper¹ linked in this blog post. Incidentally, I wrote a series of blog posts explaining differential privacy in layman's terms. The first post might be "not-technical-enough" for HackerNews, but maybe the next in the series make up for it. Feedback welcome =)
Thank you. I'll try to digest these. I barely understand the math involved. So I'm pretty sure my "intuition" is wide of the market. I'm very grateful for your efforts to explain, socialize your work.
When I studied cryptographic voting systems, my "aha" moment was realizing the magic sauce is creating hash collisions so that a secure one-way hash can be used to protect voter privacy.
Re-re-reading the differential privacy stuff, this para jumped out:
"The intuition for the 2006 definition of ε-differential privacy is that a person's privacy cannot be compromised by a statistical release if their data are not in the database. Therefore with differential privacy, the goal is to give each individual roughly the same privacy that would result from having their data removed."
Oh. The "differential" part means modeling the difference between data captured and not captured.
I think (hope) this means figuring how much to fuzz the capture data so that hash collisions will match real world fuzzing.
--
Again, I'll continue to try to grok this stuff. Real world (story book style) examples will be very helpful.
Until I do understand, I think it's crucial for crypto and privacy minded people to quantify the assumptions and context involved. When I was working on election integrity (and medical records & guarding patient privacy), all the discussions were just make believe. I did help author a govt report to meant to help quantify the attack surface area for election administration. But I don't think it did much good, nor was it replicable (to new contexts).
After reading all your links, I'm still not sure why or where Differential Privacy is needed.
1) How could aggregated data (means, average, min max) be used by attackers? Aren't aggregated data already private? For example, the Google postgres extension returns aggregated data, why is DP required here?
2) In the case of sharing entire databases, if all the PII are removed, why does it matter that we can match two records from two databases? Yes we can do correlation between 2 databases, but if PII were not gathered and stored at all in any database, there would be no privacy issue in the first place.
1) Note that the "min/max" example trivially leaks individual information: for example, releasing the max salary of employees of a company leaks the salary of the CEO. More generally, there have been numerous attacks on privacy notions purely based on aggregate data. One of my favorite is this one: https://blog.acolyer.org/2017/05/15/trajectory-recovery-from...
2) Typically, PII is not the only thing that can be used to reidentify someone, and matching records from different databases can sometimes infer sensitive information about people. One example: https://www.cs.cornell.edu/~shmat/netflix-faq.html
I'm still not convinced, but I guess I'm lacking critical technical background to grasp it.
1) The CEO example isn't really a good one to me, given the wealth inequalities in the world, leaking CEO's salary is almost desirable... I tried to read the blogpost and paper about mobile data location. At one point they talk about aggregated data, but then in the paper : "This dataset
is collected by a major mobile network operator in China.
It is a large-scale dataset including 100,000 mobile users
with the duration of one week, between April 1st and 7th,
2016. It records the spatiotemporal information of mobile
subscribers when they access cellular network (i.e., making
phone calls, sending texts, or consuming data plan). It also
contains anonymous user identification, accessed base sta-
tions and timestamp of each access.". So... the data is not really "aggregated"? The dataset literally lists some user IDs.
2) If I'm fired because my boss didn't like my history of movie, then it can probably be defended in court, depending on the country. I could also find another boss who has a natural sense of ethic and who doesn't judge me for what I watch.
Thank you for the links anyway. I will look at them again in a few day to see if I missed something
Haven't read the paper yet, but have read the blog posts (which are awesome, BTW!).
I'm wondering if you have any thoughts on Frank McSherry's old blog post expressing his distrust for approximate-DP [1]. He seems to have different intuitions than your "almost DP" post expresses and makes criticisms that aren't quite addressed in your post.
Based on a skim, it seems that this requires individuals (or whatever class of entity we don't want to leak info about) be expressly identified in each record of the queried dataset, eg with a uid column. So for a dataset that one wanted to use this method with that doesn't identify individuals (eg in the web log case, not logged in visitors, or simply not recording logged in user), one would have to heuristically assign a (potentially synthetic) identity for each record first. Is that right?
^ commenter is a security researcher who works for Google.
This whole area of research seems like it exists as a way to rationalize wide-scale data collection. Rather than focusing on the collective rights of all people being tracked, it focuses on the risk that any individual person faces from an attacker.
Apple uses differential privacy and evangelizes its use. DP is not either-or with consent.
Let's say your mapping app says 'Do you want to contribute traffic information during your drive to help provide better navigation experience for everyone' If you click "yes" and opt-in, do you, or do you not, want this to use Differential Privacy?
Excellent point! That's one of the reasons why people understood that syntactic definitions (which try and tell you "how anonymous data is" by looking at the data) were not the right approach, and that you had to look at the mechanism to determine whether the process was sufficiently private.
This line of thinking led to the definition of differential privacy, which completely changed the perspective. I wrote two articles about it, one very simple [1] and the other a bit more detailed [2].
It's doable to compare syntactic definitions (k-anonymity vs. k-map vs. δ-presence, for example) and I tried to do that a bit. Differential privacy is something pretty much fundamentally different, as it applies to the mechanism and not the output dataset.
I'm trying to fix this gap: my PhD thesis is about rephrasing syntactic definitions in terms of relaxed versions of differential privacy. Typically, I'm trying to get to a point where I can say "hey everyone, remember this old definition that is easy to use but we don't really know what guarantee it gave? Turns out it's differential privacy with weakened assumptions, so here, have a formal guarantee for free". These are (I feel) natural questions that are still open problems. I will most definitely write about them when I have time (and solid results) =)
I also run a relay, and I don't have much contact with the Tor community. I'm not sure that most of the relay operators contribute to the IRC channels or other medias used by the community. There are ~5000 Tor nodes, and I just saw ~450 nicks on #tor - and many of them are probably users and not operators.
We’re a company focused on differential privacy, a technology that allows organizations like the U.S. Census Bureau, the IRS, or the Wikimedia Foundation to safely publish statistics from very sensitive data.
To support these use cases, we’ve built Tumult Analytics, the best-in-class framework for differential privacy. And today, we’re releasing it as open-source software
You can find more information about differential privacy in the linked blog post, and you can also directly dive into the software documentation: https://docs.tmlt.dev/analytics/latest/ =)
If you have any questions, don't hesitate to post them below! We'll be monitoring this thread today and we're happy to get feedback & answer questions.