If the community demands the ability to review code and data then the usage of closed data sets will simply decrease until they comply. We should not base our standards of research on the ease of complying. We should do what we can to produce the most accurate results.

The problem is not one of convenience, the problem is that you are depriving yourself of huge amounts of valuable (closed) data, and this may cause large gaps in what is researchable.

This was the point. You have to pay the price for the principles and hope that the tide turns one day.

If you take your suggested easy route then this day may never come.

Are you a researcher and following those principles? Because it is very easy to demand other people pay the price for what you believe are the right principles.

The reality of the way research funding and academia is set up (at least in Europe) is that the price you pay is likely your career. You can go through all the work and effort to follow those principles, and all that will happen is that the politicians will give the research funding and tenure positions to competitors that published more and told more exciting stories.

Academia and research will necessarily reflect the incentives put in place by the people with the money (i.e., politicians). And right now they are giving money for exciting stories and publication counts. That is where the change must start.

I did not demand anything. I just explained that when everybody complies, nothing will probably change.

My point is that there are large fields where we may not get access to open source data in the foreseeable future; in the meantime, we are depriving ourselves of potentially useful or valuable knowledge. Imagine if you could reduce the number of traffic deaths by analyzing a closed data set from Lyft or Uber; is this worth doing, or should we just tell the PhD candidate not to bother?

The problem is that a lot of studies don't suffer from this problem, and their data/code isn't open either. What we're looking at is a default-closed mindset, which needs to be inverted ASAP if these people want us to trust them in any significant capacity.

And such data sets should be trusted...why?

Not directly relevant to the more widespread issue of how people should provide and react to criticism of scienctific work.

Then it should be required whenever and to the extent it is possible. And where it is not possible, we should be considerably more skeptical of claims made, because trusting a claim based on a proprietary dataset requires a hefty dollop of faith.

I agree, its like saying "I have proof I just cant show you".

Then the provider of the subscription must be able to vouch for it or that data should not be used. The provider should also be .... sensitive to the need for that data to be used well, for any conclusion drawn from it to be reproducible.

Presumably however other researchers could subscribe to the same source... third-party proprietary datasets, if non-partisan wrt research results, are still better than first-party proprietary datasets, right?