> A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that?

Slava at RethinkDB here. Yes, we are.

Good answer! I want to see the results.

> A negative report could generate the opposite though. Are they ready to bet the new $12M+ on that?

I disagree; it all depends on how they act upon it.

So worst case scenario he finds something deeply flawed. What does RethinkDB do? Work to fix it while communicating and being responsive to the development community. Show developers that you're humble and, more importantly, you listen and fix things. That would generate a good amount of good will, in my opinion.

Exactly! And if some company is reluctant to do this in the open, they could use him for QA. That is, pay him to do a preliminary test, then fix things, run the tests again, fix again,... until there are no more critical issues. Of course the resulting blog post should then outline the whole process.

Everyone makes mistakes - what matters is what you do to discover and fix them.

Good point. For example I remember etcd and Consul devs responded and fixed a few issues, after the test came out.

Up to this point, nothing in Aphyr's history has suggested he'd be a biased tester. When he tested Riak, he was invited to Ricon to give the results for the world to see live. Much of the Basho engineers at the time were personal friends of his. As you know if you read that analysis, Aphyr held back nothing.

Right now I'd trust Aphyr more than any other analyst, consultant, or full-time employee (with employment to lose) to give truly impartial judgement.

No offense meant to Aphyr, whose work is awesome, but conflicts of interest are incredibly corrosive. The sales cycle for consulting services requires a "gotta keep the customer happy" attitude. That's hard to square with truly impartial judgment. It's not that you give in to the dark side; once you take somebody's money you're no longer impartial, so you end up having to try to simulate it.

It's even harder to square with the appearance of impartial judgment. What if a product is honestly great? If we decide that he's being fair because he says both positive and negative things, he has an incentive to create the appearance of fairness rather than actual fairness.

There's a reason that systems where truth is important (e.g., legal proceedings, medicine, science) have complicated rules around conflicts of interest. I hope he finds a way to find funding that doesn't put either his judgment or his reputation at risk. Consider, e.g., Consumer Reports: they get money from their subscribers, not advertisers or manufacturers. Maybe he could let database users fund all his public Jepsen work.

And if he deviated from his format of brutal honesty we would know something is up and his brand would be tarnished. Getting a B on Jepsen and fixing it is probably the best that any vendor can hope for at this time.

It is about perception. I think he's unbiased, you do as well it seems. But to others it might look biased. That was my point.

Oh yeah, I'm with you completely. I trust that he will stay impartial, and I hope that vendors allow him the freedom to write as he has in the past so that all appearances reflect what I believe to be is the reality.