Hacker Newsnew | past | comments | ask | show | jobs | submit | imaginaryunit's commentslogin

"In academia, the end product is a publication and your code needs to work only once."

In my experience as ex-academic, the above is all too true in too many computational fields (and many sub-fields of CS). So I ask what scientific value (or any kind of value, for that matter) is being generated by grinding out publications with results that won't be repeatable?


Depends on the journal you're publishing in.

Lots of journals (at least in my biological sciences field) require you to submit your code along with your publication so that a) the editor can repeat your results to accept your publication and b) for other scientists to repeat your stuff. Biostatistics AFAIK has an editor responsible solely for reproduction of results.

Personally saying lots of my code is repeatedly re-used by colleagues who can't program, for example scripts that parse BLAST-output and get the best result for each query etc.


Your point is well-taken and (thankfully) there are safeguards in place at top journals. However, Rob Tibshirani examined this a while back with a case study and came up with some unsettling conclusions about reproducibility: http://www-stat.stanford.edu/~tibs/FL/report/. The case study highlights the importance of debugging not only code, but also the entire experimental process.


True!

If you want a newer example, here's a very recent paper which looks at the results of a couple of RNA-Seq-papers and after being able to reproduce only 12% of the results, concludes that they all used the wrong assumptions in their statistics: http://www.plosgenetics.org/article/info%3Adoi%2F10.1371%2Fj...

Which shows that the editor really needs to know what's going on and why in each paper - I don't think every editor can know everything about his or her field.


We've found that Bump often requires several re-bumps, whereas QR codes can be reliably scanned within a few seconds. Also, we enable referrals of contacts (and real-time notifications thereof) with a single touch & click gesture.


Wow. That's really good. Impressive. I'll try this!


My 2c as someone who has a similar background to yours (did PhD, post-doc, now an entrepreneur): a startup is in most ways more fulfilling than an academic environment. I think startups force you to be good at many things whereas in an academic environment, you're rewarded more for being the expert in a single narrowly defined subject. I do have to say though, given the way fields are getting narrower and narrower and the way the academe has devolved, I see little real connection between stuff being shown in academic research papers and the technology that we'll be hyping in 2032.


"given the way fields are getting narrower and narrower and the way the academe has devolved, I see little real connection between stuff being shown in academic research papers and the technology that we'll be hyping in 2032."

I think that's a myth. The fields I studied aren't any narrower today than they were two decades ago -- the fields I studied didn't exist two decades ago. They barely existed ten years ago. And now there's a vibrant field of research. The same is true all over the biological sciences, computer science, chemistry, biochemistry, bioengineering, etc. The fields where a PhD means something are constantly re-inventing themselves, because the frontier of technology keeps getting bigger.

In the 1980s, Rodney Brooks at MIT was doing some interesting academic work with simple, toy robots, while everyone else was building thinking machines and planning systems and industrial robots. Twenty years later, and there are multiple companies commercializing the technologies emerged from that work. Some people might look at iRobot and say "what an innovative startup!", but I look at them and see a company that has done a good job of commercializing 20-year-old technology. They've definitely refined it since then, but the core technology was invented at MIT, in the 80s. The same applies to nearly every "hot" technology that startups are pimping today -- dig a bit below the surface, and you find an iceberg of academic research.

And that's what I mean when I say that the startups of 2032 will be hyping academic research done in 2012. It's all a matter of perspective.


What I get from pg's comment is how the research experience has become less fun across the board in the last few decades (I'm focusing on post-graduate research, but a case could be made for undergrad research too). There are many reasons why this has happened, but the major outcomes are that:

-Publication pressure makes it such that research progress has become significantly harder to gauge (e.g.: it's become uncommon for results to get reproduced) across many fields (statistics, machine learning, the health sciences, which I've done work in for over 7 years at a highly respected institution)

-Balkanization of research fields where problems become increasingly exotic and narrowly focused to the point of being contrived (all the better for publishing papers of course)

-Increased difficulties in securing research funding or employment (more temp jobs though);

-Arbitrary metrics often being used to judge one's work. Example: if you do theoretical work, reviewers can reject your paper for lack of "applications"; if you do applied work, they can reject for "lack of theoretical depth" (what either of these terms mean in 90% of cases is arbitraily defined by a single reviewer who just wanted to torpedo your work).

-In some fields, entire mafia groups have formed that rigidly control the review process for that field, which means good luck to you if your work competes with theirs and you're submitting a paper/applying for grants

-Lack of accountability as to how research dollars are spent (vis-a-vis the public's interests)

-...

BTW, I say the above as someone who did a lot of enjoyable open-ended research as a grad student.


Minor point that has always bugged me: why doesn't Cocoa have native implementations of tree data structures?



We recently developed a way to get rid of paper business cards and their shortcomings (e.g.: clutter). It's called Napkkin, and it lets you create a business card on your phone and swap it with people you meet by scanning a QR Code. You can get referred by your new contacts and get notified in real-time so you can track who's helping you with word-of-mouth referrals.

We've got mobile/desktop web apps and Android/iPhone native apps set up. We're looking for feedback, so let us know what you think!


Feynman's view on CS is similar by Hal Abelson's take on the term (see http://www.youtube.com/watch?v=zQLUPjefuWA). Abelson's point about CS being more like magic has always stuck with me.


I really loved the portrayal of programmers as sorcerers and programs as spells in SICP. It's a very apt metaphor.


Though the gov't might want research to be distributed, the underlying problem here has more to do with the scientists themselves than anything else. We've had ArXiV around for a long time and yet I ask who's adopted it other than mathematicians and (many, but not all) physicists? Why hasn't it been adopted by NIH-funded life scientists?

The fact that years ago, life scientists easily could have adopted an ArXiV-like model for publishing, and chose not to, is quite telling. It suggests a far deeper problem with incentives in (general) academic culture to publish and that article availability is not going to affect that at all. As someone who spent >6 years Ph.D./PostDoc (bioinformatics, stats and CS), I can say that the vast majority of researchers have no genuine incentive to take action. Protesting against Elsevier online in the comfort of your office is one thing, but having to publish X>10 papers/yr to get tenure/brownie points within your department is another.


You're absolutely right that academics are torn between two things that they want:

1. open access for their research (both for altruistic reasons, because they want to spread knowledge as widely as possible, and for selfish reasons, because greater access to their research means more citations of their research, which helps them with career advancement)

2. publishing in the most prestigious journals they can, because that helps with career advancement.

The NIH, which funds all research into biology and medicine in the US, just cares about 1. above. They just care about maximizing distribution for taxpayer-funded research, as long as the distribution is done so in an economically sustainable way.

The NIH drove the open access mandate in 2008. My sense is that in the short term, driven mainly by the demands of funding agencies, all journals will move to 'open access models', where publishers pay to publish their research in the journals, and for readers, it's free to access. This satisfies both 1. and 2. above: academics get both open access and the validation of being published in prestigious journals.

Longer term, I think alternative credit metrics will replace the credit metric that's governed research for hundreds of years, i.e. the brand value of the journal you were published in.

I think when that happens, the power of journals will disappear altogether. Right now, the journals' role in driving the discovery of research has been disrupted a lot. In the pre-web days, people used to walk down to their libraries, and check out what had been published in the journals recently. In those days, the journals used to drive a lot of research discovery.

Nowadays, pretty much all research discovery happens online, and it's driven by things like search engines (Google, Google Scholar, Pubmed), and social platforms (Twitter, arXiv, Academia.edu; general communication technologies like email and IM are also used a lot).

The journals still have a strangle-hold on the validation of research. Which journal you were published in is still incredibly important, as far as the validation of your paper is concerned.

Other credit metrics are emerging, however. Citation counts are one metric, and that has a big role in driving research discovery on search engines like Google Scholar. Hiring committees are starting to look at the kinds of credit metrics that apply to general web content, e.g. page views.

As alternative credit metrics emerge, there will come a point when they do a good enough job at the validation of research that academics feel they no longer need to submit their papers to journals.

I wrote up some further thoughts on this here http://techcrunch.com/2012/02/05/the-future-of-peer-review/


If pre-publication peer review goes out of fashion, then another possibility if that the brand of major journals becomes more important. We still need a quick gauge of the quality of an article, other than its Google rank or number of page views. Nature can retract popular articles that are later proven flawed; I don't think Google would attempt to wield that kind of authority.

Relevant example: You published these two posts in TechCrunch to get a wide audience. (And I'm glad you did!) I read them partly because they appeared in TechCrunch.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: