I remember having two Jurassic Park velociraptor toys as a kid. One had the arms positioned as in the film but, oddly enough, the other got it right, as in the article.
I remember 2 being harder plastic and 1 being the more rubbery one. Though I could well be wrong. Sold all that stuff years ago except three or four of the bigger ones, which my kids play with.
Maybe, but it's intended to be actively grabbing with them, not at rest (so having them turned correctly is probably just an accident, not a correction of the earlier pose) so the curved-in hands might make sense—I didn't get the impression from the article that their wrists and "fingers" can't bend. The toy would move the arms in a grasping motion when, IIRC, you pulled the legs a certain way. Something like that.
I'm still not sure I understand the logic of that; the only rotation I see is in the supine in the 'correct' version. The 'wrong' version doesn't require any rotation at all, even using the illustrations. It's simply lifting the front limbs up, allowing the hands to rest.
If you want to bring it in front of you in the zombie like position most people use, you should have a notable rotation of both the elbow and the wrist.
This is true, but I notice similarly if I attempt a hands up position.
I should mention how excellent and detailed this article is. Anyone with a young child knows how fascinating they find this stuff and I have some corrections from this article to share with mine.
From experience working in this area, I believe there's a significant tradeoff between performance, flexibility, and time to delivery when it comes to consensus and the things it's used for, like database replication. It's like: "good, fast, or cheap, pick two".
As one of the core authors of the Apache Kudu Raft implementation <http://kudu.apache.org/> (which is written in C++) I know that we tried to design it to be a pretty standalone subsystem, but didn't try to actually provide a libraft per se. We wanted to reuse the Raft write-ahead log as the database write-ahead log (as a performance optimization) which is one reason that making the log API completely generic eluded us a little.
That said, I'm currently at Facebook helping to adapt that implementation for another database. We are trying to make it database agnostic, and we continue to find cases where we need some extra metadata from the storage engine, new kinds of callbacks, or hacks to deal with various cases that just work differently than the Kudu storage engine. It would likely take anybody several real world integrations to get the APIs right (I'm hopeful that we eventually will :)
Sure, here are a couple examples of things we've had to include in the log APIs:
1. Kudu uses a special "commit" record that goes into the log for crash consistency, related to storage engine buffer flushes. So we need an API to write those into the log. They don't have a term and an index, since they are a local-engine thing, so they have to be skipped when replicating data to other nodes in the case of the current node being the leader. If we were not sharing the log with the engine, we wouldn't need this.
2. Another database I'm working with requires file format information to be written at the top of every log segment, and it has to match the version of the log events following it. That info has to be communicated to the follower up-front even when the follower resumes replicating from the middle of the leader's log. So we need plugin callbacks on both sides to handle this, in terms of packing this as the leader and unpacking it as a follower into the wire protocol metadata.
Requirements like these will come up and either you hack around them by making some kind of out-of-band call (not ideal for multiple reasons) or you bake the capability into the plugin APIs and the communication protocol.
Frankly, designing generic APIs is also one of the less sexy aspects to consider because we spend so much of our time dreaming about and building all the cool distributed systems capabilities like leader elections, dynamic membership changes, flexible quorums, proxying/forwarding messages, rack/region awareness, etc etc etc. :)
The details of long-tail stuff like this is often hammered out as it comes up during implementation.
Since I mentioned performance, one other area that makes flexibility nontrivial is how you decide to serialize different types of messages on the wire and on disk. If you don't need extensibility, it's easy to keep things pretty efficient just by using e.g. gRPC and protobuf out of the box. If you want complete flexibility, the simplest thing to do is to give your plugin interfaces blobs to write into and you end up double-serializing everything.
FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.
Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability).
- Remote bootstrap (due to membership changes) also has undergone substantial changes given that YugaByte DB uses a customize/extended version of RocksDB as the storage engine and does a tighter coupling of Raft with RocksDB storage engine. (https://github.com/YugaByte/yugabyte-db/blob/master/docs/ext...)
- Dynamic Leader Balancing is also new-- it causes leadership to be proactively altered in a running system to ensure each node is the leader for a similar number of tablets.
I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern.
I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus.
We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.
I have been working full-time on Kudu since its early development. As others have mentioned, Arrow and Kudu are quite different. Despite the controversial-sounding title of Daniel Abadi's article, his content was actually reasonable and his conclusion in the final paragraph of the article is worth reading. In summary, he acknowledges that in-memory and on-disk columnar formats have different goals and both have their place (Arrow being an in-memory format).
Apache Kudu is much more than a file format - it is a columnar distributed storage engine. One way to think of Kudu is as mutable Parquet, but really it's a database backend that integrates with Impala and Spark for SQL, among other systems. It's fault tolerant, manages partitioning for you, secure, and much more. For a quick introduction to Kudu you can check out this short slide deck I put together over a year ago... it's a bit dated but a good overview: https://www.slideshare.net/MichaelPercy3/intro-to-apache-kud...
For more up-to-date information, follow the Apache Kudu Blog at http://kudu.apache.org/blog/ or follow the official Apache Kudu twitter account @ApacheKudu.
I think it's because Qt has traditionally had a pretty weird dual- or tri-licensing model: https://www.qt.io/faq/#_Toc453700684 which resulted in less adoption by developers because of the viral nature of the GPL.
IIRC, it used to be that even the core Qt libs were GPL (unless you paid for a commercial license), while now most (but still not all) libs are also available under the LGPL.
Fred, really interesting article! Thanks for posting it.
For those in the Bay Area, if you'd like to hear more about Kudu, I'm giving a talk about it in Palo Alto tomorrow (Wed) at the Big Data Application Meetup at 6PM: http://www.meetup.com/BigDataApps/events/227191025/
Also presenting are Julien Le Dem of Dremio on Apache Drill and James Taylor of Salesforce on Apache Phoenix. Should be a really interesting evening!
(Don't mean to hijack the thread, just thought this might be relevant to some folks. :)
Hrm. I'm not looking, but I like reading these things. Regarding "Objective, seriously? It's not 1992 anymore", I've always put an objective on my resume. Typically something short and sweet, like "seeking a senior software engineering position solving challenging systems problems" or whatever. Is that not useful? I wouldn't want someone to read beyond the first couple lines of my resume if they're looking for a web developer or a finance manager.
Fair enough, but here's another example. Say someone is trying to move from IC to management. They might have no mgmt experience so they put "seeking engineering management position" as their objective. Is there a better way? The other traditional thing to do would be to explain this in your cover letter.
Edit: I get that you should be shopping it around and applying to various positions but it seems inevitable that a copy of your resume will get filed into countless HR databases. It seems rational to me to indicate what you're looking for on the resume.
