There are a number of good books to explain the main concepts of statistics without getting bogged down in the technical details. As a grad student at a public university in the USA, I worked as a teaching assistant for the undergraduate course Introduction to Statistics. The course used SPSS software to enable the students to perform statistical analysis on data sets. Although we offered a good textbook (listed below), many of the students struggled with the main concepts, so my colleagues and I added a optional supplemental reading list to the syllabus. The following books are the ones we recommended to the students. I have read all these books, and they are pretty good at explaining statistics to anyone interested in learning the material at a conceptual level; anyone can read and understand these books, regardless of their educational background.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart.
Joseph Healey (2020) Statistics: A Tool for Social Research [This is the textbook used in the class, which I highly recommend].
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences.
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't.
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition.
Nassim Nicholas Taleb (2010) The Black Swan: The Impact of the Highly Improbable, second edition.
As a grad student at a public university in the USA, I worked as a teaching assistant for the sociology class "Introduction to Statistics." The course used SPSS software to enable the students to perform statistical analysis on data sets. Although we offered an excellent textbook (listed below), most of the undergraduate students struggled with the material, so my colleagues and I added a supplemental reading list to the syllabus for those students interested in studying the main concepts. The following books are the ones we recommended to the students. I have read all these texts, and they are pretty good at explaining statistics to anyone interested in learning the material at a conceptual level; anyone can read and understand these books, regardless of their educational background.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart
Joseph Healey (2020) Statistics: A Tool for Social Research
[This is the textbook used in the class, which I highly recommend]
Catherine Marsh and Jane Elliott (2008) Exploring Data: An Introduction to Data Analysis for Social Scientists, second edition
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition;
_______ (2010) The Black Swan: The Impact of the Highly Improbable, second edition
In addition to the book written by the author of this article, there is another fascinating study about the discovery of the Assyrian archives and the recovery of the Epic of Gilgamesh:
David Damrosch (2007) Buried Book: The Loss and Rediscovery of the Great Epic of Gilgamesh
> Few people doubt that this technology is going to change everything. What’s less clear is when the profits are going to start rolling in and how big they are going to be... "We are making trillions of dollars in investments on the back of tens of billions of dollars in revenues."
It seems possible that AI will eventually become an abundant commodity, with cheap alternatives available from China and elsewhere. And if that happens, profits will be competed away, just like any other commodity in a competitive market.
One would imagine that a Silicon Valley legend like Larry Ellison - who built his business empire on the basis of proprietary software products and services - would understand this better than anyone.
> One would imagine that a Silicon Valley legend like Larry Ellison - who built his business empire on the basis of proprietary software products and services - would understand this better than anyone.
When you are worth as much money as Larry Ellison is, it is nearly impossible to not believe that you've become a Master of the Universe for whom the usual rules do not apply because of your ability to bend reality to your whims.
Unfortunately that amount of money also tends to be close to self-perpetuating so they rarely feel the full brunt of comeuppance that otherwise should come from such hubris when they are wrong.
> you'd mostly care about exfiltration of data, so watching/actively managing exactly what network connections your computer/network can do, would give you upfront notification when it happens
If you have a list of good CLI utilities, you could run them in a bash script (e.g., network-monitor.sh), which would run in the background, and then redirect the output data to another file (e.g., network-monitor.txt). The key concept here is "baseline" -- you need to know what normal baseline network activity looks like, so that you can identify anomalous behavior. The way to establish a baseline is to gather a lot of data from the system.
The following are a few useful command line utilities to use for a host intrusion detection system (HIDS) using a simple network monitoring bash script. However, I am not sure exactly how to tweak the options. Also need to find a way to check for data exfiltration:
-list open ports and processes that own them: netstat -lnp
Then it would be relatively easy to write a python script with regex tools to parse the network-monitor.txt file, establish a baseline, analyze the data for patterns and search for anomalous behavior.
Besides network monitoring, there are other command line utilities you can use to check the system for possible intrusion, which you could run in a separate bash script as part of your Host Intrusion Detection System (e.g., hids-users.sh):
-show members of root group: cat /etc/group | grep root
-show users logged in: w #if you are the only user, you should not see more than one account logged in
-search for all accounts with UID of 0: grep :0: /etc/passwd #ideally there should be only one UID of 0 on the system, but attacker can create more.
-check that daemons who never log in have * or !, meaning no passwd: cat /etc/shadow
-look for orphaned files, possible sign of attacker temp account deleted: find / -nouser -print
-search for new user accounts that are not part of regular build: sort -nk3 -t: /etc/passwd #sort numerically third column (UID), colon delim (-t:)
EDITS: several small changes for clarity and also in response to comment below
> tail your network-monitor.txt file to watch for anomalies in the network connections and check for any strange outflows of data
Don't do that, you can't rely on "watch for anomalies" with your human eyes.
Either you setup something that notifies you after the fact, or you outright block all incoming/outgoing connections until you approve them. Mentioned elsewhere I think in the thread, I think both OpenSnitch, Little Snitch and PiHole can help you with all of these things.
But don't assume you can "watch for anomalies", automation and/or gated access is probably the way to go.
> you can't rely on "watch for anomalies" with your human eyes
Yes, I agree, that's a good call. I would not try to check for anomalies manually with meatware. I would parse the data with python regex tools to establish a baseline and search for anomalous patterns.
I edited my post to reflect the change you suggested.
According to an academic study, US law enforcement agencies at every level - federal, state and local - no longer need warrants in order to get access to most data on American citizens. It is easier for law enforcement to purchase data from private firms because there are fewer constitutional protections, reporting requirements and appellate checks on private sector surveillance and data collection, which enables police to circumvent privacy laws.
US law enforcement agencies get much of their information from data brokerage firms, which collect and aggregate information from public records and private sources, e.g., drivers licenses, mortgages, social media, retail loyalty card purchases, professional credentials, charities’ donor lists, bankruptcies, payday lenders, warranty registrations, and other sources who sell personal data. This information is then sold to customers willing to pay for it. Not even the FTC can find out exactly where the data brokers get their information because brokerages cite trade secrecy as an excuse not to divulge their sources.
Law enforcement agencies not only uses data collected by private firms, they also use corporate IT platforms and proprietary software applications (e.g., Palantir, Microsoft, Amazon) to store, share and analyze data. Palantir provides an interface that runs on top of other data systems, including legacy systems, making it possible to link data points across separate systems.
Source: Sarah Brayne (2020) Predict and Surveil: Data, Discretion, and the Future of Policing, Oxford University Press, pp. 24-25, 41-42.
Are you saying that Israel intentionally targets civilians? As a signatory to the Geneva Conventions, Israel is obligated to protect civilians and civilian populations from all dangers arising from military operations, and Israel is prohibited from direct attacks on civilians. The IDF must distinguish between civilian and military objectives and take all feasible precautions to avoid or minimize incidental loss of civilian life. Civilians lose protected status only if they take a direct part in hostilities.
As someone who has lost family and continues to. Have you not been paying attention to what has happened in Gaza? Conservatively.. Well over 20,000 children killed in Gaza alone. What would you call that?
I also recommend looking into an Ai system developed by the Israeli's called "Where's Daddy". Admittedly used during the the Gaza campaign by IOF themselves.
>Are you saying that Israel intentionally targets civilians?
Yes.
>As a signatory to the Geneva Conventions, Israel is obligated to protect civilians and civilian populations from all dangers arising from military operations, and Israel is prohibited from direct attacks on civilians.
They dont do it.
>The IDF must distinguish between civilian and military
All the great Western hegemonic powers have used public debt to finance the requirements and ambitions of the state. This was true of Spain in the sixteenth century, Holland in the seventeenth century, England at the height of its power during the eighteenth and nineteenth centuries, and the USA since World War II [1]. Public debt finance is a force multiplier that enables the government to undertake far greater civilian and military projects than it could otherwise achieve. Everyone knows this, including the people who are running the U.S. Treasury and the Federal Reserve.
As a percentage of GDP, government debt is high by historical standards, and it is rising at a fairly rapid pace. That could be a problem if it continues unabated. But there are two solutions: reduce spending and/or raise taxes. Most U.S. federal expenditure goes toward either defense, entitlements or interest on the debt, so it seems unlikely that spending will fall very much. That means higher taxes. At some point, the country will need to find a political reformer who can sell tax increases to the electorate. Until then, the crisis just continues to quietly grow.
[1] Giovanni Arrighi (1994) The Long Twentieth Century: Money, Power and the Origins of Our Times
Third solution is "taxation through inflation" in which the official inflation is 4%, the unofficial one is 10%, interest rates kept low, and over a few decades the debt gets nominally trimmed down against a nominally rising GDP. Real GDP growth will even accelerate this. Owners of assets that are not real-inflation protected (such as the debt owners) see their asset value go down ; people who have no net equity see their salary go up, but only in nominal value ; people who hold real-inflation protected assets that also distribute dividends are the winners in this.
"Austerity" is a red herring. I mean let's be serious, every option is austerity for someone...
Cutting spending, means those who would have received that debt spending take a loss (perhaps their job). Raising taxes means taxpayers pay. Raising productivity or inflating the debt means the young and laborers shoulder most of the burden. Financial repression means capital owners pay. Defaulting means bond holders pay.
This is what people who argue the debt doesn't matter don't understand. Sure it doesn't matter in the sense that we could always just roll the debt forward, and continue to make future generations pay. Just as it's possible to choose over and over again to wine and dine at someone else's expense, against their will.
No the debt always matters, because, it's a matter over who gets to benefit and who has to pay.
That framing makes it sound like high defense and welfare spending are laws of physics, impassable mountains you just have to navigate around. They aren't, they're very specific political choices that can be changed quickly and easily once the will is there. If you're going to pin your hope on a "political reformer" who can "sell" to the electorate, you'd do better to hope for one that can sell lower entitlements and defense spending.
In reality changing defence and welfare spending is pretty hard here in the UK. Most of the welfare budget is spent on pensioners, who vote in droves. And defence spending is non-negotiable and in fact needs to be greatly increased if we're not to get rolled over by Russia. There's no amount of will that can help.
I can understand being afraid of the Russians up until about 2025, but now that we've seen how they perform in a large war (alternative, now that we know that the state of the art in war has swing to favor the defending side) I don't get the fear.
The conventional wisdom is that the Cold War ended in 1991. George Friedman prefers to put the end of the Cold War in 2024 or 2025 when it became common knowledge that the Russians just didn't have much military strength.
Before Feb 2022 we didn't have any conclusive way to determine its strength, so a prudent military strategy would have been to assume that the Russian military was strong -- until 2024 or 2025, when a well-run country in Europe would have realized that it is safe to lower its defense spending (except Ukraine of course and to a lesser extent Moldova, Georgia and maybe the Baltic states and Finland).
I mean, the UK isn't in danger of invasion by Russia. Even Finland barely is. Ukraine is currently invaded and Russia seems to be a bit stuck in the mud.
The UKs threat from adversarys is nuclear, nearly nothing else.
Now the threats to the UKs interests abroad are larger and may necessitate a large military, of course.
>As a percentage of GDP, government debt is high by historical standards
I think this means: GDP === Debt not seen since WWII?
I mean true, that is high by historical standards, but also seems slightly not like the other times in past years when people said it's a bit high by historical standards.
Ian Ayres (2007) Super Crunchers: Why Thinking by Numbers is the New Way to Be Smart.
Joseph Healey (2020) Statistics: A Tool for Social Research [This is the textbook used in the class, which I highly recommend].
John Allen Paulos (1988) Innumeracy: Mathematical Illiteracy and Its Consequences.
Nate Silver (2012) The Signal and the Noise: Why So Many Predictions Fail, but Some Don't.
Nassim Nicholas Taleb (2005) Fooled by Randomness, second edition.
Nassim Nicholas Taleb (2010) The Black Swan: The Impact of the Highly Improbable, second edition.