This has an excellent chapter of the use of ATS software the majority of recruiters use these days.

https://en.wikipedia.org/wiki/Applicant_tracking_system

- "Immersion" by Abbie Gascho Landis

- "Welcome to the Goddamn Ice Cube" by Blair Braverman

- "Sleeping Naked is Green" by Vanessa Farquharson

- "There's No Such Thing as Bad Weather" by Linda Akeson McGurk

https://www.amazon.com/Weapons-Math-Destruction-Increases-In...

It would behoove us all to think about the consequences of the algorithms we're working on - much like this article.

You can listen to an interview she does on econtalk -- interesting to learn more about the hidden biases.

http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

Very accessible set of case studies where algorithmic management of traditionally human-run bureaucratic processes is biased and destructive.

One of the absolute WORST books I have ever read.

https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction

For those of you who like listening more than reading, she was on 99% Invisible talking about destructive algorithms and her book a few weeks ago: http://99percentinvisible.org/episode/the-age-of-the-algorit...

"Weapons of Math Destruction" presents good evidence that 90% is much lower than the actual effectiveness of X.

[1]: https://weaponsofmathdestructionbook.com/
[2]: http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

I am a software product manager with background in programming. I am creating #BooksByWomen at thebooksbywomen.com, curating a list of books written by women, especially in the tech/business category as a side project. You could head out to https://www.producthunt.com/makers-festival-2019/voting#tiny... and vote if you like it.

Made with:
Sheet2site.com.com
The love and help from the communities at WomenMake, Women in Product, Tech Ladies and a lot of my friends.

I read Cathy O’Neils book on algorithms: Weapons of Math Destruction, I absolutely loved it. It resonated with me on a lot of insights I was seeing myself in my career. So I paid attention to women authors in my network and there are so many that never would have reached me through traditional marketing and social media if I hadn’t sought them out specifically. This is why I am curating and maintaining this list and want it to succeed.

What I would ideally want to do:
* Create a page to honor my supporters
* Publish regular bi-weekly interview with women authors about their journey
* Publish featured books based on common themes, coming up
* Create a browser extension where this content can be shared effectively
I do hold a fulltime job, so carving out time would be a challenge, any support helps.
You can also support me most directly by visiting the site and buying some books from awesome female authors, I make a small affiliate commission from Amazon.

You can also support me by buying me coffee or ramen on Patreon: https://www.patreon.com/kalinaz

"Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy"

https://www.amazon.com/Weapons-Math-Destruction-Increases-In...

  Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Cathy O'Neil)

There's some good entry level books on the topic, like "Weapons of Math Destruction" that you might enjoy.

https://www.amazon.co.uk/dp/0141985410

People who work for Facebook must have to work very hard to remember the people they are experimenting on are people, not just "users", and that they don't have cart-blanch with how they treat them. Especially if they are building addiction into their service!

Also, Weapons of Math Destruction is a fantastic book.

For me, this is the key motivating point - the horse may have left the barn by the time we act. A lot of times people say this is exaggeration but "Weapons of Math Destruction" is a nice read on unintended side effects of this phenomena [0].

[0] https://www.amazon.com/Weapons-Math-Destruction-Increases-In...

- Borges' Aleph, and his essays (for example Other Inquisitions). He was fascinated about concepts programming is famous for now, he would have loved recursion

- Mann's The Wizard And The Prophet. He uses the life of two highly influential scientists (Borlaug and Vogt) as exemplars for two ways of viewing progress, one highly focused on technological progress, one being sceptical of technological progress

- Wachter-Boettcher's Technically Wrong, about implicit biases in modern technology and how that excludes or mistreats people. Some good lessons if you want to start a company and get as many customers as possible ;) Similar book with more academic stringency: Weapons Of Maths Destruction

- Roman's Writing That Works, non-fiction/memo/email advice from big advertising guys, lots of good advice on getting your point across

- Fromm's The Art Of Loving & The Sane Society, two non-fiction books from a sociologist/psychologist on how to work on your relationships (i.e., love as a movie concept doesn't exist, it's mostly very hard work and self-critique), and how society as a whole has very broken goals. Becker's The Denial Of Death (on how fear of death is a major drive in life) comes from a similar place

- Wilson's How To Teach Programming - delves deeply into the psychology of learning and how to build stable communities, it's available for free so why not

- Statistics books are always good to mend your thinking! My favorite layman's introduction is Motulsky's Intuitive Biostatistics (no formulas, plain English), for a non-practitioner Wheelan's Naked Statistics is probably better

- Seth Godin wrote lots of good business books, pick one (I liked Linchpin)

- If you haven't read them yet, get the 'classics' of software management: The Mythical Man-Month, and Facts and Fallacies of Software Engineering

- Cuckoo's Egg is a marvellously fun non-fiction book about a programmer tracking a hacker in the 80s, one of the first international computer crime cases, featuring tons of fun low-tech hacks

If that topic interests you, I can recommend Weapons of Math Destruction by Cathy O'Neil, and Algorithms of Oppression by Safiya Noble.

I'm a programmer myself, so I understand if people are initially defensive about these topics. But it's important to me that I never write software that's harmful to people, so I try to check my biases as I create software to avoid these more insidious issues. It's easier to spot issues that may arise due to programming machinery that could crush a person, for example. Or software designed to estimate fuel usage for an airplane. It's much more difficult to spot inequity of minorities in software.

There's plenty of examples out there of biased AI in response to biased training sets.

If you're more interested in the dystopia that must be created in order to "win" SimCity, here's a couple of interesting articles. (I think they've been shared here before.) By taking the game to its absolute extremes, it becomes clearer what the simulation considers most important and valuable, thereby laying bare some of the biases inherent in the game's design.

https://rumorsontheinternets.org/2010/10/14/magnasanti-the-l...

https://www.vice.com/en_us/article/4w4kg3/the-totalitarian-b...

The problem is that the vast majority of voters are swayed by emotion, not fact.

These sorts of things are only going to get worse, because it is the only way to get people to vote your way. Even if you have a good idea and a sound plan, you need to dress it up in emotion-laced slop to get people to come out and vote for you.

It also doesn't help that the mainstream media, which is a very powerful force in the US when it comes to politics, is biased toward the Democratic party (as seen in the recent Wikileaks emails from the DNC). It means that to counteract this, you need to try another tactic, like posting on the Internet.

Even our leaders are swayed by emotion. Both Obama and Hillary have commented prematurely on important events ('Clockboy' and various police shootings) without having professionals and science weigh in on the actual facts of the events after a real investigation.

This is one of the main problems with our society today: anti-science winning out over facts and assuming someone is guilty before even attempting to see if they are innocent.

There was even a book on the New York best sellers list called 'Weapons of Math Destruction' claiming that math and statistics are somehow 'racist'. Think about that for a minute to let it sink in....Facts are now racist.

Social media has made it worse because instead of just having the mainstream media feed us hyperbole and rhetoric, anybody with a Twitter account can do it too.

It has now had some real-world consequences and resulted in many people getting hurt and even getting killed in riots over half-truths, hearsay, and rumors.

If you want shit posting to stop, we have to live in a society where it has to stop working so well. Maybe even holding people responsible for posting lies that lead to riots or death.

Edit: sigh. I always try to have intellectual conversations here on HN and am always disappointed. Most people here seem to just want to hear the current San Francisco narrative about the world and live only within that bubble. It's actually really sad.

[1] https://www.amazon.com/Weapons-Math-Destruction-Increases-In...

[2] http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction

[1] https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction

The book also gives copious concrete examples of these dangers. In particular, the book describes the LSI-R (Level of Service Inventory-Revised) questionnaire and how it effectively pinpoints race even though it does not actually ask for the person's race, which is illegal.

https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction

(See Cathy O'Neil's "Weapons of Math Destruction", where she blames US News rankings specifically for the bloat of universities and the overuse of adjuncts.)

It's a question of if the outcomes are distributed differently among races because the races represent truly different probabilities, or because the inputs to the model were already tainted against some race. For example, let's say you judge creditworthiness based on membership in a particular honors society, but that honors society doesn't exist at historically black colleges.

That obviously makes a ton of sense from a business standpoint. You want to contain losses for risky borrowers but compete with other lenders for low risk borrowers.

But socially, this is perverse. People tend to be risky because they are already poor. So now money costs more for those who have the least of it. This is one of the feedback loops that makes poverty (and affluence, for that matter) so sticky.

I had this realization in my personal experience when I was able to refinance almost $100k in student loans at a crazy low interest rate. My household's finances are in great shape as my wife and I enter our prime earning years. But for us, such an opportunity is a gift, on top of an already sweet situation. The savings could be a game changer for a family whose finances are more marginal.

Something I learned from the book Weapons of Math Destruction: there's an entire industry around skimming contact information from poor people to target for certain products. People will spend heavily on AdWords advertising for terms like "apply for food stamps," and have a web page that offers the service of expediting your food stamp application for free. Of course, this website is just a form that takes your info and forwards it to the regular online food stamp application. Maybe it's got better UX, but it's not expediting anything.

As part of handling the application, the website operator can save the name, address, and phone number of someone who is highly likely to be poor and desperate. They sell this information to certain companies, predominantly diploma mills and payday loan companies, who will then call these people directly to aggressively market their services.

I don't have the book at my fingertips, but if my memory is right, the contact details for one qualified lead sells for like $75.

https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction

https://weaponsofmathdestructionbook.com/

I don't have my notes with me and I'm only 1/3 through, but the main theme is that the best predictive algorithms:

* work transparently for all parties (the creators, users, and "inputs", often people).

* Have no feedback loop (The use of data from the model should not further entrench the output of the model).

And a few others. It gets into discrimination and other major flaws of data modeling re: recidivism, school admissions, stock trading, and other things.

Not all algos are racist - but there are definite attributes to avoid, and this book (or a more rigorous version) should be mandatory reading for all "data scientists".

+1000. That book should be required reading for anyone working in machine learning. Written by a former Wall Street quant who has the math down cold.

What she knows about rampant bias in allegedly politically agnostic machine learning circles is that the formulation and production of answers is trivial when compared to the formulation and production of questions.

Super-relevant to this thread is her work on recidivism risk scoring algos run on prisoners and defendants. The feedback loops that these algos spur are seriously damaging the lives of huge numbers of persons in the criminal justice system far beyond proportionality for the offenses that brought them there.

We have told people, particularly people with low income or disadvantaged backgrounds, that college is the ticket to a high paying job. And then for-profit and other sketchy institutions prey on these people to get them to take out loans for ow value degrees. That probably aren't even completed.

The chapter in "Weapons of Math Destruction" on how for-profit colleges use "big data" to prey on low income people is terrifying. They look for people getting government assistance and other indicators of "struggle" and use facebook retargeting to bombard them with ads on why they need to go to college at a for profit institution.

You kind a have to luck out on a lot of parameters to easily pay off student loan debt. Even Barack Obama couldn't pay off his student loan debt until recently.

Weapons of Math Destruction
https://www.goodreads.com/book/show/28186015-weapons-of-math...

Automating Inequality https://www.goodreads.com/en/book/show/34964830-automating-i...

But, wouldn't collecting viewing habits and then using AI to define (and emulate) real-looking behavior immediately put the developer(s) in that moral grey area that so many algorithms occupy? Technically it could be done, and it would be fascinating to work on, but we'd have to start with a huge browsing dataset (creepy) and then process it to figure out the patterns (exactly what this tool is trying to subvert), and then feed that back as output from within the user's browser (probably feeding back indistinguishable-but-AI-driven data and creating a loop). It's a murky space to wade into, and one that needs a lot more conversation.

Instead I decided to just keep it simple. The first page is chosen randomly from a list of (user-approved) sites. A link on that page is chosen randomly from the list of links that open in the same window and point to the same domain, and that's clicked. That's repeated a somewhat-random number of times, usually about 2-7 times, before a new site is chosen from the user-approved list, and the process starts over.

Check out Cathy O'Neil's definition of "Weapons of Math Destruction" (good overview of her book here: http://money.cnn.com/2016/09/06/technology/weapons-of-math-d...) - I'd love to hear your thoughts on that framework for determining the morality of algorithms.

But to be fair, PR stunt dynamics are better understood these past two decades compared to the 80s. The last decade alone, you can see the reaction of the public to news instantly, react to it and see the public reaction-reaction instantly, with well formatted analytics. That's the only reason I don't immediately dismiss the idea like it was a flat-earther conspiracy theory. There's plenty of analytics to try making a solid plan to do something like that. At the same time, I have zero proof that it's ever been done intentionally.

There's a book called Weapons of Math-destruction. In it the author talks about Facebook's potential ability of swaying public discourse and increasing the chances of certain votes using their analytics knowledge. While the author said it's a scary thought, it could never happen and there's no evidence it has been done or would be done. This book came out 2016 (books are typically published about 6-12 months after manuscript is accepted, so written roughly 2015). So... yea.

There are several cases of problems. One is of course intentional discrimination. The larger problems are likely either a failure to care, consider, or attend to problems, or most insidious, side effects which arise entirely unintentionally.

The fact that much gradient descent machine learning is opaque to explanation means that such AI essentially becomes a new form of knowledge: like science, it provides answers, but unlike traditional Baconian scientific methods, it doesn't answer why or how, and fails to provide cause or mechanism.

Given use in increasingly complex, large-scale systems, without ready human review or oversight, this creates conditions for numerous unfortunate probably consequences.

That is what proponents of the structural racism model are doing. Here's an example I took from the book Weapons of Math Destruction:

When people are convicted of a crime, they undergo a number of personality tests, including the LSI-R (Level of Service Inventory - Revised). This is a highly detailed questionnaire that asks about prior convictions, whether the prisoner had accomplices in their crimes, whether drugs or alcohol were involved, etc.

It does not ask about race.

What it does ask about are things which highly correlate with race, such as the number of police encounters (no criminal suspicion necessary), the number of friends/family/neighbours who have committed crimes, etc. If two first-time offenders have committed identical crimes but one of them grew up in wealthy suburbs and the other grew up in the rough inner city, they will receive very different scores on the LSI-R.

So what do they use the LSI-R for? They feed it into a model which assigns the offender a recidivism risk score. Then they use that risk factor directly when determining the person's sentence, restrictions, parole eligibility, etc.

So now we're not even talking about historical injustices, we're talking about ongoing injustice based on historical injustice. It's a vicious cycle, or a negative feedback loop, if you will. This is a serious problem!

Edit: Just to add another piece of the puzzle, the reason wealthy suburbs vs rough inner cities correlate so highly with race is a direct result of the historical racist practices of redlining [1] and white flight [2]. Now combine that with grinding poverty (also a result of redlining and segregation) and the war on drugs, and the result is high-crime neighbourhoods in the inner city. Those high crime neighbourhoods attract highly increased police presence, which leads to more convictions, which leads to more patrols, etc. This is another vicious cycle which feeds into the above statistical model.

[1] https://en.wikipedia.org/wiki/Redlining

[2] https://en.wikipedia.org/wiki/White_flight

The book weapons of math destruction goes into the issues with machine learning and the prison system.

https://weaponsofmathdestructionbook.com

> Families apply for bail on the platform.

The fact that you focus on a family's ability to pay just highlights one way in which the system is biased against people without family support.

The book Weapons of Math Destruction was published in 2016, and the conclusions therein were not exactly new at the time either.[ß] The business model of ad-based social media is focused on generating engagement, above all else.

Nothing generates more engagement than outrage, fear, or playing to primal instincts. To satisfy that demand, content providers are encouraged to generate the maximal outrage and foment fear. Social media platforms are incentivised to direct people towards that content, because they make more money that way.

The end result: social media platforms automate pile-on and radicalisation at the speed of lies, at global scale. Not because they are inherently evil, but because they are making money off of people being inherently terrible.

ß: The real meat of the book is in the first two chapters, as far as I'm concerned. Everything else in there is just repeating the arguments.

Weapons of Math Destruction was one of the most eye opening books I read recently. It's all about how Machines when set up with self-reinforcing models can become a real big problem.

And how the fact nobody quite understands how The Machine makes decisions isn't helping either.

"Weapons of Math Destruction" by Cathy O'Neil delves into this in more depth. It's a very valid concern, particularly in the way non-technical people are trained over time to give deference to the algorithm.

That means if we try to tune the system to make it less racist, we’ll be making it less accurate. In essence, the system isn’t really racist, it’s a reflection of the racism in society which is leading to these outcomes. Ultimately, the problem is that putting someone in jail increases the likelihood that their relatives will commit crimes. It increases the likelihood that both they and their friends and family will reoffend. It’s a vicious cycle and it doesn’t appear to have any technical solution.

[1] https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction

This paragraph in particular is one of the worst examples I've ever seen of researchers NOT UNDERSTANDING WHAT THEY ARE DOING:

Unlike a human examiner/judge, a computer vision algorithm or classifier has absolutely no subjective baggages, having no emotions, no biases whatsoever due to past experience, race, religion, political doctrine, gender, age, etc., no mental fatigue, no preconditioning of a bad sleep or meal. The automated inference on criminality eliminates the variable of meta-accuracy (the competence of the human judge/examiner) all together.

Please, read Weapons of Math Destruction and understand how excellent machine learning is at discovering and exploiting the biases in datasets.

Edit, no, sorry, it gets worse:

the upper lip curvature is on average 23.4% larger for criminals than for non-criminals.

the distance d between two eye inner corners for criminals is slightly shorter (5.6%) than for non-criminals

Well, it depends on your breadth of understanding of the issue. If you have a limited scope, it doesn't seem like a big problem. And it also assumes that we all agree on certain basic human rights. If we don't then yes, it is a matter of perspective.

Read "Weapons of Math Destruction". It discusses how existing algorithms discriminate in the following case-studies:

- courtroom sentencing (and recidivism prediction)

- mortgage and loan rate determination

- educator performance

- job applications that use 3rd party screening tools

The book looks at specific examples of where these black-box algorithms are deployed that have ruined people's lives with no accountability. The ethics concerns that are being raised were needed a decade ago, or more, but it is only getting worse with the ad-hoc deployment of this un-baked technology to almost every industry.

I have to stress again that these black-box algorithms are ALREADY IN USE and cannot be subpoenaed by courts because it is considered intellectual property.

So yes, it is urgent to crack open this technology because it is all too easily being sold without any investigation, accountability, or thought to the consequences. "Move fast and break things" doesn't work if it ruins peoples lives by landing them in jail, or pushing them into poverty.

The book demonstrates this is not a what-if strawman, but reality.

For a fuller treatment on this I recommend Weapons of Math Destruction by Cathy O'Neil (https://www.amazon.com/Weapons-Math-Destruction-Increases-In...).

They are predatory, ill-regulated, and unstable. ITT Tech, WGU, and those all algorithmicly target audiences that are likely to get approved for Federal student loans: veterans, single-mothers, first-in-family to go to college. They get them approved for these loans and the funds go directly to the school whether they graduate or not. There's a very interesting chapter on this in the book "Weapons of Math Destruction." Don't believe their non-sense ads about being the future of education, the incentives just don't align. Go to local community or state public schools if private is not affordable. Either way, both public and private are not-for-profit.

[1] https://phys.org/news/2019-06-for-profit-america-student-deb...

I don't know where you read that but you really, really, really need to learn about that book before repeating it, much less complaining about the lack of intellectual conversation.

Cathy O'Neil has a Ph.D in mathematics (https://www.genealogy.math.ndsu.nodak.edu/id.php?id=38230), worked as a quant, and most definitely is not claiming that math is racist. Rather, she's talking about how MISUSING math – and especially machine learning – can reinforce biases which were already present or introduced by sampling error. She's actually calling for greater mathematical understanding so people keep these things in mind and avoid them:

http://www.npr.org/2016/09/12/493654950/weapons-of-math-dest...

https://mathbabe.org/2014/08/12/weapon-of-math-destruction-r...

There's a summary of it on the TED Radio Hour here:

https://www.npr.org/2018/01/26/580617998/cathy-oneil-do-algo...

TED Talk https://embed-ssl.ted.com/talks/cathy_o_neil_the_era_of_blin... (video)

Cathy O’Neil Is Unimpressed by Your AI Bias Removal Tool (A RedTail Q&A)
https://redtailmedia.org/2018/10/29/redtail-talks-about-flip...

Weapons of Math Destruction: Cathy O'Neil adds up the damage of algorithms
https://www.theguardian.com/books/2016/oct/27/cathy-oneil-we...

Human Insights missing from Big Data: https://www.ted.com/talks/tricia_wang_the_human_insights_mis...

Further references:

Weapons of Math Detruction outlines dangers of relying on data analytics: https://www.npr.org/2016/09/12/493654950/weapons-of-math-des...

Can Big Data Really Help Screen Immigrants? https://www.npr.org/2017/12/15/571199955/dhs-wants-to-build-...

The thing is, people only tend to notice it when it affects them personally (either they are the victim of the algorithm, or someone they know/like/support is). The world has long worked on irrational biases, which now are being used as the training data for decision-making systems which are subsequently declared to be "objective" because people believe an algorithm can't be biased. And increasingly, the mark of privilege is having access to a system -- applications, interviews, customer service, even courts -- which will use human judgment instead of an unreviewable algorithm.

For more on the topic I suggest the book Weapons of Math Destruction by Cathy O'Neil.

I read:

* Turn the Ship Around!: A True Story of Turning Followers into Leaders (David Marquet)

* Joy at Work: A Revolutionary Approach To Fun on the Job (Dennis Bakke)

* Ne vous résignez pas ! (Bruno Le Maire - French politician)

* Dealers of Lightning: Xerox PARC and the Dawn of the Computer Age (Michael Hiltzik)

* Disrupted: My Misadventure in the Start-Up Bubble (Dan Lyons)

* Making Things Happen: Mastering Project Management (Scott Berkun)

* Basic Economics: A Common Sense Guide to the Economy (Thomas Sowell)

* The Success of Open Source (Steve Weber)

* Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy (Cathy O'Neil)

* Programming in Lua (fourth edition - I read every edition)

I started reading (and will probably finish by the end of the year) Overcomplicated: Technology at the Limits of Comprehension (Samuel Arbesman).

As for what I recommend, it depends what you are into, but I would say I really enjoyed Making Things Happen, which is a must if you have any kind of project management to do, and Basic Economics.

Loan risk algorithms will favor people "similar to" those who have paid back loans before, a sample group biased towards people that banks have already loaned to before. As a result, a lot of the factors are biased towards "from a white upper-middle-class suburban background."

And recidivism estimators, which are used as jail sentencing guidelines in some places.

Screening algorithms for job resumes, and college applications.

Algorithms send police to where crimes are reported. Crimes are reported because the police are there to witness them. The area gets designated a high-crime area. Regular people are arrested more often because regular activity is suspicious in a high-crime area, affecting their future prospects. The higher arrest rate is used to justify this.

It's a continuous spectrum rather than a single point. But if I were to pick a single "point" where it became a self-fulling prophecy? 1994, due to the widespread passage of three-strikes laws.

https://en.wikipedia.org/wiki/Weapons_of_Math_Destruction