“The more steps I take back from researching the world of algorithms, the more I recognize a pattern in the moments and situation that a WMD will deploy. It’s not simply where there are people nearby who are gullible and intimidated by mathematics (that’s all people everywhere in the country, probably including you). Rather, a situation is ripe for a weaponized algorithm when there’s something to hide—some responsibility to be offloaded and injected into an alien black box.
That’s not to say algorithms are constructed to be evil. Many of the algorithms that end up being WMDs start out life as well-intentioned plans, efforts in education or in the justice system to establish consistent, fair, and objective criteria for decisions. Algorithms are advertised as such solutions, but they don’t become that way automatically. The problem is the blind faith; people are turning too much power over to the algorithm, without confirming that they are actually better than the previous system.
Algorithms form a constructed digital bureaucracy, where nobody in particular is to blame and everyone passively accepts their fate as directed from algorithmic gods on high. Seen that way, big data is a potent tool, and I expect insiders will make use of it whenever then can. Never mind that it can create systems that undermine their original goals.”
rest @ http://www.newamerica.org/weekly/edition-138/alien-algorithms/