Paul Nemitz on AI’s threat to democracy, Big Tech power & decentralization

This episode was automatically transcribed, there could be errors.

Pauls contributions from GDPR to AI Act

Alessandro Oppo00:00Technology which tends to centralization is never a good thing for democracy. If the technology tends to be, you know, centralized, and that is clearly the ambition of the developers of AI, then that we must insist on decentralization.

Paul Nemitz00:22Welcome to another episode of the Democracy Innovators Podcast. My name is Alessandro Oppo, and our guest of today is Paul Nemitz. Sorry for the pronunciation, and thank you for being here.

Alessandro Oppo00:33Thank you. And

Paul Nemitz00:36and, yeah, I have I mean, the first question is if you can tell us something about you because you work on many things, and so if you'd like to give us some

Democracy in the digital transition & Big Tech

Alessandro Oppo00:49Sure. I'm a visiting professor at the College of Europe in Brussels, in in in Bruges, in Belgium near Brussels, where I teach a postgrad seminar on AI law and data protection law. I do this as a retiree. I was director in the European Commission for fundamental rights, and citizen rights and special adviser, also principal adviser, on the digital transition. And in this context, I learned about the digital. I had my own playful approach to the digital as a lawyer, because I was the lead director on the GDPR, bringing GDPR onto the law books. I I led the whole conceptual preparation and and the negotiations. And, and that basically, was such a learning experience that I stayed on as, let's say, the principal adviser on on digital transition, and all the following legal acts I I was also involved in, like the AI Act, the Data Act, and so Not as a lead director. Only, the GDPR was, I was in the lead because my directorate general, DG Justice in the European Commission was in the lead. All the later acts giving shape to the digital sphere were led by a different director general, namely DG CONNECT, which I would say is, let's say, the more engineering oriented director general.

Paul Nemitz02:36And what is important to take into account when we talk about, digital trans transition or transformation?

Alessandro Oppo02:43Well, it is important, that the digital transition and also now, the transition to a world shaped by AI does not lead to a world where democracy doesn't function anymore, where fundamental rights count for nothing. I think we have to engage and revive, both fundamental rights and, democracy. And, and I think that's that's a very, very important challenge because otherwise, technology and the companies behind the technology, they will just take over. They will rule us and our lives and our societies.

Paul Nemitz03:28Yeah. There is a lot of concentration of power among who, I mean, I'm thinking about the big tech and

AI risks, political bias & polarization

Alessandro Oppo03:36Yes. There's a lot of concentration of power, and history probably has never seen such efficient mix of power both over individuals and societies. If you think about it, there are eight sources of power, which are concentrated in the big tech companies. First of all, they are the richest and best capitalized companies in the world. They're all in the top 10, of this US stock exchange, so they have a lot of money. Second, they have all the personal data about us. Every individual in the world is profiled by these companies with up to 3, 000 parameters, so they basically know everything about us. About every individual, they know the leanings, the political leanings, the sexual leanings, religious leanings. They know our fears and our ambitions, our dreams, and they know all our relationships. And this is, of course, very, very dangerous that, these profiles are at the core of the danger to democracy of fundamental rights and fundamental rights. And then, these companies have the power of innovation on their side, and not only, are they strong innovators and innovate in secrecy in to such an extent that public universities really don't know anymore exactly what is the state of the art. These companies, impose, gagging orders, on their contracts, work contracts with the gagging clauses, which means that people can't talk about, what what they are researching, inside the companies to such an extent that really, the good old tradition that, a PhD, students return to their professors from time to time once they start working and talk with the professor about what they are now working on, it's just not happening anymore. They have completely these companies have completely cut off academia and pushed forward technology in secrecy, which is, I think, a great problem. And it's a it's a huge change, let's say, to thirty or forty years ago still where there was still a very lively dialogue between universities and and private research. Okay. So we are talking about, the eight elements of power which are concentrated in big big tech and which gives them a power in society and democracy over people, but old also over whole countries, which really never existed before. And I was saying that, of course, the key element of this eight is the personal data which they collect, because this means they know everything about every individual. They know our dreams, our ambitions, our fears, and they know our political leanings, our religious leanings, our sexual leanings, they know our relationships. And of course, in the first place, they use it to sell for commerce, but this information can and is being used for political purposes. And, it's not only, about understanding people. It is used to manipulate people in their choices, and not only commercial choices, but also democratic choices. And we have seen this before. It is documented. So this is at the critical core of democracy. This concentration of knowledge in the profiles about people, I think that is a core problem. And, honestly, if we want democracy, we have to get rid of these all encompassing profiles. And I'm not talking about your playlist on Spotify or your loyalty card, with a supermarket or, with an airline. You know, I'm talking about profiles which consolidate all the information, which is in all what I just mentioned, plus everything else you can imagine. Every time you're on the Internet and often also offline, you are being profiled through a collection of your activities of your personal data. And this has to end. This this consolidation into all encompassing profiles is a huge problem, and I will explain later why it's why it's even a bigger problem in times of AI. But let's continue on, the elements of power. They have the power of innovation, I mentioned already, and they have the power because they have the money to buy up competing innovation. They are able to buy up the next logical development of their own technology and just integrate it, but they can also buy up smaller competitors, startups, new market entrants, and just kill them in order to, not have competition anymore. And they do not only, own the power of data and network technology and now AI, they also own and control the networks, the hardware, in terms of the cables which transmits, information. And they are very, very influential in politics. In particular, in American politics, you can see how they are trying to buy votes, how much money they are spending in every state and in Washington to avoid legislation which is in the public interest, to control the content of legislation and to bend it to their will with money. So I think, we have to see all this together. And the final, and also important thing is they use their money to spread narratives, to buy up philosophers, sociologists, political scientists, to develop stories, about the future, which place technology at the center of everything. And, this combination of technological power, data power, network power, power to influence the public sphere, the political choices which are made in countries by lobbying politics, but also by manipulating individuals before elections, And the power of narrative, all of this together, is a huge danger to democracy in itself. And now we enter the age of AI, And everything gets even worse. Because with AI, not only the collection of data becomes much more intense because people tell to, the AI systems, the large language models, much more than they were revealing in, the normal use of social networks and, in in their search behavior or watching videos on TikTok or YouTube. They talk to the LLM like it's a friend, so they reveal even more. And all this is being used for profiling. But also based on the profiles, the targeting of people, the individual manipulation from the strongest computers of this world with AI, of course, gets much more intense. It's quite logical. You can use AI to manipulate people in a much, let's say, more sophisticated way than you could do it before AI. So, our theme today is, is democracy being supported by AI, or is it being undermined? And my view on that is very clear. There is ample evidence already that AI undermines democracy. And there's a lot of talk and a lot of wish that it supports democracy, but that's not real. We don't have large scale examples of empirics where we can already show that AI serves democracy, but we have large scale empirics already that AI undermines democracy. So, we have to look at the empirics and not get lost in theoretical ambitions and theoretical dreams about how great AI could be for democracy if we would just be using it in the right way. Because the likelihood that AI is being used without the companies using all the other elements of power which I just mentioned in their interest and also in AI in their interest. After all, they are these companies which have the data about us are also among the top players of AI is very small. It's, very unlikely that suddenly, these huge corporates, and I would call, also now OpenAI and and and Anthropic among these, you know, why should they suddenly turn, to be good citizens and supporters of democracy? Until now, they have basically stalled any type of legislation, and, they have resisted, any type of, strict obligations being posed on them. We have seen the fight they were fighting in Europe, and this fight is by far not finished because, obviously, it continues also into enforcement activities. So in short, it's crunch time for democracy, and it's time for individuals to engage on this issue.

Paul Nemitz13:59When when you say that AI undermines democracy, do you mean AI as a technical tool or the fact that that is concentrated in just, let's say, few hands? And because I'm thinking like, could AI, let's say, if it is, I don't know, sort of public AI, European public AI, could be in that case good for democracy?

Decentralization by law & risk for human thinking

Alessandro Oppo14:28Well, I think we there are a number of constellations we have to think through. If it is true that AI takes over big part of our work, it means that people do not have, the context anymore in which they normally have their political debates, because the classical political opinion building takes place at the workplace. This is where people talk with each other. This is where they meet. This is where they make their social connections. And if that falls away because AI either takes over the job or AI allows people to work on their own at home or as digital nomads or as crowd workers, it's very difficult for them to engage in democracy and have the type of dialogues which you have at the normal workplace. So I think AI as a technology already on this point is something which will become a problem for democracy. The second point where I think AI as a technology becomes a problem is that it dequalifies us. You know, how are people going to be motivated to learn if everything seems to be known by AI? And how will people be able to check on the political bias of the answers they get when they ask questions on AI? You know, that is much more difficult than if you read your daily newspaper where you know these guys are on the right, these guys are on the left. You know, newspapers have a line normally and it is known. And we make choices on this basis. The ideology of AI is everything is scientific, everything is objective, we give you the one best answer. That's of course a huge lie. The Max Planck Institute of Intelligent Systems in Tübingen has a website where you can ask your questions and you get 21 different answers. And in the first step, the people at the Max Planck, they don't tell you who's answering. And I have tested, on this website, the political bias of AI answers. I have asked political questions. For example, I've asked the question, what is the evidence that AI undermines democracy, And what is the evidence that AI supports democracy? And the answers from the AI system were hugely different. The large majority of AI systems gave an answer which I think, is really not a good answer. It said, oh, is a technology. It's neutral, and it can be used for good purposes and for bad purposes. It all depends on the governance. Well, honestly, I don't believe that's true. AI is not a neutral technology because it is produced by people who have interests for their purposes. And, so it's too simple and under complex to say, oh, it's like any other tool. It's not any other tool. It's a, it's a technology of domination. And that's the difference to the hammer and the knife. The people who develop AI want to dominate us and many of them want to overcome the human. Many of them are posthumanists. They think that humans are basically failing information treatment systems, and they believe that AI is the better information treatment system, and therefore it's absolutely okay in their eyes to push the humans out and have it all done by AI. Certainly, Elon Musk is one of those. So what I'm saying here is, you know, with a little bit of thought, one discovers very easily that already today, AI has huge potential and does actually undermine democracy. I I would go as far as saying that the business model, and that's not an AI only argument, that's also a business model argument, the business model of social networks combined with AI even more than before, polarizes society, drives people into the extremes, and that's what we see in America, and that's what we see in Europe in the elections. More and more elections worldwide lead to a rise of extreme right wing and extreme left wing. And that's not a coincidence. But, that is, I would say, empirical evidence of the fact that the communication environment which social networks combined, with AI create is good news, for the extremists because it the business environment to make money, to keep people on the screen, to maximize clicks favors extreme views. On the other hand, you know, your question, well, you know, let's just think about the potential, of the technology. Yes. We can dream, and we can tell great stories, and that's part of the hype to tell to business, you know, everything will be more efficient when you do it with AI, and to tell to societies, oh, you know, all your problems will be solved with AI, including the problem of consensus building. I mean, imagine deep mind had the hubris to build a machine which they called Habermas Machine. Habermas, the great philosopher who, at the center of his academic work, explained to us how communication in the political environment, in the public space work, and how consensus is built. And they really went out without the agreement of Habermas, this is in my book, because we have talked to Habermas about this, without his agreement, without ever talking to him, and claimed they've created a Habermas machine which simulates the consensus building find building, which he describes in his philosophical theory, which is total hubris. And, we wrote to them, and, they said, okay. We will not use this name anymore. But, of course, the scientific, quote unquote scientific, I say, yeah, I don't think it's scientific, but but but the publication is out and is being quoted as an example of how AI can, help a democracy. Well, you know, I have heard from, Justin Hendrix, the editor of Tech Policy Press and who's a professor of informatics at Cornell University that they have tried to replicate the Habermas machine, and it didn't work. So replication was not possible. And that's just one of the many examples which I could give you where, you know, a story is created claiming that AI can help democracy, but in reality it's not happening and it's not working. In the end, you know, democracy is about people talking with each other, and I would even go as far as saying people physically talking with each other. Because, you know, democracy means sustainable long term engagement. It's often a fight. You need to rely on the others. So the type of solidarity and the type of sticking together which you need in protracted long term political fights is only built if you know the people, if you see them physically, if you speak and discuss with them not just once but many times. So it's a genuinely human thing, and I don't think, mediation helps, helps a lot in that context. Sorry.

Paul Nemitz22:51Don't worry.

Alessandro Oppo22:53Just a moment. I have to take this once.

Paul Nemitz22:55Yeah. Don't worry.

Alessandro Oppo22:57Francesco. No problem. Tell me let's be quick because I'm in a video conference. Yeah. Just send me the address or the name. I'll find it. And 01: 00 is fine for you? Okay. And 01: 00 is fine for you? Okay. See you later. Yeah. It's good. Okay. Bye. Sorry. Now I switch it off. Don't worry. Sorry. Sorry. Sorry. Okay. On we go. And,

Paul Nemitz23:35I mean, before the my the next question, I just want to ask you if we can have a little, I don't know, 10 more. Will it be fine? Okay. What can be done, let's say, a European perspective to not let technology undermine democracy? And also, do you think that old technology can undermine democracy?

GDPR and AI Act enforcement

Alessandro Oppo24:06No. I think there was, let's say, historic experience and evidence in media like television and radio. Of course, television and radio have become essential parts of public communication and the public space, like the newspapers. But if they are not regulated properly, they kill democracy. And the example is in, you know, the times of Hitler and the end of the Weimar Republic, the radio and the monopoly in the printed press in Germany helped Hitler to rise, and certainly for the Fascists, the radio was the key instrument of domination. And the consequence which was drawn from this after the Second World War in Germany and in Europe, and in fact was imposed on Germany, by the Allies, was a separation of powers, vertical and horizontal, including on media. Radio was by law and TV, by law, completely decentralized. So there's no, federal, TV domestically, like in Italy, RAI as a national station. It doesn't exist, in Germany as a result of the experience of fascism, but it has been decentralized. So that's my first point. Technology, which tends to centralization, is never a good thing for democracy. If the technology tends to be, you know, centralized, and that is clearly the ambition of the developers of AI, then that we must insist on decentralization decentralization by law, because the market is not on its own delivering this. The private TVs, you have seen this also in Italy, you know, with Berlusconi, you know, national TV, started decentralized but became very centralized. So, that's the first point, and there are many other reasons why media need to be the public space, needs to be regulated for democracy. If you leave it just to the market, it doesn't deliver democracy, and you can see this in many countries of the world. So that is my comment on the technology, which is probably closest to social networks and AI. But AI is a special case because so far, when we talk about telephone or we talk about social networks or we talk about book printing, These are distribution technologies of content. AI pushes people out of the production of content, of the thinking, and that is a completely new quality. And, so I think that the danger, to, democracy which needs thinking people, democracy only works which people which think themselves, which are responsible, which take part, and engage and inform themselves and get information that they can act, as citizens, if they are if if if the technology agenda is to push people out and to overcome them and to produce all the content they previously produced in their minds through the machine, well, where does that leave democracy? Where does that leave thinking people? Where where does that leave, responsible people? So I think, it is not correct to discuss theoretically about this technology only, but we always have to see this technology also together with the ambition which is behind it. The ambition behind AI is to demonstrate that the machine can do everything better than humans, and therefore humans should be pushed out. They should be pushed out of work, out of business processes. Mister Altman said it very clearly, the ideal company for me is me plus 1, 000, 000 agents. AI agents, no human, and, of course, push them out of democracy and let us govern. We want to govern. We are the new oligarchs. We have the technology. We want to govern. That is the ambition, and that is the agenda of most of these technology actors, and that's why they also happily line up with Trump.

Paul Nemitz29:03And and for the regulations, because before you mentioned GDPR and the AI Act, I wanted to ask you what it worked with the GDPR and what it didn't, and also for the AI Act issue that, I mean, is still new.

Can code act as law?

Alessandro Oppo29:22Yes. I think on both, we have to be realistic. They were political compromises at the time between a grand coalition of social democrats, conservatives, liberals, and greens. So they are halfway. You know? Some others, you know, if you come from the more, market neoliberal side, you would say, oh, we have avoided the worst, in terms of intervention in the market by by by the legislator. I am more on the social democratic side. I would say we have not gone far enough. So, you know, there are many things which could be done better in a second generation of these, laws in the public interest. The second problem is enforcement. For me, it's really the main point of weakness of GDPR enforcement is that none of the enforcement actions so far has gone for these all encompassing personality profiles, which are in the hands of Facebook, Meta, also Instagram contributes to that, of course, and Alphabet through, the search, now also through through Gemini and through YouTube. These two companies not only cash in 80% of all Internet based advertisement, which previously was money for the free press for journalism, and they are killing journalism, but they also constitute the core of data collection, which is then used to create this all encompassing profile of about all peoples. And I am of the firm opinion these profiles are absolutely illegal under GDPR, but the problem is the enforcement authorities are so small and so weak, and none of them dares to go for this central element of problematic data processing, because it's not only about the collection, it's also the fact that they consolidate the data, that they process it to learn about people, and that they thus do things which are completely unproportionate. And they collect too much, and keep the data too long. And so we have a huge enforcement problem in Europe also because of the fragmentation of the enforcement authorities. And, we have a very specific enforcement problem with the AI Act, which is that in contrast to GDPR, people have no rights, it seems, to invoke the AI Act when they feel that they have a problem, with how AI is being used in relation to them. Why do I say this? In GDPR it's very clear that an individual has the right to complain to a data protection authority and the data protection authority has the duty to follow-up this complaint. In the AI act it's not clear whether the authorities have a duty to follow-up or whether they can just say, yes, yes, we take this into account. Thank you very much for your letter, but then do nothing specific. You'll have to see how things develop. The proposal of the commission, of the European Commission, was clearly intended to exclude to exclude individual rights. The European Parliament added to, the AI Act a Fundamental Rights Impact Assessment, but only for the use in the public sector, so not generally, maybe one can draw honey from this addition of the parliament, namely the the fundamental rights impact assessment, and argue, well, if an act contains a fundamental rights impact assessment, it serves the protection of fundamental rights, and therefore people should have an individual right to complain, and there should be a duty of the enforcement authorities to follow-up. But for that, we need jurisprudence. And I think this argument can be made. I would be willing to make it. But I'm not sure the jurisprudence would confirm that this is the right way to interpret the AI. My view is at least the commission proposal at the outset was clearly intended to exclude individual rights. And that's a huge problem. Because it means nobody can force the enforcement authorities to do anything. And that's the difference to GDPR. You see, GDPR, there have been cases already where civil society and individuals have forced enforcement authorities which, you know, have a natural inclination not to do anything. I can say it. I was myself a civil servant. The natural inclination of the civil servant is to do nothing. But in GDPR, they can be forced. And that's important because the enforcer needs to know that also if he or she does nothing, he or she will be taken to court, not only when he or she takes a tough decision. That's the big fear. Yeah? You know, I take a tough decision, and then I'm in court for years because, big tech companies are so rich. You know, they can go always to last instance, they put all the best lawyers of the world at this, and, so my work as a regulator is bogged down for years. That is the reason why many of the enforcers don't do anything hard. At least if an enforcer knows, if I don't act, I will also be taken to court, you know, there's an incentive to do something.

Paul Nemitz35:33And in relation to enforcement, I was thinking about code is law from Lessig. And so I was wondering because law then has to be enforced. Otherwise, while, code in a way, it is enforced by itself. The rule of how people can interact with a software are written in the software itself. So I was wondering, if, regulation can also be done instead of I I mean, then it is the same in a way. Like, if you do it by policy, then you apply it also by design. So you apply also in how you write the code that then is what the people are allowed to do or not to do. I don't know if this is clear, like, as an example, like, if some people if we use a WhatsApp group and one person is the administrator, then the rule of everything are written there. And that person could be the administrator, I don't know, ban someone from the group. But the actions that the people can do in the group, let's say in the WhatsApp group are defined by the software. So I can click like, I can reply to a message, I can maybe delete a message. So the software is just giving some options. And so I wonder if there could be, like, always in relation to enforcement if we can use code as law.

Final message: Democratize Technology & AI

Alessandro Oppo37:42Yeah. Well, I mean, there is, of course, the famous book called Code as Law, and and the author, Harvard constitutional law professor, he said, 200 engineers rule how we live. And, of course, that's exactly not democracy. That's exactly what this company wants, that they set the rules in the community rules, how we communicate and how we live. So I'm not a great friend of of of this this idea. And, you know, code is written for stupids. Who are the stupids? The stupids are the computers because they cannot think themselves. That's why we always need a revision of the code, an update. We see it on our mobile phones, you know, every two months, we we need an update because the computer can't think itself. Law is written for humans. And humans can think themselves. A good law can be reinterpreted in light of developments of business model, of technology, of society by people, namely judges, who can think for themselves. And so I'm actually not convinced that to give shape and control and and direct technology, or for that matter, behavior in society, it's an advantage to put it all in code. I would say the legal system, you know, is a pretty efficient system exactly because you don't need to rewrite the code, the law, all the time when there is a change of circumstances. The trick is here that law must be of course technology neutral, and it must be not too specific, not using the buzzwords of the day, but written at a certain level of abstraction so that it keeps meaning also if circumstances change. But there's another argument against code as law, and that is if we mean it with democracy, the laws must be understandable to people. Now, I granted to you that many laws today are complicated, and if you read it as a normal person the first time, oh god, it's difficult. But at least it's written in the language which you understand. And if you make an effort, and if you research a little bit, you will understand the law. But with code, I don't think we can hope for that people understand, you know, and that we have parliamentary debates about the code and this and that. That is much more removed from the normal individual understanding code than understanding language. And so, you know, I would say, let's rehabilitate the normal human language before the code. I think it is fundamentally wrong to say children must learn coding in school. No. I would say they must learn to express themselves properly in the language. Also, by the way, in order to live together peacefully and, you know, in a way which is, you know, a good way of living together. You must master the human language, and that should be our priority. We have to rehabilitate the normal language before the code, and also for democracy, the language is key. So, I don't believe I know, also in Germany, there are, and and all over Europe, also the OECD has projects, to digitalize law. Yes, you know, that can work on some types of law, like traffic law and law which is very numerical, like tax law and so on, that can work. But digitalizing constitutions or digitalizing take AI Act or or GDPR, you know, honestly, I think that is again hubris of the engineering world of computer science to think they can do everything. They cannot do everything. For example, Stuart Russell, is a computer scientist, says it very clearly in his book, The Problem of Control. He says, every AI must be programmed in such a way that when a value judgment is necessary, the program must stop and return to the human because AI cannot do value judgments. Value judgments can only be done by humans. So I would say, you know, we have to fight the narrative that humans are useless, everything can be done by AI, and, we have to insist that democracy can never be done by the machine and that there are many things which human can do with with human language, which AI cannot do.

Paul Nemitz42:53Of course, it's very important that every person is able to think and decide, and also having a politics where there are people. I think that is very fascinating about Kodi's law is that law can be is interpreted and maybe code can seems more, like, objective in some way. I'm thinking about this. And then if you have a sort of final message also for the people who or all the people who are trying to, innovate on democracy or, yeah, working on similar things.

Alessandro Oppo43:43Yes. I would say, the challenge is to democratize the digital and democratize the AI rather than the other way around to digitalize democracy and, you know, make democracy the object of AI consensus finding or or whatever. So let's work together to democratize AI, to decentralize AI as a technology. Let's develop, and I call on you because I can't develop, I call on people who have the technological knowledge, to develop AI which is decentral, which doesn't allow power concentration, which spreads wealth in different countries and different regions, and which is also controlled by different entities and, you know, not just by five or six or 10 companies in the world. I think that's what we should be aiming for, and I'm happy to hear from time to time that edge computing and maybe also open weight AI models may go in this direction. Every type of decentralization of the technology itself and of the control of the technology is a step forward and is good for democracy. That's the first point. The second point is democracy needs the technological knowledge of the new technical intelligentsia, the people who work with digital and AI, which develop it, which use it as professionals, and so on. And I would invite all of you to participate in one way or the other through a party, through an NGO, through a trade or or or professional organization in democratic dialogue and democratic process. That's what our society needs also from technologists, not just the technology as such.

Paul Nemitz45:42Thank you a lot, Paul, and if you'd like to add anything else that maybe we didn't touch.

Alessandro Oppo45:49Thank you, Alessandro. It was good to talk to you. My pleasure.