Check out our interview with Sonia Joseph, a member of South Park Commons and researcher at Mila, Quebec's preeminent AI research community.
Topics:
- India's Joan of Arc, Rani of Jhansi [[wiki](https://en.wikipedia.org/wiki/Rani_of...)]
- Toxic Culture in AI
- The Bay Area cultural bubble
- Why Montreal is a great place for AI research
- Why we need more AI research institutes
- How doomerism and ethics come into conflict
- The use and abuse of rationality
- Neural foundations of ML
Links:
Mila: https://mila.quebec/en/
Follow Sonia on Twitter here: https://twitter.com/soniajoseph_
Follow your hosts:
John: https://twitter.com/johnvmcdonnell
Bryan: https://twitter.com/GilbertGravis
And read their work:
Interview Transcript
Hi, I'm Bryan and I'm John. And we are hosting the Pioneer Park Podcast where we bring you in-depth conversations with some of the most innovative and forward-thinking creators, technologists, and intellectuals. We're here to share our passion for exploring the cutting edge of creativity and technology. And we're excited to bring you along on the journey. Tune in for thought-provoking conversations with some of the brightest minds of Silicon Valley and beyond.
John: okay, so today I'm super excited to invite Sonia onto the podcast. Sonia is an AI researcher at Mila Quebec AI Institute and co-founder of Alexandria, a Frontier Tech Publishing house. She's also a member of South Park Commons where she co-chaired a forum on agi, which just wrapped up in December.
We're looking forward to the public release of the curriculum later this year. So keep an eye out for that. Sonia, welcome to the.
Sonia: Hi John. Thanks so much for having me. [00:01:00] It's a pleasure to be here.
Bryan: Yeah, welcome.
Sonia: Hi, Bryan.
Bryan: Yeah, so I guess for full transparency, John and I were both attendees of this AGI forum.
And I was waiting every week's I guess session with baited breath. I thought that the discussions in the forum were super interesting. There was a bunch of really prominent, interesting guests that we had come through. And yeah, it was really interesting some intersection of like practical questions with sci.
And a lot of things that are like used to be sci-fi that are getting far more practical than perhaps we ever anticipated.
John: All right. So Sonia, I feel like the question that's on everyone's mind is, Who is Rahni of Jansi ?
Sonia: Oh my gosh. Yeah. Yeah. So basically like I grew up on a lot of like Indian literature and Indian myth.
And she's considered to be India's Jonah Arc. So female leader like has a place in feminist scholarship if you look at any literature. And I [00:02:00] believe she read. Of India against the British. I actually wanna fact check that .
John: Yeah, no, that's really cool. Just we love the the recent kind of blog post that you worked on with S and you pointed out how these kind of influences like really enabled you to succeed at your current endeavors.
So we're like, just curious about maybe like how your background. Made you who you are. .
Sonia: Yeah. Yeah. No, I appreciate that question a lot. So like I, I would say I had a kinda culturally schizophrenic background in some ways where I spent a lot of time. When I was a child in India but then the other half of my life was in Massachusetts.
Which was very like a lot of Protestantism and growing up on a lot of like American history. I like I saw things in a calculation of various like cultures and religions and that has like very much impacted like my entry into AI and how I'm conceiving of ai.
John: Yeah. Something that we loved about the AGI forum is that you have this [00:03:00] kind of really critical eye towards the culture of the way that AI is practiced and the way that research is going forward.
And we can I think you really brought this kind of unique perspective that was super valuable.
Bryan: Yeah, I'm curious do you, are there any points at which you think there's like current I guess problems either in the way that research is being done or the kind of I guess the moral framework in which that research is being done?
Sonia: It's a really interesting question. I would say the AI world is like very big first of all, so it's like hard to critique the entire thing. But it. Have it, parts of it have some of the problems that physics had in the 1990s or still has in being male dominated or like focused on like certain cultures.
And the culture will generate a certain type of research. So your scientific conclusions and the community or culture you're in, you have this like reciprocal relat. For example, in like the 1990s like there's this amazing book called The Trouble with Physics, with Lee [00:04:00] Smolin that goes into sort of like the anthropology of the physics community.
And the 1990s, the physics community was deeply obsessed with string theory. If you weren't working on string theory, you just weren't cool at all and you probably weren't gonna get tenure track. Goes into how string theory wasn't empirically proven. It was like mathematically, internally consistent, but it was by no means like a theory of everything.
And how the monoculture of physics and like the intellectual conclusion of St string theory would feed off each other in this like that cycle. Lee Smolin basically created his own institute to deal with this problem cuz he got just like very frustrated.
I don't think AI is quite so bad. But there are pockets of AI that I do notice. Similar dynamics. And particular the parts of AI that were previously like more influenced by effective altruism and LessWrong in this like the AI safety and alignment camp. I don't think these fields have as bad a problem anymore.
There have been recent. Attempts [00:05:00] called the reform attempt that Scott Aaronson had a very great blog post on how AI safety is being . There's an attempt for AI safety, like a legitimate science that's like empirically grounded and has mathematical theory. But I did notice that more classical AI safety definitely had these like 1990s style string theory problems, , both in the science being like not empirically verified, but like dogmatic. And also in the community that was generating it not being fairly healthy. And I guess with the caveat, I'll say have been either adjacent to or in these communities since I was basically like 12.
So I have seen like a very long history. And I also don't mean to like unilaterally critique these communities. think they have done a lot of good work and given a lot of contributions to the field both in terms of frameworks talent funding but a am looking at these communities with a critical eye, like as we move forward.
Cause it's like, what is, what are coming. Both as a scientific paradigm and as the research community that like generates that paradigm.
Bryan: I'm curious. To me there seemed like kind of two issues. I don't know if they're orthogonal but I think like the scientific integrity of a community and the ability for that community to [00:07:00] generate and falsify hypotheses and the culture of that community and whether or not that culture is a healthy culture to be in, whether it's like a nice place to work in and all that sort of stuff. And I guess my hypothesis is like none of us wanna work in a shitty culture and none of us wanna be part of communities where insults or like abusive behavior is tolerated at all.
But I think that a lot of scientific communities can be interpreted as quite dogmatic because there's an insistence on a specific sort of intellectual lens that you need to adapt to participate in the discussion. And for me it's it always seems like there's like a balance there.
Because for instance, if you wanna be a biologist, you better accept evolution. And like you, you're you have to meet that criteria. And I'm curious, do you think that, for instance in the, is there some sort of almost Intellectual cowtowing or basically a tip of the hat that one needs to do when you're studying artificial intelligence to make it into the room to be taken seriously.
Sonia: That's a great question. Yeah and evolution is an interesting example. Cause that's one that has been empiric. [00:08:00] Verified in various places and maybe the exact like structure o of evolution is open to debate. Like we dunno if it's more gradual or happens in leap burst.
But example in some AI communities is of accepting that on oncoming AI is gonna be bad. Or like culture or more apocalyptic culture. And this is prevalent in a lot of AI safety communities where in order to. Get your research like taken seriously or to even be viewed as an ethical person.
It becomes about character. You have to view AI as inevitable. It's coming fast, and it's more likely than not to be incredibly disastrous. And to be clear, I think ai, like we should be thinking about the safety behind incoming technologies. That's obvious and good.
If AI ends the world. That would be terrible. And even if there's a very small percentage that could happen, we should like to make sure it doesn't happen. But I do think that some of these communities like overweight that and make it almost part of the sort of dogma when it's not empirically proven that this is gonna happen.
We have no evidence this is going to happen. It's like a priority argument [00:09:00] that's actually like mimicking a lot of. Student stay cults and also like death cults that have been seen throughout history. And it's absolutely fascinating that much to less now than it was before.
A lot of possibly AI safety has become like modern alignment. Or practiced in more professional spheres where I think views are a lot more, more nuanced and balanced. But there is still a shadow of Bostrom and Yudkowski and these original thinkers who were influential, e even more influential like 10 to 15 years ago.
John: Sonia sometimes when I talk to people who are really into the alignment problem there's a kind of view that like the philosophical argument that is made is just like very strong.
And so They just, people just view it as actually just like a very strong argument that this, these systems are very dangerous. If you think about when I think about Holden Karnofsky's pasta, like I, I of imagine okay, I think if the system was that powerful, it seems like it would be dangerous.
I don't know exactly how likely I think that exact version of it is to be [00:10:00] created. When you think about those, when you think about the, I guess like the content of that alignment argument? Do you think the content did you just think it's do you think the argument is is strong or do you feel like it's actually overrated?
I guess what's your view on that. .
Sonia: Yeah. Re remind me of the so my memory of pasta is that there's some like math and AI that starts like, executing on experiments or like using the results of the experiments, like feedback.
John: That's right. Yeah.
Sonia: Yeah. Yeah. This is fascinating. I love pasta.
I, I think it's absolutely fascinating as a thought experiment. My pushback here would be like all of these scenarios strike me as being slow takeoff. Opposed to, someone develops, like an agent, like a single lab develops an agent, and the agent starts like recursively, self-improving and it like takes over the world, which is like often like the classic scenario presented.
John: Yeah.
Sonia: The reason this doesn't make sense to me is that there are so many like limitations in the physical. for example, just like the speed of molecules in biology. We're gonna be limited by that. The speed of a robot, like [00:11:00] traveling across the country. We're going to be limited by that.
Like we there's one, one argument that computers think so fast. They're not going to be able, they're gonna be able to outthink us. I think this is true, but ultimately for the co computer to interface with the physical world, it is going to be dealing with the slowness of the physical world.
And that is not something the computer can artificially speed up. There are also various other constraints, like the government has a lot of red tape and bureaucracy. In order to actually run any study you have to go through a certain approval process. Maybe the AI figures out how to bypass that.
That's possible. Maybe the AI has a physical army and it doesn't care if that's also possible. But I do think that. , the real world has enough red tape and constraints where we're not gonna wake up one day and see like drones like everywhere. And some like AI has taken over. I think it'll be like slower and more subtle than that.
This is also not to say not necessarily to worry, having some sort of superhuman scientist like that gets out of her control sounds objectively bad, but I don't actually think pasta in its current form is an inevitable [00:12:00] outcome.
Bryan: Yeah, that's very interesting. I feel like there's like a dynamic with a lot of technologies, and this was true for crypto, the wave of crypto interest is that everyone that was very close to the technology assumed its expansion to take over the world was inevitable.
And then everyone far away from it like treated it as utterly irrelevant and it turns out that both are wrong. Like that there's some sort of middle ground of estimating its impact on the world where you can converge on truth.
Sonia: Yeah, exactly. And it's so funny cuz like every time I leave San Francisco and go talk to people, not in tech, like I'm in Boston right now, it's just like I, I can feel myself leaving the bubble where people are like, what?
What is ChatGPT? Bt like what is, I heard about that on in, in the New York Times. I think , I don't, it's just like a different world out here.
John: Yeah. . So maybe backing up a little bit, I'm really curious about Mila, both like the work you're doing there and how you ended up going there and what that community is like.
Sonia: Yeah. Yeah. So Mila is a deep learning institute in Montreal. It was founded by Yoshua Bengio, who's one of the fathers [00:13:00] of modern deep learning. And it's an incredible place. Like it's very international. It's the largest like academic institute for machine learning in the world. It has a lot of collaborations with industry.
It's known for a lot of researchers who are awesome in reinforcement learning. Ian Goodfellow, who invented GANs, did his PhD there, so it has this like very vibrant history in machine learning. And I went there to work with Blake Richards, who does some incredible work at the intersection of neuroscience and ai.
Can we look at the brain fiber AI algorithms and like I felt like Blake's lab had a lot of intellectual independence, which was very attractive in me partially trying to escape some Bay Area monoculture and develop my own intellectual frameworks.
I think my intellectual frameworks are very much in development. Otherwise I would be able to stay in the bay and actually interface with that. So Mila's been like a very good environment to, to work on that.
Bryan: I'm curious, do you [00:14:00] feel like it's a different bubble? Is it outside the bubble? Is it a loosely connected bubble? Because we, I think we definitely recognize that San Francisco and the Bay Area in particular has a specific ethos and take on the way that AI is affecting things.
Obviously this is the heart of the industrial applications or the commercial applications of these things. I'm curious how Mila stays connected but also separate.
Sonia: Yeah, so Mila feels like less of a bubble, I think because it's so international. There is a constant influx of students coming in from Europe, from Canada, from India, from the United States that it doesn't feel as closed as an ecosystem.
There are also like more stakeholder interests than in the Bay. I think like you have government as a stakeholder, you have industry, you have startups, you have academia itself. So as a result I don't think anyone has to... prescribe to a particular way of being. Plus Mila's culture or Montreal's culture enables this environment in that the culture is very caring and very respectful of individual differences.
One interesting thing [00:15:00] that's happening at, that was happening at Mila last year was the introduction of the scaling hypothesis. So the scaling hypothesis, just like the more data compute and parameters you have, intelligent behaviors go to emerge. This is something I think the Bay accepted much sooner and much more readily.
For better or worse in, in this case, I think for better. Cause I think the scaling hypothesis is mostly correct, but I noticed at Mila there was just like a lot more resistance to it. And this could be a variety of things. This could be coming from academia. And academia has always been a bit more anti-scaling compared to industry.
Just due to compute limitations. . But I also do think that Mila is just less susceptible to any kind of trend due to people just being very exposed or used to very different ways of thinking or more independent in their opinions.
John: Yeah, so it can often feel like a lot of research or the kind of vibe of the field right now is just this kind of like scaling hype train.
Are there kind of other lines of interest of research that are [00:16:00] you're interested in, that are going on at maybe at Mila or other places that are less affected by that specifically?
Sonia: Yeah. One area that I find interesting is some of the interpretability research on toy models or like mechanistic interpretability on, on toy models.
So basically you are looking at the structure of the neural net in order to understand what is going to output. So you're trying to reverse engineer its algorithm. And this I think is pretty suited for low compute paradigms cuz you can just have like a four layer, like neural net or some like tiny transformer and just do all sorts of fascinating visualizations to try to find first principles.
The hard part here is I do think you need to connect it to the larger models which show, like fundamentally different behaviors. But this is a fun part cuz you can come up with laws that sort of generalized the behavior of these toy models to the behavior of the large models.
And you can do a lot of work without as much compute.
Bryan: One thing that may be true in the Bay, that the, there's definitely an overrepresentation of these power, of these commercial interests and the startup [00:17:00] culture and the impact of scaling. But one might also argue that is actually the truth.
That the reality is that more and more of the innovation in this space and incredible proportion of the kind of overall power represented by these technologies is being monopolized by commercial interests. And that to not acknowledge that, is almost like silly. These aren't, these state actors haven't been very effective and academic institutions are increasingly falling behind.
A lot of the best publications are emerging from the private sector. Is there almost like a, perhaps like a blinders on in terms of not acknowledging that.
Sonia: Yeah. Yeah. I think that's mostly right. The one caveat is I think academia is still useful for this sort of theory and doing ethics and policy work where that is in conflict of interest with industries in cor in corporations as a safe haven for that.
But returning to your question, I think in academia there is, Amongst some parties, maybe some denial about scaling laws [00:18:00] because it might make your research feel less important. It might make it feel like you've been working on an inductive bias for five years or 10 years, which is a very long time
And there might be a sort of like psychological, like aversion or denial of it to protect like your research. Plus I do think there's less engineering talent. All these industrial labs are working side by side with amazing full stack engineers, and in a lot of cases, setting up these systems requires that talent and that can feel less accessible if most of your skill is like mathematics or doing stuff in a Jupyter Notebook.
John: Yeah. I've heard you ask the question no, I was trying, actually trying to see if you'd written about it, but I'm not sure you have, but about what would it take to create a new AI research group? Is that something you're exploring?
Sonia: Yeah. Yeah. That question is like so fascinating to me.
Like I, I'm very fascinated by the origin stories of Deep Mind and OpenAI and want to spiritually do something similar, but think that it's going to actually look quite different because when Deep Mind and OpenAI were [00:19:00] founded AI was like less hot, or it was like hot a, in a different way and it was considered more sci-fi and crazy.
So there was a lot more alpha in doing something like this. Like the founders of both Deep Mind and OpenAI were called crazy multiple times by both their colleagues and by professors. So if I were to. Say, all right, we're starting another deep mind. We're starting another like industrial AI research lab.
Let's go. Of course this is like a very ambitious and like difficult thing to do, but it's now considered to be like very feasible, which makes me wonder if there's like less, actually, like less alpha and doing something like this , so right now, like starting privatized science or like another privatized science.
it's this very fascinating high-dimensional problem of what is your funding model? What type of person are you gonna hire? Like how do you balance attention between research and development and product? How do you, how quickly do you spin off into applications? , how do you ensure like the right feedback loops so that your [00:20:00] research is going in directions that are both like scientifically useful and eventually commercially useful?
Yeah. Do you go straight to consumer? Do you go to businesses? Do you go to governments? How do you ensure that you're going to have enough funding? When we have like language models that are like in the billions of dollars, which I think is not crazy to happen. So these are all questions that we were actually exploring in, in, in the AGI forum that I'm still thinking quite strongly about.
John: Yeah. Is there a gap that you see where it's oh we've got open ai, we've got Deep Mind, like. Where those guys are missing something or like there, there's a need for an additional house that's gonna do stuff differently.
Sonia: Yeah. Yeah. I definitely think there could be like 10 and fif 10 to 15 more companies.
Like I think the space under-saturated actually. But when you're in the Bay Area, , it can feel saturated. I I think even if one were to try to create another philanthropic the research would inevitably take a different direction just because space of possible research is so big. So it could be another like privatized [00:21:00] science bet or some sort of b corp like anthropic.
But there are also like other niches that haven't been filled. For example, some sort of group that works closely with the government. Like Microsoft for example, I do think has people doing this. But I'm thinking about something more like Palantir or Andrew. So that's a niche that's not yet been filled in terms of doing a lot of like foundation model based stuff.
And the second thing that comes to mind is, Something that's as independent as possible from capitalism. So some sort of un for agi. Not literally the un but that's like analog where you're working primarily with governments and you're primarily getting government funded. And I think this would also be incredibly useful in like setting international standards for doing this kind of research.
Bryan: I'm curious, we've talked a little bit about maybe not being as much of a doomsday doomsdayist regarding an uncontrolled agi like taking over the universe and making us all into paperclips as the famous example is. [00:22:00] But what do you see as the kind of realistic doomsday scenarios or like the bad outcomes that could result from use by state actors?
What are the, what's the kind of geopolitical perspective that you have on the use of these technologies going forward.
Sonia: Yeah. Yeah. So I am way more afraid of people right now than I am of ai. Like I, I think powerful high leverage AI in the hands of some bad actors is more scary than potentially the AI itself being a malevolent agent.
For example one country commanding like an army of like drones and robots that are all have like very advanced, like facial recognition technologies that can invade villages or invade towns or invade cities. This is scary.
Bryan: The current reality in Ukraine, basically.
Sonia: Yeah. . Yeah. Which is already quite bad.
Like scary AI outcomes might look like everything that's scary we have now just augmented. Where that or situation in Ukraine is way more right now it's quite dire. It's way more dire and it's way more of a [00:23:00] global threat because it's AI augmented. And then of course there're like less like sci-fi or like less Hollywood friendly scenarios where all the AI we have is just loaded with weird biases.
It's like a bunch of like very biased, like very Racist or sexist humans, like everywhere, just making weird decisions that are full of all these, like structural , biases that we can't quite understand. And ...
Bryan: like nazi bots everywhere, basically.
Sonia: Yeah. Yeah. Like Nazi bots everywhere. Like in your tax software, like in all these like predictive policing algorithms in your self-driving cars, just like systemically making decisions, and this is already here in, in terms of like certain groups of humans. Which is what I mean by again, like feel like evil is already in the world and evil AI will just look like an accentuation of that evil.
John: So actually I, I did wanna ask you about your relationship with Southpark Commons. So what, what caused you to join Southpark Commons and what's your experience been like?
Sonia: Yeah, so I'm largely coming from like a research and [00:24:00] academic background, but in like really internalizing the founding story of Deep Mind and in seeing sort of academia dying as Bryan you were talking about or at least dying in certain ways.
I very much wanted to master startups and entrepreneurship. So South Park Commons was actually an ideal place to do it because the, it's so exploratory here. It's it is like a Montessori school, in some ways. That it was an ideal place to talk to a lot of founders and see what is a tick to start a company.
What sort of mindset and skills do you need? So that was like one. . And the other thing that I really liked about South Park Commons and has been incredibly useful is the open-mindedness here. In the Bay Area. We've been talking about this monoculture, and I don't think anyone's completely immune from say the generative AI hype cycle like that I think has hit almost everyone I know who is in tech. But at SPC I have not experienced a very dogmatic [00:25:00] way of thinking of ai. In fact, like the sort of modus operandi of thinking about AI has been very builder for like builder friendly, like open-minded. And this has made SPC a great place to try start developing frameworks to think about AGI I that are not biased by an a priori doomsday focus.
John: Yeah, do you actually feel like, maybe related to that, like there's this kind of like deccelerationist idea coming out of the doomer world. Do you think that might be actually hurting us in terms of our ability to handle the actual challenges that are arising around AGI stuff or AI stuff?
Sonia: Yeah, the deccelerationist of stuff is really fascinating. Like I remember at NeurIPS there were a bunch of researchers wearing "Don't build AGI" shirts, . And it's interesting, I'm not sure what it looks like in practice. Is this like a social and like artistic movement or are there actually ... being made that are decelerating ai? I'm [00:26:00] not sure.
Like the way I like to think about it is, Thinking very carefully, how are we applying this AI now and how can we be applying it in the most effective and maximally good way. Which is both a technical problem and a a social problem.
Are we applying it to. Reduce poverty in India? Or are we giving it to like farmers in India to test the soil so they can plant their crops with more intentionality as to weather patterns? Like that's like a very different use than say, like using it for social media. So I'm not sure.
I would say don't build agi, but think extremely carefully as to every single use case that you ... aGI two would be my version of.
Bryan: In recent interviews, Sam Altman has talked a lot about how as the cost of most forms of intellectual work decline --and I think that is a realistic hypothesis that like document generation, code generation, like a ton of white collar sector jobs and fairly [00:27:00] expensive skills are gonna become a lot cheaper, and that those are gonna be filled in by increasingly sophisticated models. Now, whether or not we call that AGI or just ai, I, think it's an interesting debate, but the, for me, there seems to be a bit of naivete.
On what the future looks like in this world of abundance because the the economic sort of biases seem to be that the owners of these systems which are these large commercial entities that we've been talking about, are the ones that get to absorb all that value and that the entire middle. And we're talking, career transitions that we probably have tr have trouble thinking about.
Just massive career transitions for people. And it, there's this constant cycle of destruct creation and destruction with jobs. But I'm curious, what are your own views on how society begins to change as a function of these tools getting more sophisticated?
Sonia: Yeah, it's fascinating.
One thing is I do think [00:28:00] automation of jobs is largely overplayed. This is seen historically and I think this is the case again, in that jobs will just become different. Like right now, prompt engineering is almost like a job or a thing that some people are better at than others. And when prompt engineering becomes less relevant, maybe these models become better.
I do think it'll be something else, like maybe, I don't know, navigating like the nightmare ecosystem that's about to develop around, language models or or setting up like generative AI workflows for people. I do wonder if using these tools is going to be harder than we think. For example, I was trying to use ChatGPT to help me write an essay and found it was pretty useful in helping me reword awkward sentences, but it was like pretty bad at actually getting an essay that flowed in a unique way.
And sure, maybe we need more data and GPT4 will solve this problem. But I do think that, that these are just, these are so far tools that are automating away some jobs, but not necessarily making things as easy as we would think [00:29:00] and making a bunch of new jobs arise. But say I'm to take like the strong version of what you're saying, which is yeah, jobs are actually, are getting automated away and this is actually like quite bad and these corporations are owning these algorithms.
And we're seeing like massive wealth disparity. This strikes me as being at government level of policy in terms of having some sort of like universal basic income. If we're gonna have a post scarcity world that. Pretty necessary or pretty severely taxing these organizations or treating like aspects of AI as a public good or as a public utility.
Bryan: Yeah, one part of me fears that we're headed towards Elysium, I don't know if y'all have seen that movie .
Sonia: Yeah. So that's like a terrifying AI future . I, I fear. , I fear that as well. And also very much don't want that.
John: I guess actually kinda underlying these conversations are the questions about what systems of ethics we need to be applying. So obviously there's been a lot of impact from the EA movement. Alright. Now we've had some retrenchment on some of that due to recent [00:30:00] the FTX meltdown and Sam Bankman-Fried's association with it.
What's your view on maybe both like effective altruism and also like I, what you might what alternatives you think are interesting as well?
Sonia: Yeah, I love that question. So I do think we need to be looking at every ethical system that has , that had any sort of prominence, like across cultures.
So not just ethical systems from Western cultures or Europe, but also ethical systems from Aboriginal tribes or other countries China, India, Japan. I am coming to the conclusion that there is this like meta ethical system that people have or develop in which you deploy almost like a different ethical framework in different contexts.
And it's actually like quite complex. But there is like this almost like meta ethical algorithm that's like quite sophisticated and it doesn't look like something as this as clean and decomposition as like deontology, like consequentialism, like one or the other at certain times or different ratios or it, it's not quite as simplistic as that.[00:31:00]
And I do care a ton about something that I think is very underemphasized currently, which is the sort of like personal, ethical, and like moral systems of the researchers, the scientists, the engineers, the operators, the people in this field making high leverage decisions. And I think this is controversial because in America there's often this like strong separation between like your work life and like your personal life and like what you do in your personal life like shouldn't affect your work life.
But I think when it comes to anyone in a position of leadership or high leverage, I do think that ethics of both have to be like relatively watertight or as good as possible. And ethical and moral education for these leaders is like pretty paramount and this education could look like studying like these ethical systems and moral systems across the world.
It could also look like, just like traveling a lot and like talking to a variety of very different people different ages, like from different countries, like from different like cultures and subcultures. [00:32:00] To just get a better sense of what the world is like.
Regarding ea there is just a very strong emphasis on rationality, on utilitarianism, to a degree that I don't think is functional. I think a little bit of rationality and a little bit of utilitarianism is great, but these cultures have tended to overdo it. The most sophisticated thinkers in these communities don't make these mistakes, but I think the average person in the community and the community as a whole does make this mistake.
And rationality is especially interesting because I think in India, which is the other culture I'm familiar with, I don't see this like strong clean binary between rationality and emotion. It doesn't make sense. It's like taking a neural net and like arbitrarily saying the logits are rationality and like the inner layers are like intuition, like something like that.
They're more viewed as like parts of a more cohesive system. So to like almost fetishize rationality over intuition or emotion. It doesn't make sense from, in like a more structural way. The [00:33:00] other thing is that there is this amazing book called Eurocentricism by Samir Amin, who is like this Egyptian scholar who goes into sort of the history of metaphysics and metaphysical cults.
And we'll notice that there have always been cults of reason like rationality based cults like this is not some recent phenomenon. This is a phenomenon that's been documented for the past 2000 years, and the scholar makes the hypothesis that he, he just notices that all of these cults of reason and rationality will eventually dissolve into like vision and revelation.
And I don't I don't stand strongly by this statement. I haven't seen enough case studies but it's just like an interesting, like thought experiment where what if your systems of reason are not enough to represent the world. What if in some ways emotion and intuition are actually more predictive or more useful?
And our very technocratic society has largely dismissed like huge swaths of [00:34:00] religion. When I think this is like a huge mistake and religion is functional and useful in a lot of ways.
My answer to like what moral and ethical systems are like to see in ai... I I think we need to study everything. And not just restrict ourselves to like rationality or western dominated thought.
Bryan: To maybe make one part of that more concrete. You mentioned at the beginning of this that in within our own behavior there's sort of an adoption of a specific kind of ethical framework and then we transition between those.
Can you provide an a concrete example of like where you think that, get where that happens?
Sonia: Yeah. Let me think. Say there is a research community that is working on some very high leverage technology and there's this tension in the community about sort of personal ethics and utilitarian ethics. So here the utilitarian goal is we're going to develop vaccines against viruses.
But this is very dangerous because we might accidentally develop a vaccine that doesn't work. Or we might have some sort of security [00:35:00] hazard and the virus might escape. It is our foremost goal to develop a safe vaccine and do this as quickly as possible else, or else millions of people are gonna die.
And we will do whatever it takes. We'll sacrifice our sleep schedule, we'll sacrifice our personal lives. We'll sacrifice our quality of life to ensure this will happen. Cuz we believe it's just so important. And in many ways, this is incredibly noble. Sacrificing these like more like local resources for the sake of the goal of saving millions of.
In this very like visceral kind of way. However, say some researchers start cutting cutting corners. Some of the researchers, especially the high leverage ones and the ones managing other researchers start screaming at their employees for not working more start controlling huge swaths of their employees' lives.
Start sort of manipulating their employees or even taking advantage of their positions of power to psychologically abuse their employees. [00:36:00] So here you see like this tension between utilitarianism and virtue ethics. Sorry, not virtue deontology is what one could say.
John: Maybe both.
Sonia: What
John: maybe virtue ethics is too, right? Maybe both.
Sonia: Yeah. Both. And you wonder is it worth dealing with what might be called like local problems or like problems on a shorter time horizon of this like unhealthy community? Or is, or do you say it's worth it? Like, all of this is justified the ends justify the means.
And I think there are different contexts where your answer might change. If tomorrow, like literally tomorrow, we were all going to die from a virus and we have very high confidence in this being the case. Then I would say focusing on, like emotionally fixing this community and like getting the right people therapy or kicking out bad actors. There just isn't time to do this in a way that's like effective and humane.
But in a different scenario if the virus were coming like two years from now and there is time then maybe all of this is justified .
[00:37:00] There's also another view where, abusing people close to you is like never ever, okay. Not even for the name of a utilitarian goal. So here it's like a human rights violation, as treated as unilaterally bad, human rights are sacred. You cannot ends, never justify the means. And I think one needs a maximal amount of context to make this call.
I vere on the side of human rights are sacred. Like they cannot be violated for utilitarianism, but know that there might be scenarios where things get increasingly gray. And that very much is a judgment call.
The reason I don't like the trolley problem or just I don't like trolley problems in general is because they're all very contextless. What is this trolley like, why are people like tied onto the trolley track? Like who invented this?
Bryan: Who put them there?
Sonia: Yeah, exactly. Exactly. It's like the context lessness of the trolley problem. And it's making it like overly rational in, in almost making it in this idealized like gasless, physics style chamber is actually like, not that like [00:38:00] realistic and not great, like moral training. Yeah.
John: What I love about your example with the way that you're treating people, if there's like a virus that's gonna kill everyone tomorrow or something, is you can really see how that directly applies to fast take off doomerism. Because if you really strongly believe in fast take off doomerism or convince other people of fast take off doomerism then that will like tilt, if people are very utilitarian, then it will tilt people towards excusing a lot of like short-term behaviors on the basis of that, that otherwise would never be excused essentially.
Sonia: Exactly.
Bryan: It reminds me a lot of revolutionary thinking in general of how like it is often the case that the most egregious violations of human dignity occur by people who are very persuaded in some sort of doomerist or doomsday sort of theology.
Be that. Like socialism or re reli religious a religious movement, like these sort of cultish things emerge around a really strong thesis, [00:39:00] which basically serves as some excuse or justification to commit anything anything is permitted under that strong thesis.
Sonia: Yeah, exactly. The ideology takes precedent over individual humanity.
Bryan: One point, I'm just curious to throw this out. There is, this is relates to a quote, I think it's from Mark Andreessen, and I never quote it correctly, but it's like the magic power of capitalism is to absorb all criticism about itself and then sell you a t-shirt.
And I feel the same way about like critiques of rationality, like rationalism, perhaps as a sort of like pseudo spiritual movement can be critiqued and perhaps set aside, but to explore your argument about your own moral framework there. You attempted to structure that in a kind of rational framework of thinking about a time bound and like which which moral framework might be more useful under which time bound?
And I would argue that's an application of rationality attempting to solve this ethical question.
Sonia: It could be it could be an application of rationality. It, or [00:40:00] maybe it was an intuition that I later formalized into words in order to communicate it.
I... would love to see, I guess what is your definition of rationality? Is it like using scientific language? Is it like attempting to use logic? Is it I'm going to try to event invent like words. I, is it like how you're directly perceiving the world? Is it... what do you mean? I guess .
Bryan: That's a good question. I guess I would somehow, I think I would define it by, Some attempt to transparently derive one's conclusions through causal analysis, including one's own kind of like predispositions and and thoughts.
So for example, I think the utilitarianism is often critiqued as really being not very descriptive of how humans behave and missing something about the locality of consciousness and the locality of emotions. And I believe that locality should be accounted for. That's a fair [00:41:00] criticism. And I think it's actually quite reasonable to care about the health of one's community versus the caring about an anonymous community that one never interacts with.
I think that is a rational concern and actually can be driven and explained by not just purely like emotional things, or in fact, the emotions behind it can be contextualized by, "oh yeah emotions are part of the human experience, so we should try to rationally account for our emotional disposition towards things."
Sonia: Yeah that, that makes a lot of sense. Like some sort of like integrated view. I think you can reframe a lot of like emotional things in rational language and you can reframe a lot of like religious thinking in rational language as well. And this is also actually my preferred way of being like, as, as much as I'm critiquing rationality I, rationality is great. I love rationality.
And do think that the mapping and do prefer like the mapping of emotional stuff to rationality opposed to the [00:42:00] other way around. The other way around doesn't necessarily make sense.
Bryan: Sounds like There's a shared critique of people who use this, that dress themselves up in this in order to justify being assholes.
Sonia: Yeah, for sure. And also the broader theme of just like humans in general, taking some sort of ideology and turning, converting the ideology into some sort of like power structure and using the ideology to justify like personal gain. For example, like the Catholic church might be like the egregious example of that, but I think that dynamic is seen like throughout history and cultures.
John: There's some other parts of your background that I I was really curious about that I was hoping we could touch on too. One thing is you actually have some degree of expertise in neuroscience, right?
Sonia: Yes. Yeah. So I was originally actually a neuroscience major and worked at Genia research campus doing neuro stuff.
John: How have recent advance has an AI changed your viewpoint about neuroscience?
Sonia: It's a fascinating question. I became disillusioned by neuro for a while. , [00:43:00] like a lot of neuroscientists. The brain is just so noisy and complex. And toy models weren't generalizing. When all the scaling laws stuff came out every, everything clicked cuz it's like, of course, complex systems are going to show these emergent properties.
And went very deeply into studying AI as being basically almost like studying the brain in a lot of ways, but with systems that were very clean. And in neuro there was been, there was a lot of excitement around using neural nets as these kind of artificial laboratories to make conclusions about the brain.
And I think this is still feasible, but it's not as easy as it sounds because I've tried it. And I think I still see neuro as being very relevant to AI going forward in that a lot of the interpretability stuff we did on the brain on fMRIs are useful for interpreting toy models and also models at scale.
[00:44:00] Interpreting models at scale feels a lot like neuroscience on artificial brains. And a lot of the same principles will apply. But given my frustration with neuro, I'm entering all of this interpretability stuff with a lot of weariness. Where I wonder if our ability to interpret complex systems will hit some sort of asymptote at a point and interpretability may not be as important as we think it is. And what might be more important is actually aligning these systems in a more engineering like capacity.
John: Yeah. Yeah. And I dunno if you know but I actually have a PhD in cognitive psychology and I had similar frustration.
I I was even I was definitely like, I, in some ways I feel like I was like way too early. Cuz I w like I was doing mine. I entered in 2008 and it's, People were like over neural nets. Like Connectionism was like not considered a useful model, but then people were just oh, we should just go even higher levels of Mars abstraction and do like Bayesian models and so my my thesis is all about Bayesian models, but it's just like very abstracted from what's actually happening.
Bryan: I'm curious to dive a little bit deeper there. My understanding is that there's [00:45:00] obviously this sort of branch of AI exploration that I think Bostrom calls whole brain emulation. And there's been some very simple projects of emulating. The neuro neurological systems of very simple organisms do.
Is that a branch of research that just feels very bounded and constrained by the complexity of biology? Is that a branch of research that you think will have some things unearthed from it?
Sonia: It's a fascinating branch bridge of research. I think they're definitely gonna learn something, but it's also like a different goal than building artificial systems.
Emulating biological systems or understanding the brain almost it takes on these like tones of like naturalism where it's like, what is the natural world like as it is now, and how do we form a very high resolution model of that? And that's like a very scientific and worthwhile endeavor, but it feels like a different goal than what is intelligence in the abstract and how do we build intelligence systems that benefit us.
And it's not clear to the extent that you need to look at the brain [00:46:00] .To some extent --yes, like deep boarding originally arose from looking at the brain, so I think there's still a lot of stuff there, but it's not totally clear how biological one needs to go to carry that back.
John: One thing I find kinda interesting is just that like back prop seems to not be the brain's algorithm, but it's seems unclear whether we know what the brain's even basic algorithm is, which is, it seems like it would be useful to know . .
Sonia: Yeah. That's so fascinating. And that's something that my lab, like Blake Richards has done like really interesting work on if you have like Hebbian learning and like back prop or like various other algorithms, how effective is Hebbian learning, if you like, scale it. Like what is like the scaling law of each learning rule and the scaling of Hebbian learning doesn't appear to be like very good.
So it's like, what is going on? I would love to know the learning algorithm as well. . Like right now it seems like propagation is like abstracting over various levels of learning from like individual learning to cultural learning to like civilization scale learning [00:47:00] and back prop is like the right like metaphor for all of that happening at once.
Bryan: I heard an interesting thing just to cite an example that did draw a parallel between, a discovery in the computer science setting or the artificial intelligence setting that felt like it had a corollary in biology.
And that was time difference learning. And this recently came up with an interview with between Ezra Klein and Brian Christian in which Brian Christian was taking was exploring a parallel that basically the ability of our brain to be gratified from being pleasantly surprised was just very similar to the way that a system like AlphaGo learns from its forecasts being off dramatically.
And I thought that was just like fascinating. It's oh wow. Have we threw a separate means, discovered something that has a biological corollary?
Sonia: Yeah, that's super fascinating. Like dopamine reward circuits and TD learning is very cool. . Super cool.
John: Okay, so I think we're starting to come up on time.
I thought maybe a good final question would be what advice do you have for smart [00:48:00] young people who are curious about intelligence?
Sonia: Ooh it's. broad question. Follow your instincts. Like the field of intelligence right now is somewhat constrained in my opinion. And historically has always been constrained. In terms of defining what intelligence is and the various ways in which we could study it.
Intelligence is one of those words that historically was like monopolized by like particular types of intelligence, like being super good at chess. Researchers just made a bunch of algorithms that were very good at chess. But types of intelligence that I would love to see are robots that are very good at dancing or o other forms of intelligence as well.
So whatever your intuition is just follow it. It'll lead to interesting places. The second thing that's might be more practical or actionable, is to just read a ton of textbooks. Like you can get textbooks for free online and if you don't necessarily have access to like classrooms, just like download them and just read them cover to cover.
And that's how I learned Deep [00:49:00] Learning. It wasn't through school, it was through just reading textbooks and autodidacting, and the knowledge is more accessible than one might think. And you don't have to be super intelligent to study intelligence.
John: What are some of your favorite textbooks?
Sonia: Deep learning.
It's Aaron Courville, Yoshua Bengio and Ian Goodfellow is like a classic. I've read it, like four or five times. Deep learning... I think it's one of those fields where the first time you read a textbook, like a lot of it might, fly above you, but if you just keep like staring at the same equations in a different context for many years, eventually it all starts clicking.
Bryan: Thank you so much for being our first guest on the podcast. It's awesome to talk with you.
Sonia: Thanks so much guys. That was really fun.
Alignment, risks, and ethics in AI communities with Sonia Joseph