Pioneer Park interviews Jesse Han, co-founder of Multi AI. Jesse discusses his background, experience at OpenAI, and his philosophy towards research. He draws inspiration from Alexander Grothendieck's philosophy of listening to the universe and arranging theories accordingly. He also talks about the differences between research and startup thinking, the potential for machines to inspire new algorithms, theories, and results in mathematics, and the use of language models and compute to reduce the risk of misaligned outputs. He believes that language models will become as cheap and accessible as microprocessors, and that the value will go to those who build the software and infrastructure to make them accessible to end users. He recommends that those looking to shift their career in the direction of deep learning and generative AI should work hard, find good mentors, and aim for something that will endure.
Transcript
Jesse Han
===
[00:00:00]
Bryan Davis: Welcome to Pioneer Park. My name is Bryan Davis and this is John. Today we're interviewing Jesse Han. Jesse Han is the co-founder of Multi AI, an AI startup based in San Francisco. He holds a PhD in mathematics from the University of Pittsburgh, and previously worked as a research scientist at OpenAI.
I know Jesse through working for Multi, for a few weeks last year, in which I was helping out with some of their product launches. And was thrilled to invite Jesse to talk a little bit more depth about his background, his experience at OpenAI and Multi as it goes forward. Welcome, Jesse.
Jesse Han: Thanks for having me on the podcast, guys. Thrilled to be here.
Alexander Grothendieck
---
John McDonnell: Yeah. So Jesse, like I wanted to start this off by asking about on your personal webpage you have a picture of you looking up like very thoughtfully at this picture of Alexander Grothendieck. So I was curious what that picture means to you.
Jesse Han: What that picture means to me. So I just thought that picture of him was really funny, [00:01:00] Because it's like this shrine. So for context, that hangs in Tong Long University in Vietnam, which was founded by one of the students that he mentored when he visited Vietnam during his career. And this was during a time that Vietnam was being bombed and he wrote some, like very moving recollections about how he would teach at the university.
And then they would all have to dive into an air rate shelter and they would come out and one of the mathematicians had been like, hit by the bombs. And , that student became a very prominent mathematician in Vietnam, and he had such a lasting influence that they made this, the shine and honor of him. There's this nine foot tall portrait of him there. And I just thought it would be funny to kinda like Adam and God .
But the other thing is that I take a lot of inspiration from his philosophy towards research. There's a saying that he has, a saying that he was famous for, which I think is still relevant for people working in startups today or like trying to run a company, which is that so he says that the, so I'm paraphrasing, but the mark of a good researcher is someone who listens very carefully [00:02:00] to the voices of things.
They try to listen to what the universe is trying to tell them about the structure of the universe. And they they arrange their understanding and their theories and what they're doing accordingly. And I think similarly when you are trying to build something, when you're trying to do something new, you have to listen to what the world is telling you.
You have to listen to what the market is telling you and build accordingly.
I hope that was philosophical enough for you, .
John McDonnell: Yeah, I love it. There's this there's this guy David Whyte, who's who's a poet, and he has this kind of concept that he likes to incorporate into poetry that life should be a conversation between you and the world.
And
John McDonnell: like a really meaningful life or a great life is one where that conversation is really effective and goes both ways. And so that really reminds me of that.
Jesse Han: Yeah, totally , it's a reminder to be open to what the world is telling [00:03:00] you. And I think that's really important to remember as you like go heads down and you try to make something happen in a startup. You have to be on the lookout for signals that maybe you should be doing something different.
Maybe you should be pressing something harder. It's a careful balance that you have to strike.
Research vs startup thinking
---
Bryan Davis: Do you find that the signals you're listening to or the incentives present are different in a research context versus startups? And if so, how so?
Jesse Han: To be honest, I don't really think they're that different. Like in research. So especially if you're in like a high pressure environment or if you're working in like a field that's moving really quickly, like AI , like what research looks like is taking a bunch of bets and choosing how to allocate your resources and figuring out Like what kinds of unfair advantages that you have that might make you unusually capable of capitalizing on the outcomes of some of those bets.
And so I think that a lot of. So a lot of the thinking [00:04:00] around what kinds of bets one should take in their career apply equally well to startups and similarly thinking around what kinds of activities are useful for startups to think about, apply equally well to research. So like an example is pursuing like very high impact research events.
Like you could spend a large majority of your career just pursuing incremental advances which carry less risk and are more likely to be published but which don't have an endearing legacy in terms of the research activity of others in the field. Or on the other hand, you can work on something that fundamentally changes the way that people think about some problem inside of the field. And that has a far more scalable... so I think a lot of the same thinking applies.
Jesse's path from research to startups
---
Bryan Davis: And how did you navigate with that perspective? You were previously a researcher, you were a PhD student, you recently finished your PhD, and you've obviously worked in technology prior to launching a startup. [00:05:00] But how did you, what was your own journey from going from the research context into deciding to work in industry?
Did you ever aspire to be a professor?
Jesse Han: I did at some point. So at some point I was very deeply enmeshed in the pure mathematics world. I was trained a logician for most of my undergraduate years. And then I spent my masters just studying mathematical logic and model theory. But I think that gradually shifted towards a more ambitious vision, which formed the basis for the research program which I pursued in my PhD.
Partially due to my realization that I probably didn't have what it was gonna take to become a top mathematics researcher. I simply didn't have the let's say the intellectual horsepower. Because there are a lot of very talented people working in math, and it's a super small field.
So to like really get up there, it's like being a star athlete. It's like you have to train every day, you have to study the work of the masters.[00:06:00] You have to be in the right place at the right time with the right advisor, working on the exact right field to me making that kind of impact.
And towards the beginning of my PhD I came to the realization. The more impactful thing for me to do would be to try to just automate all of mathematics instead. And so I had this grand vision of eventually building some kind of planetary scale system for automatically searching for mathematical theorem improves.
So that one day human mathematicians would just be the operators of such a machine whose details. And intricacies would be hidden from them, like an operating system hides most of its complexities from the end user. And so that was what sort of drew me towards AI and got me into more industry adjacent things because building a system like that requires a lot of engineering skill, requires some pretty compute heavy resource.
And that kind of brought me into the orbit of people trying to apply the [00:07:00] latest techniques and deep learning to automate theorem proving.
Automating mathematics
---
Bryan Davis: To anchor a little bit more in the math world, do you ever think we'll reach a point in mathematics or perhaps are we already there where we're at the limits of the capacity for human brains to comprehend and do you think that there's a zone in mathematics, in pure math where machines will begin to inspire, be the chief creators of new algorithms, new theories, new results?
Jesse Han: Yeah, I think that's a really interesting question. I think the fields where computers have a large advantage is with like really concrete kinds of combinatorics. That's one thing that stands out. Like subfields of discreet mathematics, like places where computation is really the main way to see how phenomena occur. For example, if you are like studying the dynamics of Conway's game of life, [00:08:00] then running computer simulations, or say you're just like studying say cellular automata, then running computer simulations is probably the best way to gain a good understanding of what's going on with any of the phenomena happening there.
But on the other hand, if you're working in more abstract fields that require like a large tower of definitions say algebraic geometry, then. The computer based foundations get a bit more shaky because there are many ways that you can represent various things. And there hasn't been a lot of work on shoring up, commonly accepted foundations.
Does that answer the question?
Bryan Davis: Yeah, I think it does. I remember reading, I believe it was a, or listening to an interview with Richard Fineman several years ago where he was talking about understanding the universe as peeling layers off an onion, and his hypothesis was that, there may never be an end to the layers.
We could just keep peeling and keep peeling, and eventually we might reach a boundary at which our capacity just to [00:09:00] abstract, our capacity to represent what is actually beyond the next layer, is just somehow limited or contained by the limits of, biologically based IQ or biologically based intelligence and I thought that was an interesting concept that I was, I'm curious to, to investigate whether the same thing might apply or whether you think the same thing might apply to mathematics.
Jesse Han: Oh, yeah. There are totally trivial cases, right? Like there, there are like there are prime numbers that require more bits to represent than is representable inside of the human brain?
Like that would be like a really trivial example. But you can already see this happening with the social fabric of mathematics. So what happens now is that a professional mathematician will go really deep and it's like harder and harder to become a true polymath. Like someone who's achieved mastery of like many fields of mathematics.
And so what happens is that the social fabric of like mathematics is made up of these experts who only see a [00:10:00] very narrow slice of the entire picture. Like for example, there's a vanishingly small number of people who have a complete understanding of the classification of finite groups. And that's simply one piece of mathematical lore, which has been written down in relatively, maybe not low fidelity, but written down in questionable fidelity in a constellation of like papers and preprints and surveys. But the true understanding of the proof, the the thing that is communicated from a master to a student in mathematical practice is very hard to grasp.
And it's only owned by a very small number of people today. So that's definitely happening and there are more examples in other fields of mathematics as well. This kind of phenomenon where where, you know it's becoming increasingly unclear. What parts of mathematics stand on firm foundations and what parts don't spurred a lot of research activity over the past few years in formalizing mathematics in a computer understandable way. This [00:11:00] research program was championed by Kevin Buzzard at Imperial College, where he drove a lot of people to organize a lot of mathematics in a computer understandable format, in a theorem proving language called Lean theory.
And he gave a very good talk at Microsoft Research titled The End of Mathematics where he talks about things like this where there are so many parts of math where the highest standard of proof is just social understanding between mathematicians. And when you think about it, these things are on shakier foundations than you might first believe.
John McDonnell: So where does that put us with, so you were saying that your hope was to build an, some kind of AI or automated system that could move the field forward essentially. And I think you mentioned like the foundations are firmer in a place like combinatorics. Do you feel like this is already having an impact? Or what are the kind of milestones to having impact.
Jesse Han: Do you mean the milestones to having impact in terms of in terms of [00:12:00] like fully automating a part of mathematics or just in verifying the existing knowledge?
John McDonnell: Maybe your perspective on both of those. How far is your vision from being realized even in a small way and what would it take to get there?
Jesse Han: I don't think a system like this has really been constructed for any particular field of mathematics. Of course mathematics is vast and there are many talented people working in it. Like many of whom have, are, have been schooled in the ways of formal proof. And so a system like this might have been built, but as far as I know, like nobody's built like this automated proof search thing where ... So the thing that, that I would like to be automated there is like how mathematics research is conducted by a very senior researcher, right?
Like they, they have this deep, deep understanding of the field and like what things are provable, what things should be proved, what kind of like [00:13:00] research programs should be carried out. It's kinda like building of like giant building, right? Like you say oh you can add an arch there if you like, use these tools from over here and because you have five years of experience already, it should only take you three months.
And so that's like something which requires really intense focus, incredible amounts of persistence, superhuman willpower at times. And if computers were able to do that, and we were able to scale up the amount of compute that we threw at these problems. Then a lot of this could be automated and parallized, right?
So ideally a mathematician could just come in and point to some location in the distance and say we should go there. And then an army of these AI mathematicians would go and do all the work that's needed for actually building that super structure that's needed for getting all the way over there.
So I would say it's a few years away, but there are some pretty talented people working on this problem. For example there's the N to formal group at Google Research, which has been doing a lot of cool work in enabling [00:14:00] all the fundamental technology needed for building a system like this. They've done a lot of great work in auto formalization, i e the ability to automatically translate from natural language into computer form mathematics.
Representing formal logic in language models
---
Bryan Davis: I'm very curious to pause and focus there because a lot of the technology at Multi and obviously at OpenAI is about using natural language as the primary tool of interface with computers, but as very widely known at this point, their ability to do formal reasoning and complicated mathematics is limited.
And there's a lot of things a lot of ways these things can blow up and fail. I'm curious, what do you see as being the key to weaving formal reasoning and formal logic systems together with large scale natural language models.
Jesse Han: Yeah so at Multi, our mission is to automate all knowledge work and one way in which we envision that happening is by providing developers with [00:15:00] a series of trusted software components built on top of large language model primitives that have some kind of structural or semantic guarantee on their outputs. And so when you have building blocks like that you can trust that the kind of powerful occasionally unreliable intelligence that you get from prompt engineering, a pre-trained language model.
So when you have those kinds of guarantees, then you can really begin to build sophisticated software applications that really tackle valuable real world use cases. And in that way, the massive amounts of compute that we have available today can be applied to create incredible value by being able to trust what these models are doing at a more granular level.
Bryan Davis: How do you create those guarantees? That seems like the big problem.
Jesse Han: So if you [00:16:00] look at the special case of mathematics, right? So what happens when you're using a language model to prove a theorem in some kind of formal system? So what you're really doing in that case is that you're synthesizing a program in a programming language, which might be implicit or explicit, that has a sufficiently expressive type system for capturing mathematical theorems.
And so when that program is synthesized, it's done in a way that you can guarantee that program satisfies the specification, and that's checked by a trusted component of your theorem proving system. So if you take that kind of technique and you zoom out and you apply it to the more general case of building robust trustable software on top of large language model [00:17:00] primitives.
Then how that looks is you want to synthesize code, you wanna synthesize entire data structures, which are subject to some kind of specification that is computer checkable. So what we're doing is like very related to some recent work that people in the AI community have been doing structured extraction and the creation of knowledge graphs from informal text.
But we wanna provide even stronger guarantees. We wanna provide more modular components that people can use not just as, things to enter into a SQL database or things to add into, like a knowledge graph, data structure, but components that people can build software with, software that can plug into the world and take actions.
Software that can like power the backend for a far more complex application than just a chatbot.
Solutions to producing verifiable systems
---
Bryan Davis: One strategy I've heard to resolve these issues is basically [00:18:00] a trial and error method whereby you ask a model to come up with a solution to a problem that sort of satisfies some condition , and then you check the condition and if it fails, then you just ask it to regenerate. Is there any other secret sauce that can resolve these issues, that can make these things better or more reactive to the feedback from a checker?
Jesse Han: Yeah, I have a couple tricks with my sleeve. You'll see . But so you can always just add a function that calculates a bool at the end, right? . So then you can be like, with max retries is equal 16, sample me something that matches this model. And you pray that everything actually parses. That's obviously not great.
But if you, so if you approach it the right way with all the expertise that like one might get from trying to solve Olympia math problems in a formal theory proving system, then you can get really far and you can begin to see some really cool things. Like you begin to do what feels like the future of programming, [00:19:00] right? Where where not only do you have this black box that's able to like, transform strings into strings, right? This like programmable fuzzy black box that represents some, some distribution that you're just sampling over all strings But you begin to write programs that, that reliably use language model primitives to achieve some goal that would be impossible with normal programming alone.
So we're gonna be releasing this in a few weeks. People are going to be able to access capabilities that Multi has built on top of language model primitives through an api, they're gonna be able to integrate it into their own AI applications and the kinds of guarantees that we provide make it possible for people to build AI applications that can rely on certain kinds of outputs, right? Like you can process an entire document. You can make sure that you're [00:20:00] extracting text that always support certain claims which are made. You can create so you can create data structures that are constantly accumulating like a stream of thought, a chain of thought audit trail of like why the model is doing what it's doing. And that kind of interpretability, that kind of transparency into what's happening under the hood, and that kind of sophistication is something which is only unlocked when you have a library of trusted and modular software components.
So that's something that, that we were building towards with Multi Flow, which was the first product that we released. So that's like this low code workflow builder. That prioritizes a bunch of AI capabilities. But really what we want to do is we want to expose this to everyone and not necessarily have those workflows only accessible through our low-code builder.
We believe that all developers who want to build these kinds of new AI applications, anyone who wants to build their, their own internal version [00:21:00] of chat, p t plugged into their company's internet should have these kinds of reliable software components so that they can build as confidently and as rapidly as possible.
Vision for Multi
---
John McDonnell: Maybe it's actually worth because we're alluding to it I'd be curious how you would summarize essentially what Multi is and at the big picture, what the vision is for Multi.
Jesse Han: Yeah. So we're going to automate all knowledge work , and how that's gonna happen is that we're gonna provide people with these building blocks that represent, at first relatively basic units of automation, but which will over time become more and more sophisticated units of automation in a verifiable and trusted way.
And as people continue to use Multi components in their AI applications, they'll be able to build more rapidly and more confidently. And, eventually there's going to be capabilities that we ship that are going to do [00:22:00] tasks that were once thought to be entirely under the purview of humans.
Something as complex as taking a quarterly financial report from a Fortune 500 company and compiling it down to a compressed report along with a bunch of data that's entered into a spreadsheet that kind of like structured extraction, critical thinking, reading comprehension, something that like might take someone who's working at the lower rungs of, say, an investment bank, several hours to do will be compressed down to several minutes by using software that has been built out of the components that Multi provides.
Alignment and verifiability for ambiguous goals
---
John McDonnell: Yeah, and one thing I was curious about in terms of this idea of having these components be just be very reliable at providing the desired kind of outputs, is that there's things that you can instantly check in an automated way.
So if I'm generating code, you can just immediately tell me if my python code is syntactic. And so I [00:23:00] can see how you can just really verify, okay this thing's gonna output Python code. And I'm gonna one thing I'm gonna verify is that it is, that it's syntactic. It seems a lot harder to do that for concepts that are less simple to define for a computer.
And so just like an immediate example would be for a financial report. If I'm doing, if I'm trying to read an FTC filing, and I'm gonna pull out of that certain facts about the company, there's a lot of complexity, for example, with how accounting works, and so there are things that seem really fuzzy to me about exactly what maps from what's in that report to what you're gonna want to have in the spreadsheet.
And it's, it feels to me really hard to automate the checking of the accuracy of the output.
Jesse Han: Yeah, totally. No, so you've hit the problem right on the head. Which is that because we don't have a perfect formal representation of all the concepts that we want, right, there's no principia mathematical that we can write here. There's no formal logic that we can write down that specifies that 'oh, this [00:24:00] is a good this is a good summary of this SEC filing,' right? This is what your manager at the investment bank wants. So because we don't have a kind of algorithmic check, we don't have a way to statically analyze the algorithm that produces those outputs and to guarantee that they actually satisfy that specification. There's this problem of you give instructions a natural language. How do you ensure that the outputs are actually aligned with those instructions? Because if the instructions are a natural language and the specification is a natural language then the only way that you can verify it is by having some other agent that understands natural language look at those outputs and hopefully you find a way to, to minimize the likelihood that it's not aligned.
John McDonnell: But there will be a lot of context that's not in the language, right, so it's like my boss at the iBank says summarize this FTC report for me. The instructions were to summarize this report. Like I know a lot about my boss. Like I know I have a lot of contacts for this situation.
[00:25:00] Like I, I have a lot of ability to fill in a lot of gaps about what's really needed that are not, that are just like absent from the stated instructions.
Jesse Han: Yeah. So I think that the way forward there is that you have to break down. So you have to break down the the task of whatever kind of discriminator you're applying to your inputs and outputs in such a way that you can,
so in such a way that you can apply large amounts of compute through language models, because what's gonna happen is that, so over the next few years software, like the kind of software that I'm describing, this kind of neuro symbolic software is going to be responsible for an increasingly large fraction of economically valuable activity. And in order to provide guarantees about the correctness and the alignment of the actions and outputs produced by that software, we are going to have to apply the same kinds of [00:26:00] techniques used to build that software in order to provide bootstrapped guarantees about that software as well.
So I don't think there's a way to completely guarantee the correctness of what comes out of that kind of software. Except for having humans check the outputs. But I do think that what we can do is that, that we can apply the same kinds of language model based compute to to provide soft guarantees.
So we provide enough compute and we apply it in the right way, then we can push the probability that the outputs or actions are severely misaligned way out, to the point where if there's like a 99.5% likelihood that it satisfies like the the extremely convoluted say judging process of these very these five very large language models then it probably won't [00:27:00] need human supervision.
And that's something which will benefit from a data flywheel effect as well. So as we get more and more data from people who will be judging the alignments, the alignment of the kinds of inputs and outputs that we see there. We'll be able to train better and better systems that will provide stronger guarantees, but we won't ever be able to completely get rid of the possibility that there is some extremely unlikely output that comes out, which is completely misaligned.
John McDonnell: I feel like there's also, there's a bet here, right? Which is because one way, sometimes people think about this. This is oh, I'm gonna get my model to the point where, in validation, where like I do a one-off validation. So I do a training and then maybe I do like my fine tuning and then okay. It looks like I have some kind of validation set where the model's performing up say 98% accuracy or something on some metric I came up with. And so I, I could just say oh, I'm just gonna ship that. And I know, like as long as I stay within [00:28:00] distribution like I'm just gonna expect the model to continue to perform at that level.
Versus like the alternative approach of saying actually every time I do inference, I'm gonna have a validation check on that inference. And I feel like one thing that, and I actually haven't heard people making this really strong case as many places. So it's like a really interesting contribution from the way you're describing it is that it's this bet that that doing online validation at inference time is gonna be really crucial. Is that accurate?
Jesse Han: So I suppose it depends on how strong the underlying language model is. There is a language model or some ensemble of language models. Like, for example, you don't really have to use the same language model for everything, right?
Like you can just use like super strong language models for the most cognitively taxing parts of some task that you're writing down in this knowledge work automation software. So it depends on how strong that software is originally. But to be [00:29:00] honest, my take is that , that kind of verification at runtime is just part of the execution of the program.
I think applying, so I think applying language model compute that way is simply part of the toolkit that should be taken for granted when working in this new style of programming. And the only thing that we can ever do is we can just turn up the compute to push that tail risk further and further out.
And I think that's something that's actually incredibly valuable, right? Because if it's a low stakes kind of economic activity, maybe you don't really care if the email spam that you're sending out for SEO is like not great, but if it's like a high touch oh, these, like 512 sales emails have to be really awesome so like we better...
You just have to provide a way for people to signal how much they value the correctness of the outputs. If they want to pay for more compute if they want to pay for more intelligence then in a world where intelligence is too cheap to meter, that's not really a problem for the customer.[00:30:00]
And in fact, it's actually a net positive.
Bryan Davis: I'm curious to zoom out a little bit and talk about the trends of the industry. You've spent time at OpenAI, you were there building as a research scientist. Now OpenAI obviously they're doing this sort of machine learning as a service for the industry at large, and they're also partnering very closely with Microsoft.
Where do you envision the provision of large language models and models themselves as an industry going? Do you think that there's going to be a lot of horizontal scaling? Will these things become more commoditized over time? And how does that relate to open source projects like LangChain?
Jesse Han: So the question is How does the potential commoditization of language models have an impact on open source projects like LangChain?
Commoditization of intelligence
---
Bryan Davis: I'm curious about the kind of overall take on the, maybe the topology and the players in this space, whether or not horizontal sort of expansion of people who are building models and building sort of cornerstone models like OpenAI and like Anthropic and these others, and the trend there to basically [00:31:00] provide machine learning as a service, versus the sort of vertical specific approach of building a specific product.
For one thing, I think it's interesting to that ChatGPT has really been the accelerator for a lot of OpenAI's success over the last three or four months, which is a very specific instantiation of their technology. And perhaps that'll divert them from a business perspective away from this more horizontal commoditized approach and more to a product specific approach.
Curious what you think.
Jesse Han: I obviously can't speak to precisely what they have in mind for their strategy. It's been a while since I've been there. But I do think that the world towards which we're headed is one where language models become increasingly commoditized. Compute is still a bit expensive right now.
The total number of dollars that you have to pay per unit of intelligence, so to speak, is like still relatively high. And it's still a bit cost-prohibitive to run [00:32:00] purely magical feeling apps on top of AI. But I think that's definitely going to change. I think the world towards which we're moving is where... so where language models, speaking precisely , the forward passes of language models that are equivalent to say what GPT-4 will be. I think we're moving towards a world where that kind of intelligence is going to be as cheap as water, where the availability of language models will basically feel like the availability of microprocessors right now, right?
Like right now we don't even think of microprocessors as obstacles to building, right? They're like these completely commoditized things. They sit in every smartphone. There, there are more microprocessors on earth than humans. And we're definitely moving towards a world where the same thing is going to be said of language models or their descendants.
And so I think that what's going to be really important in that case is the kinds of software and infrastructure that you can [00:33:00] build on those primitives. Microprocessors weren't useful until. The various layers of firmware and then low level and high level software were built that could make them really accessible to end users.
And I think that, similarly, a lot of the value from this existing AI wave is going to go to the people who go that last mile in building out the rest of that stack. And I think LangChain is a great example of a project that consolidates a lot of the knowledge that's needed to get builders going.
LangChain and Multi
---
Bryan Davis: Do you anticipate LangChain or projects like LangChain as being competitive with Multi or something that is integrated and part of Multi's journey?
Jesse Han: Oh, I don't think they necessarily have to be competi. I think it's natural that there are gonna be various sorts of software frameworks, software providers and services providers built on top of a new platform, like the language model [00:34:00] platforms that we're seeing right now. We're definitely not competitive currently.
The future
---
Bryan Davis: What would you say is the one and five year vision for multi? Where do you want to take it? We've gone into it a little bit, but curious to think about what are you most excited about in the next one year, in the next five?
Jesse Han: So I'd say the thing I'm most excited about for the next year is the increasing availability of language model technology. I'm really excited about some of the upcoming open source releases. And I think that once once more people have access to. This kind of commodified intelligence, the more demand that there is going to be for the applications built on top of that. And I think Multi's gonna be able to provide a lot of the tooling that people are gonna need for building really economically valuable applications.
As for where we want to be in five years. So I really want multi to [00:35:00] ...to provide intelligent automation for every sector of the economy. Like multi should be the cognitive engine, which powers like, I don't know, 60% of what used to be investment bankers, 60% of what used to be the people who would compare the terms provided by insurance providers against like certain claims. Should be like 60% of the people who just digest a bunch of information from various data feeds and prepare them into reports at various market research firms.
There are so many cognitively taxing tasks right now, which are performed predominantly by humans, which I think are right now perfectly within the range of the kind of automation that you can build with language models. The only thing that has to be provided is the infrastructure and offering of trusted components, so to speak, that lets people create that and roll it out at massive scale.
I want to be in [00:36:00] a place in five years where it's clear that we are on a path to automating all knowledge work, and we have begun to automate a serious fraction of knowledge work and at least several important parts of the economy.
Time at OpenAI
---
Bryan Davis: I'm curious, what were your takeaways from your time at OpenAI? What did you leave there feeling you needed to be done. Obviously you journeyed from working at OpenAI to starting your own company.
What did you feel like was missing from OpenAI that made you feel like there was something else you needed to.
Jesse Han: That I feel was missing at OpenAI. So OpenAI is a very special place. It's got great talent density. It's got some really motivated and hardworking people. It's got a very unique mission. There, there aren't many places in the world that have a credible claim to being the potential locus of AGI with as much real world impact as they've been able to achieve.
That being said, by virtue of the clarity and focus of their mission there are many paths [00:37:00] that they are unable to take because they're not on the critical path to AGI. And I think that the emergence of this technology created the opportunity for many trillion dollar companies. And while OpenAI is perhaps positioned to capture that value I think that it became increasingly apparent to me as I was working on this technology that there was room for many more.
And I wanted to go out and create that kind of real world. , I want it to be on the front lines. Connecting that technology to the real world and going out and automating all knowledge work. That doesn't necessarily mean scaling up to the next gigantic cluster and building GPT-5. That doesn't necessarily mean pushing the latest state of the art in theorem proving with language models.
I think what that means is doing the highest impact thing that I could for this vision [00:38:00] of trying to automate all knowledge work. And for me, that meant going out and doing Multi.
John McDonnell: What important stuff did you learn at OpenAI that helps you carry forward to, to this next journey you're doing?
Jesse Han: Things I learned? I mean I learned a lot about language models and those are sure important in our line of work. I had some really great mentors when I was there.
I think something which I was particularly inspired by was, was the strength of Ilya's conviction in whatever research program that he was championing at the time. He really taught me the value of really strong conviction, held in the right idea, compounded over a long time.
And while that doesn't sound like much, I think it's, so I think that kind of intellectual courage is really needed when you're on the forefront of something [00:39:00] like this. Because, so especially in a field like deep learning and concomitantly in the field of building startups in generative AI, there's so much happening around you at all times, there's so much flux. There are always like new papers being released, new like flashy trends being pushed by various people trying to get their careers off the ground. So many like flashy releases on Twitter that it's really important to keep your eye on the ball and to remember the principles.
From which you came to your conclusions about why you should be doing what you're doing and having the courage to pick something and to just go at it in spite of all naysayers, in spite of all setbacks, I think is really valuable.
Advice to others
---
Bryan Davis: I'm curious what your advice would be to those, perhaps new arrivals to this space, [00:40:00] either in the academia or people who are suddenly tuned in to the ChatGPT craze, people that want to use these technologies in creating new businesses and solving new problems.
What would you, what would your advice be to people who want to shift their career in this direction?
Jesse Han: Well, there's all the general advice, which is of course, work really hard, try to find some really good mentors, people who can teach you a lot. Find really ambitious people. Try to work as closely with them as you can.
In terms of like people starting companies right now, I would say again, general device, like you should swing for the fences. Cause it's so easy to pick something that's like hot. Something that might be flashy for a month or two. But I think so I think aiming for something that really endures is.... I think it's hard cuz you have to think deeply about what to work on [00:41:00] and you have to really believe in the thing that you end up doing. Yeah, if I had to sum it up my advice would be swing for the fences in every sense of the.
Bryan Davis: Love it.
John McDonnell: Yeah. It really comes across that you're doing that. Like you're gonna automate all, all knowledge work. That's like such a legit, extremely ambitious vision.
Bryan Davis: Yeah. Moonshot.
Recommendation
---
Bryan Davis: Jesse, we love to end our interviews with recommendations specifically on something to read, watch.
Is there any sort of media it could be anything from a song, a poem, a mathematical theorem that you'd recommend that our listeners sort of tune into?
Jesse Han: Song, a poem, a mathematical theorem...
one book that, that I could really recommend is...
in general, I think it's really important to remember that, it's important to remember that the dawn of computing is still in living memory, which, if you think about it, is really [00:42:00] remarkable. There are people alive today who, when they were born, you know like the Soviets were like still trying to assemble like room size computers from like glass tubes. And there was no way for this person who was like born in a Siberian Tundra to even know that something like language models would be down the road in like a sizable fraction of a century later.
One book that I found really interesting about the early history of computing and software is Steve Lohr's Go To. So there's an interesting account in there about how Ken Thompson and Dennis Richie wrote Unix in three days, cuz they were really like, Fed up with the Multics operating system and they thought why don't we just build something better with like composable components that you can pipe to each other.
So I feel like, [00:43:00] so I feel like there are moments of history where like time and technology really align and the veil grows thin, so to speak, and really talented people working together, on the right problem at the right time, can push through the curtain and build things that have incredible outsized impact.
People like Ken Thompson and Dennis Richie can build Unix in three days. And forevermore influence the history of computing. And then they can reco collaborate later on and build the B programming language and then have another crazy outsized impact on computing. And I think that so I think that we're really going through a similar period right now.
So I think that with this current wave of technology, the veil is again thin, and sufficiently talented people working [00:44:00] together, focusing on the right problem, can really go out and build something that might change the world.
So I think it's really inspiring to think about how in previous previous cycles of history, similar alignments of time and technology have enabled crazy amounts of change. And I think the dawn of computing is a great example. So the dawn of computing and software specifically.
So Steve Lohr's Go To great book.
Bryan Davis: Thanks, Jesse.
John McDonnell: Yeah, I'll check it out. Thanks so much, Jesse.
Bryan Davis: Thanks for being a part of Pioneer Park.
Jesse Han: Yeah, thanks for having me.
The limits of human-derived mathematics with Jesse Michael Han