How to Design Human-AI Interaction

00:00:07: Johannes Schartau: Welcome to Episode 20 of Transform Together, the Holisticon podcast

00:00:11: Johannes Schartau: meetup. I'm Johannes Schartau, and today I'm talking to Dr.

00:00:15: Johannes Schartau: Babak Hemmatian. Babak is a research associate at Stony Brook

00:00:19: Johannes Schartau: University's AI Innovation Institute.

00:00:23: Johannes Schartau: With a bachelor's degree in clinical psychology, a master's degree in

00:00:27: Johannes Schartau: computer science, and a PhD in cognitive science, he studies how

00:00:31: Johannes Schartau: people reason with the knowledge held by their communities.

00:00:34: Johannes Schartau: More recently, his research has focused on what changes when

00:00:38: Johannes Schartau: artificial intelligence becomes part of this collective process.

00:00:42: Johannes Schartau: He is the first author of the chapter Collective Cognition in Hybrid

00:00:46: Johannes Schartau: Groups, a Network Science Synthesis, appearing in the Springer Nature

00:00:51: Johannes Schartau: Handbook of Hybrid Intelligence. It looks at human-AI groups as

00:00:55: Johannes Schartau: networks and asks how their composition, connections, roles and

00:00:59: Johannes Schartau: cognitive differences shape what the group can achieve.

00:01:02: Johannes Schartau: Today we'll explore what that might mean for organizations, how we

00:01:06: Johannes Schartau: design teams, structure collaboration and decide which roles AI should

00:01:10: Johannes Schartau: and perhaps should not play. Babak, welcome to the podcast.

00:01:14: Dr. Babak Hemmatian: Thank you for having me.

00:01:16: Johannes Schartau: Could you briefly explain in layperson terms what the central question

00:01:21: Johannes Schartau: behind your research is?

00:01:22: Dr. Babak Hemmatian: Yeah. So, the world that we live in is very complex

00:01:28: Dr. Babak Hemmatian: as we realize more and more so, I would say, each

00:01:34: Dr. Babak Hemmatian: day. And we personally cannot hold all of the

00:01:39: Dr. Babak Hemmatian: information that we need to function in modern society by

00:01:45: Dr. Babak Hemmatian: ourselves. But fortunately, our minds have evolved

00:01:51: Dr. Babak Hemmatian: to be able to very naturally incorporate

00:01:57: Dr. Babak Hemmatian: information that other people hold.

00:02:01: Dr. Babak Hemmatian: Be it, for example, checking something on Wikipedia, asking a friend

00:02:06: Dr. Babak Hemmatian: for more information about something, and just incorporating that into

00:02:10: Dr. Babak Hemmatian: our daily thinking and daily lives.

00:02:14: Dr. Babak Hemmatian: But we do this so naturally and effortlessly that

00:02:21: Dr. Babak Hemmatian: lots of times we forget

00:02:26: Dr. Babak Hemmatian: where our knowledge ends and the knowledge that another source

00:02:31: Dr. Babak Hemmatian: provides to us begins. And that can result in certain

00:02:38: Dr. Babak Hemmatian: blind spots where we think we know certain things that we don't know

00:02:42: Dr. Babak Hemmatian: or we think we don't know certain things that we do.

00:02:45: Dr. Babak Hemmatian: And the key question that motivates my research is how

00:02:51: Dr. Babak Hemmatian: do we identify these types of blind spots?

00:02:54: Dr. Babak Hemmatian: How do they affect the outcomes of our thinking and our performance

00:03:00: Dr. Babak Hemmatian: on tasks? And more recently, how much that might have been changed

00:03:06: Dr. Babak Hemmatian: with the advent of AI?

00:03:10: Dr. Babak Hemmatian: Because you could think of the AI as an interactive personification

00:03:17: Dr. Babak Hemmatian: of a lot of that information that exists in the community, but

00:03:24: Dr. Babak Hemmatian: we need to succeed in the tasks that we perform.

00:03:28: Johannes Schartau: Yeah, you just mentioned it feels like AI is the technology of the

00:03:32: Johannes Schartau: moment. And it feels like there are two camps.

00:03:36: Johannes Schartau: So some people feel like AI is going to lead to this utopian future.

00:03:41: Johannes Schartau: Everybody's going to be happy. Everyone will be taken care of.

00:03:45: Johannes Schartau: And then we have a different camp. And people say it's going to lead

00:03:49: Johannes Schartau: to the extinction of humankind or something like that.

00:03:52: Johannes Schartau: And there's little in between. And so I was interested in what do we

00:03:57: Johannes Schartau: actually know about how people interact with AI today?

00:04:03: Dr. Babak Hemmatian: I would say less than we should.

00:04:07: Dr. Babak Hemmatian: But that's partly not a matter of the effort that is being spent,

00:04:12: Dr. Babak Hemmatian: but more a matter of where the effort is being spent.

00:04:16: Dr. Babak Hemmatian: So a lot of the recent research looking at humans

00:04:22: Dr. Babak Hemmatian: relative to AI has taken a kind of like, you know, evaluating

00:04:28: Dr. Babak Hemmatian: competence approach, where you know that humans can do

00:04:34: Dr. Babak Hemmatian: certain things.

00:04:38: Dr. Babak Hemmatian: We give a human a task, we give the AI a task, we see how well they do

00:04:43: Dr. Babak Hemmatian: it. And then we say, oh, AI is not as creative as humans or

00:04:49: Dr. Babak Hemmatian: AI is more creative than humans.

00:04:51: Dr. Babak Hemmatian: But the issue that I have with

00:04:57: Dr. Babak Hemmatian: that kind of work is that a lot of that is not how AI is actually

00:05:02: Dr. Babak Hemmatian: being used. AI is being much more interactively used.

00:05:06: Dr. Babak Hemmatian: So what matters a lot more is how do people actually

00:05:12: Dr. Babak Hemmatian: interact with the AI?

00:05:15: Dr. Babak Hemmatian: What kinds of attributes in the humans and what kinds of attributes in

00:05:19: Dr. Babak Hemmatian: the AI make the results of those collaborations better?

00:05:22: Dr. Babak Hemmatian: And this is the area where we have less knowledge, but we

00:05:28: Dr. Babak Hemmatian: are starting to learn a bit more. Like, for example, what happens to

00:05:33: Dr. Babak Hemmatian: the degree of trust that people have when

00:05:38: Dr. Babak Hemmatian: the number of AI agents with the network increases?

00:05:45: Dr. Babak Hemmatian: Or is it better to give AI feedback on

00:05:51: Dr. Babak Hemmatian: its performance or to ask it, you know, clarifying questions?

00:05:56: Dr. Babak Hemmatian: Or these types of questions that get more at the dynamic

00:06:03: Dr. Babak Hemmatian: that evolves between the human and the AI.

00:06:06: Dr. Babak Hemmatian: I would say we've just scratched the surface over the last, you know,

00:06:11: Dr. Babak Hemmatian: few years of what we should know at that intersection.

00:06:17: Johannes Schartau: Could you go a bit more deeply into any kind of different strengths

00:06:22: Johannes Schartau: that either humans or the AI has or any particular blind spots that we

00:06:27: Johannes Schartau: know about, especially when we combine the two, what are possibilities

00:06:32: Johannes Schartau: and what are difficulties?

00:06:33: Dr. Babak Hemmatian: One good way of thinking about both humans and the AI

00:06:39: Dr. Babak Hemmatian: agents is that there are three types of cognitive and by cognitive, I

00:06:45: Dr. Babak Hemmatian: mean just information processing systems that are important for how

00:06:51: Dr. Babak Hemmatian: well we perform a task. One of them is memory that is just holding on

00:06:57: Dr. Babak Hemmatian: to information. One of them is attention.

00:07:02: Dr. Babak Hemmatian: What do you keep an active eye on and, you know, adaptively

00:07:08: Dr. Babak Hemmatian: respond to? And the third one is reasoning.

00:07:12: Dr. Babak Hemmatian: How do you think? How do you think through, for example, different

00:07:16: Dr. Babak Hemmatian: solutions to a problem? If you look at these, you know, one at a time,

00:07:22: Dr. Babak Hemmatian: you immediately see strengths and weaknesses in both the human part

00:07:27: Dr. Babak Hemmatian: and the AI. And that's the important part. In terms of memory, the

00:07:30: Dr. Babak Hemmatian: amount of information that the AI agent has access to is

00:07:36: Dr. Babak Hemmatian: immense. Humans simply cannot compete with that.

00:07:41: Dr. Babak Hemmatian: But sometimes being able to forget can actually be adaptive.

00:07:47: Dr. Babak Hemmatian: And I think this is adaptive forgetting is something that humans do

00:07:52: Dr. Babak Hemmatian: very well. I feel like a lot of people who have worked a lot with AI

00:07:56: Dr. Babak Hemmatian: have had those types of situations where the AI initially gets a task

00:08:01: Dr. Babak Hemmatian: wrong. And you keep trying to correct it, but it has difficulty

00:08:07: Dr. Babak Hemmatian: forgetting what it did earlier. Even if you start a new chat with it,

00:08:13: Dr. Babak Hemmatian: it's going to look in the memories, look in the other chats, and bring

00:08:17: Dr. Babak Hemmatian: those wrong ideas back into the current conversation.

00:08:21: Dr. Babak Hemmatian: And that is an example of a situation where the forgetting is not

00:08:27: Dr. Babak Hemmatian: happening. But overall, the AI has an edge in terms of

00:08:33: Dr. Babak Hemmatian: memory. In terms of attention, again, the AI has

00:08:38: Dr. Babak Hemmatian: more of an edge just because there is this thing called working

00:08:42: Dr. Babak Hemmatian: memory, which means the number of things that you can simultaneously

00:08:48: Dr. Babak Hemmatian: pay active attention to. And in humans, this is very limited.

00:08:53: Dr. Babak Hemmatian: On average, this is four. So you can think about four different pieces

00:08:59: Dr. Babak Hemmatian: of information at the same time. Well, the AI is very different,

00:09:04: Dr. Babak Hemmatian: right? It could, if it has enough computing capacity, it

00:09:10: Dr. Babak Hemmatian: could simultaneously pay attention to a thousand different channels.

00:09:13: Dr. Babak Hemmatian: But one thing that humans do better with respect to attention is that

00:09:19: Dr. Babak Hemmatian: our attention is very holistic. So we might look at something

00:09:25: Dr. Babak Hemmatian: and without even realizing it, our brains are evaluating that in,

00:09:31: Dr. Babak Hemmatian: you know, 30, 40 different dimensions.

00:09:33: Dr. Babak Hemmatian: And the moments when you might notice this is

00:09:39: Dr. Babak Hemmatian: when the AI, for example, generates something.

00:09:44: Dr. Babak Hemmatian: And you look at it and you say, wait a moment, this aspect

00:09:50: Dr. Babak Hemmatian: of it does not feel right.

00:09:54: Dr. Babak Hemmatian: And that aspect might be a dimension of the task that you hadn't laid

00:09:59: Dr. Babak Hemmatian: out for the AI as important.

00:10:02: Dr. Babak Hemmatian: And the AI may not have identified as important while it was

00:10:07: Dr. Babak Hemmatian: generating something either. But that's a result of your attention

00:10:12: Dr. Babak Hemmatian: being very holistic and considering intuitively a lot of different

00:10:18: Dr. Babak Hemmatian: dimensions and how they fit together. And the last part for reasoning,

00:10:24: Dr. Babak Hemmatian: again, the capacity of the AI is much more.

00:10:28: Dr. Babak Hemmatian: If you try to think through an idea in a minute, you can go through

00:10:33: Dr. Babak Hemmatian: just so much in a minute. But an AI agent can do maybe

00:10:39: Dr. Babak Hemmatian: a thousand steps of thinking in that same amount of time.

00:10:43: Dr. Babak Hemmatian: But here, part of the thing that humans do really well is

00:10:48: Dr. Babak Hemmatian: keeping important threads of thoughts alive, you know, through a

00:10:54: Dr. Babak Hemmatian: long thinking process. Now, the AI agents are getting better at this,

00:10:59: Dr. Babak Hemmatian: but they still have this tendency for the meaning collapsing

00:11:05: Dr. Babak Hemmatian: in on itself. If they think too much in too many steps or if they

00:11:10: Dr. Babak Hemmatian: converse between themselves, different AI agents, for too long.

00:11:17: Dr. Babak Hemmatian: And then all of the conversation, all of the ideas become very

00:11:22: Dr. Babak Hemmatian: similar, become the same. And that is sometimes called model collapse,

00:11:28: Dr. Babak Hemmatian: sometimes called semantic collapse. But essentially, the conversation

00:11:34: Dr. Babak Hemmatian: content-wise, collapsing in on itself.

00:11:38: Dr. Babak Hemmatian: That's what happens. So, and the other thing that humans do very well

00:11:43: Dr. Babak Hemmatian: with reasoning is setting, you know, different hierarchies of goals

00:11:49: Dr. Babak Hemmatian: for the task. And keeping those in mind and, you know, holding on

00:11:54: Dr. Babak Hemmatian: to them. Because AI agents, sometimes, part of the work that I've done

00:12:00: Dr. Babak Hemmatian: has shown that AI agents can have difficulty holding on to the

00:12:04: Dr. Babak Hemmatian: personas that we assign to them.

00:12:07: Dr. Babak Hemmatian: Or the values, there's something called value drift, which means the

00:12:13: Dr. Babak Hemmatian: goals that we assign that might start changing a bit as a

00:12:18: Dr. Babak Hemmatian: conversation.

00:12:18: Johannes Schartau: Right. I'd like to quote from a paper. So, it says, over the next five

00:12:23: Johannes Schartau: to ten years, as agentic AI is woven ever more deeply into human

00:12:26: Johannes Schartau: groups, the structural choices catalogued here, the composition of

00:12:30: Johannes Schartau: teams, the placement of people, the design of the human-AI interface,

00:12:34: Johannes Schartau: and the pacing of convergence will increasingly determine whether a

00:12:38: Johannes Schartau: hybrid collective amplifies human intelligence or quietly erodes it.

00:12:41: Johannes Schartau: Could you walk us through the potential but also the risks that we

00:12:46: Johannes Schartau: should pay attention to as we integrate AI into our working

00:12:50: Johannes Schartau: environments?

00:12:50: Dr. Babak Hemmatian: And so, lots of times, lots of times, the potentials and the risks are

00:12:56: Dr. Babak Hemmatian: kind of mirror images of one another.

00:12:58: Dr. Babak Hemmatian: And just to give you an example of this by going through those

00:13:03: Dr. Babak Hemmatian: different elements, you know, one by one. And if you think of the

00:13:07: Dr. Babak Hemmatian: composition of a group, we now have some evidence that, let's say,

00:13:12: Dr. Babak Hemmatian: a group of people are debating in an online group and they're trying

00:13:17: Dr. Babak Hemmatian: to come to consensus on a debated topic.

00:13:21: Dr. Babak Hemmatian: It has been shown that if you add a few AI

00:13:26: Dr. Babak Hemmatian: agents to that group, it helps the process of moving towards

00:13:31: Dr. Babak Hemmatian: consensus. And the reason is because they are very good

00:13:36: Dr. Babak Hemmatian: at aggregating information and noting the different points of views

00:13:42: Dr. Babak Hemmatian: that have been voiced. But at the same time, if the ratio of

00:13:48: Dr. Babak Hemmatian: the AI agents to humans becomes too high, the humans

00:13:53: Dr. Babak Hemmatian: lose trust and engagement goes down.

00:13:57: Dr. Babak Hemmatian: And the humans also lose their capacity or motivation to

00:14:03: Dr. Babak Hemmatian: oversee, you know, what's happening in the network.

00:14:08: Dr. Babak Hemmatian: So, that means, you know, depending on the task at hand, there

00:14:14: Dr. Babak Hemmatian: is a balance. There's a suited spot for, you know, how much

00:14:20: Dr. Babak Hemmatian: AI should be involved and, you know, how they should be involved.

00:14:24: Dr. Babak Hemmatian: In terms of positioning of different types of people in that network,

00:14:30: Dr. Babak Hemmatian: again, it's going to really matter what the risks of the task

00:14:36: Dr. Babak Hemmatian: are. For example, if the key point is throughput, how

00:14:43: Dr. Babak Hemmatian: much can we process? You wouldn't want to make the human a gatekeeper

00:14:49: Dr. Babak Hemmatian: on, you know, what information the AI has access to.

00:14:54: Dr. Babak Hemmatian: Because then you run into the bottleneck of how much information the

00:14:59: Dr. Babak Hemmatian: human can process. But that human acting as a supervisor

00:15:05: Dr. Babak Hemmatian: is really going to be important if errors are really costly.

00:15:10: Dr. Babak Hemmatian: So, for example, think of clinical decision-making support systems.

00:15:16: Dr. Babak Hemmatian: If you make a mistake, your decision, that could really

00:15:21: Dr. Babak Hemmatian: hurt someone in terms of, you know, the treatment decision that's

00:15:25: Dr. Babak Hemmatian: being done. So, it's really important for the human to serve as a

00:15:30: Dr. Babak Hemmatian: supervisor and check what kind of, you know, decisions exit the

00:15:36: Dr. Babak Hemmatian: AI part of the system.

00:15:39: Dr. Babak Hemmatian: It's really an interaction between what the task is, what are the

00:15:45: Dr. Babak Hemmatian: concerns that we have. And humans and the AI agents should be placed

00:15:51: Dr. Babak Hemmatian: in those different places accordingly.

00:15:54: Johannes Schartau: So, let's say we want to set up a new team, for example, in an

00:15:58: Johannes Schartau: organization or just generally design the interaction between humans

00:16:01: Johannes Schartau: and AI. What are some of the factors that we should pay attention to?

00:16:05: Dr. Babak Hemmatian: The most important question I would say is how difficult is

00:16:11: Dr. Babak Hemmatian: the task? Are you trying to come up with new treatments, you know, for

00:16:17: Dr. Babak Hemmatian: cancer? Or are you trying to set up a pipeline that, you know,

00:16:22: Dr. Babak Hemmatian: generates somewhat edited, you know, personalized email drafts?

00:16:28: Dr. Babak Hemmatian: Those are very different tasks.

00:16:31: Dr. Babak Hemmatian: And the reason why that really matters is because

00:16:37: Dr. Babak Hemmatian: you want to set up the system to try harder

00:16:43: Dr. Babak Hemmatian: to find the really best answer if the task is

00:16:48: Dr. Babak Hemmatian: more difficult. And also, again, I alluded to this

00:16:54: Dr. Babak Hemmatian: in my previous response. It also matters how costly mistakes are.

00:17:01: Dr. Babak Hemmatian: So, if a mistake is really costly, for example, you have

00:17:07: Dr. Babak Hemmatian: an AI in the human system that is in a biofoundry that is trying to

00:17:12: Dr. Babak Hemmatian: set up an experiment that's going to cost a million dollars.

00:17:16: Dr. Babak Hemmatian: It's going to take, you know, weeks to run.

00:17:18: Dr. Babak Hemmatian: Then you really don't want to make any mistakes.

00:17:22: Dr. Babak Hemmatian: And that has a consequence for how you set up the network.

00:17:27: Dr. Babak Hemmatian: All of that also relates to the incentives, to the incentives in your

00:17:33: Dr. Babak Hemmatian: organization and the way those incentives are shaped by

00:17:38: Dr. Babak Hemmatian: the task. For example, if a task is really difficult, the

00:17:44: Dr. Babak Hemmatian: systems should be set up such that you allow the humans and the AI

00:17:50: Dr. Babak Hemmatian: agents to make mistakes. If they are too worried

00:17:55: Dr. Babak Hemmatian: about making any mistakes, they're going to just cling on to the best

00:18:01: Dr. Babak Hemmatian: answer that has been found so far.

00:18:04: Dr. Babak Hemmatian: And that is going to really hamper your ability to find

00:18:10: Dr. Babak Hemmatian: the best solution to a critical problem. But if the problem is a bit

00:18:15: Dr. Babak Hemmatian: easier, if errors are not so costly, then the balance shifts.

00:18:22: Dr. Babak Hemmatian: Then maybe you want to penalize mistakes more immediately and

00:18:28: Dr. Babak Hemmatian: more strongly to make the system more efficient at doing what

00:18:33: Dr. Babak Hemmatian: it's doing. So these are kind of the big high-level

00:18:39: Dr. Babak Hemmatian: questions that I would ask. Plus, how much resources do you have?

00:18:45: Dr. Babak Hemmatian: Because, of course, that's going to change the conversation about,

00:18:51: Dr. Babak Hemmatian: you know, what options are available for setting up the network.

00:18:56: Johannes Schartau: So one of the most fascinating parts in your paper is this collection

00:19:01: Johannes Schartau: of patterns that can be used for human-AI interaction.

00:19:04: Johannes Schartau: To me, it looks like at the moment what we mostly see in organizations

00:19:07: Johannes Schartau: is the human in the center or maybe like orchestrating multiple AI

00:19:13: Johannes Schartau: agents. Could you just give us some examples of other patterns that we

00:19:17: Johannes Schartau: might choose from that might be interesting in an organization?

00:19:20: Johannes Schartau: How to design this human-AI interaction?

00:19:23: Dr. Babak Hemmatian: So what you're mentioning sounds more like either, you know, human hub

00:19:28: Dr. Babak Hemmatian: or what we call human hub or recursive delegation, where I

00:19:34: Dr. Babak Hemmatian: set up a few AI agents and they might set up some AI sub-agents

00:19:39: Dr. Babak Hemmatian: to do sub-tasks, so on and so forth.

00:19:42: Dr. Babak Hemmatian: But I would actually argue that there are one or two other forms

00:19:48: Dr. Babak Hemmatian: that are very common.

00:19:51: Dr. Babak Hemmatian: And one of them is what we call the centaur.

00:19:54: Dr. Babak Hemmatian: And referring to the mythological half-human, half-horse,

00:20:01: Dr. Babak Hemmatian: you know, creature. And here the idea is that imagine I'm an employee

00:20:06: Dr. Babak Hemmatian: and I have been working with an AI agent

00:20:12: Dr. Babak Hemmatian: for months. It knows the types of tasks that I need to pursue,

00:20:18: Dr. Babak Hemmatian: the different projects that I have. And another employee at the

00:20:23: Dr. Babak Hemmatian: same company also has a separate AI agent that works

00:20:29: Dr. Babak Hemmatian: with closely. The reason why we call this centaur is because the human

00:20:35: Dr. Babak Hemmatian: and the personalized AI agent work so closely together that they kind

00:20:40: Dr. Babak Hemmatian: of merge into one unit. And there's usually communication between

00:20:46: Dr. Babak Hemmatian: these two different units under the guidance of the human.

00:20:50: Dr. Babak Hemmatian: So I, as one of the human employees, might go to the other person and

00:20:55: Dr. Babak Hemmatian: say, hey, for this task that I'm doing, I need X or Y,

00:21:01: Dr. Babak Hemmatian: and that person is going to work closely with their AI agent to

00:21:05: Dr. Babak Hemmatian: generate that. But there are a lot of other setups that

00:21:12: Dr. Babak Hemmatian: have either been explored in research or in particular domains.

00:21:17: Dr. Babak Hemmatian: And examples of these are, you know, AI moderators

00:21:23: Dr. Babak Hemmatian: and AI information brokers. So AI moderators might do

00:21:29: Dr. Babak Hemmatian: this thing where they would say, I noticed that you're working on this

00:21:34: Dr. Babak Hemmatian: particular task. Mary, another employee in the company, is

00:21:40: Dr. Babak Hemmatian: doing something that is quite related. You should go and talk to Mary.

00:21:46: Dr. Babak Hemmatian: And that is an example of essentially the AI determining the

00:21:52: Dr. Babak Hemmatian: connections between the different people in the network.

00:21:55: Dr. Babak Hemmatian: Or it might work as a kind of data broker.

00:22:01: Dr. Babak Hemmatian: Taking information from work that it has done with me and feeding that

00:22:06: Dr. Babak Hemmatian: to another person in the organization to develop the task that,

00:22:12: Dr. Babak Hemmatian: you know, they can do. And you can imagine how helpful this can be.

00:22:17: Dr. Babak Hemmatian: Lots of times in organizations, you have these kinds of situations

00:22:23: Dr. Babak Hemmatian: where something does not move along.

00:22:26: Dr. Babak Hemmatian: Because I am not exactly sure what other people

00:22:31: Dr. Babak Hemmatian: are doing, what they have already accomplished.

00:22:34: Dr. Babak Hemmatian: I might reinvent the wheel. Or I might generate something that is

00:22:39: Dr. Babak Hemmatian: incompatible with something that somebody else generates.

00:22:42: Dr. Babak Hemmatian: But I think part of the reason why these approaches so

00:22:49: Dr. Babak Hemmatian: far have remained more within research and have not really spread in

00:22:54: Dr. Babak Hemmatian: companies is because there needs to be, you know, clear-cut governance

00:23:00: Dr. Babak Hemmatian: structures in place. For, you know, clear-cutting, for, you know,

00:23:04: Dr. Babak Hemmatian: clear-cutting, for, you know, the relative privacy of the information

00:23:07: Dr. Babak Hemmatian: that each of the employees needs to do their own job

00:23:12: Dr. Babak Hemmatian: properly. Or with whom the responsibility sits if

00:23:18: Dr. Babak Hemmatian: the AI gets these types of tasks wrong.

00:23:22: Dr. Babak Hemmatian: So I think the reason why people have been using the Centaur more

00:23:27: Dr. Babak Hemmatian: often is because the responsibility is quite clear there.

00:23:30: Dr. Babak Hemmatian: People are taking advantage of the AI agents, but the responsibility

00:23:36: Dr. Babak Hemmatian: very clearly sits with the person that has turned into a unit

00:23:42: Dr. Babak Hemmatian: of the AI they work with.

00:23:44: Johannes Schartau: The companies that I work with at the moment, they all have some form

00:23:48: Johannes Schartau: of AI initiative. And when you ask what it's supposed to do, then they

00:23:53: Johannes Schartau: say, essentially, we want to have more efficiency or maybe some kind

00:23:58: Johannes Schartau: of, like, more productivity or something like this. What kind of other

00:24:01: Johannes Schartau: improvements could we imagine when we use AI in an organization?

00:24:05: Johannes Schartau: So you just mentioned, for example, different ways of making

00:24:08: Johannes Schartau: information accessible. What other things could you imagine that might

00:24:12: Johannes Schartau: be interesting to organizational leaders?

00:24:14: Dr. Babak Hemmatian: Yeah, I think there's a certain anthropomorphism in thinking about

00:24:19: Dr. Babak Hemmatian: the AI where that feeds into this simplified approach

00:24:26: Dr. Babak Hemmatian: of, okay, you now have access to AI.

00:24:30: Dr. Babak Hemmatian: If there was something that I expected you to be able to do within a

00:24:35: Dr. Babak Hemmatian: week, now you're going to do the same thing, but you're going to do it

00:24:39: Dr. Babak Hemmatian: in two days. And I think that really misunderstands

00:24:45: Dr. Babak Hemmatian: a lot of the opportunities that exist here.

00:24:48: Dr. Babak Hemmatian: And I'm just going to note a few of the possibilities.

00:24:53: Dr. Babak Hemmatian: One thing that the AI is very capable of doing

00:24:59: Dr. Babak Hemmatian: is serving as a mentor for developing new

00:25:04: Dr. Babak Hemmatian: skills, new hands-on skills. Think of someone who wanted

00:25:11: Dr. Babak Hemmatian: to learn, you know, advanced programming a few years

00:25:16: Dr. Babak Hemmatian: ago. They would have to initially have some sort of human

00:25:22: Dr. Babak Hemmatian: mentor. And then later, they would still spend half of their time

00:25:27: Dr. Babak Hemmatian: online on the website, Stack Overflow, just looking at other responses

00:25:33: Dr. Babak Hemmatian: to questions and seeing if they can solve their own problem.

00:25:36: Dr. Babak Hemmatian: But the AI can serve as a kind of tireless mentor.

00:25:43: Dr. Babak Hemmatian: But whenever you get stuck in the task, it can give

00:25:49: Dr. Babak Hemmatian: you just the right amount of help that you develop the

00:25:55: Dr. Babak Hemmatian: correct intuitions for expertise in that area.

00:25:58: Dr. Babak Hemmatian: But you also don't get stuck.

00:26:01: Dr. Babak Hemmatian: So it can be really powerful for helping,

00:26:07: Dr. Babak Hemmatian: you know, the workforce adapt to new skills that, you know, jobs

00:26:13: Dr. Babak Hemmatian: and situations require. The other thing that it's immensely

00:26:19: Dr. Babak Hemmatian: powerful for is just collecting and

00:26:25: Dr. Babak Hemmatian: aggregating an incredible amount of information.

00:26:30: Dr. Babak Hemmatian: And I think the way this has lots of times gone wrong

00:26:36: Dr. Babak Hemmatian: when people have tried to implement this is that if you're

00:26:42: Dr. Babak Hemmatian: trying to aggregate more information than a human could ever process

00:26:47: Dr. Babak Hemmatian: in their lifetime, you're going to have to put some, you're

00:26:54: Dr. Babak Hemmatian: going to have to have a very good list of constraints on what counts

00:26:57: Dr. Babak Hemmatian: as relevance. And what the goals are.

00:27:00: Dr. Babak Hemmatian: And what the form of the deliverable, you know, summary of that

00:27:06: Dr. Babak Hemmatian: information is. Because, and this is not just a matter of

00:27:12: Dr. Babak Hemmatian: us being able to understand the summary that the AI generates, but

00:27:17: Dr. Babak Hemmatian: it also matters for how well the AI can generate that type of summary.

00:27:23: Dr. Babak Hemmatian: So that's the second thing that I would highlight.

00:27:29: Dr. Babak Hemmatian: And the third thing I would say is essentially a mapping between what

00:27:35: Dr. Babak Hemmatian: the task is and what the correct level of AI involvement and,

00:27:41: Dr. Babak Hemmatian: you know, model, you know, a strength for the lack of a better word.

00:27:47: Dr. Babak Hemmatian: And that alignment needs to be better.

00:27:51: Dr. Babak Hemmatian: This is, and the example that comes to mind for that process going

00:27:57: Dr. Babak Hemmatian: terribly wrong is the relatively recent initiatives where the

00:28:03: Dr. Babak Hemmatian: companies encouraged people to use more AI tokens.

00:28:06: Dr. Babak Hemmatian: And then people were token maxing.

00:28:09: Dr. Babak Hemmatian: And that really misunderstands what the

00:28:15: Dr. Babak Hemmatian: technology does. So essentially the idea

00:28:21: Dr. Babak Hemmatian: isn't that for everything we need, we just need more of the AI.

00:28:26: Dr. Babak Hemmatian: The idea is to use it specifically for

00:28:31: Dr. Babak Hemmatian: those tasks where it's incredible amount of memory.

00:28:37: Dr. Babak Hemmatian: It's incredible capacity for attention.

00:28:40: Dr. Babak Hemmatian: It's ability to go through a thousand steps of a reasoning process.

00:28:46: Dr. Babak Hemmatian: We want to get the most out of that for the tasks that need that the

00:28:52: Dr. Babak Hemmatian: most. And I think that fit is where a

00:28:58: Dr. Babak Hemmatian: lot of improvement can be made to how AI

00:29:03: Dr. Babak Hemmatian: is actually applied in organizational settings.

00:29:06: Johannes Schartau: Just talking a bit more about this topic of efficiency, because that's

00:29:11: Johannes Schartau: usually what most organizational leaders really care about.

00:29:15: Johannes Schartau: What I found interesting was that in your paper, you highlight that

00:29:19: Johannes Schartau: maybe duplication and redundancy is a good idea.

00:29:23: Johannes Schartau: For example, you talk about repeated explanations and cross-checking.

00:29:27: Johannes Schartau: That is what keeps human-AI collaboration accurate.

00:29:31: Johannes Schartau: And I was wondering, could you explain a bit more what kind of

00:29:35: Johannes Schartau: inefficiencies might be beneficial in an organization that might be

00:29:40: Johannes Schartau: counterintuitive? That we tend to try to avoid these, but actually

00:29:45: Johannes Schartau: would need more of these because we are dealing with a different

00:29:49: Johannes Schartau: situation where we have human-AI interaction.

00:29:51: Dr. Babak Hemmatian: So I think there are two different levels

00:29:58: Dr. Babak Hemmatian: at which this efficiency principle really comes into focus.

00:30:03: Dr. Babak Hemmatian: One is in how people individually just interact with

00:30:09: Dr. Babak Hemmatian: the AI. We as humans, we are very good at capturing,

00:30:14: Dr. Babak Hemmatian: you know, the common ground of a conversation.

00:30:18: Dr. Babak Hemmatian: The shared context of things that I would expect you as

00:30:24: Dr. Babak Hemmatian: a fellow human being to know.

00:30:27: Dr. Babak Hemmatian: And as a result, I would not, you know, belabor that part of my point.

00:30:31: Dr. Babak Hemmatian: As a result, lots of times when humans talk to each other, we are

00:30:36: Dr. Babak Hemmatian: pretty efficient. Don't talk much.

00:30:39: Dr. Babak Hemmatian: We say more things in fewer words.

00:30:45: Dr. Babak Hemmatian: And that's also partly because speaking takes effort and time for us.

00:30:51: Dr. Babak Hemmatian: Lots of times people, when they try to interact with AI agents,

00:30:57: Dr. Babak Hemmatian: bring the same type of intuition to working with the AI.

00:31:02: Dr. Babak Hemmatian: But the AI does not have the exact same context as you and me.

00:31:07: Dr. Babak Hemmatian: It has not lived in our world, you know, with jobs, with,

00:31:13: Dr. Babak Hemmatian: you know, our multi-sensory experiences, with, you know, all of the

00:31:18: Dr. Babak Hemmatian: nuances of human culture and, you know, social interactions that

00:31:23: Dr. Babak Hemmatian: you're familiar with. All the information it has access to is just the

00:31:29: Dr. Babak Hemmatian: linguistic conversations that humans have.

00:31:34: Dr. Babak Hemmatian: But sometimes people end up giving it not enough context to be

00:31:39: Dr. Babak Hemmatian: able to do what it needs to do.

00:31:41: Dr. Babak Hemmatian: You know, there is this community on Reddit that's named Explain It

00:31:46: Dr. Babak Hemmatian: Like I'm Five. And I think a similar approach should be

00:31:52: Dr. Babak Hemmatian: taken to working with AI. You want to be really concrete, you know,

00:32:00: Dr. Babak Hemmatian: what the context of the task is, what the goals are, what the rules

00:32:05: Dr. Babak Hemmatian: are, at which points in the process it should stop and,

00:32:11: Dr. Babak Hemmatian: you know, check in with the human.

00:32:14: Dr. Babak Hemmatian: What kind of information should check in about.

00:32:18: Dr. Babak Hemmatian: Lots of times that actually the way that works best is

00:32:24: Dr. Babak Hemmatian: if you do this at the very beginning of starting on

00:32:30: Dr. Babak Hemmatian: a project with the AI agent.

00:32:32: Dr. Babak Hemmatian: And this is the other sense in which I think the inefficiency might

00:32:38: Dr. Babak Hemmatian: be a bit counterintuitive for people in human organizations where

00:32:44: Dr. Babak Hemmatian: you feel like if a task can be done in two days,

00:32:50: Dr. Babak Hemmatian: why should I stop on doing the actual tasks involved

00:32:57: Dr. Babak Hemmatian: in it for an extra day or two?

00:33:02: Dr. Babak Hemmatian: But lots of times when working with AI, it really pays off

00:33:08: Dr. Babak Hemmatian: to spend a day or two beforehand.

00:33:11: Dr. Babak Hemmatian: Create a constitution, if you will, for the project.

00:33:16: Dr. Babak Hemmatian: A detailed document, this is going to be a living document that lays

00:33:20: Dr. Babak Hemmatian: out all of those aspects of the context that are important.

00:33:24: Dr. Babak Hemmatian: Have it go through those, ask your questions about any part that is

00:33:29: Dr. Babak Hemmatian: unclear. And once all of that is figured out, get us started on the

00:33:34: Dr. Babak Hemmatian: actual tasks. And the counterintuitive aspect of it is that that

00:33:39: Dr. Babak Hemmatian: usually ends up making the whole process much shorter.

00:33:44: Dr. Babak Hemmatian: Because it makes it much less likely that you're going to have to redo

00:33:50: Dr. Babak Hemmatian: and redo what the AI does. And this ties into the point I

00:33:56: Dr. Babak Hemmatian: made earlier about adaptive forgetting.

00:34:00: Dr. Babak Hemmatian: If the AI gets the task wrong initially, it's

00:34:06: Dr. Babak Hemmatian: lots of times much more difficult than with the human for you to tell

00:34:12: Dr. Babak Hemmatian: it to set those ideas aside. And think about this problem in this

00:34:17: Dr. Babak Hemmatian: completely new way that is more accurate. Which is why spending that

00:34:22: Dr. Babak Hemmatian: time upfront to make sure that the shared context is

00:34:27: Dr. Babak Hemmatian: correct with the AI agent is going to be really important.

00:34:33: Johannes Schartau: I find this so fascinating that we're still absolutely learning how

00:34:37: Johannes Schartau: this technology works. And it's really, in some ways, very familiar to

00:34:41: Johannes Schartau: the way we interact. In other ways, it's very different.

00:34:44: Johannes Schartau: I thought it would be fun just to speculate a bit.

00:34:46: Johannes Schartau: So let's imagine we as humans have gotten much better at using that

00:34:51: Johannes Schartau: technology. And at the same time, the models have really improved.

00:34:54: Johannes Schartau: What do you think an organization is going to look like in maybe five

00:34:58: Johannes Schartau: to ten years when AI is deeply ingrained in daily work life

00:35:04: Johannes Schartau: and we really have made steps forward?

00:35:07: Dr. Babak Hemmatian: Yeah. I think one direction that is going to certainly

00:35:13: Dr. Babak Hemmatian: pick up a speed is earlier

00:35:19: Dr. Babak Hemmatian: we talked about certain forms of organization where the

00:35:25: Dr. Babak Hemmatian: AI is not so closely tied to individual humans, but it

00:35:30: Dr. Babak Hemmatian: acts as its own type of agent.

00:35:33: Dr. Babak Hemmatian: For example, as a facilitator, as an information broker, as kind

00:35:38: Dr. Babak Hemmatian: of a first responder that takes care of a large volume of

00:35:44: Dr. Babak Hemmatian: material, but then flags more difficult cases for

00:35:50: Dr. Babak Hemmatian: humans to make decisions on.

00:35:53: Dr. Babak Hemmatian: And I think these types of situations where the AI is

00:35:59: Dr. Babak Hemmatian: working as its own type of agent with a bit less oversight are

00:36:06: Dr. Babak Hemmatian: going to become more common. And they are going to become more common

00:36:12: Dr. Babak Hemmatian: as we develop governance structures that can accommodate

00:36:18: Dr. Babak Hemmatian: them well. That can still determine when and how

00:36:24: Dr. Babak Hemmatian: their work is going to be audited.

00:36:26: Dr. Babak Hemmatian: And how does the responsibility get distributed

00:36:33: Dr. Babak Hemmatian: in the organization now that we have these kinds of relatively

00:36:37: Dr. Babak Hemmatian: independent agents running around? I think a couple of risks that

00:36:43: Dr. Babak Hemmatian: we need to be really mindful of.

00:36:45: Dr. Babak Hemmatian: One of them is the goal setting and

00:36:51: Dr. Babak Hemmatian: the kind of like hidden work that the AI agents do.

00:36:57: Dr. Babak Hemmatian: So sometimes the way that AI work goes wrong is that you give

00:37:03: Dr. Babak Hemmatian: it a task that is a, you know, pretty difficult task and say it has

00:37:08: Dr. Babak Hemmatian: like, you know, a dozen different steps in it.

00:37:12: Dr. Babak Hemmatian: Sometimes some of the steps of the task that is assigned to the AI

00:37:16: Dr. Babak Hemmatian: agent are underdetermined. They need, you know, design decisions,

00:37:22: Dr. Babak Hemmatian: creative design decisions that are not specified in the prompt.

00:37:27: Dr. Babak Hemmatian: And the AI, especially frontier models, which means the

00:37:33: Dr. Babak Hemmatian: currently most advanced models, would go ahead and create, you know,

00:37:36: Dr. Babak Hemmatian: some sort of heuristic or, you know, rule of thumb for solving that

00:37:41: Dr. Babak Hemmatian: problem. Why? Because we ask them to generate the product and they can

00:37:45: Dr. Babak Hemmatian: generate the product unless they take care of that.

00:37:48: Dr. Babak Hemmatian: But part of the risk is us not seeing those decisions.

00:37:53: Dr. Babak Hemmatian: I have found myself sometimes after the fact interrogating the

00:37:59: Dr. Babak Hemmatian: AI agent in a moment.

00:38:02: Dr. Babak Hemmatian: What kind of role did you use to choose, you know, the items for

00:38:07: Dr. Babak Hemmatian: further analysis here? And then it would say, well, I use this rule of

00:38:11: Dr. Babak Hemmatian: thumb. I said, no, that is not defensible.

00:38:15: Dr. Babak Hemmatian: You know, that's something that needs to be better determined.

00:38:20: Dr. Babak Hemmatian: So I think that's part of the worry, that they are

00:38:26: Dr. Babak Hemmatian: going to go through processes in generating the response that they

00:38:31: Dr. Babak Hemmatian: want, that they might not necessarily agree with.

00:38:34: Dr. Babak Hemmatian: But the perhaps bigger problem is, you know, the

00:38:40: Dr. Babak Hemmatian: goals that we set for the AI agents.

00:38:44: Dr. Babak Hemmatian: We see this risk in some of the recent so-called jailbreaks

00:38:51: Dr. Babak Hemmatian: that have made the news where, you know, tasks

00:38:56: Dr. Babak Hemmatian: were assigned to models in development that were, for all intents

00:39:02: Dr. Babak Hemmatian: and purposes, impossible.

00:39:05: Dr. Babak Hemmatian: And remember that these are systems that are designed to only care

00:39:10: Dr. Babak Hemmatian: about getting the task done. And what ended up happening was,

00:39:16: Dr. Babak Hemmatian: you know, they showed all sorts of worrying behavior.

00:39:20: Dr. Babak Hemmatian: You know, they shouldn't have had access to the internet.

00:39:23: Dr. Babak Hemmatian: They essentially hacked, you know, the containers, got access to the

00:39:27: Dr. Babak Hemmatian: internet. They hacked external websites.

00:39:29: Dr. Babak Hemmatian: They changed the contents of, you know, online wikis to perform

00:39:35: Dr. Babak Hemmatian: the task. And in the former case, where the AI is

00:39:41: Dr. Babak Hemmatian: not sharing part of the process with us, I would say the

00:39:45: Dr. Babak Hemmatian: responsibility is a bit less with the humans. But in this case, the

00:39:50: Dr. Babak Hemmatian: responsibility is with the humans.

00:39:55: Dr. Babak Hemmatian: So, if you set impossible goals for the AI agents,

00:40:01: Dr. Babak Hemmatian: you should expect that they are going to act in

00:40:07: Dr. Babak Hemmatian: utterly unpredictable ways.

00:40:11: Dr. Babak Hemmatian: Doing whatever they can, especially if they are part of a swarm, part

00:40:16: Dr. Babak Hemmatian: of a group of interconnected agents to get that task done.

00:40:21: Dr. Babak Hemmatian: And that, I think, ties back in again to this idea of you

00:40:27: Dr. Babak Hemmatian: need to have good governance rules and, you know, constitutions,

00:40:33: Dr. Babak Hemmatian: if you like, for particular projects. What is the space of things that

00:40:39: Dr. Babak Hemmatian: are allowed? And you need to enforce these at human-paced checkpoints.

00:40:45: Dr. Babak Hemmatian: What do I mean by that? Because these computers, they process so much

00:40:50: Dr. Babak Hemmatian: so fast. By the time human caught

00:40:56: Dr. Babak Hemmatian: the issue, you know, with those jailbreaks, it

00:41:01: Dr. Babak Hemmatian: was already too late. Now, that would have looked very different if

00:41:06: Dr. Babak Hemmatian: human checkpoints were embedded into the process.

00:41:12: Dr. Babak Hemmatian: For example, let's say every 10,000, you know, communications between

00:41:17: Dr. Babak Hemmatian: agents, the whole network is stopped and a human examines

00:41:25: Dr. Babak Hemmatian: and tries to summarize the situation. Are they doing what they should

00:41:29: Dr. Babak Hemmatian: be doing or not? And then the task continues.

00:41:33: Dr. Babak Hemmatian: And it's important for it to be human-paced because this

00:41:39: Dr. Babak Hemmatian: is not going to naturally arise in a group of AI agents in a

00:41:44: Dr. Babak Hemmatian: modern, you know, computer version of a petri dish.

00:41:50: Johannes Schartau: There's so much information about AI. You just mentioned, for example,

00:41:53: Johannes Schartau: the jailbreaks or the models going rogue.

00:41:57: Johannes Schartau: If I'm an organizational leader and I just want to kind of keep up

00:42:00: Johannes Schartau: with what is happening, do you have any tips on where I could go to

00:42:05: Johannes Schartau: find reliable information that is not just all hype?

00:42:08: Dr. Babak Hemmatian: For the frontier models, essentially, you know, the top

00:42:14: Dr. Babak Hemmatian: of the line of, you know, capabilities.

00:42:19: Dr. Babak Hemmatian: Unfortunately, because of the, you know, cross-company competition

00:42:23: Dr. Babak Hemmatian: that is currently going on, a lot of the work is becoming less public

00:42:28: Dr. Babak Hemmatian: and, you know, much more closely guarded.

00:42:31: Dr. Babak Hemmatian: So, so, so data stuff is less accessible.

00:42:35: Dr. Babak Hemmatian: But at the same time, I personally don't think those types of models

00:42:41: Dr. Babak Hemmatian: are, you know, the core of the future of AI in organizations.

00:42:47: Dr. Babak Hemmatian: Because they require an incredible amount of, you

00:42:52: Dr. Babak Hemmatian: know, resources to run. And they are suited to a

00:42:58: Dr. Babak Hemmatian: small subset of the most difficult tasks that companies could possibly

00:43:03: Dr. Babak Hemmatian: focus on. So, I think those types of models are going to end up being

00:43:09: Dr. Babak Hemmatian: kind of like boutique products. They are expensive products for

00:43:13: Dr. Babak Hemmatian: domains where, you know, the task is really hard and errors are really

00:43:18: Dr. Babak Hemmatian: expensive. So, it's worthwhile to use those.

00:43:23: Dr. Babak Hemmatian: For a lot of the other tasks, I think the trend is going to

00:43:29: Dr. Babak Hemmatian: move towards using models that are smaller, more agile, open

00:43:35: Dr. Babak Hemmatian: source models that can easily, you know, be adjusted on.

00:43:39: Dr. Babak Hemmatian: And these models are going to be adapted to the particular tasks that

00:43:43: Dr. Babak Hemmatian: the company or organization is trying to do.

00:43:47: Dr. Babak Hemmatian: And these models, there's a lot of academic research on these.

00:43:52: Dr. Babak Hemmatian: This field moves pretty quickly, but it is also openly accessible.

00:43:57: Dr. Babak Hemmatian: There are some, you know, top conferences like advances in neural

00:44:02: Dr. Babak Hemmatian: information processing systems. Or, in short, NeurIPS.

00:44:08: Dr. Babak Hemmatian: That's one of the top conferences that shares research on these types

00:44:12: Dr. Babak Hemmatian: of studies. And there are free kind of, you know, TLDR, AI

00:44:19: Dr. Babak Hemmatian: -based tools that try to aggregate the information from these

00:44:24: Dr. Babak Hemmatian: types of sources. And they would send you kind of like weekly

00:44:29: Dr. Babak Hemmatian: newsletters based on, these are the papers that came out.

00:44:33: Dr. Babak Hemmatian: These are some of the coolest ideas.

00:44:36: Dr. Babak Hemmatian: This is the trend that, for example, you know, research is moving from

00:44:41: Dr. Babak Hemmatian: this type of analysis to this type of analysis.

00:44:44: Dr. Babak Hemmatian: So I would recommend keeping abreast with the advances using

00:44:49: Dr. Babak Hemmatian: those types of tools.

00:44:54: Dr. Babak Hemmatian: Because I also think, yeah, they are going to be the really

00:44:59: Dr. Babak Hemmatian: sustainable future of AI in organizations.

00:45:04: Johannes Schartau: Thank you. This has been such a great conversation. Where can people

00:45:07: Johannes Schartau: go to learn more about you and your work?

00:45:10: Dr. Babak Hemmatian: LinkedIn would be a good place to, you know, follow and contact me.

00:45:15: Dr. Babak Hemmatian: And I usually share any new work that comes out of my research

00:45:21: Dr. Babak Hemmatian: there. But I would also recommend, you know, just to add a

00:45:26: Dr. Babak Hemmatian: plug there. I think the AI Innovation Institute at Sønberg University

00:45:32: Dr. Babak Hemmatian: is going to be very worthwhile for a lot of people to

00:45:38: Dr. Babak Hemmatian: keep in their mind. And, you know, follow the updates from the

00:45:42: Dr. Babak Hemmatian: Institute, particularly because it is designed to be broad in focus

00:45:47: Dr. Babak Hemmatian: and, you know, oriented towards applications and, you know,

00:45:52: Dr. Babak Hemmatian: collaborations with the community in its projects.

00:45:56: Dr. Babak Hemmatian: So if people have any questions or, you know, or ideas or

00:46:02: Dr. Babak Hemmatian: potential collaborations, I would welcome them to definitely either,

00:46:08: Dr. Babak Hemmatian: you know, write to me or write to the AI Innovation Institute.

00:46:14: Dr. Babak Hemmatian: So I'm not claiming we are the only people doing, you

00:46:19: Dr. Babak Hemmatian: know, this type of work. There are, of course, plenty of others.

00:46:23: Dr. Babak Hemmatian: But I also think that there's a great sense of collaboration among

00:46:29: Dr. Babak Hemmatian: people working in this type of space.

00:46:31: Dr. Babak Hemmatian: So if you reach out to one of us, we could also probably point you to

00:46:36: Dr. Babak Hemmatian: others who are doing work that might potentially even be a better fit.

00:46:43: Johannes Schartau: Great. Thank you so much for your time.

00:46:45: Dr. Babak Hemmatian: Of course. Thank you for having me.

00:46:49: Johannes Schartau: I hope this session has given you some concrete ideas for your own

00:46:52: Johannes Schartau: transformation challenges. At Holisticon, we provide companies with

00:46:56: Johannes Schartau: holistic support from strategy and technology to organizational

00:46:59: Johannes Schartau: change. If this sounds interesting, please feel free to contact us.

00:47:03: Johannes Schartau: You can find all the information you need at holisticon.de.

00:47:06: Johannes Schartau: And don't miss our next Transform Together session.

00:47:09: Johannes Schartau: I'll see you then.