Risk, Uncertainty & Hallucinations
Risk! Engineers Talk Governance Podcast
Season 8, Episode 4
In this episode of Risk! Engineers Talk Governance, due diligence engineers Richard Robinson and (Risk Engineer Achievement Award winner!) Gaye Francis discuss Risk, Uncertainty, and Hallucinations.
Prompted by a recent course on AI and probability, Richards exaplins how large language models actually work as giant inference engines predicting the "most likely" next word, and why that's fundamentally different from genuine risk assessment.
They explore why AI can widen the number of scenarios you can process without ever solving the harder problem of criticality — the rare, high-consequence events that haven't happened before, but still need to be found and controlled by human judgement.
If you'd like us to cover a specific topic or have any feedback we'd love to hear from you: email admin@r2a.com.au.
For further information on Richard and Gaye's consulting work with R2A, head to https://www.r2a.com.au, where you'll also find their booklets (store) and a sign-up for the quarterly newsletter to keep informed of our latest news and events.
Apto PPE is also available via the R2A online store.
Show Notes
(00:56) Richard congratulates Gaye on winning the Risk Engineer Achievement Award at the Melbourne Engineering Excellence Awards, recognising 25+ years in risk due diligence, industry publications, podcasts, bushfire and public safety work, and advocacy for women in engineering
(02:27) Founding Apto (Women's) PPE and its impact at forcing the market to offer proper female PPE
(04:30) Setting up today's topic: why people still default to thinking about risk as consequence and likelihood, rather than criticality and control
(04:53) Richard's University of Helsinki AI course and the link between AI probability weighting and Markov chains, used at R2A for availability modelling
(05:32) Different types of probability — fixed-outcome events (a coin toss) versus genuine future uncertainty (geopolitical shocks, oil markets)
(06:18) How LLMs generate "hallucinations" (or Geoffrey Hinton's term, "confabulations"), tokenising context and predicting the statistically most likely next word, sometimes inventing plausible-sounding but false attributions
(07:16–08:16) AI as a Monte Carlo-style tool: useful for running large numbers of trials fast, but this doesn't eliminate criticality, it only shrinks the pool while critical outcomes still have to be identified
(09:08–09:42) The limits of AI's training cutoff; it can't flag risks that haven't happened before or reflect a rapidly changing context; identifying criticality still requires human judgement
(10:26–11:12) The risk of the next generation treating AI output as "gospel" without questioning it
(11:16–11:41) Wrap-up: risk as future uncertainty, AI as a tool for faster insight, but criticality and control remain R2A's core focus
Episode transcript
Megan (Producer) (00:00):
Welcome to Risk! Engineers Talk Governance. In this episode, due diligence engineers Richard Robinson and Gaye Francis discuss Risk, Uncertainty, and Hallucinations.
(00:14):
We hope you enjoy the discussion. If you do, please support our work by giving us a rating and subscribing on your favourite podcast platform. And if you'd like more information on R2A, our newsletter and resources, or have any feedback or topic ideas, please head to the website, www.r2a.com.au.
Gaye Francis (00:34):
Hello Richard. Welcome to another podcast session.
Richard Robinson (00:36):
Good to be here again, Gaye.
Gaye Francis (00:37):
It is good to be here. Today we're going to talk about risk uncertainty and hallucinations and also how it sort of relates to AI a little bit.
Richard Robinson (00:48):
And life in general, depending on what things you're thinking about.
Gaye Francis (00:51):
That is true. That is true. So yes, that's our topic for today.
Richard Robinson (00:56):
Yeah. Before I start, one of the things that did happen last week was that Gaye got to be announced as Risk Engineer of the Year, or more precisely I think it was, the Risk Engineer Achievement Award at the Engineering Excellence Awards in Melbourne. And the way it was expressed, so I'll just quote it: "Gaye was chosen in recognition of her distinguished contribution to risk engineering over more than 25 years. Her leadership in risk due diligence," (and note the due diligence part) "industry publications, educational podcasts, bushfire, and public safety initiatives, and work addressing risks faced by women in engineering has had significant and lasting impact on both the professional and broader community." Actually, that's actually probably truer than you realise Gaye because you sort of get embarrassed about these things.
Gaye Francis (01:41):
I do get embarrassed by these things because it's just what I do. It's a job I love doing.
Richard Robinson (01:45):
Yeah. In that sense, you do regard it slightly hallucinatory, even though that's not exactly what the subject of this thing was about.
Gaye Francis (01:51):
True.
Richard Robinson (01:52):
But I would just point out two things, because I don't even know actually if they understood this either, but when you worked in the Powerline Bushfire Safety Task Force and the continuing committee, and I do remember when you were doing it, because you asked whoever the chair was at the time saying, "Why I exactly am I here?" And I can't remember how they exactly expressed it, but they said, "You made everything work," or something like that.
Gaye Francis (02:18):
You could bring all the technical as well as the political elements and all of that aspect together and talk about in a due diligence or a risk context.
Richard Robinson (02:27):
But the other one was the Apto (Women's) PPE thing, which you genuinely undersell because of a whole lot of different reasons. But one thing we did always agree was if Apto PPE and you and Michelle and Larisse hadn't started and done it, because remember Engineers Australia tried to do it, but it couldn't work. So you actually started the company to make it happen.
Gaye Francis (02:46):
Correct.
Richard Robinson (02:46):
But the reason for doing it wasn't so much you were expecting to be a resounding financial success, which I can confirm it has not been.
Gaye Francis (02:52):
It has not been. <laughs>
Richard Robinson (02:54):
But what was to get the market and force the market to actually change their ways and take into account female PPE, which they have done.
Gaye Francis (03:01):
And I think they have, there's so many more options for females in our industry now than there was 13 years ago when we started it. And one of those is that many of our designs, well, a couple of our designs, including our maternity range, has been copied by the big guys.
Richard Robinson (03:17):
Reverse engineered I think is the technical expression Gaye.
Gaye Francis (03:21):
So yes, that's one of the things that I am proud of. But as I said, it was an honour to be recognised for it over very many years, but also a little bit, not embarrassing, but it's humbling because it's just recognition for a job that I love doing.
Richard Robinson (03:37):
Well, yeah. And we were talking before, I mean, once upon a time, and I can speak from history in here, under the Code of Ethics of Engineers Australia, you weren't meant to be self-laudatory. It was actually against the rules. And yet the world has changed so much, has it not? And those people who were sort of, shall I put it, in the old mold tend not to push themselves forward.
Gaye Francis (03:58):
That is true. That is true. And I think this award is a reflection of the R2A team in general. Over very, very many years, including yourself and past and present people that have worked through R2A. We've always worked with a pretty good group of people.
Richard Robinson (04:17):
Yes, we have. In fact, we've always enjoyed it too.
Gaye Francis (04:19):
Yeah, absolutely. So thank you for that. But yes, we might move on to the real podcast now while my cheeks settle from being a little bit red.
(04:30):
So risk, uncertainty and hallucination. And I think one of the reasons this came up was we delivered a course last week in particular and people talking about risk and as part of this award was they're still thinking about risk in terms of consequence and likelihood.
Richard Robinson (04:47):
Rather than criticality.
Gaye Francis (04:48):
So when we say due diligence, we're really talking about criticality and control.
Richard Robinson (04:53):
Yeah. Now part of the reason for this was, I can't remember if I'd said previous podcast, but I'd done that one day course from the University of Helsinki, which if you're interested in AI, you can go and do. And it was quite interesting, but they were talking about, and they were explaining how the probability and the weightings work and all those sorts of things for AI. But I was reading something a bit later and I suddenly realised they would start talking about Markov chains. Now Markov chains is something we use really for availability modeling at different times, but you can explain all this in terms of Markov chains and the probabilities. Now that sort of brought us the conversation a bit before we started the podcast about what constitutes risk and probabilities. And one of the difficulties we've always had is that there are different types of probabilities out there.
(05:32):
If you're just talking about flipping a coin toss, there are fixed outcomes. It's heads or tails. Some people have the misfortune of it landing on its edge, but there are actually only a fixed number of outcomes. When you look at the current world and you're looking at all the oil shocks and the Houthis and the different things happening in the Middle East and the things that are going on, you're looking at a future with uncertainty like that, that is a completely different kind of uncertainty. It's not just a mathematical one or the other. And this is then explaind in one way with this hallucinations things they talk about or confabulations, as Jeffrey Hinton put it, from the AI is coming because you get a lot of people using AI more or less as truth and gospel and sometimes it's not exactly precisely what's going on.
(06:18):
And the best way to explain it, I could think of it, was that if you think about it as a series of, I don't know if your memory about Markov chains is all the best, but basically it's a whole series of probabilities. In a sense, you could think of as falteries integrating below. But what it does, if you have a statement, you say, "In this context, I want you to answer this question." So it converts all the context into tokens and then that puts it into probabilities and then it looks at what the next word's going to be in terms of the question that you answer. But that means it's just acting as a giant inference engine. So if it thinks a certain author, for example, is normally for a question in that context, then it will suggest that this is the right author. And then it will say, "Well, this is what such an author would say in this context." The fact there's no particular reference to that author and that it's actually just made it up. It's just a consequence of it actually expressing the sheer probability outcome, this is the most likely thing to happen.
Gaye Francis (07:16):
And I think that's what we talked about, wasn't it? It's that risk in its context of likelihood and consequence. It will give you your most likely outcome, a bit like Monte Carlo simulations will give you your most likely outcome.
Richard Robinson (07:29):
Yes, that's correct. So it's not an error. But see, that was one of the other things that we sort of realised because the thing actually completely, I always find it incomprehensible, is in order to create these giant probability matrices of all these interlocking markov chains and put all the weights into all the vector and the numbers like that, the sheer quantum of that information, and that's why it can actually predict or say what it believes the right inference should be. My mind just basically just doesn't actually function. But in terms of the way it's working, because one of the comments we make about doing the probability add-ups and things like that is you've got to do a number of trials in order to find what the final probability distribution is. You can send an AI agent out there now to go and do those trials for you in effect.
(08:16):
And so where we previously commented that in order to find a one in a billion chance of three probability distributions, all the long tails summing together, theoretically now you've got to tell an AI agent to go and do that many trials. Now the consequence of all this though is that it doesn't actually eliminate the criticality question because you still got that. You may have reduced the size of the criticality pool, but in terms of the critical outcomes, they still just sit there waiting for you.
Gaye Francis (08:45):
So it does two things. I think the opposite is that it expands the number of cases it can look at because it can look at more scenarios faster. So the ability to process information and data is getting easier. But you're right. What it doesn't do is get rid of or be able to identify all those critical ones, which are the process...
(09:08):
Because it's doing from the background when it was last trained. I mean, Gemma four, I think it was last trained in January 25.
(09:15):
But it's no different to saying, "Oh, well, if it hasn't previously happened, then we don't know about it."
Richard Robinson (09:20):
Yes.
Gaye Francis (09:20):
But we know in projects and in organisation and in safety issues, there are some things that can happen that haven't happened for a very long time or haven't happened before because the context and our environment is changing so much. And so I think the criticality aspect of it, we still need to think. It still requires a human to think it through.
Richard Robinson (09:42):
I think that's what we keep coming back to. And the fact that if you're just using these probability things and it's a giant inference and just say this is what ought to be the case, that hasn't been the way the world's worked.
Gaye Francis (09:54):
No. And I think it's useful data, right? It's useful insight that you can, but you got to understand the limitations of it.
Richard Robinson (10:01):
Correct.
Gaye Francis (10:01):
Which is what we say about all the risk and due diligence tools that we have. They all have different purposes and provide different insights. And depending on the question you want to answer, depends on which...
Richard Robinson (10:12):
And the context of the question you want an answer on, that's correct.
Gaye Francis (10:16):
So I think, again, AI is great. It provides some insight, but you have to have an understanding of what that does.
Richard Robinson (10:26):
Well, I think it's going to be the problem is that from our point of view, so the next generation's going to adopt AI with this enthusiasm that perhaps the rest of us don't share.
Gaye Francis (10:36):
And I think we've talked about that in another podcast. I've got younger children, well, not so young anymore....
Richard Robinson (10:43):
They were both at the Excellence Awards and they both carried themselves very well, Gaye.
Gaye Francis (10:46):
Thank you, Richard. I was very proud of them. Mark (husband) and I were very proud of them. But AI is part of their life. And so if AI pops something up to them, they're taking it as gospel. And I think that's the scary thing about it, that the information's there, but how do you interpret what's real and what's not? And I don't know that we know the answer to that.
Richard Robinson (11:12):
Well, that's why being a parent's more complicated than you might think, Gaye.
Gaye Francis (11:16):
<laughs> Thanks, Richard, for that advice. So again, risk and uncertainty. We certainly use the term risk to mean future uncertainty.
Richard Robinson (11:26):
That's the way we use it. Yes.
Gaye Francis (11:27):
That's the way that R2A uses us. We still focus on that criticality and control.
Richard Robinson (11:32):
Correct.
Gaye Francis (11:33):
But tools such as AI is helping to give you more information faster, I guess.
Richard Robinson (11:41):
And spot trends and produce.
Gaye Francis (11:44):
So it's certainly providing great insight going forward. So I think we might wrap today's podcast up. Thank you for joining us, Richard, and we'll see you next time.
Richard Robinson (11:54):
Thanks, Gaye.