In episode 31, be ready to enter the realm of Embedded Machine Learning and Tiny ML with Jan Jongboom – CTO, Edge Impulse 🧙♂️⚙️.
Edge Impulse is the leading development platform for Embedded Machine Learning, used by over 1,000 enterprises across 10,000 Machine Learning projects worldwide, and helping developers leverage the power of Machine Learning.
We start the episode by discussing what Embedded Machine Learning means for IoT and connected devices, lensing in on why people may be sceptical to deploy Machine Learning on an MCU. We then explore key use cases around ML, core differences between Embedded ML and ML-based server projects and the main challenges when it comes to EML deployment. We finish the episode analysing the role of signal processing in EML and where we can go for the future with EML!
Sit back, relax, tune in and discover:
⚙️ Background in Embedded Engineering and IoT 00:30 – 04:26
⚙️ Growth for ML companies among the pandemic 04:26 – 05:28
⚙️ Why do people tend to be sceptical about Embedded Machine Learning? 05:28– 08:19
⚙️ What are the use cases around embedded machine learning and why is it important? 08:19– 10:53
⚙️ What are the core differences between ML-based server projects and embedded machine learning? 10:53– 12:17
⚙️ The main challenges in developing Embedded Machine Learning solutions and algorithms 12:17– 14:53
⚙️ Why has embedded ML not been as widely used before? 14:53– 17:55
⚙️ How do signal processing and Embedded ML work together 17:55– 20:22
⚙️ Where can Embedded Machine Learning go in the next five years, how will this affect people in the everyday life? 20:22–23:00
⚙️ Future use cases of ML: Formula 1 Cars 23:00–24:28
⚙️ Security and Machine Learning- What effects does security risk have on Machine Learning? 24:28– 31:15
Welcome to The IoT Podcast show. Today, I’m joined by Jan Jongboom. Jan is the Co-Founder and CTO of Edge Impulse. Edge Impulse are the leading development platform for embedded machine learning, used by over 1,000 enterprises and over 10,000 projects worldwide. Jan is a technology evangelist. He has a fantastic background in embedded engineering and has written thousands of patches for open source projects. Jan, thank you so much for joining us today on the show.
Wonderful intro, Tom. Thanks so much for having me.
What is your background in Machine Learning and what does Impulse Edge do?
You’re very welcome. I had a little bit of help with the intro. We have a great research team, so you’re very welcome. Jan, could you kick off by just explaining a little bit about your background and what Edge Impulse actually does?
Yeah, absolutely. So I think it’s really cool to be on The IoT Podcast. For me, the whole reason that I started doing embedded engineering was because I saw the IoT happening from somewhere behind me, and I was like okay, this train is so cool. I want to jump on it.
So I was building connected devices already when I first heard about all of this, in 2013, but mobile phones. And at some point I wanted to have something that ran longer than a week on a battery but still connected, so the only way you could go down was the embedded route.
So started building devices, smart offices was one of the solutions that we started working on with 2015. And I thought it was so cool that I really wanted to shape the future of this, so I went to work for Arm out of Cambridge in early 2016 as a principal engineer. There, I ran a software team that was forward-looking. How do we think embedded developers are going to write software 5, 10 years from now?
And all of a sudden, I realized that while we have lots of IoT devices all around us, they do very little useful stuff. There’s so many temperature sensors that send a temperature value every hour to the clouds, but there’s so much more data around this.
So we started building some embedded machine learning things to see if we can run ML on really tiny IoT devices. And yeah, after a bit of research, we actually realized we could, but it was really hard for embedded engineers to build that, so you need a team of data scientists to really build a good solution.
So about a year and a half ago, me and my co-founder decided to leave Arm and set out to build a company that lets embedded engineers really apply embedded machine learning to [inaudible 00:02:53] projects. So as you said, it’s super widely used. We’re 15,000 projects now.
That number was hit two months ago. It’s absolutely insane, the amount of volume and projects that people start creating. But Edge Impulse is an engineering tool a bit like an IDE that helps you find the correlations in sensor data and helps you understand the world a bit better through that.
That’s fantastic. Yeah, so we share a similar background. My background is in embedded engineering at university. I went one way, you clearly went the other, but I know Arm very well. And that’s really interesting. It must be quite daunting, I suppose, as well, really, Jan, to go out and fly the nest as it were, and set up a business.
Yeah, for me, I never thought of myself as an entrepreneur, as a technologist more. I really like building new things and trying to push the limit of what’s possible.
But I think the opportunity here was so big and I see it as such a potential transformation for the way that we write software, not that it will take over everything, it’s just another tool in your toolbox, but it has a potential effect on the 30 billion MCUs that ship every year that we figured let’s actually do that and let’s try it, and it worked out tremendously well. We’re a year and a half old. We’re 25 people now. Yeah, you blink your eyes and the whole world has changed.
Yeah, absolutely. And I think that’s a really nice problem to have, isn’t it? Growth. Especially right now, through the pandemic and the struggles that other people have had in different industries, it’s really heartwarming to hear success stories.
Yeah, I think for us, it sounds a bit weird, but so we signed our lead customer in November 2019, right before COVID hit, but it’s a customer in consumer health. And all of a sudden, when pandemic hit, they realized they were sitting on lots and lots of data and were using Edge Impulse to set up points to find correlations in sleep and in sleep classification.
But all of a sudden, they realized that lots of data about people getting sick and typical health patterns, so they started using Edge Impulse also to do COVID research. So that was, for us, a really great catalyst in terms of growing.
How does Impulse Edge differ to competitors?
Yeah. And Edge Impulse, how does it differ compared to peers within the same industry?
Yeah, it’s a hard question. So on one end, I think we’re still trying to show people that embedded machine learning is possible, that a tiny MCU, the little camera module here, that a chip one square centimeter is actually capable of running some machine learning. And telling people about that, that is I think the biggest challenge.
It’s not so much about the competition at this point. The real competition we see is do-it-yourself, it’s people that say, “Well, I’ve hand-tuned my DSP algorithms for the last 30 years, I can do better than an ML algorithm. And that’s fine, actually. I think for us, we’ll show you what’s possible, we’ll always try to tell you never trust what a machine learning algorithm tells you just by face value, always have some checks, et cetera, in there. And so far, that seems to resonate quite well.
Sure. And so from a technological standpoint, so the microcontroller unit, is it because people believe that ML is usually based around quantum computing and large scale devices with lots of power, and so they’re probably a little bit confused that you can actually start doing ML on an MCU?
I do think that’s the case, and that’s where I came from. When in 2017, [Miltan 00:06:57], one of my engineers at Arm, in a bar in Taipei told me, “Hey, I got this machine learning model running on a [Freedom-K64F 00:07:03].” I was like, “No way. How do you do that? It’s 256 kilobytes of RAM.” But the thing that we realized is a machine learning algorithm is not special.
During training, it requires lots of inputs and lots of outputs, and then it finds a hidden correlation, and that’s a super expensive process, but what comes out of a machine learning model is just math, and the number of parameters that you have there, you can control, so you can set the constraints of what you have in an MCU as the boundaries of the formula that you want to find.
But in the end, what a machine learning model finds is just mathematics and MCUs are incredibly good at math. A typical [inaudible 00:07:52] can do 40 million operations a second, and that’s definitely fast enough to do … And we realized that that is actually fast enough to do real-time audio classification, image classification, stuff happening all around you. And once people see that as a realization, they’ll start realizing that you can actually do this on MCU and it’s actually feasible.
What are the use cases around Embedded Machine Learning?
Yeah. And for our listeners out there that may not understand machine learning and embedded to perhaps the degree that you do, what are the use cases around embedded machine learning and why is it important?
Yeah, good question. I think what machine learning does is it finds a hidden correlation in your data. So let’s say that you’re building a control algorithm and you want to know if water is cooking, so you have a temperature sensor, you put it in a pot of water, you notice okay, well the temperature is 102 degrees and the water is cooking.
Great, now I cool it down to 98 degrees Celsius, and now the water is not cooking. Cool. I write a control loop, if temperature is above 100 degrees, then water is cooling. Works great. Now, you deploy it to the field. Someone goes mountain climbing, he’s at two kilometers height, and he looks at it, and according to your sensor, at 98 degrees, the water should not be cooking, but you observe and it actually is. So you know there’s a correlation somewhere between the altitude that you are, the temperature, and the cooking temperature of water.
Machine learning can help you unravel that hidden correlation. So it’s an engineering tool that says well, I have lots of input data, and I know what the outcome should be, but I can’t find this correlation myself, I can’t write this control loop myself, then machine learning can help you uncover that.
So you can do that on very simple use cases, so everywhere where currently you have a control loop where data comes in, and you have thresholds set up, and you want to respond to that, that’s great, but you can also do much higher … you can also do things where a human can’t really see this correlation. So audio classification. Do we hear glass breaking, yes or no? Really hard to program it out if you have a wave form, relatively easy for machine learning.
Image classification. So we have a super cool project with Smart Parks where they have a solar powered camera that takes a snapshot of the savannah, and when it sees an elephant or a poacher, it sends a signal over a satellite, because then a ranger should take a look.
And then a super wide variety of predictive maintenance and machine monitoring use cases, like does this machine oscillate differently than all the other machines in my factory, et cetera? So anywhere where you have lots and lots of data, you know that there’s somewhere a hidden correlation and you can determine it yourself but it’s too difficult to write by hand, that’s where machine learning helps.
What is the differences between Machine Learning and Embedded Machine Learning?
Yeah. And would you say that was the main difference between ML based, more service side projects and actually embedded machine learning? That real time useful information that you can get from sensor driven data?
Yeah, it’s also a bit of a personal preference, I think. I think the world around me is so incredibly fascinating, so I want to write algorithms and I want to write devices that actually interact with that environment. And a lot of the way to machine learning has grown was in server farms because they were dealing with data that was already on the server farm, like is there fraud on my credit card transaction, yes or no? And that is really easy to scale.
If you need a larger algorithm, you spin up a larger virtual machine in the cloud and you run it there. The interesting part with ML in the IoT sense is that you’re very constrained by what a chip can do that’s sitting on there, what your MCU can do. If my model doesn’t fit on this MCU, then I either need to redesign, or I can’t make my bomb, or anything like that. For me, it’s much more interesting, and the sensor data all around us, and constrained environments, which is always fun.
What are the challenges for Embedded Machine Learning?
Yeah, absolutely. And I guess on that note, Jan, there must be quite a lot of challenges in developing embedded solutions for ML projects. What are the main things that you’ve come across since starting Edge Impulse?
So the main challenge that we … I think the hard part with machine learning algorithms is that there’s always some sort of black box component, because yeah, you can ask the machine to find you a magic formula that classifies things, but can you trust it?
And that is important already if you deploy this in a server farm, we’ve seen lots of discrimination by ML algorithms, and it’s really easy to point at the computer and say the computer says no, and that’s really, really bad, but this gets even worse if you start deploying that in millions of devices where you don’t even have control over the algorithm anymore.
If I ship an ML algorithm in the control loop of a industrial system, I want to be very, very sure that it behaves exactly the way it should, even considering all the edge cases. So we try to help a little bit with that. So we try to visualize the feature space of what we have covered. So we can tell the developers, say, “Hey, there’s this whole class of potential data that you haven’t uploaded or collected in your dataset.
Maybe you should do something about it.” And we have some safeguards here. So we never trust the output of a neural network just by the face value because neural networks are very unpredictable. They work really well on data that looks like anything they’ve seen before, but very poorly, kind of random, at data they’ve never seen before. So we typically do stuff like anomaly detection models next to it to say oh, this looks like data I’ve never seen before. I’m not going to trust the rest of my ML algorithm.
I’m going to alert an operator or send something off. And that explainability is I think really important if you’re going to deploy this in embedded systems, because once you put it out there, it’s out there, you can very often never change it. There’s lots of device that are never connected. And last thing is if you have a safety critical feature, just write it out by hand, please. Even as a last safety guard. And I think that’s really important. It’s an engineering tool. Don’t treat it as a black box.
Why has Embedded ML not been as widely done before?
Yeah, got it. And it’s interesting because one of the first questions I asked you was how does your business differ from peer groups, and you mentioned that really there isn’t many peers, there isn’t many people doing this.
That’s curious because it seems obvious. And I think the age I would add it is it’s obvious when you know how, it’s easy when you know how. Why do you think that you’re quite pioneering within Edge Impulse in putting ML on embedded devices and on MCUs? And why has it not been done before?
Yeah, so it’s been done a little bit, but only by really large themes that consist of both embedded experts, as well as experts on the ML side, as well as data scientists that know how to design for constraint systems because we probably all have a device in our pocket that does embedded ML all the time.
The chip that’s responsible to respond to Hey Siri or Okay Google does not run on the main application processor, it’s a separate DSP, really small side processor, that runs all the time and just listens for the keywords. But yeah, only Google, and Apple, and Amazon have been able to really build systems that work really well in 20K of RAM.
And we realized that when we were at Arm, and the only thing, we had this amazing research team that was capable of making the actual math run incredibly fast on normal MCUs, and then the only demo that we could basically get together was oh, here’s the [inaudible 00:16:23] network, here’s some predefined inputs, and I will print out a serial or we’ll print the outputs.
And that’s because if you have this Venn diagram of data scientists know how to build these models here, and then you have embedded engineers that know how to build embedded systems here. There’s no overlap.
There’s virtually no-one that can do both of these things, but it’s required to have these two people in the same room and speak the same language to build a successful system.
So what we tried to do at Edge Impulse is say well, we’re going to skip the data scientist here, so we’re going to have a bunch of … so we’re going to talk about this from the perspective of the embedded engineer. So we always start with signal processing first.
We’ve been doing on device learning or on device analysis for a long time, we just called it the signal processing. Let’s do that first and then let’s build some tools that help you get some of this data science knowledge into your own little bubble. And that allows embedded engineers to apply their knowledge to their problems. And that is much more useful.
Then you don’t need to have to this translation layer, or the steam of PhDs or the data scientists sitting next to you to help build that. And yeah, we’re early with that, but I do think that this is going to be a tool that every embedded developer will have under his belt 5 to 10 years from now because it’s so useful. But we’re super early, absolutely.
What is signal processing?
Yeah, and that’s useful that you mentioned signal processing, because clearly this is key to embedded ML, isn’t it? And again, for the benefit of our listeners that don’t perhaps understand signal processing in the depth that you do that maybe I understand a little bit, could you go into a bit more around that?
Yeah, sensor data typically is very noisy. So let’s say I have a rotating machine, and I have a sensor on that, and that measures the acceleration. So if it’s rotating perfectly, I’ll probably see a graph like this, a sinus wave. So if I want to know how fast that machine is rotating, [inaudible 00:18:36].
So you have to sin wave, but it’s noisy because well, there’s lots of other stuff happening with the machine as well. So first I want to de-noise this signal just to make my life a little bit easier. So we can use a filter to do that. Say, I’m not checking processing things.
So we use a low patch filter to remove all those noise things. Then we’re interested in okay, well, I’m not interested in the sin wave I’ll see, but I’m interested to see how often these peaks happen. And so in signal processing, we use a Fast Fourier Transform to transform this into time frequency data, so we see a peak maybe at every two seconds.
So now we know already, we’ve de-noised the signal, removed everything we’re not interested in, and we know that there’s a rotation happening every two seconds. That’s already super interesting information. And very often that can be an indication of whether a machine is going to fail.
If instead of every two seconds, it starts to rotate every one second, there’s probably something wrong with the machine. So we can use this as well in machine learning.
So rather than give you this raw wave form, which has lots and lots of data, we clean up the data first, we do some basic analysis stuff, the stuff that we’ve already been doing for the last 30 years, and that compresses the data a lot. So typically, we have maybe 5% of the data left, of the signals left. It has cleaned up a lot of the data, so all the noise has been removed.
And this, we can deploy much, much smaller machine learning algorithms. That’s how these two things work together, and that’s how we’ve been able to do real-time audio classification on really tiny MCUs or real-time machine classification on really tiny MCUs.
Where will Embedded Machine Learning go in the future?
Yeah, fantastic. Thank you for that, Jan. And so I guess this begs the next question, where can this go now? Now that you’re doing it, now that you’re making it happen, I guess it’s almost limitless in terms of the use cases and what you could do with it, but personally for you and Edge Impulse, where can we see this going in the next five years? How is this going to affect people in their everyday life?
I think that every embedded engineer … Okay, so like how an [RTOS 00:20:51] is now a fundamental property of virtually every embedded device, an FPU is now virtually a property of every embedded device, I think that embedded machine learning will also be part of that.
It will not be something you use for literally everything, but it’s something you have in your tool belt in case you want to solve a problem. And once enough people understand that, and once enough people have the knowledge of where to apply this, they can start applying it to fields that we have never heard of or problems that we’ve never encountered.
And I think that’s where the value of all technology is sitting in because even the 25 people at Edge Impulse or the 100 people you guys have can think about all of this, but you need absolute domain knowledge on the field where you’re actually deploying your device to really see where you can add that value. But from there, really the sky’s the limit.
So I think we’re in a state where you can run, for example, vibration analysis algorithms, anomaly detection algorithms, some basic audio classification on any IoT device, any brownfield IoT device even, which is really cool, but the development in ML are not standing still either, we’re coming into the state, so this little device [inaudible 00:22:12], it’s a DSP. This thing can do 10 frames a second image classification, which is insane. It’s almost video on an MCU class device.
Can you imagine what you can do if your device can actually understand what it sees? Because right now, the whole reason why we use accelerometers, or temperature sensors, or all those surrogate sensors that’s right against the world, but if you can just look at what is happening, very often you get to see that. How many people are there? Okay, well here, snap a photo. Cool. I count the number of people. Is there a fault on my production line? I snap a photo of my product. I see if the screw is in or not. I think that capability, that is going to be super transformational for devices. If you can do that low power on a battery in real time, I think that’s going to be … that’s going to be mental.
Yeah. Well, I mean that’s a perfect example there. You have accelerometers on devices at the moment, so they’re aware of it being moved and which direction, et cetera, but potentially you could remove that through ML, couldn’t you? You could remove the need to actually have these additional features on said devices, perhaps making them smaller, cheaper to produce, perhaps.
Yeah, I like the analogy with a Formula One car. So while they’re doing all the test driving, they try to stick as many sensors as possible on their car to get information about how the car drives, but this is a really cool use case. They want to build a demo around this.
They introduced synthesized sensors at some point, so they realized that if I have sensor A, B, C, and D, and I removed sensor B, I can use machine learning to actually interpolate the value of sensor B. So I don’t need to change my control loop, but I can have one device less on the car, which means less weight, et cetera.
So that’s one part, but I think there’s lots of other things we can do exactly with this. There’s so much more things that we can sense on our devices that we’re currently either imprecise or not being able to send at all where ML can really help.
How can we make Emedded Machine Learning more secure?
Yeah. What about security, Jan? So often a lot of people see security as an afterthought. As a group of businesses, we’re very much involved with things like the IoTSF and various organizations. It does potentially pose some challenges there, because if the device can recognize people, distances from an ML base rather than it being a sensor or some physical accelerometer, or what have you. What are your thoughts on that, and as a company?
Yeah, that’s very serious concern, and my main thing here is if we’re going to collect this kind of data and analyze that, let’s make sure we actually process it on the device and don’t need to send it to a central location. I’m relatively okay with, and that’s mainly because I trust Apple, I’m relatively okay that my phone is listening for hey Siri all the time, because I know it runs on the DSP only, it never leaves the device. It would be much more … I would not like it at all if my phone was constantly recording, sending everything to Apple’s location analyzing for that keyword, and them telling that they throw the rest away.
So I think that’s one part, try not to get data into a place where it can be stolen. I think for us as well, if you deploy a model from Edge Impulse to your device, there’s no link to us anymore, we just give you source code, so there’s nothing that actually tracks back to us.
But yeah, there’s a whole bunch of new privacy constraints and security constraints that come with the data that you process there that we need to reinvent as an industry as well, but it’s hard. I think as someone said at some point, the company that invents the SSL lock for IoT will get rich, but it’s an impossible problem to solve.
Well yeah, exactly. Well, impossible, improbable. But it’s interesting what you’re saying, and again, for the benefit of people that are listening to this, so effectively what you’re saying is as long as the data is stored natively on the device, and it’s not sent up the cloud somewhere else, and that’s a layer of security by design which wouldn’t allow interpretation or vulnerability to people hacking that, because it’s staying on the device.
I think it definitely helps. It definitely help. Regardless, I mean be considerate. We have rules around where you can put cameras, et cetera, and those rules don’t fundamentally change in IT, but the challenge that your camera has all of a sudden two square centimeters large and could do all the processing on the device, yet it poses new challenges.
I think also society in, what do we accept as society? And what do we want with this type of data? That, I think we need to figure out over the next couple years.
Yeah, absolutely. But I think the technology exists is fantastic, and we shouldn’t lose sight of the technology because what we can do, and the pros and the benefits of this is enormous, but you always have to think of that undercurrent, the underbelly, the dark side of this, and what people could potentially do it for, and being attuned to that.
And it’s nice to know your thoughts on that and how your views are as a business around it, because so often sometimes, as I mentioned earlier, it’s an afterthought, it’s a gold plating exercise, isn’t it? You’re doing it retrospectively as opposed to thinking about it from the start, which I think in an ever increasing connected environment is going to pose some challenges, isn’t it?
Yeah, so we require actually everyone to accept the responsible AI license before you can use us.
We think that AI should be used for good and we try to be a company for good as well. And yeah, that also means for us that we won’t work with certain companies.
So offensive military, for example, we’ll never work with that. We try to do a lot in conservation to give back, so we do some really cool projects on the African savannah. We’re part of 1% for the Planet, so we donate 1% of our revenue to that. I started teaching Data Science Africa every year, unfortunately not this year, to help to bring some knowledge that way. So try to do our part, but it’s hard as a technology company sometimes.
Yeah, it is. But I think the major hurdle is the fact that you’re thinking about it and you’re doing these things, and that you’re wanting to work with companies that are in that mindset, being involved with 1% for the Planet, being involved with these initiatives. And often that’s enough, isn’t it? Because it means it’s resonating with you already.
Yeah, it’s never enough, but that’s the thing, like I said at the beginning of the conversation, I said I’m not an entrepreneur, but this is the part of entrepreneurship that I really love. I think Arm is one of the companies that does this kind of stuff really well, they give you a day or a couple of days a year to work on volunteer projects, et cetera, but you always need to convince someone that something is a good idea or you should sponsor something. Being able to start your own company and put these type of things into the DNA of the company from the very start is super worthwhile. I think that’s really cool if you start off your own, to be able to do that.
Yeah, absolutely. Well Jan, thank you so much for taking the time out to come onto The IoT Podcast show. I really enjoyed it, and especially understanding more about your business. It’s truly really at the cutting edge, isn’t it? And I hope things go from strength to strength. In the future, it might be nice to get you back home when you’re 50, 75 people, when it’s being used across 20,000 projects. Well, 20,000 will probably be next week at the rate you’re going.
Oh, it’s going to be very fast.
Yeah. Maybe we’ll go for 200,000, but it’s exciting, and it’s nice to hear good stories like this, so really appreciate your time.
Thanks so much.