ReRun: An Open Source Package For Beautiful Visualizations

On Apr 5, 2023

Nico, Emil, and Moritz founded ReRun with the mission of making powerful visualization tools free and easily accessible for roboticists. Nico and Emil talk about how these powerful tools help debug the complex problem scopes faced by roboticists. Tune in for more.

Nikolaus West
Co-Founder & CEO
Niko is a second-time founder and software engineer with a computer vision background from Stanford. He’s fanatic about bringing great computer vision and robotics products to the physical world.

Emil Ernerfeldt
Co-Founder & CTO
Emil fell in love with coding over 20 years ago and hasn’t looked back since. He’s the creator of egui, an easy-to-use immediate mode GUI in Rust, that we’re using to build Rerun. He brings a strong perspective from the gaming industry, with a focus on great and blazing fast tools.

Links

transcript

[00:00:00] (Edited for clarity)

Abate: Welcome to the Robohub podcast. I’m here with Nico and Emil, two of the co-founders of Rerun. Can you introduce yourselves?

Niko: Sure, I can start. I’m Nico, the CEO of Rerun and one of the co-founders. My background is as a computer vision and machine learning engineer, and prior to Rerun, I built computer vision and ML-powered products that usually worked out in the physical world. Emil and I met at a Swedish startup that did 3D scanning of feet to recommend shoes. We worked together for several years, and since then, I have been working on various mobile projects.

Emil: I’m Emil, a programmer from Sweden. I’m an engineer and have worked in different industries, from physics simulation to game engines, game programming, 3D scanning, and computer vision. I fell in love with Rust five years ago and believe everything should be rewritten in Rust.

Abate: What motivated you to start Rerun?

Niko: The story starts at the company we worked at together, Volumental. The company specialized in 3D scanning in physical retail, and it was a tough product to build. One of the reasons that the company was successful is that we built excellent internal visualization tools that made it easy to understand the input and output data at any point in the system. After I left Volumental, I realized I couldn’t live without those tools and had been building poor versions of them in other companies. The lack of good tooling was painful, and I ran another company before Rerun that failed because of it. Emil and I discussed the problem with Moritz, and we realized that there was a lack of good tools for building computer vision or perception-heavy products. We felt this was particularly true of visualization and all the data infrastructure related to it.

Abate:vAbate:What are some of the other use cases for internal visualization tools?

Niko: Well, I think with these kinds of products, it touches most people at the end of the day. Even products like self-driving cars require visualizations of the perception planning algorithms. At Volumental, we had Friday demos where we demonstrated something we built every week. The more beautiful and visual the demos, the more points we got. It made us think about how to best present to colleagues.

Emil: It improved the understanding of the product throughout the organization. It gave everyone a view into how the machine was working and thinking. It’s one of those things where you don’t anticipate the value until you have it.

There are also more obvious benefits like ops. For example, if a scanner isn’t working properly, we can use visualization data to detect issues like glare around the cameras. Once you have good visualization, anyone in the company can do it, not just techs.

Abate: Creating these tools lets people fish for themselves. They don’t need to talk to an engineer to understand the problem. They can figure it out themselves and iterate quickly, without being bottlenecked on engineering resources.

Niko: For example, we can see the quality of issues coming in from operations or support improve when they can dig into the problem. Sometimes it’s an issue with camera calibration, which is easily identifiable in the point cloud. It may take more effort to understand the cause, but it’s more helpful than a support ticket saying the scanner thinks the customer is an apple.

Emil: Or confidence below 0.95. Okay.

Niko: Yeah. Exactly. Yeah.

Abate: So, is the product now open source? How can people sign up for it?

Niko: Yes, they can sign up by PIP installing it, getting the source code, or using cargo add if they are a Rust user.

Emil: It’s fully open source at github.com/rerun.io/rerun or just rerun.io. We’re an open core company.

Abate: Recently, the beta was released, right? So, this is the first time customers are interacting with it at a somewhat larger scale.

Niko: Yes, we released mid-February, meaning we put together a demo video to show the idea, but mainly just open-sourced the repository and made sure it’s easy to install. It’s been fun seeing what people have been building with it so far and what the community is excited about.

Abate: What’s been the reception? What are people excited about and less excited about?

Niko: People love that it’s very fast, thanks to Rust. We built the stack from the ground up and were inspired by how modern game engines are built. One of our main focuses is to reduce the number of obstacles and make it as easy as possible for people to get started. You can simply PIP install rerun sdk and rerun log image log image to see the image. If you log multiple images, you can use the time slider to see what happened before projected in your space.

Emil: So, we could make it…

Abate: And, you’re building a completely open source product. How are you balancing that with creating a sustainable business that can continue to provide updates in the future?

Niko: Our approach is that everything a single developer or researcher needs for their own work, like working locally on their own machine or with a robot, goes into the “completely free and open source” bucket. There are also helpful things for teams, such as sharing and collaboration, which requires additional infrastructure and functionality. Running that infrastructure easily falls into the paid bucket. Additionally, there are many things related to handling larger data sets and improving performance, particularly around collaboration, that go into the paid bucket as well. That’s how we split the paid and free features.

Abate: So, if a developer is running it on their local machine, that’s the open-source part. They can install it using PIP install run. However, if they need to use it across a larger system that requires cloud computing, then it becomes more of an enterprise sale.

Emil: Yeah, something like that. We should note that so far we, we haven’t built our commercial thing.

Niko: Yeah. Important

Emil: is the, yeah, an important thing to note. Right now we only have the free version.

Niko: Yeah, (laughing)

Abate: What’s the ecosystem around computer vision? What are the current tools out there and how do you fit in?

Niko: I think the major comparison is with plotting libraries like Matplotlib, Open3D, or Plotly versus building something from scratch. OpenCV imshow is also a comparison. Within robotics, ROS-style tools like RVIZ, Foxglove, or Exis are used.

Abate: And where do you aim to be on the scale of visualization tools? As easy and lightweight as Matplotlib or OpenCV imshow, but still almost as powerful as the bigger products, and eventually more powerful.

Emil: So we’re taking a small, simple use case and building a complex and rich visualization from that.

Niko: Yes.

Abate: It’s a very ambitious project.

Niko: Yes, it is. But it’s also important because it brings value to communicating these things internally and externally across different departments. Everyone is building this again and again in their own companies.

Abate: So what are you guys excited about developing next?

Niko: Right now, our focus is really honing in on the super early stage of working. We want to replace imshow and simpler uses of plotting libraries like pyplot or draw geometries with something much more built for computer vision and perception robotics developers that’s easier to use and more powerful. That’s what we’re super excited about building right now.

Abate: I really appreciate the work that you guys are doing because it’s one of those things that’s super annoying to do. For example, at my previous startup Baller TV, we built an iPhone app that could detect where the players are, where the ball is, and where the court is in a volleyball, basketball, or soccer game. But everyone else in the company was kind of blind to what was happening and couldn’t tell if it was following the action. That’s why visualization is so important.

Niko: One of our early thoughts when we were thinking about this was inspired by my reading of books about gaming history, like “Masters of Doom” about ID Software and Carmack.

Emil: Masters of Doom.

Niko: Yeah, I was just thinking about how gaming companies tend to organize themselves, like there’s a clear spread of roles, such as the engine people, the artists, and the tooling team in between. It’s interesting to compare this to how product teams work in computer vision. The algorithm teams aren’t building an algorithm engine for someone else to build an app on top of; they’re just solving the problem. But this means the product people don’t always understand enough, so the algorithm people end up being the product people too. There should be more interaction and handoff between the teams.

Abate: Hmm.

Niko: The algorithm team should also hand over an understanding of how this works so that product people can make good decisions around it. Visualization is really important for this.

Abate: Yeah, so not only does it make the engineers more effective because they don’t spend as much time creating visualizations themselves, but it also makes the product people more effective because they have a better understanding of what they’re trying to spec the product for.

Niko: I think that can be a bigger value in the long term. It’s really hard to build intelligent products with complex behavior, and if you don’t understand that behavior, it’s really hard to design for another person to use that product. It’s difficult to explain why visualization is important, but it’s not just because it saves engineering time. It’s like building business intelligence, and everyone needs to be on the same page.

Emil: Yeah. It goes back to Florence Nightingale, who wanted to improve the situation for field hospitals during war. She couldn’t get the attention of the politicians until she came up with a visualization – throwing up a pie chart – showing that more people were dying in the hospitals than in battle. That got people’s attention. When you can put a picture to something, you can really convince people and help them understand.

Abate: How do you market that?

Niko: You won’t find anything about that on our website. I think you don’t necessarily need to upfront market all the values if the market isn’t ready and doesn’t understand them yet. For us, we focus on developers who know they need it. CEOs of larger companies are also thinking about this because their internal tooling and different teams don’t communicate, which is a priority for them. We did a lot of market exploration and that was one surprising part.

Emil: Visualizing data brings quality of serendipity. Once you visualize the data you have, you suddenly have ideas you wouldn’t have otherwise. This is hard to market, but engineers who work in visual fields and have done visualizations already know this.

Abate: Yeah, one of the other values is that if you don’t understand how a system works, it leads to a lot of speculation. Being able to show it in a clear visualization usually makes the answer much more apparent.

Emil: I’ve had this problem in game development many times. People speculate about a problem for 20 minutes, but once they visualize it, the problem is completely different from what anyone speculated.

Niko: People have mental models and are confident in them, even if they’re not close to the data. Any system with complex behavior will lead to this.

Emil: I think engineering as a culture has internalized the idea that you should always profile before optimizing your code, and we need to internalize the same thing for visualization. Always visualize before fixing a bug or improving an algorithm.

Niko: Yeah.

Emil: Profiling is just a way to visualize performance, right?

Abate: Absolutely, yeah.

Niko: I remember in university, I was always told to double-check my math problems by hand graphing them or drawing them in the beginning. At first, I thought it was just more work, but it actually helps a lot, especially with complex problems.

Emil: Exactly. A third or a quarter of our brain is dedicated to visual processing, so we should play to our strength as visual thinkers.

Abate: Is that why you guys chose robotics first? Because it has a big visual component and is super complex?

Niko: Well, for me, I got into it because of a practical application I really wanted to solve, but I stayed because it’s so fun to do something really advanced and cool, and see it come to life.

Emil: Our background is in robotics and computer vision, which is a subset of robotics. But I believe the tool we’re building is applicable to many other things as well. I’m interested to see how the gaming community can use our tool, Rerun, once we have C++ libraries available. Right now, our SDK is only for Python and Rust.

Abate: Awesome. Awesome. Thank you guys.

Emil: Oh, thank you. This was really fun

Niko: [00:25:00] Oh, thanks. Yeah.

transcript

tags: open source, podcast, Robotics technology, software, startup