Behavioral AI is making computer vision more impactful

For many years, computer vision innovations have focused on tasks like object detection - identifying objects such as a dog or a car in images and videos. These approaches have enabled applications in areas such as autonomous vehicles, manufacturing, and healthcare. 

However, these tasks often focus only on identifying what an object is. What if Vision AI systems could go a step further? For instance, instead of simply detecting a dog, let’s say it could understand that the dog is chasing a ball or that a car is braking suddenly because a pedestrian is crossing. This shift from basic recognition to contextual understanding represents a major shift toward smarter, context-aware behavioral AI.

At YOLO Vision 2024 (YV24), Ultralytics’ annual hybrid event celebrating advances in Vision AI, the concept of AI-driven behavior analysis took center stage during an interesting talk by David Scott, CEO of The Main Branch.

In his talk, David explored the transition from basic computer vision tasks to behavioral tracking. With over 25 years of experience in building cutting-edge tech applications, he showcased the impact of this leap. He emphasized how decoding patterns and behaviors are reshaping industries like agriculture and animal welfare.

In this article, we’ll walk through the highlights of David’s talk and explore how behavioral tracking makes AI more practical. 

Understanding AI adoption challenges

David Scott started his keynote with a bold reality check and said, “A colleague of mine often says, ‘Science doesn’t sell,’ which kind of offends many of us here because we really like science. AI is really cool - why wouldn’t people just buy it? But the reality is, people don’t want to buy it just because we think it’s cool; they need a reason to buy it.”

He went on and explained that at his company, The Main Branch, the focus is always on solving real problems with AI, not just showing off its capabilities. A lot of clients come in wanting to talk about how they can use AI in general, but he sees that as a backward approach - it’s like having a solution without a problem. Instead, they work with clients who bring specific challenges so they can create AI solutions that actually make a difference.

‍

David also shared that their work often goes beyond just recognizing objects in a scene. Spotting what’s there is only the first step. The real value comes from figuring out what to do with that information and making it useful within the larger value chain. 

Behavioral tracking technology: the key to actionable AI

A vital step in making AI truly useful is moving beyond basic computer vision tasks like object detection and using those insights for behavioral tracking. David highlighted that behavioral AI focuses on understanding actions and patterns, not just identifying objects. This makes AI capable of recognizing meaningful events and providing actionable insights.

He gave an example of an animal rolling on the floor, which could indicate illness. While people can’t watch an animal around the clock, AI-driven surveillance systems with behavioral tracking capabilities can. Such solutions can monitor objects continuously, detect specific behaviors, send an alert, and allow timely action. This turns raw data into something practical and valuable.

David also showcased that this approach makes AI not just interesting but truly impactful. By addressing real problems, like monitoring behaviors and acting on them, behavioral tracking can become a key part of effective AI solutions across various industries.

Bringing behavioral AI to life

David Scott then illustrated how Ultralytics YOLOv8, a computer vision model, was a breakthrough for his team’s behavioral tracking projects. It gave them a solid foundation for detecting, classifying, and tracking objects. His team also went a step further and custom-trained YOLOv8 to focus on monitoring behaviors over time, making it more practical and helpful for real-world situations.

Interestingly, with the release of Ultralytics YOLO11, solutions like the ones created by The Main Branch can become even more reliable and accurate. This latest model offers features like improved precision and faster processing that enhance its ability to track behaviors. We’ll discuss this in more detail after getting a better understanding of the applications behavioral AI can be used for.

Next, let’s explore the solutions David talked about and how behavioral tracking technology is being used in real-world applications to solve everyday challenges and make a meaningful impact.

HerdSense with AI-driven behavior analysis

First, David shared an exciting challenge they tackled with a project called HerdSense, which involved monitoring the health of thousands of cows on a massive feedlot. The goal was to track the behavior of individual cows to identify potential health issues. This meant keeping an eye on tens of thousands of animals at the same time, and it wasn’t a simple task.

‍

To begin solving the problem of identifying each cow and tracking its behaviors, David’s team conducted a two-day workshop to outline every possible behavior they needed to monitor. They identified over 200 behaviors in total.

Every one of the 200 behaviors depended on being able to accurately recognize individual cows, as all the data had to be tied to specific animals. One major concern was tracking cows when they grouped together in huddles, which made it difficult to see individual animals. 

David’s team developed a computer vision system to ensure each cow was consistently identified, even in tricky situations. They were able to confirm that the same cow would always be assigned the same ID, even if it disappeared from view, mingled with others, or reappeared later.

Monitoring horse health using computer vision

Moving on, David introduced another fascinating project where they applied similar behavioral tracking techniques to monitor horses. In this project, David’s team didn’t need to track individual horse IDs as closely as they did with the cows. Instead, they focused on specific behaviors and tracked details like eating patterns and general activity levels to spot any health issues early on. Identifying small changes in behavior could lead to quicker interventions to provide better care and prevent problems before they become serious.

‍

Why behavioral AI isn’t as simple as it seems

David also discussed the complexity of behavioral tracking through an intriguing example. While researching ways to improve behavioral analysis, his team came across a company claiming to detect shoplifting by analyzing specific poses, like someone having their hand in their pocket. At first, this seemed like a smart idea - certain movements could suggest suspicious behavior, right?

‍

However, as David explored further, he realized the limitations of this method. A single pose, such as a hand in a pocket, doesn’t necessarily mean someone is shoplifting. It could just indicate they’re relaxed, thinking, or even cold. The problem with focusing on isolated poses is that it ignores the larger context. Behavior is not just a single action - it’s a pattern of actions over time, shaped by context and intent.

David highlighted that true behavioral tracking is far more complex and requires a holistic approach. It’s about analyzing sequences of actions and understanding what they mean in the broader picture. While the AI industry is making strides, he noted that there’s still work to be done in advancing behavioral tracking to deliver meaningful and accurate insights.

Creating smarter vision AI models that understand actions

Subsequently, David took the audience behind the scenes to show them how his team built a computer vision solution to monitor cow health with the help of YOLOv8, and its pose estimation abilities.

They started by creating a custom dataset for pose estimation of a cow, increasing the standard number of key points from 17 to 145 to make the model better at analyzing movement. Then, the model was trained on a massive dataset of over 2 million images and 110 million behavioral examples. 

Using advanced hardware infrastructure, David’s team was able to train the model in just two days instead of the weeks it would have taken on conventional hardware. The trained model was then integrated with a custom behavior tracker that analyzed multiple video frames simultaneously to detect patterns in the cows’ actions.

The result was a vision AI-driven solution that can detect and track eight different cow behaviors like eating, drinking, and lying down to spot minor behavioral changes that could signal health concerns. This allows farmers to act quickly and improves herd management.

The road ahead for behavioral AI

David wrapped up his talk by sharing an important lesson with the audience: "If you don’t give AI room to fail, you’re setting yourself up for failure because, at the end of the day, it’s statistical." He pointed out that AI, despite its strengths, isn’t flawless. It’s a tool that learns from patterns, and there will always be times when it doesn’t get things right. Instead of fearing those mistakes, the key is to build systems that can handle them and continue to improve over time.

This is also true when it comes to computer vision models themselves. For example, Ultralytics YOLO11, the latest version of the Ultralytics YOLO models, has been built keeping in mind the need to take things to the next level compared to YOLOv8. 

‍

In particular, YOLO11 offers better performance, especially with respect to real-time applications where precision is key, like agriculture and healthcare. With its advanced features, YOLO11 is redefining how industries use AI by providing innovative real-time insights and helping them tackle challenges more effectively.

Die wichtigsten Erkenntnisse

David’s keynote at YV24 was a reminder that AI is more than just a cool innovation - it’s a powerful tool for solving real problems and improving how we live and work. By focusing on behavior, AI is already making an impact in areas like tracking animal health and recognizing meaningful patterns in everyday actions. 

The potential for behavioral AI is exciting, and we’re only at the beginning. By transforming raw data into actionable insights, behavioral AI shifts from passive monitoring to active problem-solving. As it develops further, behavioral AI is set to drive smarter decisions, streamline processes, and bring meaningful improvements to our lives.

Stay connected with our community to learn more about AI and its real-world applications. Visit our GitHub repository to discover innovations in areas like AI in agriculture and computer vision in manufacturing.