Datorios Improves Data Streaming Visibility to Support More Reliable Real-Time AI Systems

Getting started with streaming data observability Datorios Ltd. today announced the immediate availability of a new real-time observability platform for the Apache Flink big data processing framework.

With the new platform, businesses will benefit from what the startup claims are never-before-seen insights into streaming data processing. This information can help create new real-time artificial intelligence systems that can be fully audited to ensure they don’t misbehave, the startup said.

The founders of Datorios say they have leveraged years of experience researching and developing real-time military intelligence systems to create their new product, with the end goal being to help developers create intelligence applications. More advanced AI that can operate in real time. The basic idea is that Datorios provides more visibility into information fed into AI systems in real-time using data pipelines based on Apache Flink.

It can be thought of as a “real-time AI flight recorder,” the company said. By improving visibility into real-time streaming data, the startup believes, it can help simplify tasks like troubleshooting and auditing real-time AI applications.

Apache Flink has become an essential tool for creating real-time AI. It’s a open source big data processing framework which is widely used in industries such as video streaming, e-commerce, and financial services. It allows data to be processed as soon as it is created and offers use cases in data analysis, transaction processing, and AI models that need to process up-to-date information.

The Apache Flink framework is often compared to Apache Spark, which also processes information in real time. However, the main advantage of Flink is that it can perform calculations on data in motion, as it is processed, which its creators claim is true real-time data processing. This is different from Apache Spark, which processes data in small chunks using a technique known as “microbatching,” where information must be accumulated in small batches before being calculated. The microbatching approach means slightly higher latency, which can be a deal-breaker for certain types of AI systems.

Fixing real-time data disruption

Real-time AI systems rely on continuous streams of up-to-the-second information to automate processes such as fraud detection, product recommendation engines, and vehicle routing. But even though Apache Flink’s ability to process information in real time is virtually unmatched, the streaming data it manages can still be disrupted. For example, it may arrive in the wrong order, or multiple times, or in an unexpected format. When this happens, it can cause big problems for AI systems that need information to be accurate and immediately accessible.

Datorios aims to solve these problems by increasing visibility into these real-time data streams, so developers can be confident that their AI systems are processing the right information they need.

Specifically, the startup said its platform provides event visualization insights, showing the journey of data as it passes through each processing operation. It also has event search capabilities, allowing users to trace specific events for auditing and compliance purposes. Its state monitoring features allow users to see how data state changes during each processing step, while window analysis tools provide insight into areas such as the order in which data is processed, its delay, records deleted, etc.

By providing these new data observability capabilities, Datorios claims to carve out its own niche within the broader Apache Flink ecosystem as the only tool capable of identifying disruptions in streaming data.

Ronen Korman, co-founder and CEO of Datorios, said today’s most innovative companies want to use AI to run their business operations on autopilot, and to do so, they must ensure accuracy and the reliability of the data that feeds these systems. “Datorios is their flight data recorder, showing them exactly how a system behaved and why at any given time,” Korman explained. “Just like airplanes, unexpected conditions can affect performance; Datorios’ information makes this clear so they can continually improve their AI.

Datorios said he was building his platform available for free with almost no limitations for new users, to show how it can support their real-time AI systems.

Images: Datorios