Big data company dbt Labs’ new AI co-pilot to simplify data analytics lifecycles

Big Data Company Laboratoires dbt Inc. launched its annual user conference Merge 2024 in Las Vegas this week with a slew of updates to its flagship dbt Cloud product, claiming they will help cement its status as the “data control plane” for business analytics.

The company said the new services announced at the event support users at every stage of the analytics development lifecycle, introducing what the company says is unparalleled cross-platform flexibility. They include a new AI-powered co-pilot integrated throughout the dbt cloud platformenabling more people to contribute to analytics workflows by simplifying various data preparation tasks.

Dbt Labs is the creator of a cloud system data transformation toolwhich can be used by companies to modify data and facilitate its processing and analysis. It’s a complete data platform that does everything from consolidating multiple spreadsheets into a single file, filtering out inaccuracies in a data set, and changing how data is formatted across multiple database systems.

The company presents dbt Cloud as a sort of “data control plane” designed to help with every stage of the analytics development lifecycle. It is compatible with various data warehouse platforms including Snowflake, Databricks and Google BigQuery.

At Coalesce 2024, the company said the new updates aim to eliminate issues related to data quality and ownership, as well as data mastery, making its platform more accessible to users without sophisticated technical skills.

The most important new ability is co-pilot dbtwhich is described as an “AI engine” that can help workers speed up analytics processes by automating many tasks that previously always had to be done manually. For example, dbt Copilot has the ability to automatically generate tests, documentation and semantic models, the company said.

Additionally, there is a chat interface that allows users to ask questions about their data in natural language, available through the native dbt app in Snowflake, and the ability to bring your own API key. OpenAI application. The new features are all available in beta starting today, except for this last one, which is generally available now.

Dbt Labs also adds support for Apache Iceberg, a high-performance open source format for huge analytical tables. Iceberg enables the use of structured query language tables for big data while allowing engines such as Spark, Trino, Athena, Databricks, Starburst and Dremio to work securely with the same tables.

Furthermore, the dbt mesh tool supports cross-platform benchmarks using the Iceberg table format as the underlying transport layer, the company said. Essentially, this means that customers will be able to use dbt Cloud to centrally define and maintain data governance standards across various Big Data platforms.

For end users, one of the most visible new updates is the new “visual editing experience” introduced in beta. It is a low-code drag-and-drop environment that allows users to explore various dbt models to integrate data, part of the company’s efforts to democratize the analytics development lifecycle process by making it more accessible.

The new interface will allow downstream users, who typically have the greatest business context, to create analytical code in an accessible and secure manner, the company said. Meanwhile, users familiar with SQL can also use the visual editing experience to check their code and visualize how their DBT models work.

More sophisticated users will likely appreciate the introduction of a more advanced command-line interface, generally available now, designed to help detect unexpected behavior before new code is released into production. According to the company, this will achieve two goals: improving code quality and helping companies optimize their compute spend by materializing only correct models.

Other new features include data health tiles, generally available now, which can be integrated into any downstream application to provide real-time context in key trust signals such as freshness and data quality, and automatic exposures with Tableau Software, which directly integrates with Tableau dashboards. in line with the debt. With this, users can more easily orchestrate the creation of end-to-end data pipelines and be confident that their analysis uses only the most recent data.

The company also revealed an upcoming integration with Microsoft Corp.’s Power BI tools, allowing customers who have standardized on that company’s ecosystem to query and analyze their data with more consistent metrics. Finally, Teradata Corp. and Amazon Athena have become the latest databases to integrate with dbt Cloud.

Dbt Labs founder and chief executive Tristan Handy said the data industry has already made significant progress towards maturity in recent years, but challenges remain around data siloing and lack of trust.

“There is still too much tape in our operational systems,” he stressed. “Our announcements this week go a long way toward filling these gaps with a cross-platform, multi-person, trusted, AI-infused “One dbt experience.”

Image: SiliconANGLE