Microsoft launches Fabric, adds Copilot for the new platform

A new Fabric

Microsoft Fabric is a new environment for data integration, data management and analytics, bringing together a set of capabilities that enable customers to model and analyze data in myriad ways.

The suite includes Power BI, Microsoft's longstanding traditional BI platform on which users can develop and consume data products such as reports, dashboards and AI and machine learning models. In addition, Fabric includes Azure Synapse Analytics, a cloud-based service for data integration, data warehousing and big data analytics. Finally, Fabric includes Azure Data Factory, an extract, transform and load service that enables customers to integrate and transform data at scale.

In addition to the three formerly separate platforms, Fabric includes a multi-cloud data lake called OneLake that automatically connects to every data workload within Fabric. OneLake comes with shortcuts to data sources such as Azure Data Lake Storage Gen2 and Amazon S3.

The combination of the previously disparate platforms in a single environment is designed to simplify data management and analysis, according to Microsoft.

Meanwhile, simplicity is particularly relevant as organizations collect increasing amounts of data in increasingly complex forms, all of which needs to be combined to inform and train the generative AI language models that organizations want to build to improve decision-making.

As a result, Fabric was developed to meet the needs of organizations as AI becomes more ubiquitous and BI moves beyond data visualizations to model training and analysis, according to Frank Shaw, Microsoft's corporate vice president of communications.

"Microsoft Fabric reshapes how teams work with data by bringing everyone together on a single AI-powered platform built for the era of AI," he said during a virtual press briefing Nov. 10. "It creates an integrated, simplified experience that unifies your data state on an enterprise-grade data foundation."

The data integration, management and analytics capabilities included in Fabric are frequently pieced together by many organizations. They may use data ingestion and integration platforms from one vendor, storage and management tools from another, and various BI platforms on top to do the actual analysis and decision-making.

The approach helps organizations avoid vendor lock-in, freeing them from relying on one vendor for all their data management and analytics tools.

But a vendor-neutral approach comes with both a financial cost -- it's usually more expensive to put together tools from various vendors than to buy from just one -- as well as complexity. Rather than buying tools designed to integrate with one another, organizations themselves have to connect tools that weren't initially developed to fit together.

Cost is therefore one of the main benefits of Fabric, according to Microsoft.

Pricing for Fabric is based on usage, with prices starting at 36 cents per hour or $262.80 per month for two units and ranging to $368.64 per hour or $269,107.20 per month for 2,048 units.

Simplicity, meanwhile, is perhaps the biggest benefit of Fabric, according to Menninger.

"Most of the capabilities are not new, but they have all been brought together into one unified platform," he said. "Users should find it easier to combine all these capabilities in the context of Fabric rather than having to stitch things together to create their own fabric -- pun intended."

Beyond cost and simplicity, perhaps the most significant features in Fabric are OneLake and the shortcuts to data sources such as S3, Menninger continued.

"By enabling data sharing across applications, these capabilities will help reduce redundant data and all the headaches that go along with it, including extra data pipelines, data synchronization issues, and data governance challenges," he said.

Fabric and AI

While Fabric is designed to provide Microsoft customers with the tools needed to develop and manage modern data products including AI and machine learning models, the platform also benefits from AI.

Like many Microsoft features, Fabric will be getting a dedicated Copilot.

Copilots, first introduced by Microsoft in 2021, are NLP tools that assist users as they do their work. The tech giant's first Copilot was developed for joint users of Microsoft and software development platform GitHub. Since then, Microsoft has released Copilots for Office 365, Teams and a host of other tools.

Copilot in Fabric is now in public preview and will enable customers to work extensively with their data, using natural language rather than code.

Numerous data management and analytics platforms have featured NLP capabilities in recent years. But those NLP capabilities were limited. They were trained with finite vocabularies, meaning it still took data literacy training to use the tools, and they couldn't be used for modeling and deep analysis.

That has changed in the year since AI vendor OpenAI, a Microsoft partner in which the tech giant has invested $13 billion, released ChatGPT. The large language model has marked a substantial improvement in generative AI and LLM technology. ChatGPT and other LLMs including Google Bard and Microsoft's Azure OpenAI have extensive vocabularies that enable freeform natural language interactions.

By integrating those LLM capabilities with their own tools, data management and analytics vendors are now able to develop NLP capabilities that let customers ask questions of their data just as they would run searches and ask questions in Google. In addition, they can develop processes and pipelines without having to write code.

Copilots are Microsoft's versions of the tools that enable natural language interactions.

In the case of Fabric, Copilot will enable users to use conversational language to generate code, develop data pipelines and create AI and machine learning models as well as other data products.

Meanwhile, using natural language for tasks that previously required knowledge of code and training in data literacy should enable more employees within organizations to work with data, Menninger noted.

BI use within organizations has been stagnant at about a quarter to a third of employees for more than two decades. Conversational language could change that.

"Copilot is Microsoft's encapsulation of generative AI into its products," Menninger said. "The real, immediate value of generative AI is NLP. NLP can be used to make many aspects of data and analytics much easier to use, which also has the side effect of making data and analytics more accessible within organizations."

Shaw said Copilot will also help data experts be more efficient.

While generative AI-fueled natural language enables more people within organizations to use data and analytics, it also eliminates much of the time it takes data engineers and data scientists to write the copious amounts of code required to develop pipelines and create models.

"With Copilot in Microsoft Fabric, data professionals can use conversational language to quickly discover new insights and better compete at speed in this new era of AI," Shaw said.

In the year since ChatGPT's release initiated a surge of generative AI and LLM development, numerous data management and analytics vendors in addition to Microsoft have released AI assistants and text-to-code translation capabilities.

For example, Microsoft rivals AWS and Google have both added generative AI capabilities to their data management and analytics portfolios, as have many concentrated data management and analytics vendors such as Informatica and Tableau.

More new capabilities

While the general availability of Fabric is perhaps Microsoft's most significant new data offering, the tech giant on Wednesday also revealed a host of other new and updated capabilities.

Included among those is vector search in Azure Cosmos DB MongoDB vCore.

Azure Cosmos DB MongoDB vCore is an integration between Microsoft and database specialist MongoDB that enables joint customers to develop data applications.

Vector search, meanwhile, is a capability gaining importance amid the surge in generative AI development. Vectors are identifiers of data that give data structure so it can be searched and found amid a morass of other data and subsequently used to inform decisions.

Some data is structured from the start, such as financial records and point-of-sale transactions. But data such as text, photographs and video files is unstructured.

Generative AI models, meanwhile, require more data than other model types to ensure a high level of accuracy. That means organizations often need to combine structured data with unstructured data to have enough data for those generative AI models.

Therefore, unstructured data is often assigned a vector -- a numerical representation -- to give it structure that then enables it to be discovered and used to inform models.

In addition, vector databases are built to do similarity searches so that data that otherwise not be discovered can also be used to train models.

Beyond vector search in Azure Cosmos DB MongoDB vCore, new and updated data management and analytics capabilities include the following:

• Amazon S3 shortcuts that enable organizations to unify their data in Amazon S3 with their data in Microsoft's OneLake without having to make copies of the data.
• Shortcuts that enable data engineers to connect data from external Azure Data Lake Storage Gen2 data lakes into OneLake.
• Azure SQL updates that reduce cost and increase reliability and security.
• An integration between Microsoft 365 and OneLake.
• New features in Azure Cosmos DB to lower costs and improve developer productivity.
• Improved security for SQL Server.
• Performance enhancements in Azure Database.
• New AI capabilities and performance improvements for Azure Database for PostgreSQL.

Many of the new and improved features, such as shortcuts to S3 from Amazon, demonstrate that Microsoft is acknowledging that customers use tools from multiple vendors, Menninger noted.

Moving forward, Microsoft would be wise to do even more to enable users to develop a data ecosystem by connecting Microsoft tools -- especially Fabric -- to those from other vendors, he continued.

"While most of Fabric's capabilities are about unifying Microsoft products and services, Microsoft can help its customers even more by extending Fabric further," Menninger said. "Let's see how far they go in embracing the third-party data sources."

Eric Avidon is a senior news writer for TechTarget Editorial and a journalist with more than 25 years of experience. He covers analytics and data management.