Databricks buys feature engineering startup Fennel to enhance AI model development

Databricks Inc. said today it has swooped to acquire a young company called Fennel AI Inc. for an undisclosed price so it can enhance its data intelligence platform with real-time feature engineering capabilities.

According to Databricks, Fennel has developed a modern, incremental compute engine that supports the development of more refined data pipelines, for batch, streaming and real-time data. It helps to improve the efficiency and freshness of the data, enabling companies to develop more advanced artificial intelligence models.

Feature engineering is quickly becoming an important aspect of AI development as the industry evolves and large language models become more sophisticated. It refers to the process of selecting, extracting and transforming the most relevant parts, or features, of a dataset, to help create more effective AI models. It’s important because the performance of models is heavily dependent on the quality of features used in their training.

Databricks believes Fennel’s platform will be helpful for customers because feature engineering has always been extremely challenging, involving the maintenance of the complex extract, transform and load pipelines needed to compute the underlying data. It’s especially difficult when dealing with features that rely on both batch and real-time data sources, as that involves the added challenge of ensuring consistency between the training and model serving environments, Databricks said.

Fennel, founded in 2023 by its Chief Executive Nikhil Garg and Chief Technology Officer Abhay Bothra, who previously worked on AI infrastructure at Meta Platforms Inc. and Google Brain, says it’s uniquely able to simplify these challenges. It offers a fully managed platform for creating and managing both features and their data pipelines, helping to improve the efficiency and freshness of the underlying data. It does this by recomputing only the data that has changed, and ignoring everything else.

Fennel’s platform offers a Python-native user experience to make authoring complex features an easier and more accessible task for users. Notably, it eliminates the need to learn new programming lanuages or rely on teams to create data pipelines first. Moreover, Databricks said, its incremental computation engine can help to optimize costs by avoiding redundant work.

By integrating these capabilities into its Data Intelligence Platform, the company said, it will be able to help customers iterate more rapidly on their features and boost model performance with more reliable signals. It also supports the development of models with greater personalization and context understanding.

Although it was operating more or less under the radar, with no significant funding rounds under its belt, Fennel managed to attract quite a few customers, including Cricut Inc., Upwork Inc. and Rippling People Center Inc. Those companies are using its platform to build all kinds of machine learning features for use cases such as fraud detection, credit risk decisioning, trust and safety, marketplace recommendations and personalized rankings.

Databricks didn’t say how much it paid to acquire Fennel, but it’s unlikely that the transaction has made much of a dent in its wallet. In January, the company secured more than $15 billion in funding in the shape of a sizable venture capital investment and debt financing.

Following that round, Databricks said one of its intentions was to use the money to fund more strategic acquisitions and build out its AI capabilities, and it wasted little time in doing that, snapping up a data migration startup called BladeBridge Inc. in February and following with today’s acquisition.

Image: Databricks