Vast Data adds vector search and serverless functions to its scalable data platform

Vast Data Inc. today announced enhancements to its Vast Data Platform, enabling it to support structured and unstructured data in a single DataSpace with linear and secure scaling.

The enhancements are aimed at enabling enterprises to build artificial intelligence applications, agentic workflows and high-speed inferencing pipelines more easily.

The new capabilities combine real-time vector search, fine-grained security and event-driven processing into a single platform called Vast InsightEngine that uses autonomous agents to process, adapt and act on dynamic data streams with minimal human oversight. DataSpace is a data management technology that provides fast, consistent access to data across multiple locations by decentralizing lock management to the file, object or table level to allow each cluster to handle its transactions.

“This enables enterprise customers to unlock the value of all of their data and integrate it with inferencing so they can use LLMs [large language models] to query and have the models use the data in their environment as a data source,” said Aaron Chaisson, vice president of product and solutions marketing at Vast Data. “As we move to agentic AI, we need the ability to analyze data in real-time as it’s flowing into the system and convert it to vectors that can be accessed by agents and chatbots.”

AI models call for ultra-fast vectorized search, a technique that finds similar items in a dataset by comparing their mathematical representations, called vectors, instead of searching for exact matches. Though many database management vendors have recently added support for vectors, Vast Data said many AI pipelines lack enterprise-grade security, encryption and governance controls.

Trillion-vector scale

The company said its Vast DataBase, a data management system that unifies the functions of transactional databases, data warehouses and data lakes into a single platform, now is the first and only vector database that supports trillion-vector scale with the ability to search large vector spaces in constant time. This makes it possible to index all data and make it available to agentic workflows at any scale. Organizations can now automatically embed vectors for search and retrieval.

New serverless triggers and functions in the Vast DataEngine support real-time workflows and don’t require background extract/transform/load processes. Event-driven automation and real-time data enrichment allow the system to embed and serve context to agentic applications instantaneously for real-time retrieval-automated generation with support for high-speed queries, serverless processing, and automated and secure real-time pipelines. Vast Data added event streaming streaming to its platform last month.

“When a file is saved in our data store, the engine can trigger a microservice that chunks and embeds a file we can search,” Chaisson said. Chunking breaks large files into smaller, more manageable pieces to optimize storage, improve data transfer efficiency and enable parallel processing.

Vast Data doesn’t chunk files in memory but uses an exabyte-scale solid-state disk fabric to deliver “millisecond latency at any scale,” Chaisson said. Built-in security now offers advanced row- and column-level permissions for compliance and governance purposes with unified permissions for raw data and vector representations.

The parallel transactional design of Vast’s DASE Architecture make it possible to update vector spaces in real time and the architecture’s shared-everything approach allows for all servers to search the entire vector space in milliseconds for AI inferencing. DASE decouples compute logic from system state and uses shared and transactional data structures that support both capacity and high performance.

Image: kanawatTH/Canva