Data engineering workloads reshape platform strategy

Data engineering workloads are entering a new era — one defined by open architectures, flexible compute models and a demand for greater control and transparency.

As enterprises race to modernize their data stacks for AI-driven operations, they’re no longer settling for rigid systems that limit agility or lock data into proprietary formats. Instead, they’re prioritizing architectures that decouple compute from data, support diverse engineering languages and allow smarter orchestration of both internal and external assets. The shift isn’t just about efficiency — it’s about enabling new forms of governance, expanding access to data science and redefining what it means to deliver value in a platform-first world, according to Dave Vellante (pictured, right), chief analyst at theCUBE Research.

TheCUBE Research’s George Gilbert talks about data engineering workloads at Snowflake Summit.

“They have to create that open experience; their proprietary experience has to be great,” Vellante said. “They’ve got some decisions to make. Are they just basically data infrastructure that’s serving up agentic platforms or are they going to become the agentic platform?”

Vellante and George Gilbert (left), principal analyst at theCUBE Research, spoke with theCUBE host Rebecca Knight at Snowflake Summit, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how Snowflake Inc. is evolving its architecture and pricing to support more open, flexible and cost-effective data engineering workloads in response to enterprise demands for greater control, artificial intelligence readiness and workload portability. (* Disclosure below.)

Snowflake’s next act: Unlocking value through data engineering workloads

Snowflake Inc.’s recent architectural moves reflect its recognition that customers are pushing for modular control over data engineering and analytics environments. By decoupling compute from storage — and more recently from data itself — Snowflake is aligning its platform with the evolving needs of cost-conscious enterprises. That includes support for languages popular among data engineers and new pricing models that address past limitations, according to Gilbert.

“They addressed the pricing model problem, which was bundling the hardware for what were very cost sensitive workloads,” he said. “They have two options now. It’s essentially a serverless function where other vendors have to offer that. Everyone has to mark up the hardware. There’s no disadvantage there … or bring your own compute, which means you don’t have to buy the Snowflake hardware markup.”

Snowflake’s evolution is also being guided by the increasing importance of the data lineage graph — a crucial metadata backbone that originates in engineering pipelines. As analysts noted, this graph is central to operational governance, particularly as workloads diversify across proprietary and open formats.

“The data lineage comes from your data engineering workloads. The lineage graph is what you hang all the operational metadata off. That’s the backbone of your catalog,” Gilbert added.

Supporting Iceberg and Polaris, Snowflake now aims to enable seamless analytics and data science workloads across both internal and external tables. By syncing catalog permissions and advancing its engine’s flexibility, the company is expanding its reach into previously externalized pipelines, Vellante explained.

“What that means is that the external open experience is much more like the Snowflake experience,” he said. “Maybe not as controlled … maybe not fully. But as close … and that’s the game now. That’s the race.”

What’s unfolding isn’t just a response to competitor pressure, but a recalibration of how Snowflake can maintain its premium value. As enterprises build digital twins and agentic systems to act on real-time insights, the underlying engineering stack — and who owns it — becomes a defining battle line.

“One thing worth mentioning is that there was the perception that they were limited to doing really good work, really good workloads when they own the data,” Gilbert said. “By synchronizing with Polaris, their engine now competes with any other engine working on open iceberg tables for data engineering, for analytics. In other words, they’re getting net new workloads or net new data to apply their workloads too. And it’s showing up in their financials.”

Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of Snowflake Summit:

(* Disclosure: TheCUBE is a paid media partner for Snowflake Summit. Neither Snowflake, the primary sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE