DAIS Special: Salud, Mi Familia

Last week I had the distinct pleasure of attending the Databricks Data + AI summit along with some of my colleagues from Advancing Analytics. Announcements were snapped to the camera roll, potential clients and swag vultures were engaged at the booth, and karaoke was sung/yelled for which I make no apology. San Francisco is a long way from home, and that led to another pleasure: getting to watch The Fast and The Furious for the umpteenth time on the flight back. I did say I enjoyed writing through a lens, and much as La Familia manage to land a series of parachuted cars out of a cargo plane over the Caucasus Mountains, you can now watch me attempt to land this metaphor. Somewhere over the Atlantic — no parachute required — it handed me a better frame for the week than anything I'd scribbled in the dark.

The Fast and The Furious (#1) is a relatively grounded movie about a cop who gets drawn into the underground world of LA street racing so he can crack the case of who's been nicking TV/VHS players off the backs of trucks. Sure enough, one crappy tuna sandwich at a time — no crust — he manages to embed himself in that very gang of criminals, or should I say family of criminals, who in between their amateur heists are just all-around swell folks who love nothing more than cars, cookouts, and Coronas. It's safe to say the stakes for this one are pretty low, and this is objectively and recognisably a film about cars.

Fast-forward twenty-odd years and the same crew — now a few family members bigger — are skydiving cars out of planes, dragging bank vaults through city centres, and eventually driving a 1984 Pontiac Fiero into outer space. Honestly, if you've never seen these films, I'd love to say there are small, incremental logical leaps that take us there. But Dominic Toretto (Vin Diesel) sums up, pretty cleanly, in the first film exactly how the plot is going to work for the rest of this franchise:

But sometimes hopes and dreams is all it takes. Think back to the early days of Spark Summit, when we were all pretty jazzed that we could run some Spark in the cloud. Look at us now: just last week, over thirty thousand people descended on San Francisco, all clamouring to see what the next instalment of the Databricks franchise had to offer. Between the slew of announcements promising to revolutionise and unify basically everything in the platform to date, chats with Satya Nadella (Microsoft) and Greg Brockman (OpenAI) showing how deep and rich the wider partnerships landscape is, and the Chainsmokers playing the after-party at the Giants' ballpark, I came home with one question remarkably similar to my thoughts on the Fast and Furious franchise: how did we get from there to here?

The franchise recap, except it’s a data platform

The early instalments of Databricks were pretty grounded too. A better place to run Spark. Then we started to focus on lakes. Then "Lakehouses," which was the moment marketing got a budget. Parallels-wise, that makes it a bit like Fast Five (#5), where the films switched gears and evolved from a franchise about underground racing and car culture into full-on globetrotting heists. We'd get to enjoy about three of those before it eventually turned into the petrolheads' version of Avengers, but I digress.

Speaking of heists: don't you just love it when they introduce the characters one by one? There's something about pulling together just the right team to get the job done that gets me all hyped. Looking over the gamut of big announcements from DAIS, my inner Guy Ritchie is ready to roll the character montage. To pull off this job, we'll need our very own Usual Suspects — our Ocean's (Lake's?) Eleven, our non-Fabric Familia:

• The Face (a.k.a. Lakehouse//RT, powered by Reyden): Everyone in the room wants something from them at once — a question, a number, a decision — and somehow they answer every single person instantly, without missing a beat or letting the room see them sweat.
• The Architect (a.k.a. Genie One, Genie Agents, Genie Ontology): No one else has a better view of how it's all connected. Ask them a question and they'll break it down for the room in plain English, no mucking about.
• The Bookkeeper (a.k.a. Unity AI Gateway): Watches every door in the building at once — every model, every agent, every tool — and knows exactly who went through which one, when, and what it cost. Makes sure nothing happens off the books, and isn't afraid to make the go/no-go call.
• The Wheelmen (a.k.a. LTAP — Lake Transactional/Analytical Processing): One's running the job right now, the other's working out the next one — but they're reading off the exact same ledger as it's being written, not waiting on someone to radio the updates across. No copies, no lag, no two versions of the truth.
• The New Recruit, Open to Outside Work (a.k.a. Omnigent): Young, talented, and — unusually for this family — perfectly happy to go freelance on someone else's job too. The old heads aren't quite sure what to make of that yet, but they get results, and the family lets them roam further than most.
• The Muscle (a.k.a. Lakewatch): Doesn't say much, just stands at the door watching everyone who comes and goes. You won't notice them until something's gone wrong, and then suddenly they're the only one who matters.
• The Cleaner (a.k.a. Genie ZeroOps): By the time you've noticed something's wrong, they've already found it, fixed it on a copy nobody will ever see, and they're just waiting for you to say go.
• The One Who Never Forgets a Face (a.k.a. CustomerLake): Doesn't care what alias you're using this time, what city you last pulled a job in, or how long it's been — they'll tell you exactly who you're really dealing with, their whole history, in one look. Every crew needs someone who remembers, because nobody walks in for the first time twice.

And underneath all of them, running the town the way they always have, is the Guv'nor (a.k.a. Unity Catalog). There's nothing that happens in this family they don't know about — eyes and ears everywhere — and if they say you're in, you're in. The Bookkeeper answers to them, not the other way round: every entry the Bookkeeper logs for an agent or a model is still access the Guv'nor signed off on first. New faces keep joining the crew. The Guv'nor's been here since the first job.

All laid out before you, it's one formidable group, ready for any task thrown at them. At the beginning, all they could do was drive — clusters, that is, and sometimes Honda Civics. Now they're a family of computer-hacking martial arts specialists able to cheat death at any given turn. Sorry, I mean components of a fully unified data, analytics, and AI platform. You've got to wonder at this point if there's any job they can't do. When it comes to data platforms, the message coming out of DAIS was very clear: whether it's AWS, Azure, or GCP, none of that matters. As long as you've got Databricks, you can do it all.

What's going on under the hood?

One thing you come to realise about these huge franchise films is that not every character gets to be part of the character intro montage, but that doesn't make them any less memorable (anyone remember Ja Rule?).

Fun fact, and a complete aside: in a past life, I was an extra in a heist film — sadly uncredited, and shot in a nightclub where you can't really see me, but I'd like to think I brought a lot of heart to the film. They all got fun nicknames based on sweets; ten points to anyone who can give me a good one so I can feel like one of the gang...

Anyway — whilst it's always fun to see the main gang run it back on the keynote, buoyed by my own experience as the unappreciated extra, I was also on the lookout for the non-keynote but nonetheless important breakout talks. So let's pop the hood off this convention and see if Databricks is more than just a nice-looking vehicle.

Global UC

One of the very first talks I landed in, titled "Unify your global data and AI estate across clouds with Global UC," proposed huge changes to how we think about managing Databricks at scale.

Think about how Databricks works today: enterprise organisations that exist across multiple clouds and regions find themselves managing disjointed, siloed accounts, each with their own identity provider backing them. If you want to share data across metastore boundaries, you're basically limited to Delta Sharing — there's no straightforward way to share across accounts, clouds, or regions within that org. So what is Databricks proposing to solve this?

• Making metastores directly addressable containers across regions, clouds, and accounts — i.e., a four-level namespace, metastore.catalog.schema.object. Now you can let a query drift into a metastore in Tokyo (honestly, sometimes the metaphor is just presented to you) and UNION ALL it with a table in Canada, with policy enforced back at the source the whole time.
• Elevating accounts to a cross-cloud concept, where workspaces can be created in AWS, Azure, or GCP on demand within that one account.
• Unifying the management of multiple accounts under an organization, or "root account," where settings, policy, and billing can be standardised regardless of account ownership.
• Single login across all accounts, plus a single company subdomain.

Very, very cool stuff. Very, very much not available yet. A lot of this will get peppered out across various stages of preview over the next year, but it's absolutely something to get hyped about.

OpenSharing SecureConnect + Private Network Gateway

Databricks is one of those fascinating platforms that makes sharing data sound like the easiest thing in the world. "Just chuck it in a Delta Share, no sweat." "Quick job, in and out, done before you know it." And of course, in a total vacuum where everyone's accessing this over the public internet, it really is that simple. Annoyingly, though, there are all these networking teams and InfoSec people out there who just aren't thrilled with the idea of leaving the keys under a rock by the front door. A happy platform is a secure platform, after all.

Prior to these announcements, if you wanted to securely share data over a private connection, you'd need to make sure your consumers had their own connection to the underlying storage account(s) your share pointed to — per consumer, per storage account. At scale, that's a proper headache, and it requires coordination across both provider and consumer to make sure it's all working as expected.

Enter stage left: OpenSharing SecureConnect. SecureConnect offers a Databricks-managed proxy that greatly simplifies that pattern, letting multiple recipients share the same networking configuration. Set up once, share simply.

But that's not the only exciting piece of news — there's also a Private Preview of Private Network Gateway on Azure that flips the problem around. Instead of recipients reaching in to your storage, it's about your serverless workloads reaching out — to private APIs, on-prem systems, or some partner's database — all through one connection instead of bespoke plumbing for every single resource. For those of us who found managing the PE approval endpoint on NCCs a headache (it's honestly a shocking bit of UI), this is a welcome bit of housekeeping.

ABAC, RBAC, Backstreet's Back

In our current agentic era, platform administrators are governing an ever-growing estate without a proportional rise in headcount to manage it. Data, models, agents, and the people using them are multiplying faster than any team can hand-craft permissions for, and the old tools weren't built for that pace. So much of access control today is coarsely defined — a single MANAGE grant hands over the ability to view grants, grant permissions, edit metadata, and delete, all in one lump — and at scale, coarse controls are exactly how you end up either over-permissioned or underwater. The answer is to apply controls on data as efficiently as possible without loosening your posture to do it.

There were a lot of updates and roadmap items here, but to share a few:

• Granular controls for permissions[Private Preview] Splitting out the MANAGE permission into more granular grants, e.g., READ METADATA, MANAGE ACCESS CONTROL, etc.[Private Preview] Splitting out the MODIFY permission into INSERT, UPDATE, DELETE.
• ABAC Policy Types[Beta] Service Policies: filter interactions over MCP and model services (e.g., mask PII in request/response if a service call is made by an agent).[Beta] GRANT Policies: grant permissions using attributes (e.g., the Supply Chain team can READ any data tagged "supply"; the Data Science team can EXECUTE any model where the provider tag is "anthropic").[Private Preview] DENY Policies: deny permissions using attributes (e.g., the Supply Chain team cannot READ "restricted" data; table creators — and therefore OWNERS — can be denied MANAGE ACCESS CONTROL for tables they create).
• Richer Attribute Type Conditions[Private Preview] Identity Attributes based on the principal: pull in attributes set within the IdP (e.g., Entra) or by a user admin in Databricks, e.g., identity_has_attribute_value('country', 'Denmark').[Private Preview] Context Attributes based on the request: respect attributes set by external callers, e.g., has_context_attribute_value('on_behalf_of_user', 'true').
• Role-Based Access Control[Public Preview Soon] Lets a user assume a role to work in an access context and isolate their active permissions — a user's personal grants are suspended, leaving them with only the assumed role's permissions.

Stack all of that on top of metastore-level policy once Global UC lands, and admins finally get a fighting chance at governing this mess at scale.

And here's the thing worth noticing, now that the hood's been up for a while. Global UC, SecureConnect, the whole ABAC toolkit — strip away the feature names and they're all firing on the same cylinders. One policy that reaches every region instead of one per metastore. One network config instead of one per recipient. One rule instead of a thousand hand-cut grants. Three different launches. Same instinct every time: pull it all in under one roof, and make being under that roof feel effortless — unified, like family.

It's about family

Every Fast & Furious film is, in the end, about family. It's the emotional engine the whole absurd machine runs on. The data-platform translation of "family" is unified: one platform, one organisation above all your accounts, one catalogue addressable everywhere. Everybody comes to the cookout, nobody gets left behind, and — asterisk pending — with no cloud lock-in. You're free of needing specific tooling from any of the hyperscalers. The data moves across a neutral backbone. You're liberated.

Which, of course, is exactly the bit Dom already answered for us. The pitch is that you escape cloud lock-in, and the mechanism for escaping it is collapsing your entire data and AI estate onto one vendor's backbone. That isn't the absence of lock-in — it's lock-in with a better cast and a bigger budget. It can still be a perfectly sane trade: a neutral backbone that frees you from any single hyperscaler is worth real money to plenty of organisations. But it's a choice of who locks you in, sold as the end of lock-in — and the family that won't let you leave is still a family that won't let you leave.

Call it Stockholm syndrome, but this is a family I'm more than happy to be part of. Corny as it is, I'm in it. I'm pro-Databricks, I'd happily turn up to the cookout, and on the engineering side, I think it might be the best family going. But love it with your eyes open.

To their credit, the doors aren't all bolted shut. The open-source pieces — Omnigent chief among them — leave real room for others to get involved: the franchise confident enough to greenlight a spin-off. Hobbs & Shaw, if you like — different stars, their own billing, off doing their own thing. All in the one cinematic universe, mind you. The spin-off never quite leaves the family either.

So no, I'm not mad about any of it. The escalation is genuinely funny, and a little bit magnificent, the same way a car flying between two skyscrapers is magnificent — and it's all built on real engineering muscle. I'll be first in the queue for the next instalment. Just walk into the cinema knowing which film you've bought a ticket to. Keep one foot near the exit — or at least near your own storage — because that's the one part of the family they'll always let you bring with you.

We got from a tuna sandwich to outer space one set-piece at a time. The next one will be bigger. It always is. Salud, mi familia.

Liam is a Data & AI consultant at Advancing Analytics, professional fence-sitter on lock-in, and a karaoke menace who will not be taking requests. He writes about platforms, the people who run them, and the gap between the keynote and the changelog at youmecicd.com. He's still annoyed about the parachuted-cars-over-mountains physics, but not annoyed enough to stop watching.