Databricks Security Hardening: Unity Catalog, Network Controls, and Audit Logging
Retool's new app builder is where AI-generated code ships safely
Building apps with AI is easy. Getting them to production safely is another story.
Databricks deployments often start as data science sandboxes and grow into production platforms handling sensitive data -- sometimes without the security controls catching up. A workspace where analysts can create clusters with any instance profile, store database passwords in notebook cells, and access the UI from any IP address is not production-ready. This guide covers the four hardening layers that matter most: Unity Catalog for data governance and fine-grained access control, IP access lists and private link for network isolation, secret scopes for credential management, and audit logging for SIEM integration.
Enable Unity Catalog and Migrate from Legacy Hive Metastore
Unity Catalog is the prerequisite for all fine-grained data access controls. If your workspace still uses the legacy Hive metastore, migrating to Unity Catalog is the highest-impact security action available.
Create a Unity Catalog metastore at the account level
In the Databricks account console (accounts.azuredatabricks.net or accounts.cloud.databricks.com), create a Unity Catalog metastore in your primary region. Assign it to all workspaces in the account. Create a storage account or S3 bucket as the metastore's managed storage location -- this is where managed tables will store their data files.
Create catalogs per environment or business domain
Create separate Unity Catalog catalogs for dev, staging, and production environments, and optionally by business domain (finance_catalog, customer_catalog). Assign catalog-level GRANT permissions to appropriate groups. Production catalogs should be accessible only to production service principals and approved analysts -- not all workspace users.
Implement column-level security on sensitive attributes
For tables containing PII or regulated data, use column masks to restrict access to sensitive columns: `ALTER TABLE customers ALTER COLUMN ssn SET MASK mask_ssn USING COLUMNS (user_role)`. The mask function returns the unmasked value for users with the data_steward role and masks the value for all others. Combine with row-level security views for tables where both row and column restrictions are needed.
Audit Unity Catalog permission grants
Query the Unity Catalog information schema to audit current permissions: `SELECT * FROM system.information_schema.table_privileges WHERE grantee='<group>'`. Run this audit quarterly to identify over-provisioned groups and revoke permissions that are no longer needed. The `system.access` schema in Unity Catalog provides fine-grained access audit logs showing who queried which tables -- use this as the primary data access audit trail.
Restrict Workspace Access with IP Access Lists and SSO
IP access lists and SSO enforcement are workspace-level controls that restrict who can authenticate to the Databricks UI and API surface.
Configure IP Access Lists
In Admin Settings > Security > IP Access List, add your corporate egress IP CIDR ranges and CI/CD system IP ranges. Set the access list to Allowlist (block all by default except listed ranges). Test from a non-corporate IP to verify the block is working before communicating the change to users. Maintain this list in code (Terraform Databricks provider supports `databricks_ip_access_list`) rather than manually in the console.
Enforce SSO via Entra ID or Okta
Configure SAML 2.0 SSO in Admin Settings > Single Sign On. Connect to Entra ID or Okta as the identity provider. Enable SCIM provisioning to synchronize group membership from your identity provider -- this ensures that when a user is deprovisioned in Okta/Entra, their Databricks access is immediately revoked. Disable username/password authentication for non-admin accounts after SSO is confirmed working.
Enforce personal access token policies
In Admin Settings > Advanced, set a maximum token lifetime (90 days maximum, 30 days recommended). Disable the ability for users to create tokens with no expiry. Audit existing tokens via the Databricks REST API: `GET /api/2.0/token-management/tokens` returns all tokens with their creation date, expiry, and comment. Revoke tokens with no expiry or comments that do not identify the owning application.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
Manage Secrets and Cluster Security
Secrets appearing in notebook cells and overly permissive cluster instance profiles are the two most common Databricks credential exposure vectors.
Create cloud-backed secret scopes
For Azure: create an Azure Key Vault-backed secret scope with `databricks secrets create-scope --scope <name> --scope-backend-type AZURE_KEYVAULT --resource-id <key-vault-resource-id> --dns-name <key-vault-dns-name>`. For AWS: use the Databricks-backed scope with Databricks managing the encryption, or create an AWS Secrets Manager integration via the Databricks provider. Reference secrets in notebooks with `dbutils.secrets.get(scope, key)` -- the value is masked in all output.
Define cluster policies for security constraints
Create a cluster policy requiring: `autotermination_minutes` minimum 60 (prevent always-on clusters), `instance_profile_arn` fixed to an allowlist of approved instance profiles (prevent users attaching overly permissive roles), and `spark.databricks.delta.preview.enabled` set to false if not needed. Assign this policy as the default policy for non-admin users so every cluster they create inherits these constraints.
Enable audit logging and forward to SIEM
In the Databricks account console, enable audit logging delivery to an S3 bucket or Azure Storage container. Configure your SIEM to ingest from this destination. Set up alerts for the high-value event types: workspace exports, permission changes, and data access on sensitive schemas. Databricks audit logs are structured JSON -- write SIEM detection rules against the `serviceName`, `actionName`, and `userIdentity.email` fields.
The bottom line
Databricks security has a clear priority order: Unity Catalog first (without it, all data access is workspace-wide with no fine-grained control), IP access lists second (restricts the UI and API attack surface to known IP ranges), secret scopes third (eliminates credentials appearing in notebooks), and cluster policies fourth (prevents privilege escalation via instance profile selection). Audit logging should be enabled from day one -- retroactively enabling it means you have no forensic record of the data accesses that occurred before it was on.
Frequently asked questions
What is Unity Catalog and how does it differ from the legacy Hive metastore?
The legacy Hive metastore is per-workspace: each Databricks workspace has its own metastore, and data sharing across workspaces requires copying data or complex configurations. Unity Catalog provides a centralized, account-level metastore shared across all workspaces. It introduces a three-level namespace (catalog.schema.table), supports fine-grained access control at the column and row level, provides data lineage tracking, and enforces consistent permissions across all workspaces. Unity Catalog is the prerequisite for column-level security and row-level security -- neither is available with the legacy Hive metastore.
How do Databricks IP Access Lists work and what do they protect?
IP Access Lists restrict access to the Databricks workspace UI and REST API based on source IP CIDR ranges. Configure them in the workspace admin console: Admin Settings > Security > IP Access List. Add your corporate egress IP range and the IP ranges of your CI/CD systems (GitHub Actions IP ranges for GitHub-hosted runners). IP Access Lists protect against credential theft scenarios where an attacker has stolen a Databricks personal access token but is operating from an IP outside your corporate range. They do not affect cluster-to-cloud storage traffic or cluster-to-cluster communication.
How should you manage secrets in Databricks notebooks?
Never put credentials directly in notebook cells -- they appear in notebook history and can be exported. Use Databricks Secret Scopes backed by Azure Key Vault (Azure deployments) or AWS Secrets Manager (AWS deployments). In notebooks, retrieve secrets with `dbutils.secrets.get(scope='<scope-name>', key='<secret-key>')` -- this returns the value without displaying it in cell output or notebook history. The secret value is masked in Databricks logs. For cluster init scripts that need credentials, use cluster environment variables populated from secret scopes via cluster policy, not hardcoded values.
What are cluster policies and how do they restrict what users can create?
Cluster policies define constraints on cluster configuration parameters that a specific group of users can set when creating clusters. Use them to: enforce a maximum cluster size (prevent runaway costs and resource exhaustion), require specific Spark configuration parameters (e.g., encryption settings), block clusters with no autotermination (prevent idle clusters accumulating costs and attack surface), and restrict which instance profile ARNs can be attached (preventing users from creating clusters with overly permissive IAM roles). Assign cluster policies to groups via Unity Catalog or the legacy workspace admin console.
How do you configure private link for Databricks network isolation?
Databricks private link (also called No Public IP / NPIP mode) routes all cluster traffic through the customer's VPC without traversing the public internet. Configure it at workspace creation: set `no_public_ip=true` in the workspace configuration. Create a VPC endpoint for the Databricks relay service in your VPC. Clusters will not have public IP addresses and cannot be accessed from the public internet directly. Note: private link also requires Databricks UI traffic to route through a private endpoint -- configure a VPC endpoint for the Databricks web application and update your DNS to resolve the Databricks hostname to the private endpoint IP.
What Databricks audit events should you forward to your SIEM?
Enable Databricks audit logs and forward to your SIEM. Key events to alert on: `accounts.login` failures (brute-force attempts), `clusters.create` with no autotermination policy (policy bypass), `workspace.export` events on notebooks containing sensitive data indicators (potential data exfiltration), `permissions.change` on high-value tables or catalogs (privilege escalation), and `tokens.create` with long-lived or no expiry (persistent access setup). Configure alerts for any `databricksSql.execute` event from unexpected user accounts querying tables in PII or financial schemas.
How do you implement row-level security in Unity Catalog?
Unity Catalog row-level security uses dynamic views: create a view over the base table with a WHERE clause that filters rows based on `current_user()` or group membership. For example: `CREATE VIEW secure_transactions AS SELECT * FROM transactions WHERE region = (SELECT region FROM user_region_mapping WHERE user = current_user())`. Grant users READ on the view, not the underlying table. The view's filter is evaluated at query time using the authenticated user's identity, so the same view returns different rows to different users. This approach works across all Databricks SQL and notebook compute types without requiring application changes.
Sources & references
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, daily, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals every morning.
Win a $2,495 Black Hat pass.
Full-access to Black Hat USA 2026 in Las Vegas. Subscribe free to enter.