Databricks Administrator

Databricks Administrator

Location : Remote

Experience

7+ Years of IT Experience with 3 5+ years of Databricks Administration

Job Summary

We are seeking an experienced Databricks Administrator to manage, secure, optimize, and support enterprise Databricks environments. The ideal candidate will have expertise in Databricks workspace administration, Unity Catalog, cluster management, security, performance optimization, automation, and cloud platforms such as Azure, AWS, or Google Cloud Platform. The role involves working closely with data engineers, data scientists, architects, security teams, and cloud administrators to deliver a secure and scalable data analytics platform.

Key Responsibilities

Databricks Administration

Install, configure, and administer Databricks workspaces.

Manage multiple Databricks environments (Development, QA, UAT, Production).

Configure and maintain clusters and cluster policies.

Manage SQL Warehouses and Databricks Jobs.

Monitor workspace health and performance.

Upgrade and maintain Databricks Runtime versions.

Optimize cluster utilization and reduce infrastructure costs.

Security & Governance

Implement Unity Catalog governance.

Configure Role-Based Access Control (RBAC).

Manage users, groups, service principals, and permissions.

Integrate Databricks with Azure AD, AWS IAM, or Okta.

Configure Single Sign-On (SSO).

Manage secrets using Databricks Secret Scopes or cloud key management services.

Ensure compliance with organizational security policies.

Workspace Administration

Create and manage workspaces.

Configure notebooks and repositories.

Manage Git integration.

Administer workspace permissions.

Manage cluster access policies.

Configure job permissions.

Manage libraries and dependencies.

Infrastructure Management

Deploy Databricks using Terraform.

Automate infrastructure provisioning.

Configure networking

VPC/VNet

Private Link

Secure Cluster Connectivity

NSG/Security Groups

Firewalls

Configure storage accounts (ADLS, S3).

Manage DBFS and external storage.

Performance Optimization

Tune Spark clusters.

Optimize autoscaling.

Configure instance pools.

Monitor cluster performance.

Troubleshoot slow jobs.

Optimize notebook execution.

Manage caching strategies.

Monitoring & Troubleshooting

Monitor Databricks clusters.

Configure alerts.

Review audit logs.

Troubleshoot

Cluster failures

Job failures

Authentication issues

Permission issues

Connectivity issues

Library conflicts

Performance bottlenecks

Automation

Automate workspace deployment.

Automate cluster creation.

Automate user provisioning.

Automate permission management.

Automate backup processes.

Automate scheduled jobs using Terraform, Azure DevOps, GitHub Actions, or Jenkins.

CI/CD

Implement CI/CD pipelines.

Git integration.

Azure DevOps Pipelines.

GitHub Actions.

Jenkins.

Deployment automation.

Release management.

Data Governance

Unity Catalog

Data Lineage

Catalog Management

Schema Management

Access Policies

Table Permissions

Row-Level Security

Column-Level Security

Data Masking

Backup & Disaster Recovery

Backup workspace configurations.

Backup notebooks.

Backup cluster configurations.

Disaster recovery planning.

Workspace migration.

Restore procedures.

Required Skills

Databricks

Databricks Workspace Administration

Unity Catalog

Cluster Administration

SQL Warehouses

Jobs

Notebooks

Repos

DBFS

Delta Lake

Delta Live Tables

MLflow

Apache Spark

Spark Administration

Spark SQL

Performance Tuning

Cluster Optimization

Cloud Platforms

Azure

AWS

Google Cloud Platform

Infrastructure as Code

Terraform

ARM Templates

CloudFormation

Scripting

Python

Bash

PowerShell

DevOps

Git

Azure DevOps

Jenkins

GitHub Actions

CI/CD

Monitoring

Azure Monitor

CloudWatch

Datadog

Prometheus

Grafana

Splunk

Security

Azure AD

AWS IAM

OAuth

SAML

LDAP

RBAC

Secret Scopes

Key Vault

AWS Secrets Manager

Databases

SQL Server

Snowflake

Oracle

PostgreSQL

MySQL

Preferred Qualifications

Databricks Certified Data Engineer Associate/Professional

Databricks Certified Administrator (where applicable)

Azure Administrator Associate

AWS Solutions Architect Associate

Experience with Lakehouse Architecture

Experience with Unity Catalog Governance

Experience with Delta Lake Optimization

Knowledge of Apache Iceberg

Experience supporting enterprise data platforms

Soft Skills

Strong analytical and troubleshooting skills

Excellent verbal and written communication

Ability to work with cross-functional teams

Strong documentation skills

Ability to support production environments

Customer-focused mindset

Experience working in Agile/Scrum environments

Nice-to-Have Skills

Azure Synapse Analytics

Microsoft Fabric

Kafka

Airflow

dbt

Kubernetes

Docker

Python-based automation

AI/ML platform administration

Cost optimization and FinOps experience