Senior Databricks Engineer

This role involves designing, building, and maintaining batch and real-time data pipelines using Databricks technologies. The engineer will develop data products, optimize pipelines for performance, and support production workloads. They will also contribute to CI/CD and engineering best practices while using AI-assisted development tools to enhance productivity. Design, build and maintain batch and real-time data pipelines. Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL. Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines. Optimise pipelines for scalability, reliability and performance. Monitor, troubleshoot and support production workloads.

Desired Candidate Profile

Databricks , (DQX and DB genie in the essential skills) Lakeflow Declarative Pipelines Apache Spark (PySpark) Spark Structured Streaming Python