Data Engineer

Job Description Roles & Responsibilities Job Description – Data Engineer (PySpark / Databricks) Position Data Engineer – PySpark / Databricks Experience 4–8 Years Location UAE – Dubai / Abu Dhabi Employment Type Full-Time / Contract Work Mode Onsite / Hybrid – As per client requirement Role Overview We are looking for an experienced Data Engineer with strong expertise in PySpark, Python, SQL, Databricks, and cloud data platforms to design, develop, and maintain scalable data pipelines and data processing solutions. The ideal candidate should have hands-on experience working with large datasets, developing ETL/ELT pipelines, implementing data transformations, optimizing Spark workloads, and building cloud-based data platforms. Key Responsibilities Design, develop, and maintain scalable data pipelines and ETL/ELT workflows. Develop high-performance data processing solutions using PySpark and Apache Spark. Build data pipelines to ingest data from databases, APIs, files, applications, and other sources. Perform data cleansing, transformation, aggregation, and validation. Develop and optimize Spark SQL and PySpark jobs. Work with structured, semi-structured, and unstructured data. Implement data pipelines using Databricks and cloud-based data platforms. Optimize Spark jobs for performance, scalability, memory utilization, and processing time. Implement data quality, validation, monitoring, and error-handling mechanisms. Work with data lakes, lakehouses, and data warehouse environments. Develop reusable and scalable data engineering frameworks. Collaborate with Data Architects, Data Scientists, Business Analysts, and application teams. Troubleshoot production data pipeline issues and perform root-cause analysis. Participate in code reviews, testing, deployment, and production support. Maintain technical documentation for data pipelines and data architecture. Mandatory Skills 4–8 years of experience in Data Engineering. Strong hands-on experience with PySpark and Apache Spark. Strong programming experience in Python. Strong proficiency in SQL. Experience developing ETL/ELT pipelines. Hands-on experience with Databricks. Strong understanding of Spark architecture, DataFrames, Spark SQL, transformations, and actions. Experience working with large-volume datasets. Good understanding of data warehousing and data lake concepts. Experience with relational and NoSQL databases. Knowledge of data pipeline performance tuning and optimization. Experience working with at least one major cloud platform – Azure, AWS, or GCP. Cloud & Data Technologies Candidates with experience in one or more of the following will be preferred: Azure Azure Data Factory Azure Databricks Azure Data Lake Storage Azure Synapse Microsoft Fabric AWS AWS Glue Amazon EMR Amazon S3 Redshift Kinesis GCP BigQuery Dataflow Dataproc Cloud Storage Good-to-Have Skills Delta Lake / Lakehouse architecture Apache Kafka Apache Airflow Snowflake dbt Hadoop / Hive Data modelling Data governance and data quality Docker and Kubernetes CI/CD Git / GitHub / GitLab Terraform Experience with real-time / streaming data pipelines Exposure to Generative AI / ML data pipelines Preferred Technology Stack Programming: Python, SQL Big Data: Apache Spark, PySpark, Kafka Data Platform: Databricks, Delta Lake, Snowflake Cloud: Azure / AWS / GCP Orchestration: Airflow / Azure Data Factory DevOps: Git, CI/CD, Docker, Kubernetes Candidate Profile The ideal candidate should: Have strong analytical and problem-solving skills. Be experienced in designing scalable data solutions. Have strong debugging and troubleshooting capabilities. Understand data architecture and data lifecycle concepts. Be comfortable working with large and complex datasets. Have good communication and stakeholder-management skills. Be able to work independently as well as collaboratively in a multicultural environment. Have experience working in enterprise or large-scale data environments. Education Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Data Science, or a related discipline. Desired Candidate Profile Relevant Job Titles for Sourcing Data Engineer Senior Data Engineer PySpark Developer PySpark Data Engineer Databricks Data Engineer Big Data Engineer Cloud Data Engineer Data Platform Engineer ETL Developer Data Engineering Specialist Employment Type Full-time Company Industry RecruitmentPlacement FirmExecutive Search Department / Functional Area Engineering Keywords Data Pipeline EngineerData Systems EngineerData Integration EngineerData QualityBig DataETL DeveloperDatabase Developer Get real-time job updates only on our App