AI Data Engineer- Databricks & Snowflake

AI Data Engineer- Databricks & Snowflake

Location: Tallahassee, FL, USA

Duration: 12 Months + Extension

Bill Rate: $90/hr on C2C

Job Type: C2C/1099 Contract

Client: To Be Discussed Later

Work Authorization: US-Citizen, H-1B, OPT-EAD, GC-EAD

Job Description

We are seeking an experienced AI Data Engineer Databricks & Snowflake to design, build, and optimize modern cloud-based data platforms that support enterprise analytics and AI initiatives. The ideal candidate will have strong expertise in Databricks, Snowflake, Apache Spark, Python, SQL, cloud platforms (AWS/Azure/Google Cloud Platform), and Generative AI technologies. This role involves developing scalable data pipelines, integrating AI/ML capabilities, and enabling data-driven decision-making through modern data architecture and AI-powered solutions.

Key Responsibilities

Data Engineering & Platform Development

Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Snowflake.

Build batch and real-time ETL/ELT workflows for enterprise data processing.

Develop data ingestion frameworks from structured, semi-structured, and unstructured data sources.

Optimize data pipelines for scalability, reliability, and performance.

Implement Delta Lake architecture and data lakehouse best practices.

Snowflake Data Warehouse

Design and implement enterprise data warehouse solutions using Snowflake.

Develop data models including star schema, snowflake schema, and dimensional modeling.

Optimize Snowflake performance through clustering, partitioning, caching, and warehouse tuning.

Implement secure data sharing, governance, and role-based access control (RBAC).

Develop SQL-based transformations, stored procedures, streams, and tasks.

Databricks Engineering

Develop notebooks, workflows, and jobs using Databricks.

Implement Spark applications using PySpark and Spark SQL.

Build Delta Live Tables (DLT) and Auto Loader pipelines.

Optimize Spark jobs for high-performance distributed data processing.

Implement data quality validation and monitoring frameworks.

AI & Generative AI Integration

Develop AI-enabled data platforms leveraging Generative AI and Large Language Models (LLMs).

Build Retrieval-Augmented Generation (RAG) pipelines using enterprise data stored in Databricks and Snowflake.

Integrate vector databases and embedding models for semantic search.

Develop AI-powered analytics, document intelligence, and conversational AI solutions.

Implement prompt engineering techniques and LLM integrations using OpenAI, Azure OpenAI, or Google Vertex AI.

Build agentic AI workflows and intelligent automation solutions.

Cloud & DevOps

Design cloud-native data solutions on AWS, Azure, or Google Cloud Platform (Google Cloud Platform).

Develop CI/CD pipelines for automated deployment of data engineering solutions.

Implement Infrastructure as Code (IaC) using Terraform or similar tools.

Monitor cloud infrastructure, data pipelines, and platform performance.

Ensure cloud security, governance, and compliance.

Data Integration

Integrate enterprise applications using APIs, streaming platforms, and messaging services.

Develop data ingestion pipelines from ERP, CRM, SaaS, and third-party systems.

Implement Change Data Capture (CDC) solutions.

Build event-driven architectures using Kafka, Event Hubs, or Pub/Sub.

Performance Optimization

Tune Spark workloads and optimize distributed processing performance.

Optimize Snowflake queries and warehouse utilization.

Implement partitioning, caching, indexing, and workload management.

Improve overall platform scalability, reliability, and cost efficiency.

Data Governance & Security

Implement enterprise data governance and metadata management.

Ensure data quality, lineage, cataloging, and compliance.

Develop secure data access models and encryption strategies.

Implement role-based security and auditing.