Databricks Data Engineer Consultant

Role -Databricks Data Engineer Consultant

About the Role

We are seeking experienced Databricks Data Engineers to join our Data & AI. You will play a pivotal role in designing, developing, and maintaining scalable data pipelines and transformation workflows on Azure Databricks, enabling advanced analytics and data engineering initiatives.

Key Responsibilities

• Design, develop, and optimize scalable data ingestion pipelines using Azure Data Factory, Databricks, and Apache Spark.

• Build and maintain Databricks notebooks using PySpark for data preparation, transformation, and enrichment to support Data Science use cases.

• Collaborate closely with Data Scientists to understand data requirements and ensure timely delivery of clean, structured datasets.

• Implement data quality checks, validation rules, and monitoring mechanisms.

• Integrate data from various sources including structured, semi-structured, and unstructured formats.

• Ensure data security, governance, and compliance with banking regulations.

• Optimize Spark jobs for performance and cost-efficiency in Azure.

• Participate in code reviews, design discussions, and agile ceremonies.

• Document data pipelines, workflows, and technical decisions.