Senior Databricks Platform Developer

Senior Databricks Platform Developer Location: Remote Type: Contract Job Description Role Overview We are looking for a Senior Databricks Developer with strong hands-on experience in designing, developing, implementing and optimizing enterprise-scale data solutions on the Databricks Lakehouse platform. The ideal candidate will have strong development expertise in Python, PySpark, SQL, Delta Lake and Databricks, along with experience building automated workflows, CI/CD pipelines, infrastructure-as-code and AI-enabled solutions. This role will focus on developing reusable platform components, engineering automation, optimizing data processing workloads and enabling scalable, secure and cost-efficient data solutions. The candidate will work closely with data engineering, data science, cloud infrastructure and security teams to improve platform capabilities and developer productivity. Key Responsibilities 1. Databricks Development & Engineering Design, develop and implement scalable data engineering solutions using Databricks, Python, PySpark and SQL. Develop and optimize notebooks, workflows, jobs, Delta Live Tables and Lakeflow pipelines. Build reusable frameworks, libraries, templates and automation components for enterprise data engineering. Implement and optimize Delta Lake solutions, including partitioning, liquid clustering, Z-Ordering, OPTIMIZE and VACUUM. Develop solutions using Unity Catalog, including catalogs, schemas, permissions and governed data access. Work with Databricks Workflows, job clusters, SQL warehouses and serverless compute. Troubleshoot and optimize application performance, Spark execution, query processing and resource utilization. 2. CI/CD & Development Automation Design and implement CI/CD pipelines for Databricks applications using Azure DevOps or GitHub Actions. Develop and deploy Databricks solutions using Databricks Asset Bundles, Git integration and Terraform. Implement automated testing, code quality checks, deployment validation and release management. Develop reusable infrastructure-as-code modules and deployment templates. Automate environment provisioning, configuration management and application deployment across development, test and production. Implement version control, branching strategies and deployment rollback mechanisms. 3. AI & Agentic Workflow Development Design and develop AI-powered automation and agentic workflows for platform operations and engineering processes. Build AI agents using Databricks Mosaic AI Agent Framework, Model Serving and Vector Search. Integrate AI agents with enterprise applications and services using APIs and Model Context Protocol (MCP). Develop intelligent workflows for job failure analysis, root-cause investigation, operational monitoring and remediation recommendations. Leverage Generative AI tools, including GitHub Copilot and Anthropic models, to accelerate development, testing and documentation. Implement AI governance, evaluation, observability and human-in-the-loop controls. 4. Platform Performance & Cost Optimization Analyze and optimize Databricks workloads, Spark jobs, SQL queries and compute utilization. Implement cluster optimization strategies, autoscaling, auto-termination and workload management. Monitor DBU consumption, billing usage, compute utilization and query performance. Develop dashboards and automation to identify cost anomalies, idle resources and optimization opportunities. Apply Photon, serverless compute and workload consolidation strategies where appropriate. Implement cost governance and resource optimization practices. 5. Platform Integration & Governance Develop solutions leveraging Unity Catalog, Delta Lake and Databricks Lakehouse architecture. Implement secure data access, service principals, identity integration and role-based access controls. Work with cloud storage, networking, secrets management and enterprise security integrations. Support the development and deployment of enterprise ML, GenAI and RAG workloads. Collaborate with cross-functional engineering teams to establish development standards and reusable engineering practices. Required Skills Strong hands-on experience with Databricks development and engineering. Advanced programming skills in Python, PySpark and SQL. Strong experience with Apache Spark, Delta Lake and Lakehouse architecture. Experience developing Databricks notebooks, jobs, workflows and data pipelines. Hands-on experience with Databricks Asset Bundles and CI/CD automation. Experience with Azure DevOps or GitHub Actions. Strong understanding of Terraform and Infrastructure as Code. Experience with Unity Catalog and Databricks security/governance. Knowledge of Spark performance tuning and compute optimization. Experience with Git, automated testing and deployment practices. Good to Have Azure Databricks experience. Experience with Databricks Mosaic AI, Agent Framework and Model Serving. Knowledge of Vector Search, RAG and MCP integrations. Experience with Generative AI and agentic workflow development. Exposure to Databricks system tables, FinOps and DBU cost optimization. Experience with cloud infrastructure and automation. Databricks or Azure certifications. … mais Pesquise vagas mais rápido com Premium Acesse informações sobre empresas, como prioridades estratégicas, tendências de número de funcionários e muito mais Guilherme e outros milhões de usuários têm contas Premium Reative o Premium Aproveite o suporte 24 horas e cancele quando quiser. Sobre a empresa Enterprise Solutions Inc. 170.106 seguidores Seguir Atividades dos serviços de tecnologia da informação • 501-1000 funcionários • 579 no LinkedIn Enterprise Solutions, Inc. is a technology-intensive services company. We develop and deliver software and engineering solutions to our partners; we work relentlessly to keep our clients happy, and we measure our business success with the trust we earn from them. Enterprise Solutions offers implementation se