Lead Azure Data Engineer

Lead Azure Data EngineerRole: Lead Azure Data EngineerLocation: Remote (Candidate should be comfortable to follow PST Time zone)

Bachelor's or Master's degree in Computer Science, Engineering, or a related field 10+ years of experience in data engineering / data architecture, with deep hands-on expertise in Microsoft AzureExpert-level experience with Azure Databricks and Apache Spark (PySpark, Spark SQL) including Delta Lake, Unity Catalog, job orchestration, and performance tuningStrong experience with Snowflake on Azure, including schema and data model design, performance and cost optimization, RBAC, Streams, and TasksDBT (Core or Cloud) expertise including project structure and best practices, tests, exposures, and macrosStrong fundamentals in SQL and PythonProven experience building ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Synapse / Microsoft Fabric pipelinesFamiliarity with streaming platforms such as Event Hubs and/or KafkaSolid understanding of lakehouse and data warehousing architectures, dimensional modeling, medallion architecture patterns, and data quality frameworksStrong knowledge of security and governance, including Microsoft Purview (catalog, lineage), data masking and PII handling, managed identities, and private networkingExcellent communication and documentation skills, with the ability to create architecture and design documents and present to both technical and business stakeholdersConsulting/agency experience, with the ability to lead multiple concurrent projectsWillingness to travel as requiredNice to HaveExperience with Azure ML, feature stores, or serving ML pipelines from Databricks or SnowflakeInfrastructure as Code (IaC) using Terraform or BicepContainerization experience with DockerObservability and data quality tooling, such as Great Expectations / Delta Expectations, Databricks Quality Flows, Monte Carlo, and DatadogPreferred CertificationsDatabricks Certified Data Engineer Professional or Databricks ArchitectMicrosoft Azure: DP-203 – Azure Data EngineerAzure Solutions Architect ExpertAzure Security Engineer AssociateSnowflake: SnowPro CoreSnowPro AdvancedLeadership & CompetenciesTechnical leadership of cross-functional data teamsAbility to set engineering standards and coach/mentor engineersStrong bias toward documentation, automation, and measurable outcomes (SLAs, reliability, cost efficiency) Proven ability to de-risk complex deliveries, manage senior stakeholders, and drive alignment and consensus across teamsJob Description / ResponsibilitiesPartner with client stakeholders to translate business objectives into scalable Azure data architectures and delivery roadmapsArchitect and implement lakehouse and data warehouse solutions using Azure Databricks (Delta Lake, Unity Catalog), Databricks Bundle, Snowflake on Azure, dbt (Core / Cloud), ADLS Gen2, and medallion architecture patternsDesign and build robust ingestion and transformation pipelines using Azure Data Factory (ADF) and Microsoft Fabric / Synapse pipelinesOrchestrate ELT workflows using Databricks Jobs, Delta Live Tables, and dbt modelsEstablish best practices for performance and cost optimization, including cluster sizing and autoscaling, Photon and SQL Warehouses, Snowflake virtual warehouses, caching, file layout optimization, and Z-orderingDrive observability and monitoring using Azure Monitor, Log Analytics, and Databricks metricsImplement robust data governance and security using Azure Authentication (Unity Catalog, RBAC / ABAC, managed identities, private endpoints, and VNet injection), and Azure Key Vault–backed secretsLead code and design reviews, setting standards for PySpark, SQL, and dbt, unit and integration testing, and data quality checks (expectations and constraints) Implement CI/CD pipelines using Azure DevOps or GitHub ActionsGuide multi-domain programs (Healthcare, Retail, BFSI, or similar) Mentor data engineers and ensure high-quality, production-ready deliverablesEvangelize solutions through clear documentation, reference architectures, and executive and stakeholder presentations