Senior Data Engineer (Databricks & Cloud Analytics)

Responsibilities: Design, develop, test, deploy, and maintain enterprise data engineering solutions using cloud and big data tech Build scalable ETL/ELT pipelines with Databricks, Spark (PySpark), Delta Lake, and Azure Data Factory Create high-performance data ingestion, transformation, and integration frameworks for analytics and reporting Develop and optimize SQL queries, stored procedures, and data transformations Model data for BI, analytics, and AI initiatives Integrate RESTful APIs, event-driven architectures, and legacy SOAP services Develop streaming and messaging solutions using Apache Kafka Monitor and optimize production data pipelines; perform root cause analysis and long-term fixes Implement data quality, governance, security, and performance best practices Manage source code with Git; follow CI/CD and DevOps practices Collaborate with solution architects, product owners, analysts, and engineers to translate requirements Participate in Agile ceremonies and document deployments, tests, and runbooks Mentor junior engineers and contribute to engineering standards and continuous improvement

Key requirements: 6+ years designing, developing, and supporting enterprise data platforms or cloud analytics Hands-on Databricks, Apache Spark (PySpark), Python, SQL, Scala Experience with Delta Lake, Unity Catalog, Delta Live Tables, Databricks SQL, MLflow, Databricks Jobs ETL/ELT design using Azure Data Factory or similar Experience with Apache Kafka and event streaming REST APIs, JSON, XML, SOAP integration Azure Data Lake Storage (ADLS Gen2) or equivalent Git, Linux, shell scripting; IaC familiarity with Terraform preferred CI/CD with Azure DevOps or GitHub Actions; Agile (Scrum/Kanban/SAFe) strong communication collaboration problem-solving Databricks Apache Spark (PySpark) Python