Principal Architect

Experience: 15+ years

Role Overview: We are seeking a Principal Architect with 15+ years of cloud architecture experience and strong hands-on expertise designing and implementing AWS and Databricks solutions. This person will lead the architecture of scalable, secure, and high-performance cloud data platforms while working across infrastructure, data engineering, governance, and Dev Ops.

Key Responsibilities: Design scalable, secure, and high-performance AWS and Databricks solutions using AWS Networking, Kinesis, Elasti Cache, Redshift, EKS, Cognito, and other PaaS services. Build and maintain ETL/ELT pipelines using Python and PySpark within Databricks, including notebooks, jobs, Delta Lake, and Unity Catalog. Implement Bronze, Silver, and Gold medallion architecture and optimize Spark jobs for performance, scalability, reliability, and cost. Use Terraform to provision and manage Databricks workspaces, clusters, jobs, S3/ADLS storage, networking, IAM, and related infrastructure. Develop advanced SQL for data transformation, validation, analytics, and data quality. Build CI/CD pipelines using Jenkins and Git Hub Actions. Integrate data from databases, APIs, streaming platforms, and file-based sources. Optimize cloud costs through autoscaling, spot instances, scheduling, and resource management. Monitor pipelines, troubleshoot failures, and implement observability and alerting.

Required Skills: AWS Networking and VPC designAWS KinesisAWS Elasti CacheAWS RedshiftAWS Lambda, S3, and GlueAWS EKSAWS Cognito Databricks, Delta Lake, Unity Catalog, and Delta Live Tables Python and PySpark Advanced SQLTerraform Jenkins and Git Hub Actions

Preferred Skills: Azure Data Lake, Data Factory, Synapse, and Key Vault Big data and dimensional modeling Lakehouse architecture Spark and Databricks performance tuning Data governance, lineage, and compliance

Qualifications: Bachelor's degree in Computer Science, Information Technology, or related fieldAWS Certified Solutions Architect - Professional preferred Databricks Certified Data Engineer Professional or Apache Spark certification preferred