zortech solutionsScottsdale, AZ
Data Engineer-(DE)
Data Engineer-(DE)
Data Engineer-(DE)
zortech solutionsScottsdale, AZ
6 days ago
Computer Systems Design ServicesCustom Computer Programming ServicesComputing Infrastructure Providers, Data Processing, Web Hosting, and Related Services
Apply for this role (opens in a new tab)Data Engineer-(DE)
Location: Scottsdale AZ (day 1 onsite)
Duration: CTH and Fulltime
Job Description:
- Must have skill set: Spark, S3, Glue, AWS Redshift and infrastructure, AWS Data Lake Formation and Glue components, data security, SQL, and Python 6-8 years of IT experience focusing on enterprise data architecture and management.
- Experience in Conceptual/Logical/Physical Data Modelling & expertise in Relational and Dimensional Data Modelling
- Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
- Experience with Spark scala
- Data Lake concepts such as time travel and schema evolution and optimization
- Structured Streaming and Delta Live Tables with Databricks a bonusExperience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support
- Advanced level understanding of streaming data pipelines and how they differ from batch systems
- Formalize concepts of how to handle late data, defining windows, and data freshnessAdvanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc
- Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.
Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonusFamiliarity with concepts such as late data, defining windows, and how window definitions impact data freshnessAdvanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)
Indexing and partitioning strategy experience
Debug, troubleshoot, design and implement solutions to complex technical issues
Experience with large-scale, high-performance enterprise big data application deployment and solution
Architecture experience in AWS environment a bonusFamiliarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonusExperience with Gitlabs and CloudWatch and ability to write and maintain gitlabs for supporting CI/CD pipelines
Experience working with AWS Lambdas for configuration and optimization and experience with S3Familiarity with Schema Registry, message formats such as Avro, ORC, etc.
Also on the board Same function, level within a rung
Level
Lead
Location
Scottsdale, AZ
Occupation
Data Warehousing Specialists
Industry
Computer Systems Design Services
Posted
6 days ago