Principal Data Engineer (Azure)

ZipRecruiter - Dallas, TX

Full-time - Principal

Dallas, TX

About the position

As a Principal Data Engineer (Azure) at Tiger Analytics, you will play a crucial role in designing and building scalable data ingestion pipelines and analytics solutions using Microsoft Azure technologies. This position involves hands-on experience with Azure, Databricks, and data modeling, contributing to the development of modern data platforms and data lakes. You will collaborate with various teams to deliver high-performance data processing and analytical solutions that impact millions globally.

Responsibilities

• Design and build scalable & metadata-driven data ingestion pipelines for batch and streaming datasets.
• Conceptualize and execute high-performance data processing for structured and unstructured data, and data harmonization.
• Schedule, orchestrate, and validate data pipelines.
• Design exception handling and log monitoring for debugging purposes.
• Collaborate with peers to make tech stack and tools-related decisions.
• Interact with multiple teams and stakeholders to meet deadlines and bring analytical solutions to life.

Requirements

• Hands-on experience with Azure Data Factory (ADF), PySpark, Databricks, ADLS, and Azure SQL Database.
• Working knowledge of Azure Synapse Analytics, Event Hub & Streaming Analytics, Cosmos DB, and Purview.
• Strong programming, unit testing, and debugging skills in SQL, Python, or Scala/Java.
• Experience with big data technologies like Hadoop, Spark, Airflow, NiFi, Kafka, Hive, Neo4J, and Elastic Search.
• Good understanding of different file formats like Delta Lake, Avro, Parquet, JSON, and CSV.
• Experience working in Agile projects and following DevOps processes with technologies like Git, Jenkins, and Azure DevOps.

Nice-to-haves

• Experience working on Data Lake & Lakehouse projects.
• Experience building REST services and implementing service-oriented architectures.
• Certifications like Data Engineering on Microsoft Azure (DP-203) or Databricks Certified Developer (DE).

Benefits

• Significant career development opportunities in a fast-growing environment.
• High degree of individual responsibility.