Senior Data Engineer

$84,000 - $166,000/Yr

Dematic - Plano, TX

Full-time - Senior

Onsite - Plano, TX

Machinery Manufacturing

About the position

We are seeking a dynamic and highly skilled Senior Data Engineer who has extensive experience building self-service enterprise scale data platforms with microservices architecture and lead these foundational efforts. This role demands someone who not only possesses a profound understanding of the data engineering landscape but also has a very strong software engineering background, especially in building microservices frameworks and architectures. The ideal candidate will be an individual contributor as well as the technical lead and contribute significantly to platform development and actively shape our data ecosystem. As a senior engineer, you will be responsible for ideation, architecture, design, and development of our enterprise data platform. You will architect and design core components with a microservices architecture, abstracting platform, and infrastructure intricacies. Your role will involve creating and maintaining essential data platform SDKs and libraries, adhering to industry best practices. You will also design and develop connector frameworks and modern connectors to source data from disparate systems both on-prem and cloud. In addition, you will design and optimize data storage, processing, and querying performance for large-scale datasets using industry best practices while keeping costs in check. You will design and develop data quality frameworks and processes to ensure the accuracy and reliability of data, as well as develop microservices-based semantic layer and metadata management components. Collaboration with data scientists, analysts, and cross-functional teams to design data models, database schemas, and data storage solutions will be a key part of your responsibilities. Furthermore, you will design and develop advanced analytics and machine learning capabilities on the data platform, along with observability and data governance frameworks and practices. Staying up to date with the latest data engineering trends, technologies, and best practices will be essential as you drive the deployment and release cycles, ensuring a robust and scalable platform.

Responsibilities

• Ideation, architecture, design, and development of the enterprise data platform.
• Architect and design core components with a microservices architecture.
• Create and maintain essential data platform SDKs and libraries.
• Design and develop connector frameworks and modern connectors to source data from disparate systems.
• Design and optimize data storage, processing, and querying performance for large-scale datasets.
• Design and develop data quality frameworks and processes to ensure data accuracy and reliability.
• Design and develop microservices-based semantic layer and metadata management components.
• Collaborate with data scientists, analysts, and cross-functional teams to design data models and database schemas.
• Design and develop advanced analytics and machine learning capabilities on the data platform.
• Design and develop observability and data governance frameworks and practices.
• Drive the deployment and release cycles, ensuring a robust and scalable platform.

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or related field.
• 8-10 years of proven experience in modern cloud data engineering and broader data landscape experience.
• Solid software engineering experience, particularly in building microservices frameworks.
• Prior experience architecting and building successful self-service enterprise scale data platforms in a green field environment.
• Proficiency in building end-to-end data platforms and data services in GCP.
• Proficiency in tools and technologies: BigQuery, Cloud Functions, Cloud Run, Dataform, Dataflow, Dataproc, SQL, Python, Airflow, PubSub.
• Experience with Microservices architectures - Kubernetes, Docker, TypeScript, NestJS, NodeJS stack.
• Experience building semantic layers.
• Proficiency in architecting and designing batch and real-time streaming infrastructure and workloads.
• Solid experience with architecting and implementing metadata management including data catalogues, data lineage, data quality, and data observability for big data workflows.
• Hands-on experience with GCP ecosystem and data lakehouse architectures.
• Strong understanding of data modeling, data architecture, and data governance principles.
• Excellent experience with DataOps principles and test automation.
• Excellent experience with observability tooling: Grafana, Datadog.

Nice-to-haves

• Experience with Data Mesh architecture.
• Experience building Semantic layers for data platforms.
• Experience building scalable IoT architectures.

Benefits

• Career Development
• Competitive Compensation and Benefits
• Pay Transparency
• Global Opportunities