AWS / Databricks Cloud Data Engineer

Moser Consulting•Indianapolis, IN

2d•Hybrid

About The Position

Moser Consulting is seeking a Senior Cloud Data Engineer to join their consulting team. In this role, the engineer will design, build, and maintain scalable data pipelines and acquisition workflows across cloud and on-premises environments. They will contribute to the full spectrum of data engineering, from ingestion through transformation and delivery. The engineer will collaborate with cross-functional teams to translate business requirements into robust data solutions, while upholding best practices in architecture, cloud infrastructure, documentation, and continuous improvement. Moser Consulting has been recognized as one of the Best Places to Work in Indiana for 10 consecutive years and is committed to diversity, equity, and inclusion.

Requirements

Experience with dimensional modeling, SCDs, and medallion architecture
Previous cloud experience required; AWS preferred with hands-on exposure to Lambda, S3, and Docker
Knowledge of Azure Active Directory group management and role-based access control (RBAC)
Strong Python experience across pipeline development, ETL, and data engineering tasks, with exposure to complex datasets from disparate batch and streaming sources and an emphasis on low latency, high throughput, and cost-efficiency; pandas experience a plus
Leveraging SQL and python data tooling develop scalable, optimized ETL/ELT pipelines to ingest, transform, and load large, complex datasets from disparate sources (batch and streaming). Streamlining for low latency, high throughput and cost-efficiency.
Experience with API development in both consumer and producer capacities (REST APIs)
CI/CD experience with modern DevOps tooling and deployment pipelines
Demonstrated ability to learn new technologies quickly and apply them to solve problems
Strong attention to detail and commitment to producing high-quality work
Excellent oral and written communication skills
Professionalism and ethics including trust, positive attitude, commitment, honesty, collaboration, and approachability
Proven ability to adapt in a fast-paced consulting environment, including managing time effectively and handling multiple tasks simultaneously

Nice To Haves

Experience with version control systems (Git) and agile development frameworks
Design and document end-to-end testing plans for Python-based applications
PySpark experience for large-scale distributed data processing
Experience with Datadog or similar observability and monitoring platforms
Familiarity with Dagster or other modern data orchestration frameworks
Terraform or infrastructure-as-code (IaC) experience for cloud resource provisioning
Database experience with databases such as AWS RDS, MySQL, or PostgreSQL

Responsibilities

Participate in team stand-ups and other agile ceremonies to support project delivery
Assist in building and maintaining data acquisition pipelines from diverse sources including databases, flat-files, and REST APIs
Leverage development design patterns and best practices across workloads.
Support data modeling efforts including dimensional models and semantic layers for analytics under senior guidance
Proactively troubleshoot and resolve pipeline failures, query performance issues, data quality concerns, notebook timeouts, data model consistencies, semantic model refresh errors, and capacity throttling using monitoring tools and query optimization techniques
Participate in requirement gathering meetings and translate business needs into technical specifications
Engage in ongoing learning through online trainings, certifications, and mentorship opportunities
Create and maintain technical documentation, operational documentation, data flow diagrams, and data mapping documents
Collaborate effectively with cross-functional teams and communicate progress to technical and non-technical stakeholders