Data Engineer-Databricks

Invixon Technologies - Irving, TX

Full-time

Irving, TX

About the position

The Data Engineer position focuses on designing, building, and testing data pipelines across cloud and on-premise environments, utilizing Azure Databricks for data processing. The role involves collaboration with cloud architects and tech leads to facilitate the migration of on-premise data to Databricks, while also developing reference architectures and design patterns for cloud-based solutions.

Responsibilities

• Design, build, and test cross cloud and on-premise data pipelines using Azure Databricks.
• Collaborate with cloud architects and tech leads for on-premise to Databricks migration.
• Understand and automate data pipelines using cloud-based and on-premise technologies.
• Develop reference architectures and design pattern libraries for cloud-based solutions.
• Advise on cloud project setup, security, role-based access implementation, and network optimizations.

Requirements

• 10 years of experience in data engineering.
• 2+ years of experience with SQL and database table design, capable of writing structured and efficient queries on large datasets.
• 1-2 years of experience on Databricks on any cloud platform (AWS, Azure).
• 1-2 years of experience working on AWS/Azure cloud.
• 1-2 years of design and/or implementation of enterprise applications.
• Experience in migrating on-premise workloads to leading public cloud(s).
• Experience in AGILE development, SCRUM, and Application Lifecycle Management (ALM) with scripting experience in Python and shell.
• Hands-on experience in the full life-cycle of software development methodologies using Agile Scrum/Kanban.

Nice-to-haves

• Databricks certifications.