ETL/Azure Data Engineer

Cleo Consulting - Ontario, CA

Full-time - Senior

Ontario, CA

Professional, Scientific, and Technical Services

About the position

The ETL/Azure Data Engineer is responsible for designing, developing, maintaining, and optimizing ETL processes using Azure Databricks for data warehousing, data lakes, and analytics. This role involves close collaboration with data architects and business teams to ensure efficient data transformation and movement, including handling Change Data Capture (CDC) and streaming data. The position requires a strong background in ETL tools, data management, and optimization techniques, particularly within the Azure ecosystem.

Responsibilities

• Design, develop, maintain, and optimize ETL processes in Databricks.
• Work closely with data architects and business teams to ensure efficient data transformation and movement.
• Handle Change Data Capture (CDC) and streaming data.
• Create low-level design documents and test cases for ETL development.
• Implement error-catching, logging, retry mechanisms, and handling data anomalies.
• Develop and maintain pipelines from Oracle data sources to Azure Delta Lakes and FHIR.
• Perform unit testing and ensure performance monitoring and improvement.
• Review and optimize overall ETL performance.
• Conduct end-to-end integrated testing for Full Load and Incremental Load.
• Plan for Go Live and create production deployment steps.

Requirements

• 7+ years of experience using ETL tools such as Microsoft SSIS, stored procedures, T-SQL.
• 2+ years of experience with Delta Lake, Databricks, and Azure Databricks pipelines.
• Strong knowledge of Delta Lake for data management and optimization.
• Familiarity with Databricks Workflows for scheduling and orchestrating tasks.
• 2+ years of experience with Python and PySpark.
• Solid understanding of the Medallion Architecture and experience implementing it in production environments.
• Hands-on experience with CDC tools (e.g., GoldenGate) for managing real-time data.
• Experience with SQL Server and Oracle.

Nice-to-haves

• Knowledge of FHIR is an asset.
• Experience in developing in an Agile environment.
• Familiarity with external orchestration tools like Azure Data Factory.

Benefits

• Hybrid work arrangement with a minimum of 3 days per week in the office.