Data Engineer

Insight Global - Erlanger, KY

Full-time

Erlanger, KY

Administrative and Support Services

About the position

The Data Engineer position at Insight Global involves working in a hybrid onsite capacity in the greater Cincinnati area. This role is focused on ingesting data from various sources, managing ETL processes, and providing operational support for existing data jobs. A strong understanding of Azure Data Factory and Databricks is essential for success in this role, as the engineer will be responsible for optimizing data delivery and ensuring data quality.

Responsibilities

• Design and integrate fault tolerance and enhancements into data pipelines to improve quality and performance.
• Perform root cause analysis and solve problems using analytical and technical skills to optimize data delivery and reduce costs.
• Develop and implement backup and disaster recovery plans for Databricks environments.
• Implement ETL processes to integrate data from various sources into Databricks, ensuring data quality and integrity.
• Integrate Databricks with other data storage solutions and data lakes, ensuring seamless data flow and accessibility.
• Optimize data processing and query performance within Databricks to ensure efficient data retrieval and processing.
• Monitor data pipelines and workflows for performance issues or errors, and troubleshoot any problems that arise to maintain smooth operations.
• Design, develop, and maintain robust data pipelines using Databricks to process and transform large volumes of data.

Requirements

• 4+ years of experience with Databricks and Azure Data Factory (ADF).
• Strong working knowledge of ETL processes for moving data from source to source.
• 4+ years of experience with Python/PySpark/Spark, including familiarity with libraries and tools used in data processing.