Big Data Engineer

Cai - Annapolis, MD

Full-time

Remote - Annapolis, MD

Chemical Manufacturing

About the position

The Big Data Engineer is a crucial member of a collaborative team responsible for designing, engineering, maintaining, testing, evaluating, and implementing big data infrastructure and solutions. This role focuses on leveraging advanced database technologies to efficiently manage and extract insights from large datasets, ensuring data integrity and security while optimizing data pipelines for ETL/ELT processes. The position is fully remote and is classified as a full-time contract role.

Responsibilities

• Design and implement scalable big data solutions tailored to client's needs
• Maintain and enhance existing big data infrastructures to meet unique requirements
• Test and evaluate new big data technologies and frameworks for compatibility with systems and goals
• Collect, store, process, manage, analyze, and visualize large datasets to derive actionable insights
• Collaborate with team members to integrate big data solutions with existing systems
• Ensure data integrity and security across all platforms used within the client
• Develop and optimize data pipelines for ETL/ELT processes specific to data needs
• Document technical solutions and maintain comprehensive records in line with standards and protocols
• Stay updated with the latest trends and advancements in big data technology relevant to strategic initiatives

Requirements

• Thorough understanding of cloud computing technologies, including IaaS, PaaS, and SaaS implementations
• Skilled in exploratory data analysis (EDA) to support ETL/ELT processes
• Proficiency with Microsoft cloud products, including Azure and Fabric
• Experience with tools such as Data Factory and Databricks
• Ability to script in multiple languages, with a strong emphasis on Python and SQL

Nice-to-haves

• Experience with data visualization tools
• Proficiency with Excel and Power BI
• Familiarity with Delta Lake
• Knowledge of Lakehouse Medallion Architecture