Data Engineer, Staff

Bigbear.ai

Full-time

Remote

About the position

BigBear.ai is seeking a Data Engineer to design, develop, and maintain the Advana data infrastructure and systems. This role is crucial for ensuring efficient data processing and analysis, utilizing various technologies and methodologies. The position requires an active Secret clearance and is primarily onsite in Washington, DC or Norfolk, VA, with some remote flexibility as needed.

Responsibilities

• Design, develop, and implement end-to-end data pipelines using ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.
• Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing.
• Perform data cleansing, integration, and quality assurance to maintain the accuracy and integrity of large datasets.
• Leverage big data technologies to efficiently process and analyze large datasets, particularly for federal agencies.
• Troubleshoot data-related problems and provide innovative solutions to complex data challenges.
• Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices.
• Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures.
• Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes.
• Maintain documentation for software applications, workflows, and processes.
• Stay updated with emerging trends and advancements in data engineering and recommend suitable tools and technologies for continuous improvement.

Requirements

• Secret clearance
• Bachelor's Degree
• Minimum of 3 years of experience as a Data Engineer, with demonstrated experience creating data pipelines from scratch.
• High level of proficiency in ETL processes and hands-on experience with Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.
• Strong problem-solving skills and ability to solve complex data-related issues.
• Experience working with large datasets and leveraging big data technologies for efficient data processing and analysis.
• Understanding of data modeling/visualization, database design principles, and data governance practices.
• Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams.
• Detail-oriented mindset with a commitment to delivering high-quality results.

Nice-to-haves

• TS/SCI clearance or eligibility
• Knowledge of Qlik/Qlik Sense, QVD/QlikView, and Qlik Production Application Standards (QPAS) is a significant plus.
• Recent DoD or IC-related experience.
• High level of Databricks proficiency.
• Previous experience with Advana is a plus.