Data Engineer - Backend Developer

Bigbear.ai

Full-time - Senior

About the position

BigBear.ai is seeking a Senior Data Engineer to support a program in the Washington DC metro area. This position will work on site 5 days per week in an office located in the National Capital Region, with some travel requirements. The role requires an active TS/SCI clearance, making it an ideal opportunity for those looking to be part of one of the fastest growing AI/ML companies in the industry. At BigBear.ai, we emphasize a collaborative environment where employees are integral to our success. We foster growth and development, focusing on opportunity, recognition, and work-life balance. Our commitment to our employees mirrors the dedication we show to our clients, creating a thriving workplace culture. In this role, you will design, develop, and implement end-to-end data pipelines, utilizing ETL processes and technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks. You will create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing. Your responsibilities will include performing data cleansing, integration, and quality assurance activities to maintain the accuracy and integrity of large datasets. You will leverage big data technologies to efficiently process and analyze large datasets, particularly those encountered in a federal agency, and troubleshoot data-related problems while providing innovative solutions to complex data challenges. Additionally, you will implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices. Collaboration with cross-functional teams will be essential to understand data requirements and design optimal data models and architectures. You will work closely with data scientists, analysts, and stakeholders to provide timely and accurate data insights that support decision-making processes. Maintaining documentation for software applications, workflows, and processes will also be part of your responsibilities, along with staying updated with emerging trends and advancements in data engineering to recommend suitable tools and technologies for continuous improvement.

Responsibilities

• Design, develop, and implement end-to-end data pipelines using ETL processes and technologies.
• Create and optimize data pipelines from scratch, ensuring scalability, reliability, and high-performance processing.
• Perform data cleansing, data integration, and data quality assurance activities to maintain the accuracy and integrity of large datasets.
• Leverage big data technologies to efficiently process and analyze large datasets, particularly those encountered in a federal agency.
• Troubleshoot data-related problems and provide innovative solutions to address complex data challenges.
• Implement and enforce data governance policies and procedures, ensuring compliance with regulatory requirements and industry best practices.
• Work closely with cross-functional teams to understand data requirements and design optimal data models and architectures.
• Collaborate with data scientists, analysts, and stakeholders to provide timely and accurate data insights and support decision-making processes.
• Maintain documentation for software applications, workflows, and processes.
• Stay updated with emerging trends and advancements in data engineering and recommend suitable tools and technologies for continuous improvement.

Requirements

• Bachelor's Degree and 7+ years of experience; (in lieu of Bachelor's degree, 6 additional years of relevant experience)
• Must hold an active TS/SCI clearance
• Minimum of 7 years of experience as a Data Engineer, with demonstrated experience creating data pipelines from scratch.
• High level of proficiency in ETL processes and demonstrated, hands-on experience with technologies such as Databricks, Python, Spark, Scala, JavaScript/JSON, SQL, and Jupyter Notebooks.
• Strong problem-solving skills and ability to solve complex data-related issues.
• Demonstrated experience working with large datasets and leveraging big data technologies to process and analyze data efficiently.
• Understanding of data modeling/visualization, database design principles, and data governance practices.
• Excellent communication and collaboration skills, with the ability to work effectively with cross-functional teams.
• Detail-oriented mindset with a commitment to delivering high-quality results.
• Must be in the DC Metro area and available to work onsite 5 days per week.
• Recent DoD or IC-related experience.

Nice-to-haves

• Knowledge of Qlik/Qlik Sense, QVD/QlikView, and Qlik Production Application Standards (QPAS) is a significant plus.
• Previous experience with Advana is a plus.