Lead Cloud Data Engineer

It Concepts - Washington, DC

Full-time - Mid Level

Remote - Washington, DC

Professional, Scientific, and Technical Services

About the position

IT Concepts is seeking an experienced Lead Cloud Data Engineer to support the Consumer Finance Protection Bureau (CFPB). In this role, you will collaborate with a team of consultants, engineers, and analysts to design, build, and implement solutions for migrating CFPB's data to the cloud. Your primary focus will be on enhancing data quality, security, and integrity throughout the migration process. You will be responsible for the architecture, design, development, and maintenance of large-scale data and analytics platforms, system integrations, data pipelines, data models, and API integrations. As a Lead Data Engineer, you will create transformation paths for data migration from on-premises pipelines and sources to AWS. You will provide insights and feedback in conjunction with Data Architects to ensure optimal data standardization and readiness for analysis. Your role will also involve prototyping emerging business use cases to validate technology approaches and propose potential solutions. You will work closely with data engineers to ensure data integrity during large-scale migrations and deliver high-quality data assets that enable data-driven analyses for business leaders. Continuous improvement of data solutions will be a key aspect of your responsibilities, as you will strive to increase the quality and speed of delivery while building trust in the data engineering team's deliverables. You will also focus on reducing the total cost of ownership of solutions by developing shared components and implementing best practices and coding standards. This position is primarily remote, with a preference for candidates located in the DC area.

Responsibilities

• Collaborate & contribute to the architecture, design, development, and maintenance of large-scale data & analytics platforms, system integrations, data pipelines, data models & API integrations.
• Create transformation path for data to migrate from on-prem pipelines and sources to AWS.
• Provide input and insights, in conjunction with Data Architects, to the client.
• Coordinate with data engineers to provide feedback and organization to team.
• Ensure that data are optimally standardized and analysis-ready.
• Prototype emerging business use cases to validate technology approaches and propose potential solutions.
• Collaborate and ensure data integrity after large scale migrations.
• Deliver high quality data assets to be used by the business to transform business processes and to enable leaders to complete data-driven analyses.
• Continuously improve data solutions to increase quality, speed of delivery and trust of data engineering team's deliverables to enable business outcomes.
• Reduce total cost of ownership of solutions by developing shared components and implementing best practices and coding standards.
• Collaborate with team to re-platform and reengineer data pipelines from on-prem to AWS cloud.
• Work together with team members to ensure data quality and integrity during migrations.
• Lead by example and pitch in to enable successful and seamless client delivery.

Requirements

• 8+ years experience related to Data Engineering.
• AWS Cloud Certificate.
• Minimum of 3 years of experience in leading engineering teams, including task management and personnel management.
• Experience working in or managing data centric teams in the government or other highly regulated environments.
• Strong understanding of data lake, data lakehouse, and data warehousing architectures in a cloud-based environment.
• Proficiency in Python for data manipulation, scripting, and automation.
• In-depth knowledge of AWS services relevant to data engineering (e.g., S3, EC2, DMS, DataSync, SageMaker, Glue, RDS, Lambda, Elasticsearch).
• Understanding of data integration patterns and technologies.
• Proficiency designing and building flexible and scalable ETL processes and data pipelines using Python and/or PySpark and SQL.
• Proficiency in data pipeline automation and workflow management tools like Apache Airflow or AWS Step Functions.
• Knowledge of data quality management and data governance principles.
• Strong problem-solving and troubleshooting skills related to data management challenges.
• Experience managing code in GitHub or other similar tools.
• Minimum of 2 years of hands-on experience with Databricks including data ingestion, transformation, analysis and optimization.
• Experience designing, deploying, securing, sustaining and maintaining applications and services in a cloud environment (e.g., AWS, Azure) using infrastructure as code (e.g., Terraform, CloudFormation, Boto3).
• Experience with database administration, optimization, and data extraction.
• Experience using containerization technology such as Kubernetes or Mesos.
• Minimum of 1 year of hands-on experience migrating from an on-premise data platform(s) to a modern cloud environment (e.g. AWS, Azure, Google Cloud Platform).
• Linux/RHEL server & bash/shell scripting experience in on-prem or cloud environment.

Nice-to-haves

• Bachelors Degree in related field.
• Previous experience with large-scale data migrations and cloud-based data platform implementations.
• Prior experience with Databricks Unity Metastore / Catalog.
• Familiarity with advanced SQL techniques for performance optimization and data analysis.
• Knowledge of data streaming and real-time data processing frameworks such as Spark Structured Streaming.
• Experience with data lakes and big data technologies (e.g., Apache Spark, Citus).
• Familiarity with serverless computing and event-driven architectures in AWS.
• Certifications in AWS, Databricks, or related technologies.
• Experience working in Agile or DevSecOps environments and using related tools for collaboration and version control.
• Extensive knowledge of software and data engineering best practices.
• Strong communication and collaboration skills with internal and external stakeholders.
• Experience establishing, implementing and documenting best practices, standard operating procedures, etc.

Benefits

• Competitive Paid Time Off
• Medical, Dental and Vision Insurance
• Identity Theft Protection
• Legal Resources Coverage
• 401(k) with company matching with NO vesting period
• Education reimbursement for certifications, degrees, or professional development
• Flexibility for professional growth and networking
• Funds for activities - virtual and in-person
• Charity galas/events
• Positive workspace for creativity and innovation