Lead GCP Data Engineer

V2Soft - Allen Park, MI

Full-time - Senior

Allen Park, MI

Professional, Scientific, and Technical Services

About the position

As a Lead GCP Data Engineer at V2Soft, you will play a pivotal role in designing and implementing data-centric solutions on the Google Cloud Platform (GCP). Your primary responsibility will be to leverage various GCP tools, including BigQuery, Google Cloud Storage, Cloud SQL, and others, to create robust data pipelines and ensure efficient data processing. You will be tasked with building ETL pipelines that ingest data from diverse sources, transforming and loading it into our systems using programming languages such as Java and Python. In this role, you will also be responsible for creating and maintaining data models that facilitate the efficient storage, retrieval, and analysis of large datasets. You will deploy and manage both SQL and NoSQL databases, optimizing data workflows for performance, reliability, and cost-effectiveness on the GCP infrastructure. Your expertise in implementing version control and CI/CD practices will be crucial for ensuring reliable deployments of data engineering workflows. Monitoring and logging tools will be utilized to proactively identify and address performance bottlenecks and system failures. You will troubleshoot and resolve issues related to data processing, storage, and retrieval, while also addressing code quality issues throughout the development lifecycle. Security measures and data governance policies will be implemented to maintain the integrity and confidentiality of data. Collaboration with stakeholders to gather and define data requirements will be essential, ensuring alignment with business objectives. You will develop and maintain documentation for data engineering processes, facilitating knowledge transfer and ease of system maintenance. Participation in on-call rotations to address critical issues will be expected, along with providing mentorship and guidance to junior team members, fostering a collaborative and knowledge-sharing environment.

Responsibilities

• Design and implement data-centric solutions on Google Cloud Platform (GCP) using various GCP tools.
• Build ETL pipelines to ingest data from heterogeneous sources into our system.
• Develop data processing pipelines using programming languages like Java and Python.
• Create and maintain data models for efficient storage, retrieval, and analysis of large datasets.
• Deploy and manage databases, both SQL and NoSQL, based on project requirements.
• Optimize data workflows for performance, reliability, and cost-effectiveness on GCP infrastructure.
• Implement version control and CI/CD practices for data engineering workflows.
• Utilize GCP monitoring and logging tools to identify and address performance bottlenecks.
• Troubleshoot and resolve issues related to data processing, storage, and retrieval.
• Address code quality issues using tools like SonarQube, Checkmarx, Fossa, and Cycode.
• Implement security measures and data governance policies.
• Collaborate with stakeholders to gather and define data requirements.
• Develop and maintain documentation for data engineering processes.
• Participate in on-call rotations to address critical issues.
• Provide mentorship and guidance to junior team members.

Requirements

• 8 years of professional experience in data engineering, data product development, and software product launches.
• At least three of the following programming languages: Java, Python, Spark, Scala, SQL, with experience in performance tuning.
• 4 years of cloud data/software engineering experience building scalable, reliable, and cost-effective production batch and streaming data pipelines.
• Experience with data warehouses like Google BigQuery and workflow orchestration tools like Airflow.
• Familiarity with relational database management systems like MySQL, PostgreSQL, and SQL Server.
• Experience with real-time data streaming platforms like Apache Kafka and GCP Pub/Sub.
• Knowledge of microservices architecture for large-scale real-time data processing applications.
• Experience with REST APIs for compute, storage, operations, and security.
• Familiarity with DevOps tools such as Tekton, GitHub Actions, Git, GitHub, Terraform, and Docker.
• Experience with project management tools like Atlassian JIRA.
• Automotive experience is preferred.
• Support in an onshore/offshore model is preferred.

Nice-to-haves

• Automotive experience is preferred.
• Support in an onshore/offshore model is preferred.