Lead Big Data Engineer

$200,000 - $300,000/Yr

Unreal Gigs - San Francisco, CA

Full-time - Senior

Remote - San Francisco, CA

About the position

As the Lead Big Data Engineer, you will spearhead our big data engineering initiatives, guiding a team of skilled engineers in the design, construction, and maintenance of our big data infrastructure and pipelines. This role is pivotal in developing scalable and efficient data solutions that cater to the needs of our data-driven organization, leveraging your expertise in big data technologies and leadership experience to drive successful data projects.

Responsibilities

• Lead a team of big data engineers, providing guidance, mentorship, and technical leadership in big data engineering best practices and technologies.
• Lead the design and architecture of our big data infrastructure, including data lakes, data warehouses, and streaming data platforms, ensuring scalability, reliability, and performance.
• Lead the development of scalable and efficient data pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data from diverse sources, ensuring reliability, scalability, and performance.
• Lead the design and implementation of data models and schemas to support analytical and operational requirements, ensuring data integrity, consistency, and performance in a distributed environment.
• Lead efforts to integrate data from disparate sources and systems, ensuring data consistency, quality, and completeness throughout the data lifecycle.
• Lead optimization efforts for data pipelines and queries, identifying and addressing bottlenecks and inefficiencies to improve system scalability and reliability.
• Lead the implementation of monitoring and alerting systems to track big data platform performance and health, detecting and mitigating issues proactively to minimize downtime and data loss.
• Lead efforts to implement security controls and data governance policies to ensure data security, privacy, and compliance with regulatory requirements.
• Define and promote best practices for big data engineering, documentation, and usage, ensuring clear and comprehensive documentation to facilitate understanding and collaboration among team members.
• Collaborate with cross-functional teams, including data scientists, software engineers, and business stakeholders, to understand requirements and deliver big data solutions that meet business needs.
• Mentor junior engineers, providing guidance, support, and opportunities for growth and development in their big data engineering careers.

Requirements

• Bachelor's degree or higher in Computer Science, Engineering, Mathematics, or related field.
• 7+ years of experience in big data engineering, with a focus on designing, building, and maintaining big data infrastructure and pipelines.
• Leadership experience, with a demonstrated ability to lead and mentor a team of engineers.
• Proficiency in programming languages such as Java, Scala, or Python, and expertise with big data technologies such as Hadoop, Spark, Kafka, Hive, HBase, and more.
• Strong understanding of distributed systems and parallel processing, with experience designing and optimizing data pipelines for performance and scalability.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform, and familiarity with cloud-based big data services such as Amazon EMR, Azure HDInsight, or Google Dataproc.
• Experience with SQL and NoSQL databases, data warehousing, and ETL/ELT processes.
• Strong problem-solving skills and analytical thinking, with the ability to troubleshoot complex data issues and optimize system performance.
• Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and communicate technical concepts to non-technical stakeholders.

Benefits

• Competitive salary ranging from $200,000 to $300,000 per year, depending on experience and qualifications.
• Comprehensive health, dental, and vision insurance plans.
• Flexible work hours and remote work options.
• Generous vacation and paid time off.
• Professional development opportunities, including access to training programs, conferences, and workshops.
• State-of-the-art technology environment with access to cutting-edge tools and resources.
• Vibrant and inclusive company culture with opportunities for growth and advancement.
• Exciting projects with real-world impact at the forefront of big data innovation.