Databricks Platform Architect

$117,000 - $210,000/Yr

Publicis Groupe - New York, NY

Full-time - Manager

New York, NY

10,001+ employees

Professional, Scientific, and Technical Services

About the position

Publicis Sapient is seeking a Manager of Data Engineering specializing in Databricks Platform Architecture. This role involves leading the development of global Databricks platforms to drive significant business outcomes for enterprise clients. The successful candidate will leverage advanced data technologies to create transformative solutions that provide valuable insights, thereby helping clients navigate their digital transformation journeys.

Responsibilities

• Act as a Databricks global platform lead as part of large digital transformation journeys.
• Advance the application of Databricks data platforms as a core building block to enable true business transformation.
• Lead Data Migration and Data Modernization projects to Databricks.
• Build complex data ingestion, processing and consumption storage and pipelines on Databricks.
• Work closely with clients to understand their needs and translate them into technology solutions.
• Provide expertise as a technical resource to solve complex business issues that translate into data integration and Databricks systems designs.
• Shape opportunities and create execution approaches throughout the lifecycle of client engagements.
• Ensure all deliverables are of high quality by setting development standards, adhering to the standards, and participating in code reviews.
• Mentor, support, and manage team members.

Requirements

• Deep understanding of Databricks architecture, including clusters, notebooks, jobs, and the underlying compute and storage layers with 5+ years of hands-on experience.
• Experience building Databricks as a global platform across multiple regions supporting various Lines of Business (LOB).
• Proficiency in Apache Spark, including core components like Spark SQL, Spark Streaming, and MLlib.
• Knowledge of Delta Lake features such as ACID transactions and time travel.
• Experience using Databricks SQL for data querying, analysis, and visualization.
• Ability to create and manage complex data pipelines and workflows using Databricks Jobs.
• Understanding of cluster configurations, autoscaling, and performance optimization.
• Experience with Unity Catalog.
• Deep understanding of AWS or Azure cloud essentials, including Storage, Networking, Identity and Access Management, and data security compliance.
• Understanding of network configurations, VPCs, and security groups for Databricks deployments.
• Ability to analyze and optimize Databricks costs using features like spot instances and cluster policies.
• Infrastructure experience and familiarity with Terraform scripts.

Nice-to-haves

• Certifications for any of the cloud services like Azure, AWS, or GCP.
• Certifications for any Machine Learning/Advanced Analytics Courses.
• Experience working with code repositories and continuous integration pipelines using AWS code build/code pipelines or similar tools.
• Experience in data governance and lineage implementation.
• Multi-geo and distributed delivery experience in large programs.

Benefits

• Flexible vacation policy; time is not limited, allocated, or accrued.
• 16 paid holidays throughout the year.
• Generous parental leave and new parent transition program.
• Tuition reimbursement.
• Corporate gift matching program.