SquareShift.co

<html><body> <h1 dir="ltr"> Lead Data Engineer - Job Description<\/span> <\/h1><h2 dir="ltr">Overview<\/span> <\/h2>We are seeking a highly skilled <\/span>Lead Data Engineer<\/span> to architect, build, and scale enterprise-grade data platforms and pipelines. The ideal candidate has deep expertise in modern data engineering, cloud-native services, real-time data ingestion x26amp; CDC frameworks. This role involves leading complex data initiatives, guiding engineering teams, and collaborating with cross-functional stakeholders to deliver robust, scalable, and secure data solutions.<\/span> <\/p><div> <\/div><h2 dir="ltr">Key Responsibilities<\/span> <\/h2><h3 dir="ltr">Data Architecture x26amp; Pipeline Development<\/span> <\/h3><ul><li dir="ltr">Design and build scalable <\/span>ETL/ELT pipelines<\/span> using Python / PySpark.<\/span> <\/p><\/li><li dir="ltr">Architect and implement <\/span>batch and real-time data pipelines<\/span> using Kafka, Kafka Connect, and Debezium.<\/span> <\/p><\/li><li dir="ltr">Develop and optimize data ingestion workflows across multiple databases (MySQL, Postgres, MongoDB).<\/span> <\/p><\/li><li dir="ltr">Build and manage <\/span>Apache Airflow<\/span> DAGs for end-to-end pipeline orchestration.<\/span> <\/p><\/li><\/ul><h3 dir="ltr">Cloud Data Engineering<\/span> <\/h3><ul><li dir="ltr">Lead data platform development on <\/span>GCP<\/span> (BigQuery, Data Proc, Data Stream ).<\/span> <\/p><\/li><li dir="ltr">Implement data ingestion, orchestration, and transformation at scale using Pub/Sub, Cloud Functions/Cloud Run, Dataflow, and GKE.<\/span> <\/p><\/li><li dir="ltr">Own data warehouse design and optimization across <\/span>Snowflake<\/span> and <\/span>BigQuery<\/span>.<\/span> <\/span><\/p><\/li><\/ul><\/span> <h3>Requirements<\/h3><h2 dir="ltr">Required Skills x26amp; Experience<\/span> <\/h2><h3 dir="ltr">Core Technical Skills<\/span> <\/h3><ul><li dir="ltr">Strong proficiency in <\/span>Python, SQL, PySpark<\/span>.<\/span> <\/p><\/li><li dir="ltr">Hands-on expertise with <\/span>Kafka, Kafka Connect, Debezium, Airflow, Databricks<\/span>.<\/span> <\/p><\/li><li dir="ltr">Deep experience with <\/span>BigQuery, Snowflake, MySQL, Postgres, MongoDB<\/span>.<\/span> <\/p><\/li><li dir="ltr">Solid understanding of <\/span>vector data stores<\/span> and search indexing.<\/span> <\/p><\/li><li dir="ltr">Knowledge of GCP services like <\/span>Big Query, Cloud Functions, Cloud Run, Data Flow, Data Proc, Data Stream<\/span>, etc..x26nbsp;<\/span> <\/p><\/li><\/ul><h3 dir="ltr">Good to have<\/span> <\/h3><ul><li dir="ltr">Certifications:<\/span> <\/p><\/li><ul><li dir="ltr">GCP Professional Data Engineer<\/span> <\/p><\/li><li dir="ltr">Elastic Certified Engineer<\/span> <\/p><\/li><\/ul><li dir="ltr">AI<\/span> <\/p><\/li><ul><li dir="ltr">Gemini Enterprise<\/span> <\/p><\/li><li dir="ltr">Vertex AI Agent Builder<\/span> <\/p><\/li><li dir="ltr">ADK<\/span> <\/p><\/li><\/ul><\/ul><div> <\/div><h2 dir="ltr">Non-Technical x26amp; Leadership Skills<\/span> <\/h2><ul><li dir="ltr">Communication:<\/span> Exceptional verbal and written communication skills, with the ability to articulate complex technical concepts to both technical and non-technical audiences.<\/span> <\/p><\/li><li dir="ltr">Mentorship x26amp; Coaching:<\/span> Proven experience in mentoring junior and mid-level engineers, fostering a culture of continuous learning and growth.<\/span> <\/p><\/li><li dir="ltr">Problem-Solving:<\/span> Strong analytical and debugging skills, with a proactive approach to identifying and resolving technical roadblocks.<\/span> <\/p><\/li><li dir="ltr">Ownership x26amp; Accountability:<\/span> Demonstrates a high level of responsibility for project outcomes, system reliability, and code quality.<\/span> <\/p><\/li><li dir="ltr">Agile Proficiency:<\/span> Deep understanding and practical experience with Agile methodologies (Scrum/Kanban).<\/span> <\/p><\/li><li dir="ltr">Stakeholder Management:<\/span> Ability to effectively manage expectations and build consensus across different teams.<\/span> <\/p><\/li><\/ul><div> <\/div><h2 dir="ltr">Qualifications<\/span> <\/h2><ul><li dir="ltr">Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).<\/span> <\/p><\/li><\/ul><div>Typically <\/span>7+ years<\/span> of progressive experience in data engineering, with <\/span>2+ years<\/span> in a technical leadership or lead engineer role.<\/span> <\/div><\/span> </div></li></ul></h2></div></li></li></li></li></li></li></ul></h2></div></li></li></li></ul></li></li></li></ul></li></ul></h3></li></li></li></li></li></ul></h3></h2></h3></li></li></li></ul></h3></li></li></li></li></ul></h3></h2></div></h2> </h1></body></html>

Job Details

Skills Required

More Details

Apply