Big Data Developer

Job Title: Senior Big Data SME (Subject Matter Expert) Location: Remote Experience: 9 to 14 years CTC: 35 Lpa Education: Bachelors in Science (IT/Computer Science/Engineer) Employment Type: Full-Time About Us :We are a dynamic team focused on building scalable, high-performance data platforms across the cloud ecosystem (Azure, AWS, GCP). As part of an exciting growth phase, we are looking for a Senior Big Data SME to join our team. This role will allow you to leverage your expertise in cloud technologies, big data, and data engineering to drive innovation and excellence in data architecture and solutions .

Job Descriptio n:We are seeking a Senior Big Data Subject Matter Expert (SME) to help us architect, build, and optimize large-scale data platforms using leading cloud technologies and Big Data tools. As an SME, you will bring deep technical expertise and strategic vision to our cloud data initiatives, guiding the team on best practices, ensuring the successful delivery of data projects, and improving operational efficiency. You’ll work closely with engineers, architects, and leadership to ensure the implementation of cutting-edge solution s.

Key Responsibiliti es: Architect and Design Scalable Data Platforms: Leverage your expertise in Azure, AWS, and GCP to design and build scalable, high-performance data platforms for analytics and real time process ing.Cloud Technologies Expertise: Drive the adoption of cloud-native tools and services like Azure Synapse, AWS Glue, Redshift, and Google BigQuery, and ensure alignment with industry best practi ces.Big Data & Streaming Solutions: Provide hands-on support and guidance on big data technologies including Apache Spark (PySpark, Scala, Java), Hadoop (HDFS, YARN, Hive, Impala), and streaming frameworks like Kafka, Kinesis, and Fl ink.Data Pipelines & ETL Frameworks: Lead efforts in creating and optimizing ETL pipelines using tools like DBT, Apache Airflow, and cloud-native data orchestration tools (e.g., AWS Glue, Azure Data Facto ry).Containerization & Orchestration: Lead efforts in containerizing applications using Docker and orchestrating them with Kubernetes (AKS, EKS, GKE) for scalable and reliable deploym ent.Data Security & Governance: Ensure data security, governance, and compliance using IAM, RBAC, and data lineage tracking tools (Azure Purview, AWS Glue Data Catal og).CI/CD & DevOps: Contribute to the development of CI/CD pipelines with Terraform, Jenkins, Azure DevOps, and GitHub Actions to automate the deployment of cloud infrastructure and data workfl ows.Cost Optimization: Implement strategies for optimizing cloud resource utilization and reducing operational costs, particularly in storage (AWS S3, Azure Blob, GCS) and compute resour ces.Team Collaboration & Mentorship: Provide mentorship and guidance to junior engineers, and lead initiatives to build and maintain high-performing teams through knowledge-sharing sessions and best practice adopt ion. Requirem ents:Experience: 14+ years in Data Engineering or related roles, with a focus on cloud technologies and big data solut ions. Technical Expe rtise:Deep knowledge of cloud platforms such as Azure, AWS, and GCP.

Hands-on experience with Big Data technologies like Apache Spark, Hadoop, Scala, Mapreduce, Kafka, and Flink.Solid understanding of ETL frameworks (DBT, Apache Airflow, AWS Glue, etc.). Expertise in containerization with Docker and Kubernetes (AKS, EKS, GKE). Proven experience in Data Warehousing & Modeling (Snowflake, Redshift, BigQuery, Syn apse).Strong background in Data Security and Governance (IAM, RBAC, encryption, data lineage).

Experience with CI/CD pipelines using Terraform, GitHub Actions, and Azure DevOps. Programming & Scripting S kills:Proficiency in Python, Bash, or PowerShell for automation tasks. Cloud Architecture: Experience designing hybrid/multi-cloud architectures to ensure high availability and fault tolerance across Azure, AWS, an d GCP.Leadership & Mentorship: Proven ability to lead teams, mentor junior engineers, and collaborate effectively with cross-functional teams.

Preferred Skills:Familiarity with Machine Learning pipelines and predictive ana lytics.Experience with Serverless Computing (AWS Lambda, Azure Functions, Google Cloud Func tions).