Hire Big Data Engineers within a week
Hire Top Remote Software Dev Whizards!
Exp : 16+ Years
$60 / hr
Deepak K
Big Data Engineer
Project Manager with 16+ years of expertise in designing and developing customer-centric financial technology products. Proven leadership in cloud solutions, big data streaming platforms, scalable system architecture, and SDLC automation.
Key Skills
- Java
- Python
- C#
- Big Data
- Apache
- ElasticSearch
- OpenSearch
- NoSQLHibernate
Deepak K
Project manager
Exp : 16+ Years
$60 / hr
Experienced Engineering Director with over 16 years of expertise in designing and developing customer-centric financial technology products. Proven leadership in cloud solutions, big data streaming platforms, scalable system architecture, and software development lifecycle automation. Skilled in driving innovation and delivering high-performance, low-latency systems for financial institutions.
Educational Qualification:
- Master of Technology (M.Tech) in Computer Science & Engineering – Indian Institute of Technology, Kanpur (2005-2007)
Technical Skills
Languages: Java, Scala, Python, C#
Big Data & Distributed Systems: Apache Kafka, Apache Storm, Apache Flink, Apache Cassandra, Elasticsearch, OpenSearch, Apache Ignite, NoSQL Databases
Frameworks & Tools: Hibernate, Spring Boot, Docker, Kubernetes, Microservices
Cloud Platforms: AWS
Development & Deployment: Continuous Deployment, Scalable System Design, High Throughput Systems
SLC & Agile Practices: Agile, Scrum, Build Automation, Code Reviews, Security Vulnerability Analysis, Defect Reviews
Expertise
Big Data Streaming Platforms: Extensive experience in architecting and developing big data streaming analytics platforms for financial institutions, focusing on real-time and batch processing solutions.
Software Architecture & System Design: Expertise in designing scalable, low-latency systems for large-scale applications using distributed systems architecture.
Cloud Solutions & Microservices: Led the design and deployment of microservices-based architectures in cloud environments like AWS, with expertise in Docker and Kubernetes for container orchestration.
Leadership & Team Management: Proven track record in leading geo-distributed teams, providing technical direction, and mentoring individuals to foster growth and innovation.
Innovation & Automation: Spearheaded efforts in automating software lifecycle processes, ensuring performance testing, security vulnerability management, and build automation.
Work Experience:
Principal Engineer (Aug 2023 – Jan 2024):
- Led the development of a big data streaming analytics platform for financial institutions, providing technical guidance and future strategy to a global team of 15+ members.
- Architected product features and managed design and implementation processes.
- Mentored and coached junior team members.
Director (Dec 2021 – Aug 2023):
- Led a team of 4 in developing a big data analytics platform, providing technical leadership and shaping future strategies.
- Managed feature design, implementation, and team mentoring to drive innovation and delivery.
Senior Manager (Dec 2017 – Dec 2021):
- Led the design and development of new product features for a big data analytics platform, providing technical guidance to a team of 4.
- Contributed to overall system design and helped foster team growth and technical proficiency.
Lead Engineer (Dec 2013 – Dec 2017):
- Managed a team of engineers developing new features for a big data streaming platform and web applications used by financial institutions.
- Delivered key features using Java and the Spring framework.
Software Engineer (Nov 2009 – Dec 2013):
- Implemented new features in web application-based products for financial institutions, focusing on ETL and real-time solutions using Java and Spring framework.
Software Engineer (Jul 2007 – Nov 2009):
- Developed middleware applications for ETL and real-time solutions, utilizing C# and .NET frameworks for financial and healthcare data providers.
Exp : 15 Years
$70 / hr
Uday Kiran
Big Data Engineer
Highly experienced IT professional with over 15 years in application software design, development, and integrations across Big Data, Cloud, Data Warehousing, Data Analytics, CRM, ERP, and Business Intelligence. Proven expertise in leading and managing IT teams for over 11 years.
Key Skills
- Big Data
- Hadoop
- AWS
- Azure
- Google cloud
- Scala
- Python
- Java
Uday Kiran
Project manager
Exp : 15 Years
$70 / hr
Highly experienced IT professional with over 15 years in application software design, development, and integrations across Big Data, Cloud, Data Warehousing, Data Analytics, CRM, ERP, and Business Intelligence. Proven expertise in leading and managing IT teams for over 11 years, driving the successful delivery of projects in industries such as Retail, FinTech, E-commerce, Supply Chain, Banking, Marketing, and Healthcare. Skilled in innovation, R&D, project roadmaps, and open-source contributions. Strong leadership and strategic capabilities in data management, architecture, and business optimization.
Educational Qualification:
- Master’s in Data Science – BITS Pilani, 2020
- Data Analytics & Machine Learning (Executive Program) – IIT Roorkee, 2017
- Bachelor’s in Computer Science – JNTU University Hyderabad, 2010
Technical Skills
Big Data & Hadoop : HDFS, MapReduce, Hive, Pig, Sqoop, Spark, Oozie, Flume, Kafka, Ambari, Falcon, Kylin, Sparklens
Cloud Technologies: AWS (EC2, S3, EMR, Redshift, Glue, RDS), Azure (Data Factory, Synapse Analytics, HDInsights, CosmosDB), GCP (BigQuery, Dataproc)
Data Warehousing & BI Tools: IBM Datastage, SAP BODs, Informatica, SSIS, SSAS, Tableau, IBM Watson
Databases: Oracle, Teradata, DB2, MS SQL Server, NoSQL (Cassandra, MongoDB, HBase, Redis, Neo4j)
Programming Languages: Scala, Python, Java, Dask, GoLang, Groovy
Scripting: Unix Shell Scripting, Perl, Python
Frontend: Node.js, Angular
Reporting Tools: IBM Watson, SSRS, Tableau, Power BI, Cognos
Methodologies: Agile/Scrum, Waterfall
Other Tools: GitHub, Jenkins, Kubernetes, Jira, Restful APIs, Databricks, Google Analytics, SAP Hana, Adobe Analytics, OneStream
Expertise
Leadership & Project Management: Over 11 years of experience leading teams, managing projects, and ensuring high-quality, on-time delivery. Certified in PMP (Agile), TOGAF, and Blockchain.
Cloud & Big Data Solutions: Extensive experience in architecting and implementing cloud-based and big data-driven platforms, especially in AWS, Azure, and GCP environments.
Data Analytics & Machine Learning: Deep expertise in data strategy, machine learning implementation, and deriving business insights from data for customer behavior predictions, marketing personalization, and business optimization.
Cross-Industry Domain Knowledge: In-depth experience in FinTech, E-Commerce, Retail, Supply Chain, Banking, Marketing, and Healthcare, with a strong focus on data-driven decision-making.
Innovation & R&D: 7 years in R&D, with contributions to open-source communities such as Apache Kylin and Qubole (Sparklens). Holds a patent in Virtual Clothing Customization and Recommendation Engine.
Work Experience:
Associate Director/SEM II – Engineering & Product Management (Apr 2021 – Present)
- Leading a data platform team of 162+ members in the FinTech space, managing data strategies, cloud infrastructure, cost optimization, and data governance.
- Implemented machine learning algorithms for customer engagement, contributing to a 13% improvement in customer engagement and a 20% reduction in operational costs.
Senior Manager – Web & Marketing Analytics (Jan 2020 – Mar 2021)
- Led a 43-member analytics platform team for B2C & B2B initiatives.
- Delivered solutions for Virtual Clothing, Pricing & Promotions, and Clickstream applications, optimizing operational efficiency and boosting business KPIs.
Senior Architect/Manager – Data Analytics & Customer Insights (Nov 2016 – Dec 2019)
- Managed a 38-member team, driving R&D efforts, analytics, and customer insights in retail, focusing on inventory management and predictive analytics, resulting in a 15% cost reduction.
Subject Matter Expert (SME) – Clinical Data (Apr 2013 – Oct 2016)
- Led B2B eCommerce initiatives in healthcare, designing strategies for big data applications and managing Hadoop-based data lakes for clinical data analysis.
Tech Lead – Hadoop CoE (Sep 2012 – Mar 2013)
- Provided data warehousing analytics for payments, credit scoring, and fraud detection projects, migrating ETL jobs to the Hadoop ecosystem for increased efficiency.
ETL Developer (Feb 2011 – Sept 2012)
- Developed ETL processes for healthcare analytics, designing data models and performing optimization for large datasets.
Junior Analyst (Oct 2009 – Jan 2011)
- Worked on automation and data testing processes for Gameloft’s Vivendi platform.
Exp : 5+ Years
$30 / hr
Manikanta K
Big Data Engineer
Data Engineer with 5+ Years of experience in BI development using Big Data and Cloud services
Key Skills
- Python
- Big Data
- MS-SQL Server
- Azure SQL
- TFS
- VSTS
- Azure Data Lake
Manikanta K
Data Engineer
Exp : 5.5 Years
$30 / hr
Key Skills
- Python
- Big Data
Additional Skills
- MS-SQL server
- Azure SQL
- TFS
- VSTS
- Azure data lake
- Data factory
- SSIS
Detailed Experience
- Extensive experience working on Azure cloud and providing solutions involving several services like Datalake, VM, ADF, Azure Function, Databricks etc
- 2 years of experience working on AWS cloud and providing solutions involving several services like S3, EC2, Glue, Lambda, Athena etc
- Capable of writing complex SQL queries and able to tune the performance
- Design and Development of Big data Applications in Apache Spark, Azure
- Experience in utilizing MSSQL, Azure SQL, and Redshift.
- Excellent verbal and written communication skills and proven team player.
Exp : 5 Years
$30 / hr
Shashank
Big Data Engineer
Data Engineer with 5 Years of experience in Python, Big Data and Cloud services
Key Skills
- Python
- SQL
- AWS
- Big Data
- Oracle
- MySQL
- SQL Server
- PostgreSQL
Shashank
Data Engineer
Exp : 5 Years
$30 / hr
Key Skills
- Python
- SQL
- AWS
- Big Data
Additional Skills
- Oracle
- MySQL
- SQL Server
- Postgres
- Apache Spark
- Pyspark
- DMS
- RDS
- Glue
- Lambda
- Dynamo
- Cloudwatch
Detailed Experience
- Proficient with AWS cloud services to develop cost-effective and accurate data pipelines and optimize them.
- Capable of handling multiple data sources like DynamoDB, RDS, JSON, text, CSV.
- Developed Pyspark scripts in Databricks to transform data and load them into data tables.
- Good experience in the creation of pipelines for loan audits, and risk analysis for RBI compliance.
- Automated the generation of PMS reports using Pyspark.
- Involvement in data migration activities and data validation post data migration.
- Expert in developing Pysprak scripts to transform data to new data models.
- Created a data pipeline for a client to price their products and an ETL pipeline to compare the pricing of their product with their direct competition.
Exp : 4+ Years
$25 / hr
Vivekanand C
Big Data Engineer
Data Engineer with 4+ years of experience in ETL development and crafting robust Data
Warehouse solutions.
Key Skills
- AWS Services
- Python
- SQL
- Big Data
- Airflow
- Github
- JIRA
- Oracle SQL
Vivekanand C
Data Engineer
Exp : 4+ Years
$25 / hr
Key Skills
- AWS services
- Python
- SQL
- Big Data
Additional Skills
- Airflow
- Github
- JIRA
- Oracle SQL
- Jupyter
- V S Code
Detailed Experience
- Capable of leveraging a suite of technologies, including Python, SQL, PySpark, and AWS services like EMR, Glue, Redshift, Athena, EC2, and S3, to transform raw data into actionable insights.
- Development and implementation of ETL solutions using Python, PySpark, SQL, and AWS services, particularly AWS Glue and AWS EMR.
- Proficient in orchestrating ETL Data Pipelines using Apache Airflow, integrating S3 as a Data Lake, Glue for Data Transformation, and Redshift for Data Warehousing to create end-to-end ETL pipelines.
- Testing and data validation using Athena to ensure data accuracy and reliability after transformation.
- Successfull implementation of robust Data Warehousing solutions with Redshift to streamline downstream data consumption.
- Building Data Pipelines, Data Lakes, and Data Warehouses while demonstrating strong knowledge of normalization, Slowly Changing Dimension (SCD) handling, Fact and Dimension tables.
- Extensive familiarity with a range of AWS services, including EMR, Glue, Redshift, S3, Athena, Lambda, EC2, and IAM, facilitating comprehensive data engineering solutions.
- Expertise in Oracle Database, adept at crafting complex SQL queries for data retrieval and manipulation.
- Sound understanding of SQL concepts such as views, subqueries, joins, string, window, and date functions.
- Proficient in PySpark concepts, including advanced joins, Spark architecture, performance optimization, RDDs, and Dataframes.
- Skilled in performance tuning and optimization of Spark jobs, utilizing tools like Spark Web UI, Spark History Server, and Cluster logs.
Exp : 4 Years
$25 / hr
Rohit M
Big Data Engineer
Data Engineer with 3+ years of relevant experience on the Big Data platform and AWS services.
Key Skills
- AWS Services
- Python
- PySpark
- Flask
- Django
- REST APIs
- MySQL
- MongoDB
Rohit M
Data Engineer
Exp : 4 Years
$25 / hr
Key Skills
- Python
- PySpark
- AWS
Additional Skills
- Flask
- Django
- REST APIs
- MySQL
- MongoDB
- PostgreSQL
- GIT
- Docker
- Bamboo
- Bit Bucket
- Spark Streaming
Detailed Experience
- Experience in building data pipelines using AWSservices such as EC2, ECS, Glue and Lambda.
- Involved in writing Spark SQL scripts for data processing as per business requirements.
- Exception Handling and performance optimization techniques on python scripts using spark data frames.
- Expertise in developing business logic in Python, PySpark.
- Good experience in writing queries in SQL.
- Proficient in working with data storage and retrieval using AWS S3 and integrating it with Spark and PySpark for efficient data processing.
- Development of ETL workflows using PySpark and Glue to transform, validate, and load large amounts of data from various sources to the AWS data lake.
- Expertise in designing and implementing scalable data architectures in AWS, including data modeling and database design using technologies like Redshift and RDS.
- Strong experience in using tools like GIT, Docker, JIRA
- Proficient in programming by using the IDE’s such as Eclipse, PyCharm, VS Code
- Hands-on experience in spark Streaming.
- Usage of Databricks for a variety of big data use cases, such as data preparation, ETL, data exploration and visualization, machine learning, and real-time analytics.