I have 10 years of experience in Big data engineering, during this tennure I have created quite a few robust data pipelines which includes pulling data from various sources like RDBMS, ftps, streams e.g. Kafka, aws sqs and process them on fixed infra like CDH and transient resources like EMR in highly optimized cost effective manner.
