big data engineer

What is big data?

Big data is a label that describes massive volumes of customer, product and operational data, typically in the terabyte and petabyte ranges. Big data analytics can be used to optimize key business and operational use cases, mitigate compliance and regulatory risks and create net-new revenue streams.

Data sources include:

• credit card and point-of-sales transactions;
• e-commerce transactions;
• social media engagements;
• smartphone and mobile device engagements; and
• sensor readings generated by the internet of things (IoT).

Insights that can be gained by big data include:

• optimizing key business and operational use cases;
• mitigating compliance and regulatory risks;
• creating net-new revenue streams; and
• building compelling, differentiated customer experiences.

What is the role of a big data engineer?

It is the role of a big data engineer to build, maintain and ensure a production-ready big data environment. This environment this role works in will include architecture, technology standards, open source options, as well as data preparation and data management processes. The big data engineer's role is to:

• Design, construct and maintain large-scale data processing systems. This collects data from various data sources -- structured or not.
• Store data in a data warehouse or data lake repository.
• Handle raw data using data processing transformations and algorithms to create predefined data structures. Deposit the results into a data warehouse or data lake for downstream processing.
• Transform and integrate various data into a scalable data repository (such as a data warehouse, data lake, cloud).
• Understand different data transformation tools, techniques and algorithms.
• Implement technical processes and business logic to transform collected data into meaningful and valuable information. This data should meet the necessary quality, governance and compliance considerations for operational and business usage to be considered trustable.
• Understand operational and management options, as well as the differences between data repository structures, massively parallel processing (MPP) databases and hybrid cloud
• Evaluate, compare and improve data pipelines. This includes design pattern innovation, data lifecycle design, data ontology alignment, annotated data sets and elastic search approaches.
• Prepare automated data pipelines to transform and feed the data into dev, QA and production environments.

What are big data engineer skills and responsibilities?

Big data engineers gather, prepare and ingest an organization's data into a big data environment. They prepare and create the data extraction processes and data pipelines that automate data from a wide variety of internal and public source systems. Big data engineers also create the algorithms that transform the data into an operational or business format.

Obtaining a successful big data engineer position requires an understanding of:

• Common data archetypes, writing and coding functions, algorithms, logic development, control flow, object-oriented programming, working with external libraries and collecting data from different sources. This includes having knowledge of scraping, APIs, databases and publicly available repositories.
• Structured (such as RDBMS, spreadsheets), semistructured (such as log files, XML, JSON) and unstructured (such as text, video, audio, images, vibration) data sources.
• Relational databases (such as SQL, entity-relationship diagrams, dimensional modeling) and NoSQL databases (such as Hadoop, Spark, massively parallel processing databases).
• SQL-based querying of databases using joins, aggregations and subqueries.
• Open source tools, which can include real-time data processing products such as Beam, Kafka, Spark Streaming; time series databases like InfluxDB; a relational database like Postgres, graph database like Neo4j; and development environments such as Git and GitHub.
• Abstraction tools like Kubernetes.
• Mastery of computer programming and scripting languages (C, C++, Java, Python). As well as an ability to create programming and processing logic.
• Experience with machine learning algorithms and automated machine learning (AutoML) to automate and build continuously learning data processing streams and pipelines.

How does one become a big data engineer?

Formal training in computer science, math or engineering principles is the foundation for any successful big data engineer. These teach needed concepts such as functional decomposition, logical thinking, problem resolution, solution engineering, abstraction and creating repeatable processes.

A successful big data engineer must have solid data processing experience and a willingness to learn new tools and techniques. They must be willing to discard their current tool sets and embrace new, more powerful tool sets as they become available. Big data engineers need to have a natural curiosity and a desire to learn about the continuously changing open source landscape.

Ideally, a perspective big data engineer has working experience with both business intelligence (BI) and data warehousing, as well as with data science and data lake projects.

Big data engineers must have strong communication skills. They must feel comfortable interviewing and collaborating with business subject matter experts, business analysts and data science teams. This will help to identify, validate, value and prioritize business and operational requirements.