I come from a non-tech background, content marketing specifically. For over eight years, I’ve focused solely on helping my clients craft thoughtful, well-written and polished articles that appeal to their target audience….Content that made their readers stop what they were doing to spend time and energy reading what I’ve cooked for them. I’ve been the storyteller and content editor behind brands, ensuring that every word resonates.
You might wonder, “If you’ve achieved all these, why transition to data engineering?” Good question! Data fascinated me some years ago when I discovered that I got ads, blog posts, ebooks, and other marketing resources related to a term I searched online or whenever I discussed a topic with a friend. Upon further research, I discovered that when you conduct online research and start seeing targeted ads related to that subject or topic, data is in play.
Now, let’s tie this scenario to how it might be related to data engineering:
First off, when you explore a topic online, various platforms collect data about your search behaviour. This includes the keywords you imputed, the websites you visited, the content you engaged with and even the average time spent. In this case, data engineers design and implement systems that efficiently collect and track this information; while also ensuring it is processed accurately and securely.
Because our data collected is usually unstructured and huge, data engineers develop systems and algorithms to process and analyze this data; and ultimately extract relevant insights. Since our online activities are analyzed to understand our preferences, a profile is created for advertisers to use to tailor their marketing messages.
Finally, data engineers construct and recommend systems that predict user preferences based on historical data(past information or records of events, activities, or observations that have occurred over a certain period). In the case of targeted ads, algorithms are crafted to identify patterns in our online behaviours and serve us ads aligned with our interests. Therefore, the more accurate these systems (designed) are, the more effective and personalized the ads we see will become.
Researching data and becoming fascinated by it made me decide to study data engineering. However, that doesn’t mean I wasn’t afraid. I was! Transitioning from content marketing to data engineering was like taking a daunting leap into the unknown, or charting into unknown waters. If you are like me, the fear of the unknown, the concern about mastering technical jargon and intricacies, the imposter syndrome, the tiny voice in your head telling you you’re making a regrettable decision, the anxiety and concerns about adapting to and seamlessly transitioning into a different professional territory would send shivers down your spine.
However, the truth is that growth often lies outside our comfort zones. Change is inevitable and one of the major thoughts that gave me a paradigm shift was “acknowledging that every expert I see today was once a beginner. They weren’t born naturally with coding languages or knew databases or the intricacies of data architecture. If they could study and become who they are today, I would do the same.”
That was all I needed to help allay my fears and inspire me to approach learning with curiosity and hunger, rather than fear. And guess what? It’s working!
So, if you are new to tech, looking to transition to tech, transitioning to tech from a non-tech background, or considering Data Engineering as a new career path, consider this a hand-holding guide for you.
P.S. As someone who struggled with understanding basic terminologies when I started, partly due to fear and imposter syndrome, I’ll be using analogies (which have worked for me so far) in this post and in the future to help you better understand my posts. Let’s leave the “industry jargon” for the more experienced folks, shall we? 😅
Get ready to dive in!
What is data engineering?
Data engineering involves the collection, organization, and processing of data to make it accessible, usable, and meaningful. In simple terms, data engineering involves creating a solid foundation for data so that it can be effectively utilized to derive insights and support business goals.
Now, let’s use a simple analogy to explain data engineering.
Imagine you run a bakery, and one of your staff is responsible for ensuring that all the ingredients needed for baking delicious pastries are available, well-organized, and ready for use. In this scenario, data engineering is like being the master organizer and facilitator in the bakery.
Let’s break it down further:
1. Ingredients (Data): In the bakery, ingredients are like data. You have flour, sugar, eggs, and various other components. Similarly, in data engineering, you have different types of data — customer information, sales numbers, product details, supplier information, employee scheduling and productivity (work hours, shifts and schedules), inventory, etc.
2. Organization (Data organization): In the bakery, you ensure that flour is in one place, eggs in another, and sugar in its designated spot. Likewise, data engineering organizes data. It structures and arranges data so that it’s easily accessible and usable.
3. Preparation (Data processing): Before baking, you might need to process ingredients — like cracking eggs, melting butter or sifting flour. In data engineering, there’s a process called ETL (Extract, Transform, Load). It’s like preparing data for analysis: Extracting it from various sources, Transforming it into a usable format, and Loading it where it’s needed.
4. Recipes (Data systems): The bakery has recipes that guide how ingredients are used. Similarly, in data engineering, some systems and frameworks guide how data is processed and utilized. Think of these as the step-by-step instructions for handling data.
5. Baking (Data analysis): Once your ingredients are organized and prepared, you bake. In data engineering, this corresponds to the analysis phase. Now that your data is well-prepared, analysts and data scientists can “bake” insights and valuable information from “what you’ve prepared”.
6. Continuous improvement (Optimization): Just as a baker or a chef might tweak a recipe for better results, data engineers continuously optimize their processes. They might find ways to make data processing faster, more efficient, and adaptable to changing needs.
So, in a nutshell, you can liken a data engineer to being the behind-the-scenes expert, ensuring that all the ingredients (data) are well-organized, prepared, and ready for the chefs (analysts) to create something delightful.
What is the difference between data engineering and related fields like data science and data analysis?
Data engineering, data science, and data analysis are closely related fields, but they have distinct focuses and responsibilities. Here are the key differences between them:
Data Engineering: Think of data engineers as construction workers. They lay the foundation of “the house”, construct the framework and ensure all important infrastructure is put in place (plumbing, wiring, water, electricity, etc.). They focus on designing, constructing and maintaining the data architecture (hence, the word data engineer), similar to constructing the physical structure of a building. They ensure that the data pipelines created are robust, scalable and can handle the load.
Responsibilities:
- Building data pipelines for data extraction, transformation, and loading (ETL).
- Managing databases and data warehouses.
- Ensuring data quality and reliability.
- Designing and optimizing data architecture.
Data Science: Data scientists are the architects responsible for envisioning and designing the building’s structure. They analyze requirements, plan how every component will interact, and develop a blueprint. Just as an architect uses design principles to create functional and aesthetically pleasing spaces, data scientists use statistical and machine learning strategies to garner insights.
Responsibilities:
- Developing machine learning models for predictions and classifications.
- Analyzing complex data sets to identify patterns and trends.
- Extracting meaningful insights to inform business decisions.
- Often involves coding in languages like Python or R.
Data Analysis: Think of a data analyst as an interior decorator. They use statistical tools and visualization techniques to make data more appealing and understandable — — hence, providing insights for decision-making for relevant stakeholders.
Responsibilities:
- Descriptive statistics to summarize data.
- Exploratory data analysis to understand patterns and relationships.
- Creating visualizations to communicate findings.
- Often involves using tools like Excel, SQL, or statistical software.
Summarily, a data engineer builds robust infrastructure and pipelines for handling data (similar to constructing a building’s foundation), a data scientist designs the overall structure and plans, and uses advanced techniques to extract valuable insights, while a data analyst interprets and presents insights from the existing data.
The role of data engineering in the data lifecycle
Let me start by sharing real-world examples that illustrate how data engineering improves business intelligence and decision-making. I’ll use three scenarios: a retail company, a manufacturing company and a healthcare provider.
For a retail company looking to analyze sales performance and customer behaviour, the role of data engineers is to build pipelines to extract data from online transactions, point-of-sales systems, and customer databases. This dataset collected (from online transactions, point-of-sales systems, and customer databases) lays the foundation for business intelligence and decision-making for this retail company.
As for the manufacturing company looking to optimize its supply chain, a data engineer would design processes to collect and process data from sensors on inventory databases, production lines, and even logistics systems. This dataset collected would then empower business intelligence tools and business intelligence analysts to identify bottlenecks, optimize inventory, and most importantly, enhance supply chain efficiency.
A healthcare provider who wants to improve patient outcomes and operational efficiency would need a data engineer to build pipelines to gather data from medical devices, electronic health records, and patient management systems.
This dataset would enable the healthcare provider to gain insights into areas such as assessing and improving patient satisfaction and experience, patient treatment effectiveness and outcomes, resource allocation optimization, predicting and preventing the onset of diseases, identifying opportunities for enhancing hospital operations, understanding cost drivers and revenue opportunities, etc.
Dope, right? You bet!😉
Keeping the examples above in mind, the role of data engineering or a data engineer in the data lifecycle includes:
1. Data collection: This involves designing and executing systems to collect and extract data from different sources. These sources could be social media, transactional databases, sensor data from IoT devices, maps, texts, documents, images, sales figures, stock prices, etc.
2. Data storage: Using data warehouses or data lakes to store large volumes of data and ensuring data is organized for easy accessibility.
3. Data processing: Creating distributed processing systems to clean, aggregate, and transform data, ensuring it’s ready for analysis.
4. Data integration: Developing data pipelines that integrate data from various sources to create a comprehensive view.
5. Data quality and governance: Ensuring that data is of high-quality, reliable and adheres/complies with regulatory standards.
6. Data provisioning: Ensuring the processed data is available to end users and applications.
Why should I consider a career in data engineering?
Apart from the love for data, the demand for data engineers is consistently high as organizations seek professionals to manage and optimize their data infrastructure. This trend is driven by the increasing reliance on data-driven decision-making.
According to statista.com, the volume of data/information created/captured, copied, and consumed worldwide from 2020 to 2025 is forecasted to increase from 64.2 to 181 zettabytes. This can only mean one thing: there’s no shortage of data to be collected, mined and analyzed.
In 2020, LinkedIn listed Data Engineering as an emerging job in that year’s Emerging Jobs Report, which revealed that the hiring growth rate of professionals in this job has increased by nearly 35% since 2015. GenSigma also shares a similar opinion in its 2021 The State of Data Engineering Talent study, where it cited, “For every 3 Data Scientists in an organization, there is a need for at least 10 Data Engineers. The global market for big data and data engineering services is undoubtedly in high demand.”
Furthermore, data engineering is an ever-evolving field. It’s dynamic and offers professionals the opportunity to engage in continuous learning to stay at the forefront of technological advancements and maintain a competitive edge in the job market. A data engineer can also pivot to other roles such as Analytics Engineering, Data Architecture, Machine Learning Engineering, BI Developer, Data Science, Big Data Engineering, Cloud Solutions Architect, DevOps Engineering, etc.
Additionally, data engineering is very versatile and empowers you to work across different industries or niches — from finance and banking to healthcare, technology, e-commerce, telecommunications, government and public sector, manufacturing transportation and logistics, agriculture, media and entertainment, non-profit organizations, education, etc. There’s almost no industry where the services of a data engineer aren’t needed.
Finally, data engineering is innovative. Each organization has complex problems related to data storage, processing, and integration that a data engineer should solve. Working in this role requires creativity and innovation to design solutions tailored to meet the unique needs of different organizations.
With these in mind, “what characteristics should you possess to become a data engineer?”
Characteristics needed to become a data engineer
1. Analytical mindset: An analytical mindset is important for designing and optimizing data infrastructure since data engineers need to analyze data requirements, understand system architectures, and identify efficient solutions.
2. Programming skills: You need to gain proficiency in programming languages like Python, C++, Java, or Scala for building and maintaining data pipelines and infrastructure. Python is the world’s most popular programming language because it can be adapted across different applications, it has clear and readable syntax (hence, making it learnable for beginners), it’s open source and free; plus it’s the language of choice for data engineers. I recommend learning Python before taking on other programming languages.
3. Database management knowledge: It’s a no-brainer: Data engineers should have a strong understanding of database management systems (e.g., SQL, NoSQL) to design efficient data storage solutions.
4. Attention to detail: Now, this is my fave. 😁 As a content editor, I have a keen eye for detail to ensure accuracy, consistency, and clarity in any written content. This means that I’m accustomed to meticulously reviewing and correcting details.
Why is “attention to detail” important in data engineering? For starters, data engineers need to ensure the accuracy and reliability of data (hello “Veracity” 👋🏽) by identifying and rectifying inconsistencies, errors, or missing values in datasets.
Additionally, precision is required when writing code for data pipelines. Syntax errors or incorrect coding practices can impact the functionality of data processing scripts, making your attention to detail a valuable asset.
5. Problem-solving skills: Strong problem-solving skills are needed by data engineers when encountering various challenges, such as optimizing query performance or addressing scalability issues.
6. Communication skills: Effective communication is crucial for collaborating with cross-functional teams. Therefore, as a data engineer, you need to be able to convey technical concepts to non-technical stakeholders in understandable formats and work collaboratively to meet business objectives.
What career paths and opportunities are available within data engineering?
As mentioned in the early parts of this post, data engineering is an interesting career path. It offers diverse opportunities, allowing professionals to specialize in various areas based on their interests, skills, and the specific needs of organizations. Some of the common career paths within data engineering are:
1. Data Engineer:
- Responsibilities: Designing, building, and maintaining data architectures, pipelines, and systems.
- Skills: Database management, ETL processes, programming, data modelling.
- Career opportunities: Senior Data Engineer, Lead Data Engineer.
2. Cloud Data Engineer:
- Responsibilities: Utilizing cloud platforms (AWS, Azure, GCP) to build scalable and flexible data solutions.
- Skills: Cloud services, serverless computing, data storage, and processing on the cloud.
- Career opportunities: Cloud Data Architect, Cloud Data Engineer Specialist.
3. Big Data Engineer:
- Responsibilities: Working with large-scale (big data) and distributed data processing frameworks, such as Apache Spark or Hadoop.
- Skills: Distributed computing, big data technologies, performance optimization.
- Career opportunities: Senior Big Data Engineer, Big Data Architect.
4. Data Integration Engineer:
- Responsibilities: Integrating data from diverse sources, while ensuring consistency and reliability.
- Skills: Data integration tools, ETL processes, data transformation.
- Career opportunities: Integration Specialist, Data Integration Manager.
5. Streaming Data Engineer:
- Responsibilities: Handling real-time data processing and streaming technologies.
- Skills: Apache Kafka, Flink, Spark Streaming, event-driven architectures.
- Career opportunities: Real-Time Data Engineer, Streaming Architect.
6. Data Pipeline Architect:
- Responsibilities: Designing end-to-end data pipelines, optimizing data flow, and ensuring efficiency.
- Skills: Pipeline architecture, workflow orchestration, automation.
- Career opportunities: Data Pipeline Manager, Pipeline Architect.
7. Database Administrator (DBA):
- Responsibilities: Managing and optimizing database systems for performance and reliability.
- Skills: Database tuning, backup and recovery, security.
- Career opportunities: Senior DBA, Database Architect.
8. Data Governance and Quality Engineer:
- Responsibilities: Ensuring data quality, adherence to governance standards, and regulatory compliance.
- Skills: Data governance frameworks, quality assurance, metadata management.
- Career opportunities: Data Governance Specialist, Data Quality Manager.
9. Machine Learning Engineer (with a focus on Data Engineering):
- Responsibilities: Integrating machine learning models into data pipelines, and optimizing for production.
- Skills: Feature engineering, model deployment, data preprocessing.
- Career opportunities: ML Engineering Specialist, ML Infrastructure Engineer.
10. DevOps Engineer (Data focus):
- Responsibilities: Bridging the gap between development and operations, ensuring smooth deployment and management of data systems.
- Skills: Continuous integration/continuous deployment (CI/CD), automation, infrastructure as code.
- Career opportunities: DevOps Specialist, Data Ops Engineer.
11. Analytics Engineer:
- Responsibilities: Integrating data from various sources to create a unified dataset for analysis.
- Skills: SQL and query optimization, ETL tools, data modelling, database management systems, data warehousing.
- Career opportunities: Data Engineer (focusing on designing and maintaining data infrastructure), BI Engineer, Analytics Manager or Lead.
Are there any resources for learning data engineering?
Sure thing! There are numerous resources available online to learn data engineering as a beginner. Here are some recommendations:
1. Free online courses/certifications:
Coursera:
edX:
Google:
Meta:
UC San Diego:
Data Engineering Zoomcamp:
- It is a free course taught by Ankush Khanna, Victoria Perez Mola, and Alexey Grigorev for any beginner. P.S. To get the best out of this course, I recommend learning Python first before signing up. I recommend learning Python on W3Schools or GeeksforGeeks.
Data camp:
2. Practice platforms:
A). W3Schools: W3Schools will always be one of my favourites of all time. It is a popular online platform that provides web development tutorials and resources for beginners — ranging from Python to Java, Excel, SQL, NoSQL, Pandas, Numpy, Machine Learning, Git, Cyber Security, Statistics, Data Science, PostgreSQL — you name it! Check out their website and explore their vast library of unlimited tutorials/resources.
I love this platform because:
- It helped me learn Python easily compared to other platforms. It offers clear and concise tutorials that break down Python concepts into easy-to-understand sections/step-by-step approach, which is helpful for beginners. It applies this same methodology to any programming concepts you learn on the platform.
- It provides an interactive learning experience with live examples that you can modify and run directly on the website.
- The “Try It Yourself” editor allows you to experiment with code snippets in an online environment, fostering a practical and interactive learning experience.
- The core content on W3Schools is free to access. You can learn at your own pace without the need for a subscription.
B). LeetCode: LeetCode allows you to practice coding challenges related to data structures and algorithms.
C). HackerRank: HackerRank provides challenges and exercises on database management, SQL, and other data-related skills to help you sharpen your tech skills and pursue job opportunities.
D). GeeksforGeeks: GeeksforGeeks not only provides a wide range of tutorials, articles, and coding challenges covering various aspects of computer science and programming, including data engineering topics; but it also has an extensive collection of coding challenges and interview preparation material. Practice coding problems related to data structures and algorithms.
You can also learn from their blog section (which is an amazing source of knowledge 👌🏾) and there are tons of tutorials on various miscellaneous topics relevant to data engineering, such as Apache Kafka, Apache Spark, and more.
3. Blogs/Documentation
- Analytics Insight shares the best news and posts regarding Artificial Intelligence, Big Data, and Analytics. They also share the latest news on tech internships and tips for landing one.
- KDnuggets is an amazing resource for learning all things related to Data Science, Machine Learning, AI and Analytics.
- Medium is a good platform for learning more about data engineering. Data engineers share their knowledge, insights, and experiences that explore various aspects of data engineering, from fundamental concepts to advanced techniques.
4. YouTube Channels
- Data School: I’ve watched some of Kevin Markham’s videos, and I highly recommend his channel. Data School offers tutorials on data science, various aspects of data engineering, machine learning, and Python programming in a beginner-friendly manner.
- Corey Schafer: If you aren’t following Corey Schafer or watching his videos, you are missing out on a whole lot. His channel covers a wide range of Python-related topics, including database management, SQL, and other aspects relevant to data engineering. I love the way his tutorials are well-explained and suitable for different skill levels.
- Real Python: If you want to watch high-quality tutorials, articles, and videos covering a wide range of Python-related topics, including data engineering; Real Python is your go-to.
- Code basics: Codebasics provides tutorials on Python programming, data science, and machine learning. The channel covers topics related to data engineering, including SQL, pandas, and more.
- Sentdex: Hosted by Harrison Kinsley, Sentdex has tutorials on Python programming, machine learning, and data analysis; hence, making it useful for aspiring data engineers.
5. Online Communities
Stack Overflow — Data Engineering Section: You can engage with the data engineering community, ask questions, and learn from others’ experiences.
No comments:
Post a Comment