Data engineers play an essential role at any tech company. Amid seismic advancements in data, data science and machine learning engineering, industry leaders are increasingly recognizing the primacy of data management — and turning to professionals in this field for guidance.
That’s because data engineers are constant problem-solvers, helping a company to both make sense of its raw data and make the most of it. Data engineers build the pathways for data scientists to navigate, identifying trends in data sets then developing algorithms that allow for more useful analysis of those trends. Without these professionals designing, constructing and managing data infrastructure such as pipelines, any company would have trouble processing or interpreting data.
At Analytics8, a data and analytics consulting firm, data engineers are crucial to the company’s twin specialties of data strategy and business intelligence implementations. In his role there, consultant John Barcheski sees the value of his data engineering work daily.
“I really enjoy digging into unused data and discovering ways to unlock powerful insights that previously seemed impossible,” he said of the technical skills he relies on to succeed and his path to finding a perfect fit in this thriving field.
What led you into a career as a data engineer?
In undergrad, I majored in finance and economics. Although work in these fields was interesting, I was limited in what I could analyze within the data. I also didn’t enjoy doing manual work in Excel, and I knew there were more advanced ways I could work with data. After taking a few more advanced classes such as econometrics, I knew I wanted to focus more specifically on data for my career.
With this in mind, I pursued a master’s degree in business analytics and explored all the career options within that field. Data engineering specifically caught my eye because of the problem-solving aspect of the job; I was able to rely on my creativity and ability to code to find solutions to problems. Also, data engineers are foundational to the success of the rest of any organization, so I knew it would be exciting and impactful to work with many different teams.
What are the key technical skills that you use most often during your work day?
The responsibility of software engineers varies quite a bit depending on the job. But as a general trend, they focus more on supporting holistic environments, such as the data infrastructure — whereas data engineers are more closely focused on specific tasks, such as data transformations and pipelines that support the business.
Within a typical work day, I focus on building and maintaining the data ingestion process, modeling data and transforming data for analytics. I work with newer cloud technology like Fivetran, dbt, Snowflake and BigQuery, also building custom processes that can perform similar value when needed. The most important technical skill sets I leverage are Python, SQL and data modeling to support the data stacks I work with. I have found that when someone is a great problem solver, and they know how to put their ideas into re-usable portions of code, they are an incredible asset to an organization.
Describe a project you’re working on right now.
I am architecting a new data warehouse for a customer in the HR industry. I’m developing the conceptual architecture into actual code and infrastructure that can support analytics and new insight, enabling them to look at applicants under one unified view across different systems for various metrics.
The current processes to get this information are manually maintained, so we are working to automate the process as much as possible. We worked closely with business users to understand how they use the data today, what insights they need to fulfill requirements, what their pain points are and what insights they want to get for internal reporting.
Some of the challenges included combining customer source data that contains a variety of disparate information and identifying parts of the old process that can be automated, as well as identifying sources that hold valuable information that aren’t being leveraged. We’ve had conversations to fully understand the needs, risks and path forward. It is extremely rewarding to solve these issues.
