While big data has long been harnessed by leaders across virtually every industry to make key business decisions, today, the field is a proven and established subset of tech. With an ever-growing list of professions and use-cases surrounding big data, trends have emerged in how that data is collected, organized and used.
Obviously, developments in big data mean different things to different companies. We spoke to data pros at three Chicago tech companies — working on everything from building digital assistants to improving employee retainment strategies — to learn more about the trends they’re keeping up with and how they’re impacting business operations.
Trusting one’s gut is a practice that can lead to success in many professions. But data science is not typically seen as one of them, since trusting numbers is generally considered safer than trusting instincts. However, Data Science Manager David Thompson at Pareto Intelligence said that sentiment is changing. The leader at the healthtech company said intuition plays an increasingly large role in how industry pros work with big data.
What are the top big data trends you’re watching that are significantly impacting the industry?
Lately, and rightfully so, there has been a significant focus on social determinants of health — the conditions in which people are born, grow up, live, work and age that impact health status. Healthcare has been slow to arrive here, but trends are showing that nearly everyone in the industry is agreeing that there are benefits to considering everything about a person when assessing overall health and developing strategies for improving health. There is already an overwhelming volume of health-related data about each person, and SDOH adds even more layers to that.
As we continue to nearly drown in messy and erroneous data, I believe intuition will begin to play an even larger role in analytics.”
What under-the-radar big data trends are you watching that the industry isn’t talking about?
I have been watching the re-emergence of intuition in developing analytic strategies. While originally an enemy of data science, intuition seems to be acting more like an ally. Data scientists and analysts that have been working with big data for a while seem to have developed a recalibrated intuition that is rooted in deep experience with the data rather than general experience within the industry only. And as we continue to nearly drown in messy and erroneous data, I believe intuition will begin to play an even larger role in analytics.
How are these trends affecting the future of your company?
I believe these trends mean two things for Pareto Intelligence, and both are related to talent and human capital.
Weaving in SDOH data means we need more hands on deck. Additionally, we need to develop deep experience. If we are going to acknowledge the quiet undercurrent of intuition and leverage the benefits it can bring, we have to foster a culture that makes a long-term commitment to the company and the industry an easy and attractive decision for our data scientists and analysts. We are doing that every day through the benefits we offer, the community we cultivate, values we commit to, and so on.
When you’re working with datasets so complex they can predict and analyze consumer behaviors and even emotions while they’re online, simplifying workflows can go a long way for increasing productivity. Machine Learning Engineer Drew Boshardy told us about some tech tools that were put in place at marketing tech company Networked Insights to do just that.
What are the top big data trends you’re watching that are significantly impacting the industry?
I’m closely following the ongoing development of Kubernetes and its related technologies such as Skaffold, Kubeflow and Helm. It may seem like overkill for some use cases, and it likely is for many basic applications, but for a moderately to very complex system, it really helps keep everything running and easy to deploy to. Similarly, orchestration technologies such as Airflow are going to be very big going forward to help simplify more complicated and essential workflows, making them easier to maintain and diagnose.
Orchestration technologies such as Airflow are going to be very big going forward to help simplify more complicated and essential workflows.”
What under-the-radar big data trends are you watching that the industry isn’t talking about?
I think it’s being talked about a lot in the sphere of Google Cloud Platform-based companies, but Apache Beam is a real lifesaver for big data jobs. Both persistent and ephemeral self-managed Hadoop clusters still seem to be very common and, in many cases, unwieldy.
How are these trends affecting the future of your company?
We are currently using Google’s hosted Kubernetes solution Cloud Container Engine for our core infrastructure and other products. Our data infrastructure for our new DaaS product is running on Airflow and Apache Beam in the Google Cloud Dataflow to keep data up to date and reliable. We have the flexibility necessary to work with our many data partners, as well as the confidence that we can handle any irregularities or downtime easily and quickly thanks to Airflow.
What good is having all the data in the world if you can’t make good use of it when you need it? GreenKey Technologies provides entities in the OTC capital markets field data based on their audio and text communications. Chief Data Scientist Tejas Shastry said front-office staff at client companies could use quick, on-demand access to their comms data without tapping a data scientist for help. So, Shastry sees digital assistants as the next big wave in big data.
What are the top big data trends you’re watching that are significantly impacting the industry?
We use natural language processing to help financial firms structure and search their millions of voice and chat conversations for insights. Over the last year, the biggest shift we’ve seen is a rise in the need to query big data in the front office. Instead of using data scientists and business intelligence analysts to scour their data, firms want chat-bot interfaces so their front-office workers can get insights from their data directly as they’re doing their jobs. We think this trend will continue, with digital assistant use growing across many industries to help surface specific insights from internal data lakes.
Instead of using data scientists and business intelligence analysts to scour their data, firms want chat-bot interfaces.”
What under-the-radar big data trends are you watching that the industry isn’t talking about?
The implementation of the data protection and privacy regulations under the General Data Protection Regulation in Europe have changed the way many of our clients think about leveraging their data. With tight regulations, we see firms increasingly requiring solutions that can be deployed on-premise or in a private cloud and co-located with privacy-sensitive data. This means technology companies will be increasingly offering solutions that don’t rely on a public cloud, and similarly, companies with GDPR-related data will need to embrace cloud-standard deployment models like Kubernetes in their own environments.
How are these trends affecting the future of your company?
Being cognizant of these trends has helped us deliver better products to our clients faster. We’ve adapted our NLP that initially focused on voice conversations to process chat as well, and doubled-down on our on-premise and private-cloud deployment models. As we move forward, we will continue to think of ways we can better expose big data directly to front-office workers without firms needing to build their own machine learning models or hire their own data scientists. We think both of these trends point to companies wanting out-of-the-box, on-premise solutions to make sense of their data.
