Streamline Business Data and Forecasting

Author: Matt Paterson
Last modified: June 7, 2021

Challenge:

Does your company collect data? It’s OK, you can be honest. I do. Whether you are an ecommerce firm whose lifeblood is understanding your customers’ online behavior, or a manufacturing company trying to stay ahead of machine maintenance, you collect data on your business that helps your company perform. Cloud Brigade is setting out to answer the question: Can AI and Machine Learning help uncover actionable insights on the health and direction of your business using the data that you already own?

Benefits:

  • Gives you a real time or near-real time dashboard to monitor your business and identify issues
  • Smart data visualizations compare sales and supply cycles to budget and headcount forecasts
  • Applies NLP to your customer feedback and quantifies general attitudes (happy, frustrated, etc)
  • Allows you to run the business instead of creating manual forecasts and repetitive reports
  • CB Abi can make predictions based on your behaviors and specific market position

About/Background

There seem to be new business intelligence software released all the time. There are great apps that track your marketing spends and trends, apps that predict customer churn and lifetime value, inventory apps that help your managers better understand the product pipeline; but very few handle all of the above, and those come at a dear cost. What if you want to measure them all at the same time while saving tens or even hundreds of thousands of dollars in annual labor costs normally associated with bookkeeping, forecasting, and reporting? With CB Abi, this can be your reality.

Challenges

  • Irresolvable Complexity – With all of the different software out there, how can you decide what data to track in your business, and what does any of it even mean? Do your many tools integrate well?
  • Improving Systems and Processes – CB Abi allows you take action instead of spending all of your time learning what has already happened by providing dashboards to your business’s health and deep-dives into the predicted outcomes of various action options
  • Antiquated Technology – With the arrival of IoT devices and machine maintenance sensors, your business should be using predictive maintenance to save money and reduce downtime
  • Antiquated Technology – With the ability to use Unsupervised Learning ML Models to cluster your customer base, you no longer have to trust the gut of the highest paid person in the marketing team
  • Skills & Staffing Gaps – If you’re already collecting more data than you can use, hiring a team of three or four software engineers to sift through everything and create a new system to maintain will not pay for itself for many years. Spending a fraction of that to hire a consulting team to do it will start earning you money without the downstream cost liabilities associated with added staff

Where it Began

In the winter of 2020-2021, Chris Miller, founder and CEO of Cloud Brigade in Santa Cruz decided that it was time to get a little healthier. As a data geek and inventor, Chris was excited to see how tracking his progress using his gadgets might help him stay on track or be more successful in his workout and diet plan. He asked me to build out a Data Science pipeline that took all of the various sensor data and health reports, do some predictive modeling with them, and build dashboards with the results.

Before I went back to school for Computer Science I was a Sales Director for an online wine retailer. It was a startup that sold for $100M based mostly on their ability to dial in the important Key Performance Indicators, or KPI’s. They modeled them out with accurate forecasting, anchoring on the Lifetime Value of each customer (LTV) and the Return on Investment of each employee (ROI). I built a dashboard for that in excel, which is what compelled me to learn how to do it in python, javascript, and to use Machine Learning Models and Statistical Analysis for even better prediction accuracy than multiplication factors alone. 

My project back then at the wine startup saved me 30 hours a week (that’s right–an entire person’s workweek) which reduced my job to a mere 45 weekly hours. It was clear that in creating this new workflow now, I had come to the marriage of preparation and opportunity, and thus CB Abi was born.

Solution and Strategy

Cloud Brigade is no stranger to finding the best technologies for your business. As an AWS Partner, Cloud Brigade has experience pairing the right technology with the right problem, and CB Abi allows us to use this same process to manage your company’s data. Working with your IT or Data professionals, we can custom-build ETL scripts or connect to existing API’s. Then we are able to restructure quantitative data and interpret qualitative data to recommend best practices for your data warehousing. Finally we use Amazon Web Services (AWS) to handle our workload in the cloud.

The end result is a branded interface that can live on a big screen in your office or boiler room like an ESPN reel or a CNBC Ticker, keeping you and your teams aware and excited about your daily business performance. You can also log in to this dashboard and drill-down into each segment of your business for a closer and more in-depth look at what is happening.

This Is No Ordinary Project

Building Machine Learning solutions that deploy on the edge is not as easy as it sounds. Sure, using advanced algorithms that in many cases were developed at Google or Amazon and applying them to your business using python libraries and a little statistical knowhow seems pretty straightforward. However, it takes a lot of time-consuming data engineering before you even get to the Random Forests and Neural Networks. 

If you don’t already have a handle on your company’s data and how to access it, that will add time and complexity to the job. However, getting into the weeds creating scalable systems for your business is what we do best.

Smarty-Pants Technology

Before we can tackle this issue, it is important to build solid data pipelines using AWS Glue (as is described in another article here), or Amazon Data Pipeline or EMR. Glue allows us to take data from our data lake in AWS S3 where our devices have been programmed to send the data on a daily schedule, run an ETL job (Extract, Transform and Load) on the data, and insert it into our Amazon RDS (Relational Database System) or Amazon Redshift Data Warehouse. We can do this in PySpark, utilizing the RDD and making copious amounts of data far more manageable. But at the end of the day, the microservice that delivers a scalable solution that is the right fit for your business is the one that we will choose. Or if need be, we’ll invent one. 

Then we have to explore all of the data, normalize it for comparison, and then re-write our pipeline. Along with restructuring the database to hold it in its proper form without duplicating, truncating, or clobbering other data. The objective is to have a self-contained software that can handle the forecasting and data reporting for your business that is easy to use so that nobody has to become an expert in using the software–you’re already an expert at what you do. CB Abi lets you spend your time doing what you do best.

Photo Credit: https://medium.com/codefully-io/aws-glue-for-loading-data-from-a-file-to-the-database-extract-transform-load-fe6b722e11b8

From there, we can use a data dashboard and visualization software that renders comparative views of each of the different metrics. We may choose to build a visualization solution in-house using a python library such as streamlit, but for most use cases, Amazon Quicksight is a robust solution with a very low cost when it stays inside an organization. 

Finally, we deploy our in-house Data Science and Machine Learning team to run exploratory data analysis, predictive modeling, and more data visualizations in Amazon Sagemaker. For this step we’ll have questions for your marketing, sales, manufacturing, and IT teams in order to make sure that the models we are building properly speak to the specifics of your business.

So what does this mean? This means that we can build out CB Abi to monitor your sales and supply chain, track your customer segmentation trends and marketing effectiveness, forecast your budget and hiring needs, and even run predictive maintenance on your machinery or equipment. The actionable insights that CB Abi’s dashboards leave you with will free up time to do the important things, like coaching your staff, innovating to get ahead of your competition, and even taking a vacation to be with your family where you don’t have to constantly check your cell phone for peace of mind. 

Building the Model

Since every business will have its unique obstacles to overcome, CB Abi is not a simple one-day solution. We’ll ask to spend a little time with your decision makers and stakeholders to gain some knowledge into what currently informs your forecasts. We’ll want to get a sense of what questions you already ask yourselves every month. We’ll ask what KPI’s you are currently working with, and what rates of change you are expecting to see on a month to month or quarter to quarter basis. With a little domain knowledge and historical data, we can turn around predictive models that will live inside CB Abi with insights ready to help you plan changes for the future.

Technical Hurdles to Overcome

Since every system is different, there may be some data migration and normalization hurdles that will arise. Provided our teams are on the same page from the start, there’s no hurdle we cannot get over as we stay on track toward your custom solution.

A Successful Integration and a Technology Handoff

One thing we have learned through our work in Machine Learning is just how time consuming and intricate each step can be. A lot of times, the work of building up the infrastructure and lining up the data so all the moving parts can communicate with each other is actually more challenging than creating and operating Artificial Intelligence. This product is no different. The live application must be programmed to dynamically handle new data fields, data types, and data sources whenever the user wants to add them. The data pipeline must be programmed with the proper error handling and notification systems, and the data reports must be connected to the alert systems with bullet-proof security. However once the data science has been fully integrated, very little regular maintenance is required to keep the system working.

Opportunities

With a solution like this for your business, we can also build you a virtual assistant that can interact with your managers to alert them to oddities or answer quick questions. This assistant can utilize Amazon Lex, the software behind Alexa, to give a conversational overview of the top-line numbers every day. From threshold notifications to quickly processing how a new idea will change your profitability in a coming quarter, we’re here to let CB Abi keep working for you even after we’ve handed the code repos over to your teams.

CB Abi also integrates with Cloud Brigade’s SmartDispatch which labels, prioritizes, and delegates your incoming customer emails so that you can save time and money. The data from SmartDispatch can inform your customer segmentation models, your employee productivity metrics, and even your budget forecasts. With CB Abi you’re not just getting a report, you’re gaining a team member that never needs a coffee break.

WHAT’S NEXT

If you like what you read here, the Cloud Brigade team offers expert Machine Learning as well as Big Data services to help your organization with its insights. We look forward to hearing from you.

Please reach out to us using our Contact Form with any questions. 

If you would like to follow our work, please sign up for our newsletter.

About Matt Paterson

Matt has over 12 years of business experience in sales, entrepreneurship, machine learning and data science, and he earned his AWS Machine Learning Specialty Certification in the summer of 2021.

With a BA in English from the University of New Hampshire, Matt began building his professional foundation by surpassing expectations in sales and leadership roles before starting his own small business.

Through building data dashboards, Matt saw the importance and opportunity around this data, and opted to obtain his second degree in Computer Science. Matt’s highly technical engineering skills allow him to analyze and forecast data using Python, Scikit-Learn, Tensorflow, PyTorch and Django, while also looping in his creative problem solving and people/project management abilities to deliver solutions that people are able to use and enjoy.

When Matt’s not geeking out over data at work, he’s learning how to play the piano, running on the beach, reading fiction, or identifying exoplanets with Neural Networks and deep-diving in to other outer-space exploration topics.

CONTACT US

Would you like more information, or have an idea we can help become a reality?

Contact Us
Email Us
Call Us 831-480-7199
Newsletter Subscription