Positive impact on business processes with correct data matching

Challenge

Data matching and deduplication require careful planning and business understanding. When implementing these use cases, we at Scigility consider the following when applying our frameworks:

• Plan the data engineering steps. How much data is there? How often is it updated? Does the data require cleaning and pre-processing?
• Understand what the business considers a “match.”  Is a labeled dataset available? What is the cost of collecting reliable labeled data?
• How is the output from the application used? Is it more important to match as many records as possible, or to match only clear duplicates and avoid edge cases?

Data matching can have a considerable impact on many parts of a business. It is crucial to involve stakeholders throughout the development process and to design a pipeline that can scale and deliver reliable data.

Solution

Scigility supports and enables data science and data engineering by:

• Building scalable and distributed data pipelines using solutions such as Apache Spark and Dask.
• Using a data-centric approach to model development that fits the business needs. Labeling data is a time-consuming but extremely valuable process. Using active learning approaches, we can speed up the time to acquire new labels and improve the ML model.
• Deploying models and operations (monitoring, versioning, validation, etc.) with MLOps tools such as MLflow.