Scaling Data Sourcing with ML-Powered Web Scraping

By combining machine learning with automation, Arrk helped the customer improve data sourcing, harmonisation and operational efficiency using AWS SageMaker.

The Customer provides research, market analysis, and consulting for technology and business sectors, relying on outsourced data harvesting and entry from a range of complex sources, resulting in a labour-intensive process.

Manual data entry posed high costs and delays, while diverse, unstructured data sources complicated automation. The customer aimed to automate at least 50% of the process to reduce costs and improve operational efficiency.

We recommended an iterative approach, testing different technologies and refining the solution as requirements and limitations became clearer. The automation journey included three phases:

• Stage 1: A rule-based mapping approach was used, but it struggled to accommodate inconsistencies in data formats.
• Stage 2: NLP and NER techniques were used for free-text extraction but lacked sufficient accuracy.
• Stage 3: Integrated rule-based processing with machine learning.

Using AWS SageMaker for training and hosting, advanced models like BERT and Decision Forest were implemented to address complex classification challenges. This hybrid approach significantly reduced manual intervention, streamlined operations and lowered costs.

The machine learning solution streamlined operations and delivered measurable results:

• Model Accuracy: Five machine learning models, trained and hosted using cost-efficient AWS SageMaker resources, achieved 95%-98% accuracy, surpassing human performance in some cases.
• Efficiency Gains: Automation reduced manual data entry by 41%, saving 500,000 entries annually and eliminating 2,000 person-days of effort.
• Future Prospects: The models can be continually retrained to adapt to changing data patterns, helping maintain accuracy and efficiency over time.

Working with Arrk, the customer exceeded its original automation target and implemented a solution that improved accuracy, efficiency and operational performance.