top of page
Writer's pictureMolekiu Limited

AI Applications | Navigating the Challenges of Data Drift in Real-World AI Applications

Introduction

Two years ago, our team embarked on an ambitious project: developing an AI-powered receipt information extraction and validation system with no human intervention. The concept was straightforward: users would photograph their receipts, and our intelligent system would process these images through an intricate pipeline—transforming photos into actionable data using image processing, OCR engines, and information extraction algorithms.

In our office’s controlled environment, everything seemed promising. Our model performed well, and we were optimistic about its potential to revolutionize receipt management. However, shortly after launch, reality hit us hard. The model’s performance degraded rapidly, and we quickly realized that real users captured receipts dramatically differently from how we had trained our system.

The problem lay in our meticulously curated training images—well-lit, clear, and free from shadows that could obscure vital information. But as you can imagine, not all users take the time to perfect their receipt photos. Many snap quick pictures in dim lighting or from awkward angles, leading to a disconnect between our model’s training data and real-world inputs.

The above shows an mock-up example on how different receipt images we used for training and real-world receipt images can be. (Obtained via the large-receipt-image-dataset-SRD imageset)


This experience taught us a vital lesson about the importance of iterative processes in machine learning systems. These iterations are necessary not only to improve performance but sometimes simply to deliver the expected value. We had failed to incorporate these iterative deployment cycles into our timeline and budget planning, resulting in higher costs and longer timelines than anticipated. This phenomenon occurs when changes in input data lead to a model that is no longer relevant, ultimately resulting in performance degradation.

Why does data drift occur?

Data drift can be caused by several situations:

  1. Differences in data quality between controlled environments and real-world settings (as demonstrated in our example)

  2. Changes in the real-world environment (e.g., major receipt printer suppliers altering their formats or seasonal changes)

  3. Shifts in user behavior (e.g., users rapidly switching to online grocery shopping during COVID-19)


How do we solve this issue?

To tackle this problem, we developed a phase-based approach, drawing ideas from books and courses, to maximize our models’ performance and effectiveness in delivering expected values to our clients:


Phase 0: User Acceptance Test

Users test the system with data they provide to observe the effectiveness of the algorithms and models, as well as other software functionalities.


Phase 1: Shadow Deployment

Although our model is deployed, its predictions are not used. Instead, human/staff outputs are saved for an initial round of real-world data collection.

Phase 2: Partial Automation

The model provides predictions to staff/users, allowing them to make changes to those predicted outputs. Both the predictions and final information approved by staff/users are collected for further retraining and model evaluation.


Phase 3: Full Automation

If we achieve the agreed-upon accuracy percentage with our client on real-world data, we proceed to full automation. Users/staff are no longer required to edit output, and the system relies solely on predictions generated by our models.


Phase 4: Maintenance

Data is saved locally or in the cloud, depending on the project infrastructure. Real-time monitoring of the model’s performance (accuracy, inference time) and the computing environment (disk space, memory usage, etc.) is implemented. Data is downloaded at least twice a year for model fine-tuning as part of the annual maintenance service, ensuring the model’s performance remains as expected.


Conclusion

By implementing our phase-based approach, we’ve not only improved our ability to deliver reliable AI solutions but also enhanced our client relationships through transparent, step-by-step progress. This methodology allows for continuous improvement and adaptation to changing real-world conditions, ensuring that our AI systems remain effective and valuable over time.

As we move forward, we remain committed to refining our processes, staying informed about the latest developments in AI deployment strategies, and sharing our knowledge with the wider tech community. By doing so, we hope to contribute to the advancement of AI applications that can truly meet the challenges of real-world deployment and deliver lasting value to users.




Our Molekiu STEM Ohbot Course, provide the BEST interaction with robots, hands on learning and coding experience to secondary students. LEARN MORE



**Ohbot is our stem education course robot used!

7 views0 comments

Comments


bottom of page