Context and objectives

An international public organization lacked a centralized way to monitor and process publications from its partner institutions. Without a structured system, teams struggled to index, search, and extract relevant information from a large and growing body of reports and policy documents.

The organization needed to scrape partner websites at multiple levels (international, regional, and local) and classify the collected documents according to dozens of policy-relevant topics.

Reliable document processing methods and natural language processing (NLP) algorithms were essential to allocate research resources effectively and support more informed policymaking.

Approach

1. From proof of concept to prototype

After delivering a serverless, cost-efficient proof of concept, the client asked Agilytic to develop a full prototype. The team ran agile implementation rounds directly in the client's environment, ensuring the solution met real-world needs at every stage.

2. Building the document processing pipeline

Each document entering the pipeline required the extraction of:

• A summary

• A title

• Keywords

• A classification across dozens of interest areas

There was no common structure or format across the documents, and they arrived in any of the languages spoken in the European Union. Key extracts were translated into English to ensure efficient and consistent processing.

3. Infrastructure and deployment

The solution was deployed in the client's AWS environment, with cloud infrastructure designed and managed using Terraform for ease of maintenance and scalability.

The prototype was coded in Python, using Docker for containerization and SQL databases for storage. An API was implemented to manage the organizations to be scraped and to feed specific documents into the pipeline.

Results

The main deliverables were:

• Code to deploy infrastructure and perform scraping, extraction, and NLP analysis

• Documentation on the deployment of the infrastructure as code (IaC) solution within the client's environment

• Knowledge-sharing sessions with the client's team for complete ownership of the pipeline, including the capacity to extend it with new features

With efficient automation, this document processing pipeline improved the organization's ability to find and use publications that directly support policymaking and decisions.

They cited the solution as bringing quality and speed, flexibility, security, and cost-effectiveness to their search and decision processes.