CUR: Intelligent Data Curation Through Natural Language

In modern dataspaces, data from disparate sources often remains siloed, requiring manual curation, format conversion, and complex programming to extract meaningful insights. Organizations need a practical way to query federated data sources without understanding database schemas, SQL syntax, or data format differences. This is where the DS2 CUR module comes in.

What is CUR?

CUR (Data Curation) is a DS2 module that enables automatic curation and federation of data through machine learning. It allows users to query data through natural language, without needing to understand how to connect or convert data obtained from different sources.

At its core, CUR harnesses the power of relational databases and Large Language Models (LLMs) to transform natural language questions into executable SQL queries. The module stores data from various sources in a relational database, then uses machine learning combined with knowledge of database schemas to automatically determine how to match columns across different tables—even when column names or data formats differ.

CUR enables participants to:

• Query federated data sources using natural language

• Automatically match and join data from disparate sources

• Support queries in multiple languages (Greek, Slovenian, English, and more)

• Access OpenStreetMap data through natural language queries

• Analyze time series data with temporal context

• Learn from corrections through human-in-the-loop intervention

• Leverage RAG technology for improved query accuracy

• Access data without programming or SQL knowledge

Why CUR?

Dataspaces require more than just data connectivity. They need intelligent systems that can understand user intent, navigate complex data relationships, and deliver accurate results without requiring technical expertise from end users.

Without such capabilities, extracting insights from federated data sources becomes labor-intensive and costly. Users must understand database schemas, write complex SQL queries, handle data format conversions, and manually curate datasets—tasks that are both time-consuming and require specialized skills.

CUR addresses this challenge by providing an intelligent layer between users and data. It uses several advanced techniques to increase query success rates: evaluating generated queries before execution, using RAG technology to seed SQL generation with examples from previous successful queries, and supporting human-in-the-loop corrections that allow the system to learn and improve over time.

By providing this intelligent curation layer, CUR helps participants move from isolated, hard-to-query data systems toward accessible, user-friendly dataspace environments where anyone can extract insights through simple questions.

CUR Architecture Overview

At the heart of CUR is a sophisticated pipeline that transforms natural language into actionable database queries. The module consists of two main components: a configuration system for setting up database tables from data sources, and a query processing pipeline that handles natural language input.

The CUR architecture includes several key stages:

1. Natural Language Processing Pipeline, which converts user questions into structured queries through multiple stages:

   • GUI for user input and interaction

   • Entity extraction and translation for cross-language support

   • SQL query generation using LLM reasoning

   • Query execution and result validation

   • Human-in-the-loop correction when needed

   • Result reformulation for readable answers

2. Translation and Curation Services, enabling cross-language queries through:

   • Multilingual semantic embeddings using BGE-M3

   • Translation cache with vector similarity search

   • Automatic value mapping across different languages

   • Support for Greek, Slovenian, Romanian, and other languages

3. OpenStreetMap Integration, providing access to geographic data through:

   • Natural language to Overpass QL query conversion

   • Dual-mode query handling (SQL + Overpass)

   • Context fusion engine for merging database and OSM results

   • Spatial matching algorithms for entity correlation

   • Interactive map visualization with source attribution

   • RAG-based example storage for common OSM queries

4. Time Series Support, enabling temporal analysis through:

   • PostgreSQL time series data storage

   • Temporal query understanding and generation

   • Support for time-based filtering and aggregation

   • Historical data analysis capabilities

   • Integration with weather stations and sensor data

   • NDVI and agricultural monitoring over time

5. RAG-Enhanced Query Generation, improving accuracy through:

   • Vector store of successful query examples

   • Hybrid retrieval combining semantic and keyword search

   • Context-aware prompt construction

   • Confidence scoring and validation

6. Database Federation, allowing seamless access to multiple data sources:

   • PostgreSQL with PostGIS for geospatial queries

   • Automatic schema discovery and understanding

   • Dynamic data source configuration

   • Support for REST API data ingestion

CUR also integrates with other DS2 capabilities. It can be deployed on IDT infrastructure, uses standard DS2 connectors for data access, and supports the DS2 federation model for multi-dataspace queries. The module's web-based interface makes it accessible from any browser, while its containerized architecture enables flexible deployment across cloud, on-premise, and edge environments.

Key Features

CUR provides several distinctive capabilities that set it apart:

• Natural Language Query Interface - Users can ask questions in plain language without knowing SQL or database structure. For example, "Find apple fields in Imathia with NDVI > 0.7" is automatically converted to the appropriate SQL query with correct Greek values.

• Multi-Language Support - The translation system supports queries in multiple languages, automatically mapping English terms like "apple" to Greek "Μήλο" or Slovenian equivalents, enabling truly international dataspace participation.

• OpenStreetMap Integration - Users can query OpenStreetMap data using natural language, such as "get all restaurants" or "find schools near me." The system converts these queries to Overpass QL, executes them against the Overpass API, and displays results on an interactive map with distinct styling and source attribution.

• Time Series Analysis - CUR supports temporal queries and analysis, enabling users to track changes over time. Questions like "Show me fields where NDVI decreased during the past two months" are automatically converted to time-based SQL queries, allowing for historical trend analysis and monitoring.

• Automatic Data Federation - CUR automatically determines how to join data from different sources, matching columns even when names differ, and handling format conversions transparently. The context fusion engine can merge results from SQL databases and OpenStreetMap, enriching data with complementary information.

• Self-Learning Capability - Through human-in-the-loop intervention, domain experts can correct failed queries. These corrections are stored and used to improve future query generation, allowing the system to continuously learn.

• RAG-Powered Accuracy - The system maintains vector stores of successful queries for both SQL and Overpass operations, using retrieval-augmented generation to provide relevant examples to the LLM, significantly improving query success rates.

• Geospatial Query Support - Built on PostgreSQL with PostGIS, CUR handles complex spatial queries, enabling questions about geographic relationships, proximity, and spatial analysis. Integration with OpenStreetMap provides additional geographic context and point-of-interest data.

By combining these capabilities, CUR transforms the challenge of querying federated, multilingual, temporal, and geospatial data sources into a simple, conversational experience. Users can focus on their questions rather than the technical complexity of data access, while the system handles translation, federation, temporal analysis, and query optimization automatically.

CUR is part of the DS2 project, developed by IBM Research to enable intelligent, accessible dataspaces.