Transforming Unstructured Data into Structured Using AI

Written by Martyna Slawinska, Technical Product Manager @ MindsDB

In this digital age, the vast majority of data generated is unstructured, ranging from emails and social media posts to audio and video content. This presents a significant challenge for organizations seeking to harness this data for insights and decision-making. Leveraging artificial intelligence (AI) to structure unstructured data is crucial, as AI can process and analyze large volumes of data quickly and accurately.

Traditional methods for structuring unstructured data involve manual processes or rule-based systems, which can be time-consuming, error-prone, and difficult to scale. While these methods can provide some level of organization, they lack the efficiency and adaptability of AI-powered solutions. AI, or specifically large language models (LLMs), simplifies the process of extracting relevant information from unstructured data. This accelerates data processing and enhances accuracy, enabling real-time data analysis.

One notable solution in this space is provided by MindsDB, an open source AI platform that brings structured and unstructured data together for enterprise AI. The solution utilizes MindsDB's integration with Large Language Models to extract relevant data from unstructured text and insert it into a database. This way you can convert large amounts of unstructured data into structured formats using only a few SQL commands. Read along to follow the tutorial.

Tutorial

Start by setting up MindsDB locally via .

Connect your data source that contains unstructured data to MindsDB.

Firstly, we create an OpenAI engine providing the OpenAI API key.

CREATE ML_ENGINE openai_engine

FROM openai

USING

    openai_api_key = 'your-openai-api-key';

Secondly, we create an AI model to extract relevant data values from descriptions.

CREATE MODEL extract_data_model

PREDICT json

USING

    engine = 'openai_engine',

    json_struct = {

        'nobath': 'number of bathrooms',

        'nobed': 'number of bedrooms',

        'sqft': 'square footage of the property (only integer)',

        'address': 'address includes house number and street name',

        'city': 'city',

        'zipcode': 'zipcode includes two letters (state) and five digits',

        'country': 'country'

    },

    prompt_template = '{{description}}';

The json_struct parameter stores data fields and their descriptions; these data fields will be extracted from unstructured data. MindsDB has custom-implemented it to ease the process of extracting desired data from unstructured volumes of data.

The prompt_template parameter stores the column name that contains property descriptions in double curly braces, which will be replaced by property descriptions from the input data upon joining the model with the input data.

SELECT d.description, m.json

FROM data_source.property AS d

JOIN extract_data_model AS m;

Here is the output:

Conclusion

Transforming unstructured data into valuable insights doesn’t have to be complex. With MindsDB, you can quickly extract desired details from text using just a few simple SQL commands – making the process fast and straightforward. It’s also highly adaptable: customize the json_struct settings to align perfectly with your data requirements.

If you have any questions, join our Community Slack — we’re here to help!