Data Storage
Structured data refers to information that is organized in a format that is easily understandable by both humans and machines. In structured data, the elements or fields are clearly defined, and there is a well-defined schema or model that governs the relationships and properties of these elements. This organization allows for efficient storage, retrieval, and analysis of data.
Structured data is typically found in relational databases, spreadsheets, and other tabular formats. Each piece of data is assigned to a specific category or field, and relationships between different pieces of data are explicitly defined. The use of schemas ensures that the data adheres to a specific structure, which simplifies operations like querying, filtering, and aggregating data.
Examples of structured data include tables with rows and columns, where:
Each column represents a specific attribute or property (e.g., "Name" "Age", "Salary").
Each row corresponds to a unique record or entry (e.g., an employee's details in a company's database).
Structured data is commonly used in a variety of applications, including business databases, financial systems, and information management systems, where organization and consistency are critical. This format allows for easy reporting, automation, and analysis through tools like SQL, ensuring data integrity and seamless interaction between systems.
Unstructured data refers to information that does not have a predefined schema or structure, making it more difficult to organize, search, and analyze compared to structured data. Common examples include text files, PDFs, images, videos, audio files, and other media types. Unlike structured data, unstructured data doesn’t fit neatly into a traditional row-and-column database model.
Although unstructured data can technically be stored in relational databases as Binary Large Objects (BLOBs), it is generally more suitable for file systems or object storage systems, especially due to its large size and unique requirements for backup and compliance.
However, metadata and vector embeddings associated with unstructured data still need to be stored in databases to make this data discoverable and usable.
Metadata typically includes information such as file name, URI, size, type, owner, and creation date. It may also contain deeper details like extracted text, object boundaries, and other context-relevant data.
This metadata can be stored in either a structured or semi-structured format, such as a JSON column or a combination of both.
To further enhance the usability of unstructured data, machine learning models can be employed to generate metadata or vector embeddings. These embeddings are useful for searching, analyzing, and building real-time AI applications.
Semi-structured data exists in the gray area between structured and unstructured data. It has some organizational structure but leaves room for flexibility and undefined elements. Common formats for semi-structured data include XML, JSON, Avro, and Parquet. Data from sources like sensors and server logs can easily fall into this category, as it often appears in or can be converted to formats like JSON or CSV.
Some data vendors even classify HTML code and emails as semi-structured. For example, an email can be represented as a JSON object with fields like sender, recipient, subject, and timestamp. However, if the email includes attachments such as media files or PDFs, it may also be considered unstructured data.
Last updated