Datasets:

microsoft

/

kitab

like 9

Overview

🕮 KITAB is a challenging dataset and a dynamic data collection approach for testing abilities of Large Language Models (LLMs) in answering information retrieval queries with constraint filters. A filtering query with constraints can be of the form "List all books written by Toni Morrison that were published between 1970-1980". The dataset was originally contributed by the paper "KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval" Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Ghosh Peshawaria, Ranjita Naik, and Besmira Nushi. 2023. The dataset is named after the word kitab, which is the word for "book" in Arabic, Swahili, Urdu, Hindi and various Indian and Turkic languages.

KITAB consists of book-related data across more than 600 authors and 13,000 queries with varying number of constraints and complexity. In each query in the dataset, the first constraint is always fixed to an author and the following can vary among the following types of book constraints to test for different constraint satisfaction capabilities:

• lexical (title starts or ends with a letter, word count in title)
• temporal (published between start and end year)
• named entity (city or human name present or not present in title)

What is available in this repository?

This repository contains the following artifacts:

• All data for the KITAB sample used in the original paper. This consists of the set of authors, their corresponding books, and the set of queries with constraints.
• Example code for generating a new sample with a different set of authors. Here the sampling and data collection steps do not include the generation of queries as these may change according to the evaluation usage needs for the data. The example code also shows how to evaluate a potential model output with a list of books against the provided ground truth in KITAB, by following the same evaluation process as in the original paper. Note that this evaluation tends to relax some of the constraint satisfaction requirements in particular when the model may come up with only a partial title.
• All prompts that were used in the original paper to evaluate GPT-4 and GPT-3.5.

Data

• KITAB-ONE-BOOK-CONSTRAINTS.json and KITAB-TWO-BOOK-CONSTRAINTS.json - correspond to queries with one and two book constraints. Each file has all the sufficient information that can be used to recreate a prompt query including the author, their birth year, number of sitelinks on WikiData, the constraint type(s), the constraint(s) expressed in natural language, the list of all books by the author, and the mapped list of books by the author that satisfy the constraint(s).

KITAB-ONE-BOOK-CONSTRAINTS_features = {
    "Author": "author name",
    "Birth Year": "author birth year",
    "# of sitelinks": "number of external links related to the author",
    "constraint_id": "unique id for the constraint",
    "constraint_type": "type of the constraint",
    "constraints": "the constraint",
    "mapped_books": "list of books by the author mapped to the constraint",
    "all_books": "full list of books by author post cleaning from openlibrary",
    "raw_books": "raw list of books by author from openlibrary",
}

• KITAB-author-metadata.json - contains the set of 611 authors along with their birth year, the number of sitelinks in Wikidata, and their corresponding Open Library and WikiData identifiers.
• KITAB-book-metadata.tar.gz - contains a json file per author with all books retrieved from OpenLibrary for that author. The files contain the following information per title: the Open Library Id for the book, the Wikidata ID (if it exists), list of languages in which it was published, number of editions, number of words in the title, the earliest publishing year, city names found in the title (if any), a modified version of the title in lowercase that stripes stop words like "A" and "The" from the title, a set of of other redundant versions of the same title as found in Open Library (if any).

Code and evaluation scripts

Example notebooks included in this repository:

• collect_authors_from_wikidata.py and wikidata_open_library_author_profiling.ipynb - example code for generating a new author sample from WikiData and OpenLibrary. Here, we also make available the longer list of authors that was originally sampled from WikiData to facilitate the sampling process although future work may also choose to repeat this step as needed. The full list can be found in: wikidata_authors_crawl.csv.
• fetch_book_data.py - example code for collecting book data for the set of authors sampled in the previous steps. Pulls data from OpenLibrary and WikiData to curate and clean the sample.
• evaluation.ipynb - example code for evaluating model outputs from our prompts against ground truth KITAB data. Here, we also make available the GPT-4 output on human name detection, although as models improve future work may also choose to repeat this step as needed. Results can be found in: gpt_4_name_data_processed.csv.

Prompts

We use the following prompt templates for different experimental conditions on the KITAB data:

ALL-BOOKS (Template 1): List all books from the author. This condition enables us to estimate an upper bound of model performance in retrieving relevant information for all queries, regardless of other constraints.

NO-CONTEXT (Template 2a): List all books from the author that also satisfy other book constraints.

WITH-CONTEXT (Template 2b): First, provide a full list of books from the author as input context to the model. Then, ask the model to list all books from the author that also satisfy other book constraints.

SELF-CONTEXT (Template 3): Ask the model to first self-retrieve all books from the author, and then use that list to find those that also satisfy book constraints.

NAME-CHECK (Template 4): Ask the model to find all book in a given list that contain a human name.

Data Collection and Statistics

The author list was initially randomly sampled from WikiData and then filtered down to 611 authors to avoid potentially inaccurate data and extreme outliers. For example, this involved removing authors that have very few or too many books and authors that were born before 1850. The collected book data was derived from Open Library and contains all books from the author that are tagged to be in English by Open Library or detected to be in English by the Language Detection service from the Azure Cognitive Services API. More details about author sampling and book data collection and cleaning are present in the paper.

Since there exists a large number of constraint instances depending on their cardinality, we subsample from the potential large set of queries in a way that ensures a balanced representation across constraint types, and a variety of constraints that have different constrainedness (i.e., defined as the complement of the ratio between the number of books that satisfy the constraints with the total number of all books from the author). The dataset also contains “unsatisfiable” constraints, which do not match any book titles in our data. This constitutes 7.99% of the queries with only one book constraint. The final dataset contains 8239 single-constraint queries and 4750 double-constraint queries. The table below shows how these queries are distributed across different constraint types. For all double-constraint queries, both constraints are individually satisfiable and generated by combining our single constraint data. Only 0.76% of the queries are jointly unsatisfiable across both constraints.

Responsible AI Considerations

Data Cleaning: Despite our best efforts in collecting a complete and accurate set of books, we also faced a variety of challenges in retrieval and cleaning, which we further describe in Appendix C.1 in the paper. To estimate the extent of which potential data cleaning issues may impact the data quality of KITAB and further evaluation, we also undertook a manual data annotation exercise during which we searched on the web for titles provided by GPT4 and GPT3.5 but that were marked as not from the author in our dataset. In summary, we find that based on a manual annotation of a subsample of queries, less than 5% of the queries to GPT4 and less than 6% of the queries to GPT3.5 may potentially be affected by cases where the model finds a book title that is not in KITAB and that will consequentially be marked as not from the author during our evaluation. While this can be remediated by using further data sources, the impact of missing information on model comparison is minor.

Human Names: Entity recognition for human names was done using both Azure Cognitive Services API and GPT4 (Template 4 in Appendix D in the paper), as we found the two approaches to be complementary for detecting names from different cultures. Note that even after using both these resources, there may still be names that are not recognized by either of these APIs, which is a testimony that more work is required in improving the quality of service of entity recognition for fairness across different languages and cultures.

City Names: For city names, we use Azure Cognitive Services API along with Geonames, a database of cities with more than 1000 inhabitants.

Author representation: The list of authors in KITAB was sampled randomly from a large set of authors present in Open Library. We see that the rate of irrelevant information generated by current models increases with a lower number of sitelinks in Wikidata. Since the number of sitelinks may also correlate with the age (birth year) of the author or even their nationality and how well their community is linked to the World Wide Web, this observation has important implications on model quality of service across different geographical regions and author popularity and age. While KITAB naturally does contain more authors with a lower number of sitelinks (as indicated by its long-tail distribution of author count vs. their popularity), future fairness measurement investigations in this regard may also need to oversample explicitly from cohorts belonging to given demographic and geographical attributes.

State-of-the-art results on KITAB

How to cite

@inproceedings{abdin2023kitab,
  title={KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval},
  author={Abdin, Marah I and Gunasekar, Suriya and Chandrasekaran, Varun and Li, Jerry and Yuksekgonul, Mert and Peshawaria, Rahee Ghosh and Naik, Ranjita and Nushi, Besmira},
  journal={arXiv preprint arXiv:2310.15511},
  year={2023}
}

Contributors

Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Ghosh Peshawaria, Ranjita Naik, Besmira Nushi