Catalog Documentation

The Catalog directory serves as the core component for interacting with the Catalog API. It supports querying, updating, and managing both datasets and data sources. This directory provides the structure for different levels of data exploration, filtering, and management.

Page: Catalog

Description: This page provides an overview of the available data sources and datasets, with basic filtering and pagination.

ResultsCard: Displays individual catalog items.

CenteredCard: Provides a "View All" option to navigate to dedicated pages for data sources or datasets.

Page: Datasets

Description: A dedicated page for exploring datasets, offering advanced filtering, sorting, and pagination options.

Components:

  • DatasetsTable: Displays dataset details in a tabular format.

  • MultiDatasourceInfo: Shows detailed information about datasets.

  • DatasetContextFilter: Provides filtering options for datasets.

  • Pagination: Manages pagination for datasets.

Page: Datasource

Description: Dedicated to managing data sources, including sorting, filtering, and adding new sources.

Components:

  • DataSourceInfo: Displays data source details.

  • DataSourceTable: Renders data sources in a grid or list view.

  • AddDataSourceModal: Allows users to add new data sources.

  • DatasourceContextFilter: Provides filtering options for data sources.

Catalog Dynamic Directories

These dynamic route pages facilitate advanced interactions with the catalog, enabling users to search, filter, and view details about data sources and datasets. They use Next.js’s dynamic routing capabilities to handle user-specific queries and parameters.

  • Dynamic Routing: Uses Next.js’s dynamic routing to handle [Query], [datasource], and [dataset] paths.

  • State Management: Uses React’s useState, useReducer, and context (UserContext) for managing data loading, filtering, and user-specific group access.

  • Pagination & Filtering: Pagination and filtering are handled on dataset and data source pages, allowing users to customize the display and sort results based on their preferences.

Pages: Catalog [datasource] and Catalog [dataset]

Description: These pages provide detailed information about individual data sources or datasets, including an overview and related datasets. The pages are dynamic, adapting to the specific catalog entry that the user selects.

  • SourceHeader: Displays header information about the data source or dataset.

  • Tabs: Allows navigation between "Overview" and "Datasets" sections.

  • AccordionSection: Provides expandable details about specific sections, such as stewardship or requirements.

  • EnablementsSelection: Dropdown for selecting enablements related to the data source or dataset.

  • DatasetInfo: Renders detailed information about datasets.

Page: Dynamic Dataset Pages

These pages allow users to interact with specific datasets within a given data source, providing detailed information, management options, and enablement features. The pages are organized into four sub-sections: Overview, Bucket, Topics, and Enablements.

  • Dynamic Routing: Uses Next.js dynamic routing to handle routes like [datasource]/dataset/[dataset].

  • State Management: Uses useReducer for managing page-specific state, with initial states and reducers defined for each sub-page.

  • Pagination & Filtering: Provides pagination for topics and enablements, allowing users to control rows per page.

  • Enablement Toggling: Allows users to enable or disable dataset-related enablements, with real-time updates and success/error notifications.

Page: [Dataset] Overview

Description: Provides a detailed overview of the selected dataset, including general information, enablement details, and storage options.

  • SourceHeader: Displays the dataset’s header information.

  • PageNavigation: Navigation tabs for switching between Overview, Bucket, Topics, and Enablements.

  • DatasetBanner: Displays additional information about the dataset.

  • AccordionSection: Contains expandable sections for more information about enablements.

  • SetStorage: Manages the dataset’s storage settings.

Page: [Dataset] Bucket

Description: Displays the storage details for the dataset, specifically S3 buckets and Delta storage options.

  • SourceHeader: Displays dataset header information.

  • PageNavigation: Navigation tabs for switching between sections.

  • DatasetBanner: Displays additional information about the dataset.

  • SetCloudStorage: Manages cloud storage settings for the dataset.

Page: [Dataset] Topics

Description: Displays and manages Kafka topics related to the dataset.

  • SourceHeader: Displays dataset header information.

  • PageNavigation: Navigation tabs for switching between sections.

  • DatasetBanner: Displays additional information about the dataset.

  • SetTopics: Manages the topics related to the dataset.

Page: [Dataset] Enablements

Description: Provides a comprehensive view of all enablements for the selected dataset, including management options to toggle enablement statuses. Users can enable or disable specific enablements and receive feedback on success or failure.

  • SourceHeader: Displays dataset header information.

  • PageNavigation: Navigation tabs for switching between sections.

  • DatasetBanner: Displays additional information about the dataset.

  • SetEnablements: Manages enablement statuses for the dataset.