Superset

A comprehensive guide to understanding and utilizing Superset’s capabilities within Raft’s Data Fabric and the SOF Data Layer.

Data Stewards, Data Analysts

The Superset documentation for a data platform provides users with detailed guidance on using Apache Superset for data exploration and visualization. It explains how to connect to various data sources, create and customize interactive dashboards, and visualize data through a wide range of chart types. The documentation covers essential functionalities like building queries, filtering data, and setting up permissions for different user roles. Additionally, it offers best practices for optimizing performance, configuring Superset’s architecture, and integrating it with other platform components. This ensures users can effectively leverage Superset to analyze and present data insights in real-time.

Superset enables a system to provide powerful data exploration and visualization capabilities, allowing users to analyze and interact with data through customizable dashboards and charts. It connects seamlessly to various databases, enabling real-time querying and data analysis without the need for code. Users can build complex SQL queries using an intuitive drag-and-drop interface or write custom queries for more advanced analytics.

The system’s integration with Superset facilitates collaboration by allowing users to share dashboards, insights, and reports within the platform. It supports a wide range of visualizations, from bar charts and pie charts to geospatial maps and time-series graphs, empowering users to make data-driven decisions. Additionally, Superset’s role-based access controls and data caching mechanisms ensure secure, efficient access to data, enhancing system performance and governance. This capability makes the system a robust platform for exploring, visualizing, and communicating data insights.

df-delta is the main data store where all data is persisted after being ingested through Kafka. Delta is suitable for structured, analytics-friendly data and supports real-time analysis. It is the primary destination for most datasets in Data Fabric. Data remains here after the Kafka time-to-live (TTL) limit is reached.

Kafka is used as a message broker to stream data in real-time before it is persisted in Delta. Data in Kafka is transient, with a TTL of 1 hour, meaning it gets dropped after 1 hour but is retained permanently in Delta. Kafka is optimal for real-time event streaming, though it is not used for long-term storage.

Pinot is a real-time analytics engine and OLAP store optimized for low-latency queries. It automatically pulls real-time data from Kafka streams for immediate complex analysis, making it ideal for dashboards or maps requiring real-time views, like monitoring events or detecting anomalies. While Pinot excels in fast, time-critical queries, large-scale data storage is handled by Delta Lake. Pinot is best used for scenarios requiring quick, real-time analysis.

PostgreSQL is used mainly for manual data uploads, allowing users to store relational data in a structured, SQL-compatible database. This source is suited for users who prefer uploading CSV files or working with custom schemas.

PostGIS is used for storing and querying spatial data. It supports case-by-case geospatial analyses and can be used when working with maps and spatial datasets.

ArcadeDB is a multi-model database that can handle graphs, documents, and other non-relational data models. It is used in specific cases where these data models are required, though it is not the default destination for most datasets.

There are two ways to navigate to Superset within Data Fabric: - From the Data Fabric Home page, select the Superset card at the bottom left. image::/assets/images/home.png[alt="Data Fabric Home"] - You can also access Superset via the Data Insights tab directly, selecting Visualizations. image::/assets/images/nav_data_insights.png[alt="Data Fabric Insights Tab"]

Log in to Superset using your Keycloak credentials, which manages access to the application and ensures that your account has appropriate access levels.

To add or verify data sources: - Navigate to the Datasets tab. - Click on + Dataset to start the process. - Select a Data Fabric-managed database, then choose the appropriate schema and table.

Superset allows users to combine data from different databases (e.g., df-postgres and df-kafka) in dashboards and queries for cross-analysis.

Analysts can upload local datasets (e.g., CSVs) to databases like df-postgres, then connect them to Superset for querying and visualization.

After saving the dataset, choose a chart type that fits your data, such as a Bar Chart, Line Chart, or Pie Chart.

Discrete data represents distinct, separate values, while continuous data can take any value within a range. Choose your chart type accordingly.

Select columns to display under Dimensions, and apply filters or aggregations under Metrics.

Select numeric columns for the X and Y axes, and customize the points on the scatterplot by size, color, and filters.

Select latitude and longitude columns for geospatial data, and customize the map with WMS endpoints and layers.

Use time filters, aggregate data at query level, and set row limits to improve performance.

Limit visualizations per dashboard, and select appropriate chart types for your data.

Verify datasets in SQL Lab before building dashboards, and monitor data lifecycle with Kafka TTL.

Verify dataset connectivity and check SQL queries for errors.

Reduce data volume by applying filters or setting row limits.

Check for data ingestion issues or schema changes.

Check SQL syntax and correct table/schema selection.

Verify correct latitude/longitude columns and WMS endpoint configurations.

Choose appropriate chart types and reduce the number of metrics.

Clear the cache and verify data ingestion.