Skip to main content
Version: Next

JSON Schemas

Incubating

Important Capabilities​

CapabilityStatusNotes
Descriptions✅Extracts descriptions at top level and field level
Detect Deleted Entities✅With stateful ingestion enabled, will remove entities from DataHub if they are no longer present in the source
Extract Ownership❌Does not currently support extracting ownership
Extract Tags❌Does not currently support extracting tags
Platform Instance✅Supports platform instance via config
Schema Metadata✅Extracts schemas, following references

This source extracts metadata from a single JSON Schema or multiple JSON Schemas rooted at a particular path. It performs reference resolution based on the $ref keyword.

Metadata mapping:

  • Schemas are mapped to Datasets with sub-type Schema
  • The name of the Schema (Dataset) is inferred from the $id property and if that is missing, the file name.
  • Browse paths are minted based on the path

CLI based Ingestion​

Install the Plugin​

pip install 'acryl-datahub[json-schema]'

Starter Recipe​

Check out the following recipe to get started with ingestion! See below for full configuration options.

For general pointers on writing and running a recipe, see our main recipe guide.

pipeline_name: json_schema_ingestion
source:
  type: json-schema
  config:
    path: <path_to_json_file_or_directory or url> # e.g. https://json.schemastore.org/petstore-v1.0.json
    platform: <choose a platform that you want schemas to live under> # e.g. schemaregistry
    # platform_instance: <add a platform_instance if there are multiple schema repositories>
    stateful_ingestion:
      enabled: true # recommended to have this turned on

# sink configs if needed

Config Details​

Note that a . is used to denote nested fields in the YAML recipe.

FieldDescription
path ✅
One of string(file-path), string(directory-path), string(uri)
Set this to a single file-path or a directory-path (for recursive traversal) or a remote url. e.g. https://json.schemastore.org/petstore-v1.0.json
platform ✅
string
Set this to a platform that you want all schemas to live under. e.g. schemaregistry / schemarepo etc.
platform_instance
string
The instance of the platform that all assets produced by this recipe belong to
use_id_as_base_uri
boolean
When enabled, uses the $id field in the json schema as the base uri for following references.
Default: False
env
string
The environment that all assets produced by this connector belong to
Default: PROD
uri_replace_pattern
URIReplacePattern
Use this if URI-s need to be modified during reference resolution. Simple string match - replace capabilities are supported.
uri_replace_pattern.match ❓
string
Pattern to match on uri-s as part of reference resolution. See replace field
uri_replace_pattern.replace ❓
string
Pattern to replace with as part of reference resolution. See match field
stateful_ingestion
StatefulStaleMetadataRemovalConfig
Base specialized config for Stateful Ingestion with stale metadata removal capability.
stateful_ingestion.enabled
boolean
Whether or not to enable stateful ingest. Default: True if datahub-rest sink is used or if a datahub_api is specified, otherwise False
Default: False
stateful_ingestion.remove_stale_metadata
boolean
Soft-deletes the entities present in the last successful run but missing in the current run with stateful_ingestion enabled.
Default: True

Configuration Notes​

  • You must provide a platform field. Most organizations have custom project names for their schema repositories, so you can pick whatever name makes sense. For example, you might want to call your schema platform schemaregistry. After picking a custom platform, you can use the put platform command to register your custom platform into DataHub.

Code Coordinates​

  • Class Name: datahub.ingestion.source.schema.json_schema.JsonSchemaSource
  • Browse on GitHub

Questions

If you've got any questions on configuring ingestion for JSON Schemas, feel free to ping us on our Slack.