Data Source API Reference Guide
This page describes the dataSource
endpoint, through which users can subscribe to data sources,
make unmasking and query debug requests, and manage data source tasks. To create data sources, see the specific handler
endpoints.
Note
Additional fields may be included in some responses you receive; however, these attributes are for internal purposes and are therefore undocumented.
Data source workflow
- Search data sources and data source details.
- Access data sources and make data source requests.
- Manage data source requests.
- Update data sources.
- Delete a data source.
Search data sources and data source details
Method | Path | Purpose |
---|---|---|
GET | /dataSource |
Search for data sources. |
GET | /dataSource/{dataSourceId} |
Get a data source based on the ID. |
GET | /dataSource/name/{dataSourceName} |
Get data source based on the name. |
GET | /dataSource/sqlTableName/{shortName} |
Get data source based on the short name. |
GET | /dataSource/{dataSourceId}/lineage/{type} |
Get parent and child relationship records for derived data sources using a specified data source ID. |
GET | /dataSource/{dataSourceId}/blob/{blobId*} |
Retrieve a blob. |
GET | /dataSource/{dataSourceId}/access |
Get all users with the provided access level for this data source. |
GET | /dataSource/{dataSourceId}/users/{profileId}/policyInfo |
Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source. |
GET | /dataSource/{dataSourceId}/users/{profileId}/visibilityReport |
Retrieves a summary of total records, total visibilities, and visibilities a given user has access to. |
GET | /dataSource/{dataSourceId}/visibilityReport |
Retrieves a summary of total records, total visibilities, and visibilities the current user has access to for a specified data source. |
Search for data sources
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource |
Search for data sources. |
Query Parameters
Attribute | Description | Required |
---|---|---|
blobHandlerType | array[string] Describes the type of underlying blob handler that will be used with this data source (e.g., Custom, MS SQL). |
No |
subscription | array[string] The requesting user's subscription status: pending , owner , subscribed , not_subscribed , expert , or ingest . |
No |
status | array[string] The data source status: passed or failed . |
No |
tag | array[string] Filters data sources by tags associated with the data sources. |
No |
searchText | string Searches for data source names using the provided string . |
No |
column | array[string] Searches for data source column names. |
No |
connectionString | array[string] Searches by connection string. |
No |
schema | string Searches for data source schema. |
No |
nameOnly | boolean When true , searchText will only search data source names. Default is false . |
No |
idOnly | boolean When true , only returns the ID Of the data source and the user's subscription status. |
No |
dataSourceIds | array[integer] Searches for the provided data source IDs. |
No |
selectFields | array[string] This field accepts the values id , name , and columnEvolutionEnabled . When id or name are provided, the request will return only the ID or name of the data source and the subscription status. If columnEvolutionEnabled is provided, the response will also include information about the policies, policy conflicts, and workspaces associated with the data sources. |
No |
offset | integer Used in combination with size to fetch pages. Default is 0 . |
No |
size | integer The number of results to return per page. Default is 10 . |
No |
sortField | string Used to sort results by field, which must be createdAt , name , blobHandlerType , subscriptionStatus , recordCount , status , policy , or editable . |
No |
sortOrder | string Sorts results by order, which must be asc or desc . |
No |
excludedProjects | array[integer] Filter out any data sources that belong to the specified projects. |
No |
ephemeral | boolean When true , returns ephemeral data sources. |
No |
clusterName | string The name of the remote cluster the data source is connected to. |
No |
mode | integer Specifies the query mode, which must be 0 (FULL ), 1 (COUNT ), 4 (TAG ), 5 (MIN_MAX ), or 6 (STATUS ). |
No |
globalPolicy | string Filter by data sources that have this Global Policy applied. |
No |
hostname | string Searches data sources by hostname. |
No |
determinePolicyContflicts | boolean When true , filters results to return the data sources with policy conflicts. |
No |
collectionId | string Filter by data sources in the domain with this ID. |
No |
collectionExclude | string When true , filters results to return the data sources in the domain of the provided ID. |
No |
sddTemplateName | string Filter by data sources that have the specified SDD identification framework applied. |
No |
excludeSddTemplateName | string Filter by data sources that do not have the specified SDD identification framework applied. |
No |
Response Schema
Attribute | Description |
---|---|
name | string Data source name. |
id | integer Data source ID. |
deleted | boolean If true the data source is a deleted data source. |
description | string The data source description. |
createdAt | timestamp The date and time the data source was created. |
subscriptionPolicy | array Details the type of Subscription Policy applied to the data source. |
schemaEvolutionId | integer The schema evolution ID. |
recordCount | integer The record count. |
status | array[string] Accepted statuses are passed or failed . |
subscriptionStatus | array[string] Accepted statuses are subscribed or unsubscribed . |
blobHandlerType | array[string] Describes the type of underlying blob handler of this data source (e.g., Custom, MS SQL). |
subscriptionType | string The type of subscription policy on the project. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added). |
connectionString | string The connection string information. |
sqlSchemaName | string The schema name. |
policy | string When this value is none , there are no data policies applied to the data source. Otherwise, this field indicates whether or not there are policy conflicts among the data policies applied to the data source. |
policyHandlerType | string The policy handler type, such as None or Builder . |
Request example
The following request returns 2 data sources.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource?size=2
Response example
{
"name": "Public Barfoo",
"id": 22,
"recordFormat": "Not Provided",
"deleted": false,
"description": null,
"createdAt": "2021-07-22T14:11:55.539Z",
"subscriptionPolicy": {
"type": "subscription",
"automaticSubscription": true
},
"schemaEvolutionId": 1,
"recordCount": 0,
"status": "passed",
"subscriptionStatus": "subscribed",
"blobHandlerType": "Snowflake",
"subscriptionType": "automatic",
"connectionString": "test@immuta.connection.source.com:###/test",
"sqlSchemaName": "public",
"policy": "None",
"policyHandlerType": "None",
"native": null,
"workspace": null
},
{
"name": "Public Aaa Tpc",
"id": 39,
"recordFormat": "Not Provided",
"deleted": false,
"description": null,
"createdAt": "2023-08-21T10:39:00.250Z",
"subscriptionPolicy": {
"type": "subscription",
"exceptions": {
"operator": "and",
"conditions": [
{
"type": "groups",
"group": {
"name": "alpha"
}
}
]
},
"allowDiscovery": true,
"automaticSubscription": true
},
"schemaEvolutionId": 1,
"recordCount": 0,
"blobHandlerType": "Snowflake",
"subscriptionType": "policy",
"sqlSchemaName": "public",
"status": "passed",
"subscriptionStatus": "owner",
"connectionString": "test@immuta.connection.source.com:###/test",
"remoteTable": "tpc",
"remoteSchema": "public",
"domainId": null,
"domainName": null,
"policy": "None",
"policyHandlerType": "None",
"native": null,
"workspace": null
}
Get a data source by ID
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId} |
Get a data source based on the ID. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID . |
Yes |
Response Schema
Attribute | Description |
---|---|
name | string The data source name. |
recordFormat | string The data format of blobs in the data source, such as json , xml , html , or jpeg . |
description | string The description of the data source. |
policyHandler | array The ID of the policy handler and details about the data policies enforced on the data source. |
sqlSchemaName | string A string that represents this data source's schema name in the Query Engine. |
sqlTableName | string The SQL table name in the Query Engine. |
blobHandler | array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source. |
blobHandlerType | string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL ). |
createdBy | integer The ID of the profile creating the data source. |
deleted | boolean If true, the data source was deleted. |
type | string The data source type, such as queryable or ingested . |
rowCount | integer The number of rows. |
documentation | string Documentation associated with the data source. |
id | integer The data source ID. |
policyHandlerType | string The type of policy handler applied to the data source: Builder . |
subscriptionType | string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added). |
subscriptionPolicy | array Details about the Subscription Policy applied to the data source. |
globalPolicies | string Details about the Global Policies applied to the data source. |
status | string The data source health status. |
Request example
The following request gets a data source based on the ID 22
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/22
Response example
{
"name": "Public Barfoo",
"recordFormat": "Not Provided",
"description": null,
"policyHandler": null,
"sqlSchemaName": "public",
"sqlTableName": "barfoo",
"blobHandler": {
"url": "https://your-url/snowflake/handler/22",
"ca": {
"name": "Certificate Authority Bundle"
},
"manualDictionary": false
},
"createdBy": 2,
"deleted": false,
"type": "queryable",
"recordCount": 0,
"rowCount": 6,
"documentation": "# Public Barfoo",
"statsExpiration": "2021-08-27T16:34:47.846Z",
"id": 22,
"blobHandlerType": "Snowflake",
"policyHandlerType": "None",
"subscriptionType": "automatic",
"subscriptionPolicy": {
"type": "subscription",
"automaticSubscription": true
},
"globalPolicies": null,
"status": "passed",
"statusInfo": {
"sql": {
"status": "passed",
"message": "Passed"
}
}
}
Get data source by name
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/name/{dataSourceName} |
Get a data source based on the name. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceName | string The data source name. |
Yes |
Response Schema
Attribute | Description |
---|---|
name | string The data source name. |
recordFormat | string The data format of blobs in the data source, such as json , xml , html , or jpeg . |
description | string The description of the data source. |
policyHandler | array The ID of the policy handler and details about the data policies enforced on the data source. |
sqlSchemaName | string A string that represents this data source's schema name in the Query Engine. |
sqlTableName | string The SQL table name in the Query Engine. |
blobHandler | array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source. |
blobHandlerType | string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL ). |
createdBy | integer The ID of the profile creating the data source. |
deleted | boolean If true, the data source was deleted. |
type | string The data source type, such as queryable or ingested . |
rowCount | integer The number of rows. |
documentation | string Documentation associated with the data source. |
id | integer The data source ID. |
policyHandlerType | string The type of policy handler applied to the data source: Builder . |
subscriptionType | string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added). |
subscriptionPolicy | array Details about the Subscription Policy applied to the data source. |
globalPolicies | string Details about the Global Policies applied to the data source. |
status | string The data source health status. |
Request example
The following request gets a data source based on the name Public Barfoo
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/name/Public%20Barfoo
Response example
{
"name": "Public Barfoo",
"recordFormat": "Not Provided",
"description": null,
"policyHandler": null,
"sqlSchemaName": "public",
"sqlTableName": "barfoo",
"blobHandler": {
"url": "https://your-url/snowflake/handler/22",
"ca": {
"name": "Certificate Authority Bundle"
},
"manualDictionary": false
},
"createdBy": 2,
"deleted": false,
"type": "queryable",
"recordCount": 0,
"rowCount": 6,
"documentation": "# Public Barfoo",
"statsExpiration": "2021-08-27T16:34:47.846Z",
"id": 22,
"blobHandlerType": "Snowflake",
"policyHandlerType": "None",
"subscriptionType": "automatic",
"subscriptionPolicy": {
"type": "subscription",
"automaticSubscription": true
},
"globalPolicies": null,
"status": "passed",
"statusInfo": {
"sql": {
"status": "passed",
"message": "Passed"
}
}
}
Get a data source by the short name
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/sqlTableName/{shortName} |
Get a data source based on the SQL table name. |
Query Parameters
Attribute | Description | Required |
---|---|---|
shortName | string The data source SQL table name. |
Yes |
Response Schema
Attribute | Description |
---|---|
name | string The data source name. |
recordFormat | string The data format of blobs in the data source, such as json , xml , html , or jpeg . |
description | string The description of the data source. |
policyHandler | array The ID of the policy handler and details about the data policies enforced on the data source. |
sqlSchemaName | string A string that represents this data source's schema name in the Query Engine. |
sqlTableName | string The SQL table name in the Query Engine. |
blobHandler | array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source. |
blobHandlerType | string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL ). |
createdBy | integer The ID of the profile creating the data source. |
deleted | boolean If true, the data source was deleted. |
type | string The data source type, such as queryable or ingested . |
rowCount | integer The number of rows. |
documentation | string Documentation associated with the data source. |
id | integer The data source ID. |
policyHandlerType | string The type of policy handler applied to the data source: Builder . |
subscriptionType | string The type of subscription policy on the data source. The type can be automatic (which allows anyone to subscribe), approval (which requires the subscriber to be manually approved), policy (which only allows users with specific groups or attributes to subscribe), or manual (which requires users to be manually added). |
subscriptionPolicy | array Details about the Subscription Policy applied to the data source. |
globalPolicies | string Details about the Global Policies applied to the data source. |
status | string The data source health status. |
Request example
The following request gets a data source based on the SQL table name customer_data
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/sqlTableName/customer_data
Response example
{
"name": "Dbo Customer Data",
"recordFormat": "Not Provided",
"description": null,
"policyHandler": {
"visibilitySchema": {
"fields": [],
"version": "2021-10-06T19:39:39.145Z"
},
"handlerId": 55,
"dataSourceId": 26
},
"sqlSchemaName": "dbo",
"sqlTableName": "customer_data",
"blobHandler": {
"url": "https://1.1.1.1.1/snowflake/testhandler/26",
"ca": {
"name": "Certificate Authority Bundle"
},
"manualDictionary": false
},
"createdBy": 2,
"deleted": false,
"type": "queryable",
"recordCount": 0,
"rowCount": 1000,
"documentation": "# Dbo Customer Data",
"statsExpiration": "2021-11-05T19:37:43.270Z",
"id": 26,
"blobHandlerType": "Snowflake",
"policyHandlerType": "Builder",
"subscriptionType": "automatic",
"subscriptionPolicy": {
"type": "subscription",
"automaticSubscription": false
},
"globalPolicies": null,
"status": "passed",
"statusInfo": {
"sql": {
"status": "passed",
"message": "Passed"
},
"stats": {
"status": "passed",
"lastAttempted": "2021-10-06T19:37:43.337Z"
},
"lastAttempt": {
"date": "2021-10-06T19:39:39.821Z"
},
"highCardinality": {
"status": "passed",
"lastAttempted": "2021-10-06T19:37:43.337Z"
}
},
"expiration": null,
"catalogMetadata": null,
"workspace": null,
"seeded": false,
"schemaEvolutionId": 4,
"columnEvolutionEnabled": true,
"createdAt": "2021-10-01T14:23:27.225Z",
"updatedAt": "2021-10-06T19:39:39.145Z",
"subscribedAsUser": true,
"subscriptionId": 45,
"acknowledgeRequired": false,
"subscriptionStatus": "owner",
"requestedState": "owner",
"approved": true,
"subscriptionExpiration": null,
"filterId": null
}
Get data source relationships
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/lineage/{type} |
Get parent and child relationship records for derived data sources using a specified data source ID. |
Query Parameters
Attribute | Description | Required |
---|---|---|
type | string The type of lineage records to return. Options include: parents , children , and all . |
Yes |
dataSourceId | integer The target data source ID. |
Yes |
Response Schema
Attribute | Description |
---|---|
children | array Details of the child data source, including dataSourceId , dataSourceName , projectId , policyHandlerDiff , deleted , createdBy , and createdAt . |
parents | array Details of the parent data source, including dataSourceId , dataSourceName , projectId , policyHandlerDiff , deleted , createdBy , and createdAt . |
Request example
The following request gets the parent relationship records for the derived data source with the data source ID 4
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/4/lineage/parents
Response example
{
"parents": [{
"dataSourceId": 3,
"dataSourceName": "Public Healthcare Data",
"deleted": false,
"createdAt": "2022-08-17T13:41:38.381Z",
"projectId": 2,
"projectName": "Derived Data Source",
"createdBy": "Your Username",
"policyHandlerDiff": {
"dsType": "queryable",
"currentHandlerId": null,
"previousHandlerId": null
}
}]
}
Retrieve a Blob
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/blob/{blobid*} |
Retrieve a blob. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
blobId | string The blob ID. |
Yes |
Response Schema
The response will download the blobs in a file you specify.
Request example
The following request retrieves a blob.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--output ./the-blobs-will-be-saved-here
https://your-immuta-url.com/dataSource/22/blob/22
Response example
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 215k 100 215k 0 0 541k 0 --:--:-- --:--:-- --:--:-- 541k
Get users by access level
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/access |
Get all users with the provided access level for this data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
states | Array[string] The status levels to include when querying for user access. |
No |
approved | boolean Denotes whether the returned access objects should be approved. |
No |
searchText | string A string used to filter returned users. The query is executed with a wildcard prefix and suffix. |
No |
size | integer The number of results to return. |
No |
offset | integer The number of results to skip (for paging). |
No |
sortField | string The field on which to sort the result set. |
No |
sortOrder | string The order in which to sort the results. |
No |
expandGroups | boolean If true will return individual members of any group subscribed. |
No |
ignoreSystemGenerated | boolean If true, will not return system generated accounts. |
No |
filterBySchemaEvolution | boolean If true, will only return users who have the specified level of access across ALL data sources within the same schema evolution group as this one. |
No |
Response Schema
Attribute | Description |
---|---|
count | integer The number of users with access to the data source. |
users | string The metadata regarding the users with access to the data source. |
Request example
The following request gets all users with the provided access level for this data source.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/22/access?sortOrder=desc
Response example
{
"count": 2,
"users": [
{
"profile": 2,
"name": "First Last",
"iamid": "bim",
"userid": "first.last@immuta.com",
"email": "first.last@immuta.com",
"type": "user",
"admin": "First Last",
"approved": true,
"state": "owner",
"systemGenerated": false,
"lastExternalRefresh": "2021-10-06T14:58:46.983Z",
"subscriptionId": 586,
"createdAt": "2021-10-05T14:33:01.518Z",
"updatedAt": "2021-10-05T14:33:01.518Z",
"approvals": [
{
"requiredPermission": "OWNER",
"state": "approved",
"approverId": null,
"ownerModelId": null,
"approver": "First Last",
"ownerModelName": null
}
],
"currentUserCanApprove": false
},
{
"profile": 3,
"name": "Tommy Test",
"iamid": "bim",
"userid": "tommy.test@immuta.com",
"email": "tommy.test@immuta.com",
"type": "user",
"admin": "First Last",
"approved": true,
"state": "subscribed",
"systemGenerated": false,
"lastExternalRefresh": "2021-09-07T16:16:29.957Z",
"subscriptionId": 649,
"createdAt": "2021-10-06T14:58:31.366Z",
"updatedAt": "2021-10-06T14:58:31.366Z",
"approvals": [
{
"requiredPermission": "OWNER",
"state": "approved",
"approverId": null,
"ownerModelId": null,
"approver": "First Last",
"ownerModelName": null
}
],
"currentUserCanApprove": false
}
]
}
Get user access info for a data source
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/users/{profileId}/policyInfo |
Retrieves the visibilities, masking information, and filters that the passed in user has access to in the specified data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
profileId | integer The profile ID of the user. |
Yes |
projectId | integer The project ID. If provided, this project will be used when evaluating the user's visibilities. |
No |
Response Schema
Attribute | Description |
---|---|
visibilities | array Details of the user's visibilities, including anyKey . |
visibilityRuleApplies | boolean If true , a visibility rule exists and the user is not excepted from it. |
masked | array Masking information for the data source, including metadata , name , type , and actionType . |
additionalFilters | array Policy information for the data source, including customWhere , differentialPrivacy , eventTimeColumn , minimization , time , filterSeconds , and isOlderOrNewer . |
Request example
The following request gets the visibility information for the user with the profile ID 2
on the data source with
the data source ID 16
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/16/users/2/policyInfo
Response example
{
"visibilities": [],
"visibilityRuleApplies": false,
"masked": [
{
"type": "Consistent Value",
"metadata": {
"constant": null
},
"name": "WS_SOLD_DATE_SK",
"actionType": "Nullify"
},
{
"type": "Consistent Value",
"metadata": {
"constant": null
},
"name": "WS_BILL_CUSTOMER_SK",
"actionType": "Nullify"
}
],
"additionalFilters": {}
}
Get user visibility info for a data source
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/users/{profileId}/visibilityReport |
Retrieves a summary of total records, total visibilities (the unique values contained in a column protected by a row-level security policy that allow Immuta to determine whether or not a user can see a given row if they possess an attribute that matches the visibility of that row), and visibilities a given user has access to. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
profileId | integer The profile ID of the user. |
Yes |
informationOnly | boolean If true , the query will just return information for the UI and will skip running some queries for ephemeral data sources. |
No |
includeNestedColumns | boolean If true , the query will return just information for the dictionary page, including the masking policies for nested columns. |
No |
Response Schema
Attribute | Description |
---|---|
noVisibilities | boolean If true , the data source has no row-level security or purpose-based restriction policies applied to it. |
dataSourceVisibilitiesCount | integer The total number of possible visibilities the given data source has. |
userVisibilitiesCount | integer The number of visibilities the current user can see for the given data source. |
masked | array Masking information for the data source, including metadata , name , type , and actionType . |
dataSource | integer The data source ID. |
dataSourceName | string The data source name. |
additionalFilters | array Policy information for the data source, including customWhere , differentialPrivacy , eventTimeColumn , minimization , time , filterSeconds , and isOlderOrNewer . |
allowMaskedJoins | boolean If true the data source allows masked joins. |
policySet | array Details about the policies on the data source. |
Request example
The following request gets all of the visibility information for the user with the profile ID 2
on the data source
with the data source ID 16
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/16/users/2/visibilityReport
Response example
[
{
"noVisibilities": true,
"dataSourceVisibilitiesCount": 0,
"userVisibilitiesCount": 0,
"masked": [
{
"type": "Consistent Value",
"metadata": {
"constant": null
},
"name": "WS_SOLD_DATE_SK"
},
{
"type": "Consistent Value",
"metadata": {
"constant": null
},
"name": "WS_BILL_CUSTOMER_SK"
}
],
"dataSource": 16,
"dataSourceName": "Web Sales",
"additionalFilters": {},
"allowMaskedJoins": false,
"policySet": "[{\"type\":\"masking\",\"rules\":[{\"type\":\"masking\",\"config\":{\"fields\":[\"WS_BILL_CUSTOMER_SK\"],\"maskingConfig\":{\"type\":\"Consistent Value\",\"metadata\":{\"constant\":null}}},\"exceptions\":null,\"ruleAppliedForUser\":true}],\"global\":{\"id\":9,\"tag\":\"Confidential\",\"live\":true,\"name\":\"Data Policy\",\"reason\":null,\"staged\":false,\"deleted\":false,\"conflict\":null,\"disabled\":false,\"metadata\":null,\"template\":false,\"createdAt\":\"2022-08-09T17:23:02.634Z\",\"createdBy\":\"jane\",\"policyKey\":\"Data Policy\",\"updatedAt\":\"2022-08-09T17:23:02.634Z\",\"clonedFrom\":null,\"certification\":null,\"createdByName\":\"jane\",\"changedOnApply\":[\"dataTypeMismatch\"],\"parentPolicyId\":null,\"systemGenerated\":false,\"originalPolicyId\":\"masking(^_^)data policy(^_^)ws_bill_customer_sk(^_^)consistent value(^_^)(^_^)(^_^)\",\"ownerRestrictions\":null},\"description\":null,\"policyHash\":\"3a2faac13e332cca1829ed773afa298a5455ac5bb54e68c53ae00991575d2a4b\"},{\"type\":\"masking\",\"rules\":[{\"type\":\"masking\",\"config\":{\"fields\":[\"WS_SOLD_DATE_SK\"],\"maskingConfig\":{\"type\":\"Consistent Value\",\"metadata\":{\"constant\":null}}},\"exceptions\":null,\"ruleAppliedForUser\":true}],\"global\":{\"id\":9,\"tag\":\"Confidential\",\"live\":true,\"name\":\"Data Policy\",\"reason\":null,\"staged\":false,\"deleted\":false,\"conflict\":null,\"disabled\":false,\"metadata\":null,\"template\":false,\"createdAt\":\"2022-08-09T17:23:02.634Z\",\"createdBy\":\"jane\",\"policyKey\":\"Data Policy\",\"updatedAt\":\"2022-08-09T17:23:02.634Z\",\"clonedFrom\":null,\"certification\":null,\"createdByName\":\"jane\",\"changedOnApply\":[\"dataTypeMismatch\"],\"parentPolicyId\":null,\"systemGenerated\":false,\"originalPolicyId\":\"masking(^_^)data policy(^_^)ws_sold_date_sk(^_^)consistent value(^_^)(^_^)(^_^)\",\"ownerRestrictions\":null},\"description\":null,\"policyHash\":\"2877a1ace4cfa6427370fd39b254ce0ea75dc22cb024a2f857e033c82a987f9a\"}]"
}
]
Get current user visibility info
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/visibilityReport |
Retrieves a summary of total records, total visibilities (the unique values contained in a column protected by a row-level security policy that allow Immuta to determine whether or not a user can see a given row if they possess an attribute that matches the visibility of that row), and visibilities the current user has access to for a specified data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Response Schema
Attribute | Description |
---|---|
noVisibilities | boolean If true , the data source has no row-level security or purpose-based restriction policies applied to it. |
dataSourceVisibilitiesCount | integer The total number of possible visibilities the given data source has. |
userVisibilitiesCount | integer The number of visibilities the current user can see for the given data source. |
denialReason | string Reason the user was denied visibility. |
masked | array Masking information for the data source, including metadata , name , type , and actionType . |
dataSource | integer The data source ID. |
dataSourceName | string The data source name. |
additionalFilters | array Policy information for the data source, including customWhere , differentialPrivacy , eventTimeColumn , minimization , time , filterSeconds , and isOlderOrNewer . |
allowMaskedJoins | boolean If true the data source allows masked joins. |
policySet | array Details about the policies on the data source. |
Request example
The following request gets all of the visibility information for the current user on the data source
with the data source ID 16
.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/16/visibilityReport
Response example
{
"noVisibilities": true,
"dataSourceVisibilitiesCount": 0,
"userVisibilitiesCount": 0,
"masked": [
{
"type": "Consistent Value",
"metadata": {
"constant": null
},
"name": "WS_SOLD_DATE_SK"
},
{
"type": "Consistent Value",
"metadata": {
"constant": null
},
"name": "WS_BILL_CUSTOMER_SK"
}
],
"dataSource": 16,
"dataSourceName": "Web Sales",
"additionalFilters": {},
"allowMaskedJoins": false,
"policySet": "[{\"type\":\"masking\",\"rules\":[{\"type\":\"masking\",\"config\":{\"fields\":[\"WS_BILL_CUSTOMER_SK\"],\"maskingConfig\":{\"type\":\"Consistent Value\",\"metadata\":{\"constant\":null}}},\"exceptions\":null,\"ruleAppliedForUser\":true}],\"global\":{\"id\":9,\"tag\":\"Confidential\",\"live\":true,\"name\":\"Data Policy\",\"reason\":null,\"staged\":false,\"deleted\":false,\"conflict\":null,\"disabled\":false,\"metadata\":null,\"template\":false,\"createdAt\":\"2022-08-09T17:23:02.634Z\",\"createdBy\":\"jane\",\"policyKey\":\"Data Policy\",\"updatedAt\":\"2022-08-09T17:23:02.634Z\",\"clonedFrom\":null,\"certification\":null,\"createdByName\":\"jane\",\"changedOnApply\":[\"dataTypeMismatch\"],\"parentPolicyId\":null,\"systemGenerated\":false,\"originalPolicyId\":\"masking(^_^)data policy(^_^)ws_bill_customer_sk(^_^)consistent value(^_^)(^_^)(^_^)\",\"ownerRestrictions\":null},\"description\":null,\"policyHash\":\"3a2faac13e332cca1829ed773afa298a5455ac5bb54e68c53ae00991575d2a4b\"},{\"type\":\"masking\",\"rules\":[{\"type\":\"masking\",\"config\":{\"fields\":[\"WS_SOLD_DATE_SK\"],\"maskingConfig\":{\"type\":\"Consistent Value\",\"metadata\":{\"constant\":null}}},\"exceptions\":null,\"ruleAppliedForUser\":true}],\"global\":{\"id\":9,\"tag\":\"Confidential\",\"live\":true,\"name\":\"Data Policy\",\"reason\":null,\"staged\":false,\"deleted\":false,\"conflict\":null,\"disabled\":false,\"metadata\":null,\"template\":false,\"createdAt\":\"2022-08-09T17:23:02.634Z\",\"createdBy\":\"jane\",\"policyKey\":\"Data Policy\",\"updatedAt\":\"2022-08-09T17:23:02.634Z\",\"clonedFrom\":null,\"certification\":null,\"createdByName\":\"jane\",\"changedOnApply\":[\"dataTypeMismatch\"],\"parentPolicyId\":null,\"systemGenerated\":false,\"originalPolicyId\":\"masking(^_^)data policy(^_^)ws_sold_date_sk(^_^)consistent value(^_^)(^_^)(^_^)\",\"ownerRestrictions\":null},\"description\":null,\"policyHash\":\"2877a1ace4cfa6427370fd39b254ce0ea75dc22cb024a2f857e033c82a987f9a\"}]"
}
Access data sources and make data source requests
Method | Path | Purpose |
---|---|---|
POST | /dataSource/subscribe |
Subscribe to a data source. |
POST | /dataSource/featureStore |
Create a SQL account to access data sources through your analytic tool. |
PUT | /dataSource/featureStore/password |
Modify SQL account password. |
POST | /dataSource/featureStore/temporary |
Create a temporary SQL account. |
POST | /dataSource/{dataSourceId}/reverseMask |
Make a request for values to be unmasked. |
POST | /dataSource/{dataSourceId}/access |
Add a user to a specific data source. |
Subscribe to a data source
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/subscribe |
Subscribe to a data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer Data source ID number. |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
dataSourceIds | array The ID of the data source the user is subscribing to. |
No |
approvals | array Includes details about the Subscription policy on the data source: requiredPermissions , specificApproverRequired , specificApprover , and ownerModelId . |
No |
Response Schema
Attribute | Description |
---|---|
body | array Contains details about the data source, including the data source ID, subscription status of the user, the profile ID of the user, and the dates the data source was created and updated. |
Request example
The following request subscribes to the data source with ID 22
.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/subscribe?dataSourceId=22
Payload example
{
"dataSourceIds": [
22
],
"metadata": {},
"approvals": [
{
"requiredPermission": "Owner",
"specificApproverRequired": false,
"specificApprover": 2,
"ownerModelId": 23
}
],
"groupId": 12
}
Response example
{
"inError": [],
"success": [{
"id": 64,
"modelId": "22",
"modelType": "datasource",
"state": "subscribed",
"metadata": {},
"admin": null,
"denialReasoning": null,
"profile": 2,
"group": null,
"expiration": null,
"acknowledgeRequired": false,
"createdAt": "2021-08-26T16:36:09.587Z",
"updatedAt": "2021-08-26T16:36:09.587Z",
"approved": true
}]
}
Create a SQL account
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/featureStore |
Create a SQL account. |
Query Parameters
None.
Payload Parameters
Attribute | Description | Required |
---|---|---|
username | string The SQL account username. |
Yes |
password | string The SQL account password. |
Yes |
Response Schema
Attribute | Description |
---|---|
status | boolean If true , the SQL account was created. |
Request example
The following request creates a SQL account.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/featureStore
Request payload example
{
"username": "jane@example.com",
"password": "your-new-sql-password"
}
Response example
{
"success": true
}
Modify a SQL account password
Endpoint
Method | Path | Purpose |
---|---|---|
PUT | /dataSource/featureStore/password |
Modify SQL account password for the requesting user. |
Query Parameters
None.
Payload Parameters
Attribute | Description | Required |
---|---|---|
password | string The new SQL account password. |
Yes |
Response Schema
Attribute | Description |
---|---|
success | boolean If true , the password was successfully changed. |
sqlUser | integer SQL user ID. |
Request example
The following request modifies a SQL account password.
curl \
--request PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/featureStore/password
Payload example
{
"password": "testpassword"
}
Response example
{
"success": true,
"sqlUser": "jane@example.com"
}
Create a temporary SQL account
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/featureStore/temporary |
Create a temporary SQL account. |
Query Parameters
None.
Payload Parameters
Attribute | Description | Required |
---|---|---|
projectId | string The ID of the project to be applied. |
No |
accountPrefix | string The prefix to use for the SQL username. |
No |
expiresIn | integer The number of minutes the account should be valid. |
No |
forceDirectAudit | boolean If true , will force direct audit of queries run by this account. |
No |
Response Schema
Attribute | Description |
---|---|
success | boolean If true , a temporary SQL account was created. |
sqlUser | string The SQL username. |
sqlPassword | string The SQL password. |
port | integer The port number. |
database | string The database name. |
host | string The hostname. |
sslMode | string Indicates whether or not SSL is required. |
connectionString | string The connection string. |
Request example
The following request creates a temporary SQL account.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/featureStore/temporary
Payload example
{
"projectId": 1,
"accountPrefix": "temp",
"expiresIn": 60,
"forceDirectAudit": false
}
Response example
{
"success": true,
"sqlUser": "temp1635366310923d8b65qm6279",
"sqlPassword": "************",
"port": 5432,
"database": "test",
"host": "your.host.io",
"sslMode": "require",
"connectionString": "postgresql://temp1635366310923d8b65qm6279:************@your.host.io:5432/test?sslmode=require"
}
Request to unmask values
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/{dataSourceId}/reverseMask |
Makes a request for values to be unmasked. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
column | string The column to unmask. |
Yes |
unmaskingReason | string The reason the values need to be unmasked. |
Yes |
unmaskingUsers | array[integer] The profile ID of the users who can unmask the values for the requestor. |
Yes |
projectId | integer The ID of the associated project. |
No |
dataSourceId | integer The data source ID. |
No |
Response Schema
Attribute | Description |
---|---|
id | integer The ID of the request. |
requestingUserProfile | integer The requesting user profile ID. |
dataSourceId | integer The data source ID. |
reason | string The reason for the unmasking request. |
metadata | string Metadata regarding the masking, such as the column , values , and maskingConfig . |
type | string The type of request. |
state | string The state of the task, such as pending . |
createdAt | timestamp The date and time the task was created. |
updatedAt | timestamp The date and time the task was updated. |
Request example
The following requests for values to be unmasked.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/23/reverseMask
Request payload example
{
"column": "cc_county",
"values": ["WlRObU9ERTVNVE0yTVdabU9XVXdPQT09OktYN2lkNEZqZjlaWUluck8xTnVHOGlSN25wWmdudVZjbnZES1ArUkxhMGc9"],
"unmaskReasoning": "Marketing project",
"unmaskingUsers": [1]
}
Response example
{
"id": 1,
"requestingUserProfile": 13,
"dataSourceId": 12,
"reason": "Marketing Campaign",
"metadata": {
"salt": "**********87",
"column": "cc_county",
"values": ["WlRObU9ERTVNVE0yTVdabU9XVXdPQT09OktYN2lkNEZqZjlaWUluck8xTnVHOGlSN25wWmdudVZjbnZES1ArUkxhMGc9"],
"maskingConfig": {
"type": "Reversible",
"metadata": {}
}
},
"type": "unmask",
"state": "pending",
"createdAt": "2021-10-27T20:35:48.253Z",
"updatedAt": "2021-10-27T20:35:48.253Z"
}
Add a user to a data source
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/{dataSourceId}/access |
Add a user to a specific data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
state | string The status of the user: subscribed , owner, expert , or ingest . |
Yes |
profileId | integer The profile ID of the user being added to the data source. |
Yes |
groupId | integer The ID of the group being added to the data source. |
No |
approvals | array Details about the user approving access: requiredPermission , specificApproverRequired , and specificApprover . |
No |
expiration | date The date the user's data source subscription ends. |
No |
Response Schema
Attribute | Description |
---|---|
id | integer The user's subscription ID. |
modelId | integer The model ID. |
modelType | string The model type. |
state | string The user's data source role, such as subscribed . |
denialReasoning | string If the user was denied access, the reason for denial. |
profile | integer The user's profile ID. |
group | integer If a group was added, the group ID. |
expiration | date The date the user's subscription to the data source will expire. |
acknowledgeRequired | boolean If the data source is associated with a project, this value will be true if the user needs to confirm they have read the project acknowledgment. |
createdAt | timestamp The date and time of creation. |
updatedAt | timestamp The date and time of update. |
approved | boolean When true , the user's request has been approved. |
Request example
The following request adds a user (saved in example-payload.json
) to this data source.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/22/access
Request payload example
{
"profileId": 3,
"state": "subscribed"
}
Response example
{
"id": 19,
"modelId": "1",
"modelType": "datasource",
"state": "subscribed",
"metadata": null,
"admin": 2,
"denialReasoning": null,
"profile": 3,
"group": null,
"expiration": null,
"acknowledgeRequired": false,
"createdAt": "2021-09-21T14:24:12.528Z",
"updatedAt": "2021-09-21T14:24:12.528Z",
"approved": true
}
Manage data source requests
Method | Path | Purpose |
---|---|---|
GET | /dataSource/tasks/pending |
Get all pending tasks for this user and pending tasks this user has created. |
GET | /dataSource/tasks/{taskId} |
Handles the given task and marks it as complete. |
GET | /dataSource/{dataSourceId}/tasks |
Returns all tasks the user has made/can approve or deny for this data source. |
PUT | /dataSource/{dataSourceId}/access/{id} |
Change user status for a specific data source. |
Get pending tasks by user
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/tasks/pending |
Get all pending tasks for this user and pending tasks this user has created. |
Query Parameters
Attribute | Description | Required |
---|---|---|
searchText | string If specified, will filter results using the specified string. |
No |
searchModel | string Will filter the results by model type: dataSource or schemaEvolution . |
No |
offset | integer integer The number of results to skip (for paging). |
No |
size | integer The number of results to return per page. |
No |
schemaEvolutionConnectionString | string The schema evolution connection string to filter by. |
No |
countBySchemaEvolution | boolean Iftrue , will only return the number of tasks, grouped by schema evolution. |
No |
countByDataSource | boolean Iftrue , will only return the number of tasks, grouped by data source. |
No |
countOnly | boolean When true , will only return a count of the pending tasks. |
No |
groupByDataSource | boolean If true , will return the results as an array of { dataSourceId: , rows: } . |
No |
types | Array[string] Filters the results by the type of task: queryDebug , unmask , dataSourceCreated , columnAdded , columnDeleted , or columnTypeChanged . |
No |
Response Schema
Attribute | Description |
---|---|
outgoing | array Includes details of the tasks or requests created by the user, such as the count , type , and targetEmails . |
incoming | array Includes details about the tasks received by the user, such as the count , type , and targetEmails . |
Request example
The following request gets all pending tasks for a user and pending tasks the user has created.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/tasks/pending
Response example
{
"outgoing": [],
"incoming": [
{
"fullCount": 1,
"id": 7,
"state": "pending",
"type": "queryDebug",
"reason": "Please help me with this query.",
"targetNames": [
"Katie"
],
"targetEmails": [
"katie@example.com"
],
"requester": {
"name": "User 1",
"id": 3,
"email": "user@example.com"
},
"dataSource": {
"id": 3,
"name": "Public Fake Medical Claims 2017"
},
"createdAt": "2021-10-12T18:56:22.954Z"
}
]
}
Mark tasks as complete
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/tasks/{taskId} |
Handles the given task and marks it as complete. |
Query Parameters
Attribute | Description | Required |
---|---|---|
taskId | integer The task ID. |
Yes |
Response Schema
Attribute | Description |
---|---|
result | array Includes details about the task. |
Request example
The following request handles the given task and marks it as complete.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/tasks/7
Response example
{
"result": {
"unmaskPairs": [
{
"masked": "TnpjNFpXSXdOR1pqWkdNeE5EYzJPQT09OktaWjJEVldXZVluTmQ2SUVQdW1MajJtVTdqL2ZBT1JlaGFUUHJidmhkWmM9",
"unmasked": "jalekseev2@phoca.cz"
}
],
"column": "email"
}
}
Return tasks for a data source
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/tasks |
Returns all tasks the user has made/can approve or deny for this data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
states | Array[string] The state of the tasks: pending or completed . |
No |
targetProfileId | integer Only returns tasks where the target user has this profile ID. |
No |
requestingUserProfileId | integer Only returns tasks where the requesting user has this profile ID. |
No |
profileId | integer Returns tasks where either the target or requesting user has this profile ID. |
No |
searchText | string A string used to filter returned users. The query is executed with a wildcard prefix and suffix. |
No |
searchModel | string A string used to determine how results should be filtered using searchText. |
No |
types | Array[string] The type of task: unmask , queryDebug , dataSourceCreated , columnAdded , columnDeleted , or columnTypeChanged . |
No |
size | integer The number of results to return. |
No |
offset | integer The number of results to skip (for paging). |
No |
sortField | string The field by which to sort the result set. |
No |
sortOrder | string The order in which to sort the results. The default is desc . |
No |
countOnly | boolean If true , will only return the number of tasks. |
No |
Response Schema
Attribute | Description |
---|---|
hits | array Includes details about each task, such as the id , state , type , and requestor . |
count | integer The total number of tasks. |
Request example
The following request returns all tasks the user has made/can approve or deny for this data source.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/23/tasks?sortOrder=desc
Response example
{
"hits": [
{
"fullCount": 2,
"id": 6,
"state": "completed",
"type": "queryDebug",
"reason": "I don't understand why my query isn't returning any results.",
"targetNames": [
"John"
],
"targetEmails": [
"john@example.com"
],
"requester": {
"name": "User 1",
"id": 3,
"email": "user@example.com"
},
"dataSource": {
"id": 3,
"name": "Public Fake Medical Claims 2017"
},
"createdAt": "2021-10-12T15:48:23.095Z"
},
{
"fullCount": 2,
"id": 7,
"state": "completed",
"type": "queryDebug",
"reason": "Is my query written properly?",
"targetNames": [
"John"
],
"targetEmails": [
"john@example.com"
],
"requester": {
"name": "User 1",
"id": 3,
"email": "user@example.com"
},
"dataSource": {
"id": 3,
"name": "Public Fake Medical Claims 2017"
},
"createdAt": "2021-10-12T18:56:22.954Z"
}
],
"count": 2
}
Change user status
Endpoint
Method | Path | Purpose |
---|---|---|
PUT | /dataSource/{dataSourceId}/access/{subscriptionId} |
Change user status for a specific data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | Integer The data source ID. |
Yes |
subscriptionId | Integer The data source member's subscription ID. |
Yes |
Attribute | Description | Required |
---|---|---|
state | string The new status for the user: subscribed , owner , expert or ingest . |
Yes |
Response Schema
Attribute | Description |
---|---|
id | integer The data source member's subscription ID. |
modelId | integer The model ID. |
modelType | array The model type (i.e., datasource ). |
state | array The current state of the user's role: subscribed , owner , expert , or ingest . |
profile | integer The profile ID. |
group | integer If a group's status is being updated, this is the group ID. |
expiration | 'timestamp' The date the user will no longer have access to the data source. |
acknowledgeRequired | boolean This attribute is specific to projects. When true the user needs to confirm they have read the project acknowledgement statement. |
createdAt | timestamp The date and time created. |
updatedAt | timestamp The date and time updated. |
originalState | array The user's previous status for the data source. |
approved | boolean If true , the status is approved. |
Request example
The following request changes the user status to subscribed
for the specified data source.
curl \
--request PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/23/access/95
Payload example
{
"state": "subscribed"
}
Response example
{
"id": 95,
"modelId": "3",
"modelType": "datasource",
"state": "subscribed",
"metadata": {},
"admin": 2,
"profile": 3,
"group": null,
"expiration": null,
"acknowledgeRequired": false,
"createdAt": "2021-10-12T15:40:13.878Z",
"updatedAt": "2021-10-12T16:10:46.801Z",
"originalState": "expert",
"approved": true
}
Update data sources
Method | Path | Purpose |
---|---|---|
PUT | /dataSource/{dataSourceId} |
Update a data source. |
PUT | /dataSource/bulk/{type} |
Update multiple data sources. |
POST | /dataSource/bulkRefreshViews |
Refresh native views. |
POST | /dataSource/{dataSourceId}/blobs |
Save blob metadata to Immuta. |
POST | /dataSource/{dataSourceId}/persistBlob |
Save blob metadata to Immuta and store raw content in local blob store. |
PUT | /dataSource/detectRemoteChanges |
Trigger the schema monitoring job for the specified detection group, or all groups if no ID is given. |
Update a data source
Endpoint
Method | Path | Purpose |
---|---|---|
PUT | /dataSource/{dataSourceId} |
Update a data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
blobHandler | array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source. |
No |
blobHandlerType | string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL ). |
No |
recordFormat | string The data format of blobs in the data source, such as json , xml , html , or jpeg . |
No |
type | string The type of data source: ingested (metadata will exist in Immuta) or queryable (metadata is dynamically queried). |
No |
name | string The name of the data source. It must be unique within the Immuta tenant. |
No |
sqlTableName | string A string that represents this data source's table in the Query Engine. |
No |
organization | string The organization that owns the data source. |
No |
category | string The category of the data source. |
No |
description | string The description of the data source. |
No |
hasExamples | boolean When true , the data source contains examples. |
No |
Response Schema
Attribute | Description |
---|---|
private | boolean When false , the data source will be publicly available in the Immuta UI. |
blobHandler | array[object] A list of full URLs providing the locations of all blob store handlers to use with this data source. |
blobHandlerType | string Describes the type of underlying blob handler that will be used with this data source (e.g., MS SQL ). |
recordFormat | string The data format of blobs in the data source, such as json , xml , html , or jpeg . |
type | string The type of data source: ingested (metadata will exist in Immuta) or queryable (metadata is dynamically queried). |
name | string The name of the data source. It must be unique within the Immuta tenant. |
sqlTableName | string A string that represents this data source's table in the Query Engine. |
organization | string The organization that owns the data source. |
description | string The description of the data source. |
policyHandler | array The ID of the policy handler and details about the data policies enforced on the data source. |
subscriptionPolicy | array Details about the subscription policy enforced on the data source, including the type of policy and exceptions. |
Request example
The following request updates the data source's documentation (saved in example-payload.json
).
curl \
--request PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/22
Request payload example
{
"documentation": "# Public Credit Accounts\nThis request updates the data source documentation."
}
Response example
{
"name": "Public Credit Accounts",
"recordFormat": "Not Provided",
"description": null,
"policyHandler": {
"handlerId": 21,
"visibilitySchema": {
"fields": [],
"version": "2021-09-15T17:44:20.678Z"
},
"previousHandlerId": 20,
"dataSourceId": 1
},
"sqlSchemaName": "public",
"sqlTableName": "credit_accounts",
"blobHandler": {
"url": "https://1.1.1.1:1111/snowflake/handler/1",
"ca": {
"name": "Certificate Authority Bundle"
},
"manualDictionary": false
},
"createdBy": 2,
"deleted": false,
"type": "queryable",
"recordCount": 0,
"rowCount": "100000",
"documentation": "# Public Credit Accounts\nThis request updates the data source documentation.",
"statsExpiration": "2021-09-22T13:51:46.646Z",
"id": 1,
"blobHandlerType": "Snowflake",
"policyHandlerType": "Builder",
"subscriptionType": "approval",
"subscriptionPolicy": {
"type": "subscription",
"approvals": [{
"requiredPermission": "OWNER",
"specificApproverRequired": false
}]
},
"globalPolicies": null,
"status": "failed",
"statusInfo": {
"sql": {
"status": "passed",
"message": "Passed"
},
"stats": {
"status": "passed",
"lastAttempted": "2021-09-21T13:51:46.674Z"
},
"fingerprint": {
"status": "passed",
"lastAttempted": "2021-09-09T14:12:25.177Z"
},
"lastAttempt": {
"date": "2021-09-20T19:35:21.929Z"
},
"globalPolicy": {
"status": "passed",
"lastAttempted": "2021-09-17T19:07:38.092Z"
},
"highCardinality": {
"status": "failed",
"message": "Error could not connect to remote database",
"lastAttempted": "2021-09-20T16:43:19.426Z"
}
},
"expiration": null,
"catalogMetadata": null,
"workspace": null,
"seeded": false,
"schemaEvolutionId": 1,
"columnEvolutionEnabled": true,
"createdAt": "2021-09-09T14:12:09.511Z",
"updatedAt": "2021-09-21T13:52:42.908Z"
}
Update multiple data sources
Endpoint
Method | Path | Purpose |
---|---|---|
PUT | /dataSource/bulk/{type} |
Update data sources. |
Query Parameters
Attribute | Description | Required |
---|---|---|
type | string The action to perform on the data sources: add-users , disable , restore , delete , or tags . |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
ids | array[integer ] The IDs of the data sources to update. |
Yes |
update | array[object] Only required for add-users (includes metadata about the users' profiles : id and state ) and tags (includes metadata about the tags : name and source ) types. |
No |
Response Schema
Attribute | Description |
---|---|
bulkId | string The ID of the bulk data source update. |
jobsCreated | integer The number of jobs created. |
Request example
The following request adds the Address.email
tag to two data sources.
curl \
--request PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/bulk/tags
Payload example
{
"ids": [49, 50],
"update": {
"tags": [{
"name": "Address.email",
"source": "curated"
}]
}
}
Response example
{
"bulkId": "bulk_ds_update_4896d300e04c4a8f89493ebf125c5c6b",
"jobsCreated": 2
}
Refresh native views
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/bulkRefreshViews |
Refresh native views. |
Query Parameters
None.
Payload Parameters
Attribute | Description | Required |
---|---|---|
dataSourceIds | array[integer] The IDs of the data sources of the native views to update. |
Yes |
Request example
The following request with the payload below refreshes the view for the data source with the ID 202.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer <TOKEN>" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/bulkRefreshViews
Payload example
{
"dataSourceIds": [202]
}
Save blob metadata to Immuta
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/{dataSourceId}/blobs |
Save blob metadata to Immuta. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
blobId | string The unique ID used to identify this blob within its data source. |
Yes |
file | string The binary file to add to the data source. |
Yes |
filename | string The name that will display in the filesystem. |
No |
tags | array[string] Tags to apply to the blob. |
No |
date | data A date that corresponds to a date within the record itself. |
No |
filesize | integer The size of the file in bytes. |
No |
Response Schema
Attribute | Description |
---|---|
blobsWithoutIds | integer The number of blobs added without IDs. |
blobsInError | array The blobs that were not added because of an error. |
blobsInserted | array The blobs added to the data source. |
tags | array[string] Tags applied to the blobs. |
Request example
The following request saves blob metadata to Immuta.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/23/blobs
Payload example
[
{
"blobId": "testblob",
"filename": "testfile",
"tags": [
"update"
],
"visibility": {},
"metadata": {},
"date": "2021-10-21",
"filesize": 2
}
]
Response example
{
"blobsWithoutIds": 0,
"blobsInError": [],
"blobsInserted": [
"string"
],
"tags": [
"string"
]
}
Store blob metadata locally
Endpoint
Method | Path | Purpose |
---|---|---|
POST | /dataSource/{dataSourceId}/persistBlob |
Save blob metadata to Immuta and store raw content in local blob store. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Payload Parameters
Attribute | Description | Required |
---|---|---|
blobId | string The unique ID used to identify this blob within its data source. |
Yes |
file | string The binary file to add to the data source. |
Yes |
filename | string The name that will display in the filesystem. |
No |
tags | array[string] Tags to apply to the blob. |
No |
date | data A date that corresponds to a date within the record itself. |
No |
filesize | integer The size of the file in bytes. |
No |
Response Schema
Attribute | Description |
---|---|
blobsWithoutIds | integer The number of blobs added without IDs. |
blobsInError | array The blobs that were not added because of an error. |
blobsInserted | array The blobs added to the data source. |
tags | array[string] Tags applied to the blobs. |
Request example
The following request saves blob metadata to Immuta and stores raw content in local blob stores.
curl \
--request POST \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json
https://your-immuta-url.com/dataSource/23/persistBlob
Payload example
{
"blobId": "test/pov-queries-2.dbc",
"filename": "pov-queries",
"date": "2021-10-26",
"file": "pov-queries.dbc"
}
Response example
{
"blobsWithoutIds": 0,
"blobsInError": [],
"blobsInserted": ["test/pov-queries.dbc"],
"tags": []
}
Trigger schema monitoring jobs
Endpoint
Method | Path | Purpose |
---|---|---|
PUT | /dataSource/detectRemoteChanges |
Trigger the schema monitoring job for the specified detection group, or all groups if no payload parameters are given. |
Attribute | Description | Required |
---|---|---|
dataSourceIds | array[integer] The data source IDs to run the column detection job on. Leave empty to run this job globally on all data sources. This parameter cannot be included in the payload if schemaEvolutionId or any combination of hostname, database, port, or table is included. |
No |
hostname | string The hostname of the data sources. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included. |
No |
port | integer The port used to connect the data sources to Immuta. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included. |
No |
database | string The database name. This runs schema monitoring on the database provided. If data sources were initially registered via the V2 API, including this parameter will locate new schemas that contain tables Immuta has the ability to access, and Immuta will create a new schema project associated with these newly discovered schemas and create data sources for each table located. If data sources were initially registered via the V1 API, including this parameter will only update the columns and tables of registered schema and tables of the specified database; it will not register any new schemas. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included. |
No |
table | string The table name. This will run column detection to just update the columns in this table. This parameter cannot be included in the payload if dataSourceIds or schemaEvolutionId is included. |
No |
schemaEvolutionId | integer The ID of the schema to run the schema monitoring job on. This will run on all tables associated with the specified ID. The schema ID can be found in the response body of /dataSource/{dataSourceId} . This parameter cannot be included in the payload if dataSourceIds or any combination of hostname, database, port, or table is included. |
No |
skipColumnDetection | boolean When true , Immuta will only pull new tables from the source server. This parameter can only be paired with schemaEvolutionId. |
No |
overrides.httpPath | string If Databricks ephemeral overrides are configured, provide the alternative HTTP path to trigger schema monitoring on that ephemeral cluster. |
No |
Response Schema
Attribute | Description |
---|---|
schemaDetection | object Includes details about the resulting schema detection jobs. |
columnDetection | object Includes details about the resulting column detection jobs. |
bulkId | string The unique identifier of the jobs running schema monitoring and column detection. |
Reponses will include either
bulkId
,schemaDetection
, andcolumnDetection
(whentable
ordataSourceIds
is provided in payload) orbulkId
(payloads that do not includedataSourceIds
ortable
)
Request example
The following request triggers the schema monitoring job for the specified detection group.
curl \
--request PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
--data @example-payload.json \
https://your-immuta-url.com/dataSource/detectRemoteChanges
Payload example
The tabs below illustrate payloads for triggering schema monitoring on a host, database, or table.
Host
The request will run schema monitoring for all databases registered under the hostname provided in the payload.
{
"hostname": "organization.us-east-1.snowflakecomputing.com"
}
Database
The request will run schema monitoring on the database provided in the payload. If data sources were initially registered via the V2 API, this request will locate new schemas that contain tables Immuta has the ability to access, and Immuta will create a new schema project associated with these newly discovered schemas and create data sources for each table located. If data sources were initially registered via the V1 API, this request will only update the columns and tables of registered schema and tables of the specified database; it will not register any new schemas.
{
"database": "public"
}
Table
The request will run column detection and update the columns on the table specified in the payload.
{
"database": "public",
"table": "healthcare"
}
Response examples
The tabs below illustrate the example response for each example payload provided above.
Host example response
{
"bulkId": "31ab4312-b90a-49a6-baf8-70f87cd92a89"
}
Database example response
{
"bulkId": "5d129011-6254-413d-a365-6e394c06e277"
}
Table example response
{
"bulkId": "5d129011-6254-413d-a365-6e394c06e277",
"schemaDetection": {
"jobs": ["7d129033-6254-413d-a345-9o3242c06f242"]
},
"columnDetection" : {
"jobs": ["8d129033-6254-413d-a345-9o3123c06f123"]
}
}
View and Review Data Sources
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/test |
Refresh tags from an external catalog on a data source. The external catalog must be linked to the data source. |
GET | /dataSource/blobHandlerTypes |
Retrieve all blob handlers the current user is allowed to create. |
GET | /dataSource/byPurposes |
Get data sources that match a set of purposes. |
GET | /dataSource/featureStore |
Return the user's SQL connection parameters. |
GET | /dataSource/featureStore/connection |
Return the user's SQL connection string. |
GET | /dataSource/rpc/mine |
Retrieve all the data sources the current user has access to. |
GET | /dataSource/{dataSourceId}/activities |
Get all of the recent policy activities for a given data source. |
GET | /dataSource/{dataSourceId}/contacts |
Get the profiles for the data source owners and experts. |
GET | /dataSource/{dataSourceId}/tags |
Get the tags for a data source. |
GET | /dataSource/{dataSourceId}/{columnName}/unmaskUsers |
Return the users who can unmask the given column. |
Refresh external catalog tags on a data source
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/{dataSourceId}/test |
Refreshes tags from an external catalog on a data source. The external catalog must be linked to the data source. |
Query Parameters
Attribute | Description | Required |
---|---|---|
dataSourceId | integer The data source ID. |
Yes |
Response Schema
Attribute | Description |
---|---|
blob | object Indicates whether or not the blob was successfully crawled. |
columnEvolution | object Indicates whether or not the job run to check for columns added or removed from the data source passed and when it was last run. |
externalCatalog | object Indicates whether or not the external catalog was successfully linked to the data source. |
fingerprint | object Indicates whether or not the fingerprint job was successful (passed ) and when it was last run. The fingerprint captures summary statistics of the data source. |
framework | object Indicates whether or not the classification was successfully run on the data source to determine its sensitivity. |
globalPolicy | object Indicates whether or not global policies were successfully applied to the data source. |
highCardinality | object Indicates whether or not the job run to calculate the data source's high cardinality column passed and when it was last run. |
schemaEvolution | object Indicates whether or not the job run to check if a new table had been added in the remote database passed and when it was last run. If a new table was added, Immuta automatically creates a new data source. Correspondingly, if a remote table is removed, that data source will be disabled in the console. |
sdd | object Indicates whether or not sensitive data discovery was successfully run on the data source. |
sql | object Indicates whether or not the SQL query run to check the data source's health passed and when it was last run. |
stats | object Indicates whether or not the job run to calculate the number of rows in the data source passed and when it was last run. |
Request example
The following request refreshes external catalog tags on the data source.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/23/test
Response example
{
"sql": {
"status": "passed",
"message": "Passed"
},
"stats": {
"status": "passed",
"lastAttempted": "2021-09-09T13:43:43.948Z"
},
"fingerprint": {
"status": "passed",
"lastAttempted": "2021-07-22T14:12:01.525Z"
},
"lastAttempt": {
"date": "2021-09-09T16:47:05.173Z"
},
"columnEvolution": {
"status": "passed",
"lastAttempted": "2021-09-08T16:36:05.557Z"
},
"highCardinality": {
"status": "passed",
"lastAttempted": "2021-07-22T14:11:58.439Z"
},
"schemaEvolution": {
"status": "passed",
"lastAttempted": "2021-09-08T16:35:57.867Z"
},
"status": "passed"
}
Retrieve blob handlers
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/blobHandlerTypes |
Retrieve all blob handlers the current user is allowed to create. |
Query Parameters
None.
Response Schema
Attribute | Description |
---|---|
name | string The name of the blob handler. |
baseUrl | string The base URL for the data source. |
config | array Includes information about the connection configuration. |
port | integer The port number. |
driver | string The name of the driver. |
Request example
The following request retrieves all blob handlers the current user is allowed to create.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/blobHandlerTypes
Response example
{
"name": "Azure Synapse Analytics",
"pluginType": "odbcHandler",
"clientUrl": "/asa",
"baseUrl": "https://0.0.0.0:8823/asa",
"config": {
"port": 1433,
"allowSSLToggle": false
},
"displayOrder": 41,
"requiresHashPhraseForDP": true,
"driver": "Azure Synapse Analytics"
}
Get data sources by purpose
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/byPurposes |
Get data sources that match a set of purposes. |
Query Parameters
Attribute | Description | Required |
---|---|---|
purposes | array[string] The purposes to filter the data sources by. |
Yes |
excludedProjects | array[integer] Excludes data sources associated with specified project IDs. |
No |
Response Schema
Attribute | Description |
---|---|
id | integer The data source ID. |
name | array The name of the data source. |
policyId | integer The policy ID. |
restrictions | array Details regarding the operator (and or or ) and purposes . |
Request example
The following request gets data sources that match a set of purposes.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/byPurposes?purposes=Data+Analysis
Response example
[{
"id": 42,
"name": "Army Army Records",
"policyId": 56,
"restrictions": [{
"operator": "and",
"purposes": ["Data Analysis"]
}]
}]
Return SQL connection parameters
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/featureStore |
Return the user's SQL connection parameters as JSON. |
Query Parameters
None.
Response Schema
Attribute | Description |
---|---|
host | string The host of the Immuta database. |
port | integer The port number of the Immuta database. |
database | string The name of the Immuta database. |
user | string The SQL username. |
sslmode | string Indicates whether or not SSL is required. |
Request example
The following request returns the user's connection parameters for the feature store as JSON.
curl \
--request GET \
--header "Content-Type: application/json" \
--header "Authorization: Bearer dea464c07bd07300095caa8" \
https://your-immuta-url.com/dataSource/featurestore
Response example
{
"protocol": "postgresql://",
"host": "last-test-immuta-query-engine.iim",
"port": 5432,
"database": "immuta",
"user": "first.last",
"ssl": {
"rejectUnauthorized": false
},
"sslmode": "require"
}
Return SQL connection string
Endpoint
Method | Path | Purpose |
---|---|---|
GET | /dataSource/featureStore/connection |
Return the user's SQL connection string. |
Query Parameters
None.
Response Schema
Attribute | Description |
---|---|
connectionString | string The connection string. |
host | string The host for the Immuta database. |
port | integer The port number for the Immuta database. |
database | string The Immuta database name. |
sslMode | string Indicates whether or not SSL is required. |
sqlUser | string The SQL username. |
Request example
The following request returns the user's connection string or the feature store.
curl \
--request GET \
--header