Data Automation

Overview

Data Automations in Marq allow you to connect external data sources to your templates, enabling dynamic content generation at scale. This guide covers how to set up data sets, sources, collections, and items using the Marq Data API to automate your content workflows.

Understanding Data Automation Architecture

Marq's Data Automation follows a hierarchical structure:

Data Set: The top-level container that holds all related data
Data Source: Defines how data flows into the system (API, CSV upload, etc.)
Collection: A table-like structure that defines the schema for your data
Data Items: Individual records within a collection
Data Set Grants: Permission controls for accessing data

Quick Start: Creating Your First Data Automation

Follow these steps to create a complete data automation setup:

Step 1: Create a Data Set

A data set acts as the main container for your automation:

POST https://data.app.marq.com/dataSets
Accept: application/json;v=1.0
Content-Type: application/json

{
  "name": "Property Listings",
  "properties": {
    "Type": "USER_PUSHED"
  }
}

Example Response:

{
  "id": 6001,
  "name": "Property Listings",
  "uri": "https://data.app.marq.com/dataSets/6001",
  "properties": {
    "Type": "USER_PUSHED"
  },
  "createdAt": "2024-07-09T10:30:00Z"
}

Step 2: Create Data Set Permissions

Grant access to the data set for your account or specific users:

POST https://data.app.marq.com/dataSetGrants
Accept: application/json;v=1.0
Content-Type: application/json

{
  "dataSet": "https://data.app.marq.com/dataSets/6001",
  "permissionType": "account",
  "identifier": "164874301",
  "role": "edit"
}

Permission Types:

account: Grant access to an entire account
user: Grant access to a specific user

Available Roles:

view: Read-only access
edit: Full read/write access

Step 3: Create a Data Source

Define how data flows into your data set:

POST https://data.app.marq.com/dataSources
Accept: application/json;v=1.0
Content-Type: application/json

{
  "name": "Property Feed",
  "product": "press",
  "adapterType": "USER_PUSHED",
  "dataSet": "https://data.app.marq.com/dataSets/6001"
}

Adapter Types:

USER_PUSHED: Manual data entry via API
CSV_UPLOAD: Data from CSV file uploads
API_INTEGRATION: Third-party API connections

Step 4: Create a Collection

Define the schema for your data:

POST https://data.app.marq.com/collections
Accept: application/json;v=1.0
Content-Type: application/json

{
  "name": "Properties",
  "dataSource": "https://data.app.marq.com/dataSources/715970",
  "schema": [
    {
      "name": "mlsNumber",
      "fieldType": "STRING",
      "isPrimary": true,
      "order": 1
    },
    {
      "name": "address",
      "fieldType": "STRING",
      "isPrimary": false,
      "order": 2
    },
    {
      "name": "price",
      "fieldType": "NUMBER",
      "isPrimary": false,
      "order": 3
    },
    {
      "name": "propertyImage",
      "fieldType": "IMAGE",
      "isPrimary": false,
      "order": 4
    }
  ],
  "properties": {
    "ImageFields": "propertyImage"
  }
}

Field Types:

STRING: Text data
NUMBER: Numeric values
IMAGE: Image URLs
DATE: Date values
BOOLEAN: True/false values

Step 5: Add Data Items

Populate your collection with actual data:

POST https://data.app.marq.com/collections/23624512/items
Accept: application/json;v=1.0
Content-Type: application/json

[
  {
    "mlsNumber": "MLS123456",
    "address": "123 Main Street, Anytown, CA 90210",
    "price": 750000,
    "propertyImage": "https://images.example.com/property1.jpg"
  },
  {
    "mlsNumber": "MLS789012",
    "address": "456 Oak Avenue, Somewhere, CA 90211",
    "price": 950000,
    "propertyImage": "https://images.example.com/property2.jpg"
  }
]

Working with Data Sets

List All Data Sets

Retrieve all data sets accessible to your account:

GET https://data.app.marq.com/dataSets
Accept: application/json;v=1.0

Get a Specific Data Set

GET https://data.app.marq.com/dataSets/6001
Accept: application/json;v=1.0

Update Data Set Properties

PATCH https://data.app.marq.com/dataSets/6001/properties
Accept: application/json;v=1.0
Content-Type: application/json

{
  "Type": "PropertyListing",
  "LastUpdated": "2024-07-09T15:30:00Z"
}

Managing Collections

Get Collection Schema

Retrieve the structure of a collection:

GET https://data.app.marq.com/collections/23624512/schema
Accept: application/json;v=1.0

Update Collection Schema

Add or modify fields in your collection:

PATCH https://data.app.marq.com/collections/23624512/schema
Accept: application/json;v=1.0
Content-Type: application/json

[
  {
    "name": "bedrooms",
    "fieldType": "NUMBER",
    "collection": "https://data.app.marq.com/collections/23624512",
    "order": 5
  },
  {
    "name": "bathrooms",
    "fieldType": "NUMBER", 
    "collection": "https://data.app.marq.com/collections/23624512",
    "order": 6
  }
]

Create Metadata Collections

For advanced data partitioning (useful for multi-tenant scenarios):

POST https://data.app.marq.com/collections/23602919/metadata
Accept: application/json;v=1.0
Content-Type: application/json

{
  "name": "Office Metadata Collection",
  "metadataType": "PropertyListingsPartition",
  "schema": [],
  "properties": {
    "InheritsSchemaChanges": "true",
    "ExternalOfficeId": "office123",
    "ExternalCompanyId": "company456"
  }
}

Working with Data Items

Get All Items from a Collection

GET https://data.app.marq.com/collections/23624512/items
Accept: application/json;v=1.0

Find Items by Key-Value Pairs

Search for specific data items:

POST https://data.app.marq.com/collections/23624512/itemsByKey
Accept: application/json;v=1.0
Content-Type: application/json

{
  "keySchemaFields": ["mlsNumber"],
  "targetValues": [
    {
      "fieldValues": ["MLS123456"]
    }
  ]
}

Update Multiple Items

PUT https://data.app.marq.com/collections/23624512/items?safe=true
Accept: application/json;v=1.0
Content-Type: application/json

[
  {
    "mlsNumber": "MLS123456",
    "address": "123 Main Street, Anytown, CA 90210",
    "price": 775000,
    "propertyImage": "https://images.example.com/property1-updated.jpg"
  }
]

Note: The safe=true parameter prevents accidental overwrites by requiring exact matches.

Update a Single Item

PATCH https://data.app.marq.com/collections/23624512/items/12471
Accept: application/json;v=1.0
Content-Type: application/json

{
  "uri": "https://data.app.marq.com/collections/23624512/items/12471",
  "collection": "https://data.app.marq.com/collections/23624512",
  "fields": {
    "price": 800000,
    "propertyImage": "https://images.example.com/property1-new.jpg"
  }
}

Delete Data Items

DELETE https://data.app.marq.com/collections/23624512/items/12471
Accept: application/json;v=1.0

Using Data Automation in Projects

Apply Data to a Project

When creating a project, use dataSmartFields to automatically populate template fields:

POST https://api.marq.com/v1/projects
Content-Type: application/json

{
  "templateId": "499caa4f-34df-4e7a-b903-e27af918b545",
  "creatorId": 164252231,
  "title": "Property Listing Flyer",
  "dataSmartFields": [
    {
      "strict": false,
      "dataSetName": "Property Listings",
      "primaryKey": "MLS123456"
    }
  ]
}

Using Column-Based Filtering

Instead of a primary key, filter by specific column values:

POST https://api.marq.com/v1/projects
Content-Type: application/json

{
  "templateId": "499caa4f-34df-4e7a-b903-e27af918b545",
  "creatorId": 164252231,
  "title": "Property Listing Flyer",
  "dataSmartFields": [
    {
      "strict": true,
      "dataSetName": "Property Listings",
      "columns": [
        {
          "column": "address",
          "value": "123 Main Street, Anytown, CA 90210"
        }
      ]
    }
  ]
}

Strict Mode:

strict: false - Create project even if no matching data is found
strict: true - Only create project if matching data exists

Best Practices

Data Schema Design

Use descriptive field names: Make field names clear and consistent
Set primary keys: Always define a primary key for unique identification
Optimize field types: Use appropriate field types for better performance
Plan for growth: Design schemas that can accommodate future data needs

Data Management

Regular updates: Keep data current with scheduled updates
Batch operations: Use bulk operations for large data sets
Error handling: Implement robust error handling for data operations
Backup strategies: Maintain backups of critical data

Performance Optimization

Index primary keys: Ensure primary keys are properly indexed
Limit payload size: Keep API payloads under 1MB when possible
Use pagination: Implement pagination for large result sets
Cache frequently accessed data: Cache commonly used data items

Common Use Cases

Real Estate Listings

{
  "name": "Property Listings",
  "schema": [
    {"name": "mlsNumber", "fieldType": "STRING", "isPrimary": true},
    {"name": "address", "fieldType": "STRING"},
    {"name": "price", "fieldType": "NUMBER"},
    {"name": "bedrooms", "fieldType": "NUMBER"},
    {"name": "bathrooms", "fieldType": "NUMBER"},
    {"name": "squareFootage", "fieldType": "NUMBER"},
    {"name": "propertyImage", "fieldType": "IMAGE"},
    {"name": "agentName", "fieldType": "STRING"},
    {"name": "agentPhoto", "fieldType": "IMAGE"}
  ]
}

Product Catalogs

{
  "name": "Product Catalog",
  "schema": [
    {"name": "sku", "fieldType": "STRING", "isPrimary": true},
    {"name": "productName", "fieldType": "STRING"},
    {"name": "price", "fieldType": "NUMBER"},
    {"name": "description", "fieldType": "STRING"},
    {"name": "category", "fieldType": "STRING"},
    {"name": "productImage", "fieldType": "IMAGE"},
    {"name": "inStock", "fieldType": "BOOLEAN"}
  ]
}

Employee Directory

{
  "name": "Employee Directory",
  "schema": [
    {"name": "employeeId", "fieldType": "STRING", "isPrimary": true},
    {"name": "fullName", "fieldType": "STRING"},
    {"name": "title", "fieldType": "STRING"},
    {"name": "department", "fieldType": "STRING"},
    {"name": "email", "fieldType": "STRING"},
    {"name": "phone", "fieldType": "STRING"},
    {"name": "profilePhoto", "fieldType": "IMAGE"},
    {"name": "startDate", "fieldType": "DATE"}
  ]
}

Error Handling

Common Error Responses

Data Set Not Found (404):

{
  "error": "Data set not found",
  "code": "DATASET_NOT_FOUND",
  "message": "The specified data set does not exist or you don't have access to it."
}

Invalid Schema (400):

{
  "error": "Invalid schema",
  "code": "INVALID_SCHEMA",
  "message": "The provided schema contains invalid field types or missing required fields."
}

Permission Denied (403):

{
  "error": "Permission denied",
  "code": "PERMISSION_DENIED",
  "message": "You don't have permission to perform this operation on the specified resource."
}

Data Validation Error (422):

{
  "error": "Data validation failed",
  "code": "VALIDATION_ERROR",
  "message": "One or more data items failed validation.",
  "details": [
    {
      "item": 1,
      "field": "price",
      "error": "Value must be a number"
    }
  ]
}

Advanced Features

Metadata Collections

Use metadata collections for multi-tenant data partitioning:

POST https://data.app.marq.com/collections/23602919/metadata
Accept: application/json;v=1.0
Content-Type: application/json

{
  "name": "Regional Office Data",
  "metadataType": "PropertyListingsPartition",
  "properties": {
    "InheritsSchemaChanges": "true",
    "ExternalOfficeId": "west-coast-office",
    "ExternalCompanyId": "realty-company-123"
  }
}

Data Source Grants

Control access to specific data sources:

POST https://data.app.marq.com/sourceGrants
Accept: application/json;v=1.0
Content-Type: application/json

{
  "dataSource": "https://data.app.marq.com/dataSources/715970",
  "permissionType": "account",
  "identifier": "164874301",
  "role": "edit"
}

Integration Examples

Webhook Integration

Set up webhooks to automatically sync data when external systems change:

POST https://api.marq.com/v1/webhook
Content-Type: application/json

{
  "url": "https://your-system.com/webhooks/marq-data",
  "name": "Data Sync Webhook",
  "eventTypes": ["project.created", "data.updated"]
}

Automated Project Generation

Create projects automatically when new data items are added:

// Example webhook handler
app.post('/webhooks/marq-data', (req, res) => {
  const { eventType, data } = req.body;
  
  if (eventType === 'data.updated') {
    // Create project for new property listing
    createPropertyFlyer(data.mlsNumber);
  }
  
  res.status(200).send('OK');
});

Notes

Data sets support up to 100,000 items per collection
Image fields should contain publicly accessible URLs
Primary keys must be unique within a collection
Field names are case-sensitive
The safe=true parameter prevents accidental data overwrites
Collections inherit permissions from their parent data set