MuleSoft Integration With Marketo – Part-2 (Using Bulk API’s)

Author: Ashish Singh Chauhan

Marketo, Inc. is an American software company, headquartered in San Mateo, California. Marketo develops and sells marketing automation software for account-based marketing and other marketing services and products including SEO and content creation

Marketo is Software-as-a-Service (SaaS)-based marketing automation software owned by Adobe and built to help organizations automate and measure marketing engagement, tasks and workflows.

Based in San Mateo, Calif Marketo software aims to allow B2B and B2C marketers target qualified leads, produce lead-to-revenue opportunities and execute automated, personalized marketing campaigns across multiple digital channels. 

BULK API’s OF MARKETO

Marketo provides interfaces for retrieval of large sets of person and person related data, called Bulk Extract.  Currently, interfaces are offered for three object types:

• Leads (Persons)
• Activities
• Program Members
• Custom Objects

Bulk extract is performed by creating a job, defining the set of data to retrieve, enqueuing the job, waiting for the job to complete writing a file, and then retrieving the file over HTTP.  These jobs are executed asynchronously, and can be polled to retrieve the status of the export.

NOTE → Bulk API endpoints are not prefixed with ‘/rest’ like other endpoints.

For our integrations we are mainly concerned with following endpoints →

1)  Token – {{base_url}}/identity/oauth/token – GET

2) BULK Activities – Consist of 4 API’s –

1. Creating Job – {{base_url}}/bulk/v1/activities/export/create.json – POST
2. Enqueue Job – {{base_url}}/bulk/v1/activities/export/{exportId}/enqueue.json – POST
3. Status Check – {{base_url}}/bulk/v1/activities/export/{exportId}/status.json – GET
4. Download Job – /bulk/v1/activities/export/{exportId}/file.json – GET 

Prerequisites for Integration in Mule

1) The base_url (Marketo domain) must be provided by Clients..

2) All the API’s which are required for integrations must be put in properties files.

3) During integration keep in mind the following limitations of Marketo BULK API’s→ 

• Max Concurrent Export Jobs: 2
• Max Queued Export Jobs (inclusive of currently exporting jobs): 10
• File Retention Period: 10 days
• Default Daily Export Allocation: 500MB (which resets daily at 12:00AM CST).  Increases available for purchase.
• Max Time Span for Date Range Filter (createdAt or updatedAt): 31 days

4) For generating the Token the client credentials( client id and client secret) must be provided.

Integration in Mule

Step 1 -> Generate the Authorization token because it is required to access Marketo.

URL – {{base_url}}/identity/oauth/token

Set this url and query parameters in variables of transform message and pass them in Http Request.

You will get a token like this →

Step 2 → Attach this token with “Bearer” into the Headers section for future requests.

Step 3 → Set the payload, url, method, headers for creating job.

Url – {{base_url}}/bulk/v1/activities/export/create.json

Method – POST

Payload → 

Format → Specify the type of exported file.

activityTypeIds → 10 indicates the open email scenario.

For more information on activityTypeIds visit on this link →

https://developers.marketo.com/rest-api/bulk-extract/bulk-activity-extract/

Pass all this information to Http Request.

The response will be like ->

Step 4 → The next step is to Enqueue the Job. Catch the exportId from the previous step and append in the Url of Enqueue Job.

Url – {{base_url}}/bulk/v1/activities/export/{exportId}/enqueue.json

Method – POST

Pass this to Http Request. This will Enqueue the job with Particular export Id.

The response will be like →

Status – Queued indicates that marketo started generating the export file.

Step 5 → Now check the status of the job which you enqueued in the previous step. Pass the export id in the url to check its status.

The status might be →

Queued – indicates that marketo is still generating file.

Completed – indicates that file generation is completed.

Url – {{base_url}}/bulk/v1/activities/export/{exportId}/status.json

Method – GET

Pass this to Http Request with corresponding url and method.

The response will look like →

The response displays the File size, Number of records and File checksum once it status becomes Completed.

Step 6 → The final step is to download the exported file. This step is complicated because the download will happen in chunks so it needs to be streamed. For this either you will use a python script to download the file or write lambda expression in AWS.

You can download the file from Postman directly by selecting the Send And Download button.

Url –  {{base_url}}/bulk/v1/activities/export/{exportId}/file.json

Method – GET

It will pop up a screen. You will select where you want to store your downloaded file and also change the name if you want.

NOTE → We are demonstrating bulk extract activities in this document. For leads and program members you can change the url and it will work in the same way as mentioned in above steps.