Product Data with Node.js and JSON

For this guide, we're going to assume the following:

Your goal:
You're interested in using Datafiniti's product data to analyze trends in the women's luxury shoe market. You're a data scientist that's been tasked with the following:

Collect pricing data on women's luxury shoes from multiple online retailers.
Sort the data by brand.
Determine the average price of each brand.

Your environment and data needs:

You're working with node.js.
You want to work with JSON data.

Here are the steps we'll take:

1. Install the request module for Node

In your terminal, run the following to install the request module for Node:

npm install request

2. Get your API token

The next thing you'll need is your API token. The API token lets you authenticate with Datafiniti API and tells it who you are, what you have access to, and so on. Without it, you can't use the API.

To get your API token, go the Datafiniti Web Portal (https://portal.datafiniti.co), login, and click on your account name and the top-right. From there, you'll see a link to the "My Account" page, which will take you to a page showing your token. Your API token will be a long string of letters and numbers. Copy the API token or store it somewhere you can easily reference.

📘
For the rest of this document, we'll use AAAXXXXXXXXXXXX as a substitute example for your actual API token when showing example API calls.

3. Run your first search

The first thing we'll do is do write some code that will run a test search. This test search will give us a sense for what sort of data might be available. Eventually we'll refine our search so that we get back the most relevant data. Since we want women's luxury shoes, let's try a simple search that will just give us listings for shoes sold online.

Write the following code in your code editor (replace the dummy API token with your real API token):

/*
  Illustrates an API call to Datafiniti's Product Database for shoes.
*/
var request = require('request');

// Set your API parameters here.
var APIToken = 'AAAXXXXXXXXXXXX';
var view = 'products_all';
var format = 'JSON';
var query = encodeURIComponent('categories:shoes');
var records = '1';
var download = 'false';

// Construct the API call.
var APICall = 'https://' + APIToken + ':@api.datafiniti.co/v3/data/products?'
				+ 'view=' + view
				+ '&q=' + query
				+ '&format=' + format
				+ '&records=' + records
				+ '&download=' + download;

// Make the API call.
request(
  {
    url : APICall
  },
  // Do something with the response.
  function (error, response, body) {
  	console.log(body);
  }
);

You should get a response similar to this (although it may not be as pretty in your terminal):

{
  "estimated total": 884885,
  "records": [
    {
      "asins": [
        "B010XYTFM6"
      ],
      "brand": "inktastic",
      "categories": [
        "Novelty",
        "Tops & Tees",
        "Novelty & More",
        "Women",
        "Clothing, Shoes & Jewelry",
        "T-Shirts",
        "Clothing"
      ],
      "dateAdded": "2015-11-09T01:55:09Z",
      "dateUpdated": "2016-04-12T16:19:26Z",
      "imageURLs": [
        "http://ecx.images-amazon.com/images/I/41W6xSgsBbL._SX342_QL70_.jpg",
        "http://ecx.images-amazon.com/images/I/41StiamgorL._SR38,50_.jpg",
        "http://ecx.images-amazon.com/images/I/41StiamgorL._SX342_QL70_.jpg",
        "http://ecx.images-amazon.com/images/I/41W6xSgsBbL._SR38,50_.jpg"
      ],
      "keys": [
        "inktasticwomensbutterflybanjochickjuniorvnecktshirts/b010xytfm6"
      ],
      "name": "Inktastic Women's Butterfly Banjo Chick Junior V-neck T-shirts",
      "sourceURLs": [
        "http://www.amazon.com/Inktastic-Womens-Butterfly-Junior-T-Shirts/dp/B010XYTK1W",
        "http://www.amazon.com/Inktastic-Womens-Butterfly-Junior-T-Shirts/dp/B016HKYYN0",
        "http://www.amazon.com/Inktastic-Butterfly-T-Shirts-Athletic-Heather/dp/B016HKYPYS"
      ],
      "id": "AVkzMzokUmTPEltRlaJ_"
    }
  ]
}

Let's break down what the API call is all about:

API Call Component	Description
`https://`	This is the communication protocol used by the API. It's the same one used when you visit a secure website.
`AAAXXXXXXXXXXXX:@`	Your API token. You're telling the API who you are so it can respond to your request.
`api.datafiniti.co`	The location of the API.
`/v3`	You're telling the API which version to use. `v3` is our most recent and current API version.
`/data`	You're telling the API you're interested in querying data.
`/products`	Specifically, you're interested in product data.
`view=products_all`	The `view` tells the API in which fields you want your response. `products_all` will show all available fields in a record.
`format=JSON`	The `format` tells the API which data format you want to see. You can set it to `JSON` or `CSV`.
`q=categories:shoes`	The `q` tells the API what query you want to use. In this case, you're telling the API you want to search by `categories`. Any product that has `shoe` listed in its `categories` field will be returned.
`records=1`	The `records` tells the API how many records to return in the its response. In this case, you just want to see 1 matching record.
`download=false`	The `download` tells the API if you want to initiate a download request or not. Setting it to `false` means you don't, so it will show the matching records immediately in the response.

Now let's dive through the response the API returned:

Response Field	Description
`"estimated_total"`	The total number of available records in the database that match your query. If you end up downloading the entire data set, this is how many records you'll use.
`"records"`	The first available matches to your query. For most queries, you'll see 1 to 10 example records. If there are no matches, this field will be empty. Within each record returned, you'll see multiple fields shown. This is the data for each record.

Within the records field, you'll see a single product returned with multiple fields and their values associated with that product. The record shows all fields that have a value. It won't show any fields that don't have a value. E.g., the example product shown has no prices field showing, so that means the database doesn't have any pricing information for it (we can update our request later to make sure we only get products that do have prices).

Each product record will have multiple fields associated with it. You can see a full list of available fields in our Product Data Schema.

4. Refine your search

If you take a look at the sample record shown above, you'll notice that it's not actually a pair of shoes. It's actually a shirt. It was returned as a match because its category keywords included Clothing, Shoes & Jewelry. If we downloaded all matching records, we would find several products that really are shoes, but we'd also find other products like this one, which aren't.

We'll need to refine our search to make sure we're only getting shoes. To do that, we can add additional filters to the q parameter to narrow down the results. For example, we could change our query value to the following:

/*
  Illustrates an API call to Datafiniti's Product Database for shoes,
  with some refinements to the query to get more relevant results.
*/
var request = require('request');

// Set your API parameters here.
var APIToken = 'AAAXXXXXXXXXXXX';
var view = 'products_all';
var format = 'JSON';
var query = encodeURIComponent('categories:shoes AND -categories:shirts AND categories:women AND (brand:* OR manufacturer:*) AND prices:*');
var records = '10';
var download = 'false';

// Construct the API call.
var APICall = 'https://' + APIToken + ':@api.datafiniti.co/v3/data/products?'
				+ 'view=' + view
				+ '&q=' + query
				+ '&format=' + format
				+ '&records=' + records
				+ '&download=' + download;

// Make the API call.
request(
  {
    url : APICall
  },
  // Do something with the response.
  function (error, response, body) {
  	console.log(body);
  }
);

This query is different in a few ways:

It uses AND -categories:shirts to filter out any products that might be shirts. Note the - in front of categories.
It adds AND categories:women to narrow down results to just products for women. (We were interested in just women's shoes.)
It adds AND (brand:* OR manufacturer:*). This ensures the brand or manufacturer field is filled out in all the records I request. We call the * a "wildcard" value. Matching against a wildcard is a useful way to ensure the fields you're searching aren't empty.
It adds AND prices:*. Again, matching against a wildcard here means we're sure to only get products that have pricing information.
It changes records=1 to records=10 so we can look at more sample matches.

Notice how Datafiniti lets you construct very refined boolean queries. In the query above, we're using a mix of AND and OR to get exactly what we want.

You can run the code above to see the difference in the results.

5. Initiate a full download of the data

Once we like what we see from the sample matches, it's time to download the entire data set! To do this, we're going to update our code a fair bit (an explanation follows):

var request = require('request');
var fs = require('fs');

// Set your API parameters here.
var APIToken = 'AAAXXXXXXXXXXXX';
var view = 'products_all';
var format = 'json';
var query = encodeURIComponent('categories:shoes AND -categories:shirts AND categories:women AND (brand:* OR manufacturer:*) AND prices:*');
var records = '10';
var download = 'true';

// Construct the API call.
var APICall = 'https://' + APIToken + ':@api.datafiniti.co/v3/data/products?'
				+ 'view=' + view
				+ '&q=' + query
				+ '&format=' + format
//			+ '&records=' + records
				+ '&download=' + download;

// A function to check if a download request has completed
function checkDownloadUntilComplete(options, callback) {
	var downloadRequestAPICall = 'https://' + APIToken + ':@api.datafiniti.co/v3/requests/' + options.requestID;

	request({url : downloadRequestAPICall}, function(error, response, body) {
		var downloadRequestResponse = JSON.parse(body);
		if (downloadRequestResponse[0].status !== 'COMPLETED') {
			// NEED A SLEEP FUNCTION HERE!
			console.log('Checking on status: ' + downloadRequestAPICall);
			checkDownloadUntilComplete(options, callback);
		} else {
			callback(null, downloadRequestResponse);
		}
	});
}

// Initiate the download request.
request({url : APICall}, function (error, response, body) {
  	var downloadResponse = JSON.parse(body);
  	var requestID = downloadResponse[0].id;

  	// Check on status of the download request.
  	checkDownloadUntilComplete ({requestID : requestID}, function (error, response) {
  		var resultsAPICall = 'https://' + APIToken + ':@api.datafiniti.co/v3/results/' + requestID;

  		// Once the download is complete, get all the links to result files and write those to local files.
  		request({url : resultsAPICall}, function(error, response, body) {
			var resultsResponse = JSON.parse(body);
			for (var i = 0; i < resultsResponse.length; i++) {
				var file = fs.createWriteStream(requestID + '_' + i + '.' + format);
				request(resultsResponse[i].url).pipe(file);
			}
		});
  	});
  }
);

A few things to pay attention to in the above code:

Remove &records=10. This tells the API to request all matching data.
Change download=false to download=true.
We've added code to (a) make the initial request, (b) check the status of the download, and (c) download the result files once they're ready.

Since we've handled multiple steps of the download process in this code, we won't go into the details here, but we do recommend you familiarize yourself with those steps. Checking them out in our Getting Started Guide for Product via a Web Browser will be helpful.

6. Parse the JSON data

The download code will save one or more result files to your project folder.

📘
The JSON data will actually be a text file, instead of a single JSON object. Each line in the text file is a JSON object. We format the data this way because most programming languages won't handle parsing the entire data set as a JSON object with their standard system calls very well.

We'll need to parse the file into an array of JSON objects. We can use code similar to this to handle the parsing:

var fs = require('fs');

// Set the location of your file here.
var file = 'xxxx_x.txt';

// A function to read in each line of the file and pass that line to func
function readLines(input, func) {
	var records = [];
	var remaining = '';

	input.on('data', function(data) {
 		remaining += data;
		var index = remaining.indexOf('\n');
		var last  = 0;
		while (index > -1) {
 			var line = remaining.substring(last, index);
 			last = index + 1;
 			func(line, records);
			index = remaining.indexOf('\n', last);
 		}

 		remaining = remaining.substring(last);
	});

	input.on('end', function() {
 		if (remaining.length > 0) {
 			func(remaining, records);
 		}
		processData(records);
	});
}

// A function that converts a line from the file, parses it to JSON, and stores it an array
function func(data, records) {
	var json = JSON.parse(data);
	records.push(json);
}

// This function is called once all the data has been read from the file.
function processData(records) {
	// Edit these lines to do more with the data.
	console.log(records);
}

var records = [];
var input = fs.createReadStream(file);
readLines(input, func);

You can edit the code in processData above to do whatever you'd like with the data, such as store the data in a database, write it out to your console, etc.