Product Data with Web Browser and JSON

For this guide, we're going to assume the following:

Your goal:
You're interested in using Datafiniti's product data to analyze trends in the women's luxury shoe market. You're a data scientist that's been tasked with the following:

1. Collect pricing data on women's luxury shoes from multiple online retailers.
2. Sort the data by brand.
3. Determine the average price of each brand.

Your environment and data needs:

1. You're working with your web browser
2. You want to work with JSON data.

Here are the steps we'll take:

1. Use Chrome and add the JSONView extension

If you want to use your web browser to access the Datafiniti API, we recommend using Chrome. In our experience, Chrome is the most stable browser. It does a good job of displaying results from the Datafiniti API, particularly when coupled with the JSONView extension, which you can install here. This extension will make responses from the Datafiniti API look pretty and more readable!

2. Get your API token

The first thing you'll need is your API token. The API token lets you authenticate with Datafiniti API and tells it who you are, what you have access to, and so on. Without it, you can't use the API.

To get your API token, go the Datafiniti Web Portal (https://portal.datafiniti.co), login, and click on your account name and the top-right. From there, you'll see a link to the "My Account" page, which will take you to a page showing your token. Your API token will be a long string of letters and numbers. Copy the API token or store it somewhere you can easily reference.

📘

For the rest of this document, we'll use AAAXXXXXXXXXXXX as a substitute example for your actual API token when showing example API calls.

3. Run your first search

Open a new window in your browser. The first thing we'll do is do a test search that will give us a sense for what sort of data might be available. Eventually we'll refine our search so that we get back the most relevant data.

Since we want women's luxury shoes, let's try a simple search that will just give us listings for shoes sold online.

Copy and paste the following URL into your browser (replace the dummy API token with your real API token):

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/data/products?view=products_all&format=JSON&q=categories:shoes&records=1&download=false

You should get a response similar to this:

{
  "estimated total": 884885,
  "records": [
    {
      "asins": [
        "B010XYTFM6"
      ],
      "brand": "inktastic",
      "categories": [
        "Novelty",
        "Tops & Tees",
        "Novelty & More",
        "Women",
        "Clothing, Shoes & Jewelry",
        "T-Shirts",
        "Clothing"
      ],
      "dateAdded": "2015-11-09T01:55:09Z",
      "dateUpdated": "2016-04-12T16:19:26Z",
      "imageURLs": [
        "http://ecx.images-amazon.com/images/I/41W6xSgsBbL._SX342_QL70_.jpg",
        "http://ecx.images-amazon.com/images/I/41StiamgorL._SR38,50_.jpg",
        "http://ecx.images-amazon.com/images/I/41StiamgorL._SX342_QL70_.jpg",
        "http://ecx.images-amazon.com/images/I/41W6xSgsBbL._SR38,50_.jpg"
      ],
      "keys": [
        "inktasticwomensbutterflybanjochickjuniorvnecktshirts/b010xytfm6"
      ],
      "name": "Inktastic Women's Butterfly Banjo Chick Junior V-neck T-shirts",
      "sourceURLs": [
        "http://www.amazon.com/Inktastic-Womens-Butterfly-Junior-T-Shirts/dp/B010XYTK1W",
        "http://www.amazon.com/Inktastic-Womens-Butterfly-Junior-T-Shirts/dp/B016HKYYN0",
        "http://www.amazon.com/Inktastic-Butterfly-T-Shirts-Athletic-Heather/dp/B016HKYPYS"
      ],
      "id": "AVkzMzokUmTPEltRlaJ_"
    }
  ]
}

Let's break down what the API call is all about:

Now let's dive through the response the API returned:

Within the records field, you'll see a single product returned with multiple fields and their values associated with that product. The JSON response will show all fields that have a value. It won't show any fields that don't have a value. E.g., the example product shown has no prices field showing, so that means the database doesn't have any pricing information for it (we can update our request later to make sure we only get products that do have prices).

Each product record will have multiple fields associated with it. You can see a full list of available fields in our Product Data Schema.

4. Refine your search

If you take a look at the sample record shown above, you'll notice that it's not actually a pair of shoes. It's actually a shirt. It was returned as a match because its category keywords included Clothing, Shoes & Jewelry. If we downloaded all matching records, we would find several products that really are shoes, but we'd also find other products like this one, which aren't.

We'll need to refine our search to make sure we're only getting shoes. To do that, we can add additional filters to the q parameter to narrow down the results. For example:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/data/products?view=products_all&format=JSON&q=categories:shoes AND -categories:shirts AND categories:women AND (brand:* OR manufacturer:*) AND prices:*&records=10&download=false

This API call is different in a few ways:

1. It uses AND -categories:shirts to filter out any products that might be shirts. Note the - in front of categories.
2. It adds AND categories:women to narrow down results to just products for women. (We were interested in just women's shoes.)
3. It adds AND (brand:* OR manufacturer:*). This ensures the brand or manufacturer field is filled out in all the records I request. We call the * a "wildcard" value. Matching against a wildcard is a useful way to ensure the fields you're searching aren't empty.
4. It adds AND prices:*. Again, matching against a wildcard here means we're sure to only get products that have pricing information.
5. It changes records=1 to records=10 so we can look at more sample matches.

Notice how Datafiniti lets you construct very refined boolean queries. In the API call above, we're using a mix of AND and OR to get exactly what we want.

Update your API call in Chrome and hit enter to see the updated results.

5. Initiate a full download of the data

Once we like what we see from the sample matches, it's time to download the entire data set! To do this, we're going to make three changes to our API call:

1. Remove &records=10.
2. Change download=false to download=true.

After those changes, the API call looks like:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/data/products?view=products_all&format=JSON&q=categories:shoes AND -categories:shirts AND categories:women AND (brand:* OR manufacturer:*) AND prices:*&download=true

When you make this API call, you'll see a response similar to:

[
  {
    "id": 6073,
    "records": "",
    "query": "categories:shoes AND -categories:shirts AND categories:women AND (brand:* OR manufacturer:*) AND prices:*",
    "dataType": "products",
    "dataFormat": "json",
    "accountId": "AAAXXXXXXXXXXXX",
    "planId": "free",
    "view": "products_all",
    "async": 1,
    "status": "STARTED",
    "error": "",
    "numDownloaded": 0,
    "numFound": 7537,
    "numRequested": 7537,
    "date_started": "2017-1-11 15:52:42",
    "date_completed": "2017-1-11 15:52:43"
  }
]

Here's what these fields mean:

6. Monitor the status of the download

As the download request runs, you can check on its status by using this API call:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/requests/XXXX

You'll want to replace XXXX with the id value for your request. If you keep making the above API call, you'll see numDownloaded gradually increase.

7. Download the result file(s)

Once the download finishes, the /requests/ call will show COMPLETED for status. You can then get links to the data files by making this API call:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/results/XXXX

(Again, replace XXXX with the actual request id.) The response will look similar to:

[
  {
    "id": 120138,
    "url": "http://datafiniti-downloads-qa.s3.amazonaws.com/AAAXXXXXXXXXXXX/6073_1.txt?AWSAccessKeyId=AKIAIXQMCWHOZB3O35SA&Signature=2YtBsW9xY8CZrWDECcdLzyx4Jlk%3D&Expires=1484754763",
    "request_id": 6073,
    "downloaded": 0,
    "imported": 0,
    "date_created": "2017-1-11"
  }
]

If you've requested a lot of records (i.e., over 10,000), you may see more than 1 result object shown.

To download the result files, copy each url value and paste it into your browser. Your browser will initiate a download to your computer.

8. Open the result file(s)

Navigate to the file you downloaded and open it.

📘

The JSON data will actually be a text file, instead of a single JSON object. Each line in the text file is a JSON object. We format the data this way because most programming languages won't handle parsing the entire data set as a JSON object with their standard system calls very well.

From here, you'll most likely want to process the data using the programming language of your choice. Our other JSON guides provide instructions on how to process this data.