Business Data with Postman and CSV
For this guide, we're going to assume you're interested in using Datafiniti's business data to do some marketing analysis on hotels. Let's say you're a data scientist that's been tasked with the following:
- Collect data on hotels.
- Sort the data by state.
- Find which states have the most hotels.
Your environment and data needs:
- You're working with Postman.
- You want to work with CSV data.
Here are the steps we'll take:
1. Download Postman
We recommend using Postman when accessing the Datafiniti API. It's a great client for using any sort of RESTful API and works particularly well with Datafiniti. You can download Postman here.
2. Download Postman collection
To get your API token, go the Datafiniti Web Portal (https://portal.datafiniti.co), login, and click your settings in the left navigation bar. From there, you'll see a page showing your token. Your API token will be a long string of letters and numbers. Copy the API token or store it somewhere you can easily reference.
The collection will be pre-populated with several API calls that use your API token.
Import this collection into Postman. You can do this by opening Postman and clicking on "Import" at the top-left of your screen.
For the rest of this document, we'll use
AAAXXXXXXXXXXXX
as a substitute example for your actual API token when showing example API calls.
3. Run your first search
The first thing we'll do is do a test search that will give us a sense for what sort of data might be available. Eventually we'll refine our search so that we get back the most relevant data.
Since we want hotels, let's try a simple search that will just give us online listings for hotels.
With Postman open, click on the Datafiniti API V4 collection. Then navigate to Secured Endpoints > Search > Search Business Data. Click on "Body" in the main screen.
Enter the following in the text area:
{
"query": "categories:hotels",
"num_records": 1
}
Click the blue "Send" button. You should get a response similar to this:
{
"num_found": 139666,
"total_cost": 1,
"records": [
{
"address": "7030 Amin Dr",
"categories": [
"Hotels"
],
"city": "Chattanooga",
"country": "US",
"dateAdded": "2016-11-04T23:50:14Z",
"dateUpdated": "2016-11-04T23:50:14Z",
"descriptions": [
{
"dateSeen": [
"2016-11-09T19:26:53Z"
],
"sourceURLs": [
"http://www.hotels.com/ho141351/?locale=en_US&pos=HCOM_US"
],
"value": "No-frills hotel in Hamilton Place with health club"
}
],
"features": [
{
"key": "Services",
"value": [
"24-hour front desk, Dry cleaning/laundry service, Laundry facilities, Free newspapers in lobby"
]
},
{
"key": "Nearby Attractions",
"value": [
"[In Hamilton Place, Hamilton Place Mall (1.1 mi / 1.7 km), Dragon Dreams Museum (2.7 mi / 4.4 km), Tennessee Valley Railroad Museum (3.6 mi / 5.8 km), Concord Golf Club (3.8 mi / 6 km), Brown Acres Golf Course (4 mi / 6.5 km)]"
]
}
],
"imageURLs": [
"https://exp.cdn-hotels.com/hotels/1000000/120000/117400/117332/117332_95_n.jpg",
"https://exp.cdn-hotels.com/hotels/1000000/120000/117400/117332/117332_107_n.jpg"
],
"keys": [
"us/tn/chattanooga/7030amindr/1104338646"
],
"latitude": "35.04281",
"longitude": "-85.158",
"name": "Mainstay Suites Chattanooga",
"numRoom": 77,
"phones": [
"8004916126"
],
"postalCode": "37421",
"province": "TN",
"reviews": [
{
"date": "2013-11-11T00:00:00Z",
"dateAdded": "2016-11-04T23:50:14Z",
"dateSeen": [
"2016-11-06T00:00:00Z",
"2016-08-12T00:00:00Z"
],
"rating": 2,
"sourceURLs": [
"https://www.hotels.com/hotel/141351/reviews%20/"
],
"text": "Hotel was ok. Room not as up to date as I expected.",
"title": "Was ok",
"username": "A Traveler"
},
{
"date": "2014-11-03T00:00:00Z",
"dateAdded": "2016-11-04T23:50:14Z",
"dateSeen": [
"2016-08-09T00:00:00Z",
"2016-08-27T00:00:00Z",
"2016-07-18T00:00:00Z"
],
"rating": 3,
"sourceURLs": [
"https://www.hotels.com/hotel/141351/reviews%20/"
],
"text": "It was adequate for our one night stay there. Staff was very friendly and the room was clean but not very big. I would recommend to someone for a very short stay.",
"username": "scott"
}
],
"sourceURLs": [
"http://www.hotels.com/ho141351/?locale=en_US&pos=HCOM_US",
"https://www.hotels.com/hotel/141351/reviews%20/"
],
"id": "AVwcsllU_7pvs4fzx-yW"
}
]
Let's break down each of the parameters we sent in our request:
API Call Component | Description |
---|---|
"query": "categories:hotels" | query tells the API what you want to search. In this case, you're telling the API you want to search by categories . Any business that has hotels listed in its categories field will be returned. |
"num_records": 1 | num_records tells the API how many records to return in its response. |
Now let's dive through the response the API returned:
Response Field | Description |
---|---|
"num_found" | The total number of available records in the database that match your query. If you end up downloading the entire data set, this is how many records you'll use. |
"total_cost" | The number of credits this request has cost you. Business records only cost 1 credit per record. |
"records" | The first available matches to your query. If there are no matches, this field will be empty. Within each record returned, you'll see multiple fields shown. This is the data for each record. |
Within the records
field, you'll see a single business returned with multiple fields and the values associated with that business. The JSON response will show all fields that have a value. It won't show any fields that don't have a value.
Each business record will have multiple fields associated with it. You can see a full list of available fields in our Business Data Schema.
4. Refine your search
If you think about the original query we made, you'll realize we didn't really specify which geography we're interested in. Since we only want US hotels, we should narrow our search appropriately. Modify your request body to look like this:
{
"query": "categories:hotels AND country:US",
"num_records": 10
}
This API call is different in a couple ways:
- It adds
AND country:US
to narrow down results to just US hotels. - It changes
records=1
torecords=10
so we can look at more sample matches.
Datafiniti lets you construct very refined boolean queries. If you wanted to do more complicated searches, you could use OR operations, negation, and more.
Hit "Send" to see the updated results.
If you would like to narrow your search to just exact matches you can place the search term in quotation marks.
{
"query": "name:\"Datafiniti Inc\"",
"num_records": 10
}
The above query will only return businesses with the name Datafiniti Inc. The quotation marks need to be escaped using back slashes since they are already within another set of quotation marks.
5. Initiate a download of the data
Once we like what we see from the sample matches, it's time to download a larger data set! To do this, we're going to further modify our request body to look like this:
When using the API, you will not receive any warning if you are going past your monthly record limit. Keep a track on how many records you have left by checking your account. You are responsible for any overage fees if you go past your monthly limit.
{
"query": "categories:hotels AND country:US",
"format": "csv",
"view": "business_basic",
"num_records": 50,
"download": true
}
Here's what we changed:
- We added
"format": "csv
. If you don't specifyformat
, it will default tojson
. Using the CSV format will make analyzing our data easier for this example. - We change
"num_records": 10
to"num_records": 50
. This will download the first 50 matching records. If we wanted to download all matching records, we would removenum_records
. - We added
"view": "business_basic"
. If you don't specifyview
, it will use the default view, which shows all available fields. Usingbusiness_basic
will show a more limited set of fields and nest fields like categories and features into a single cell (instead splitting them across multiple rows and columns). - We added
"download": true
. This tells the API to issue a download request instead of a search request.
If num_records is not specified, ALL of the records matching the query will be downloaded.
When you make this API call, you'll see a response similar to:
{
"id": 7,
"results": [],
"user_id": 15,
"status": "running",
"date_started": "2017-11-16 17:46:06.0",
"num_downloaded": 0,
"data_type": "business",
"query": "categories:hotels AND country:US",
"format": "csv",
"num_records": 50,
"total_cost": 50
}
We'll explain each of these fields in the next section.
6. Monitor the status of the download
As the download request runs, you can check on its status by using the "Get Download" call in your Postman collection. Just replace :id
with the id
value you saw in the previous step.
If you keep running this call, you'll see some of the values update. Once the download completes, it will look something like this:
{
"id": 7,
"results": [
"https://datafiniti-downloads.s3.amazonaws.com/15/7_1.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20171116T174607Z&X-Amz-SignedHeaders=host&X-Amz-Expires=604800&X-Amz-Credential=AKIAJYCTIF46QVBTXWYA%2F2017xxxx%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=ecf13f1bb4b7adfdde1a99143541afd1d12347292eb9ec3f6ed1316c64d4eekf"
],
"user_id": 15,
"status": "completed",
"date_started": "2017-11-16 17:46:06.0",
"date_updated": "2017-11-16 17:46:07.0",
"num_downloaded": 50,
"data_type": "business",
"query": "categories:hotels AND country:US",
"format": "csv",
"num_records": 50,
"total_cost": 50
}
Here's what these fields mean:
Response Field | Description |
---|---|
id | This is a unique identifier for the request. |
results | This is a list of links for all the result files generated for this data set. When you first issue the download request, it will be an empty list, but it will populate as the download progresses. |
user_id | This is an internal id for your user account. |
status | This indicates the status of your download. It will be set to completed once the download has finished. |
date_started | The date and time the download started. |
date_updated | The last time the download information was updated. |
num_downloaded | The number of records that have been downloaded so far. |
data_type | The data type you queried. |
query | The query you ran. |
format | The data format you requested. |
num_records | The total number of records that will be downloaded. |
total_cost | The total number of credits used by this download. Business records only cost 1 credit per record. |
7. Download the result file(s)
Once the download response shows "status": "completed"
, you can download the data using the URLs in the results
field.
If you've requested a lot of records (i.e., over 10,000), you may see more than 1 result object shown.
To download the result files, copy each url
value and paste it into your browser. Your browser will initiate a download to your computer.
8. Open the result file(s) in Excel
Navigate to the file you downloaded and open it. Since it's a CSV file, it should open in Excel automatically. It will look something like:
9. Analyze the results
Using Excel, we can easily count the total number of hotels on a state-by-state basis. The province
column gives us where each hotel is located, so we can use it to tally up the numbers.
Updated about 1 year ago