Property Data with Postman and CSV

For this guide, we're going to assume you're interested in using Datafiniti's property data to do some marketing analysis on residential home inventory. Let's say you're a data scientist that's been tasked with the following:

1. Collect data on homes.
2. Sort the data by state.
3. Find which states have the most properties for sale.

Your environment and data needs:

1. You're working with Postman.
2. You want to work with CSV data.

Here are the steps we'll take:

1. Download Postman

We recommend using Postman when accessing the Datafiniti API. It's a great client for using any sort of RESTful API and works particularly well with Datafiniti. You can download Postman here.

2. Get your API token

The next thing you'll need is your API token. The API token lets you authenticate with Datafiniti API and tells it who you are, what you have access to, and so on. Without it, you can't use the API.

To get your API token, go the Datafiniti Web Portal (https://portal.datafiniti.co), login, and click on your account name and the top-right. From there, you'll see a link to the "My Account" page, which will take you to a page showing your token. Your API token will be a long string of letters and numbers. Copy the API token or store it somewhere you can easily reference.

📘

For the rest of this document, we'll use AAAXXXXXXXXXXXX as a substitute example for your actual API token when showing example API calls.

3. Run your first search

The first thing we'll do is do a test search that will give us a sense for what sort of data might be available. Eventually we'll refine our search so that we get back the most relevant data.

Since we want homes in the US, let's try a simple search that will just give us online listings for properties in the US.

Open Postman. You should see a new tab open.

Copy and paste the following URL into where it says "Enter request URL". (replace the dummy API token with your real API token):

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/data/properties?view=properties_all&format=JSON&q=country:US&records=1&download=false

Click the blue "Send" button. You should get a response similar to this:

{
  "estimated total": 7983205,
  "records": [
    {
      "address": "711 Kent Ave",
      "brokers": [
        {
          "agent": "Raj Singh",
          "company": "YOUR REALTY INC.",
          "dateSeen": [
            "2016-06-06T18:09:28Z"
          ],
          "sourceURLs": [
            "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970"
          ]
        }
      ],
      "city": "Catonsville",
      "country": "US",
      "dateAdded": "2016-06-06T18:09:28Z",
      "features": [
        {
          "key": "Air Conditioning",
          "value": [
            "Heat Pumps"
          ]
        },
        {
          "key": "Sewer Type",
          "value": [
            "Public"
          ]
        }
      ],
      "keys": [
        "us/md/catonsville/711kentave",
        "mlsnumber/us/md/bc9677283",
        "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970/homepath.com-46273970"
      ],
      "latitude": "39.284462",
      "listingName": "711 Kent Ave, Catonsville, Md 21228",
      "longitude": "-76.734069",
      "lotSizeValue": 0.16,
      "lotSizeUnit": "Acres",
      "mlsNumber": "BC9677283",
      "numBathroom": 2,
      "numBedroom": 4,
      "postalCode": "21228",
      "prices": [
        {
          "amountMax": 199900,
          "amountMin": 199900,
          "currency": "USD",
          "dateSeen": [
            "2016-08-08T00:00:00Z",
            "2016-08-03T00:00:00Z"
          ],
          "isSale": "false",
          "sourceURLs": [
            "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970"
          ]
        },
        {
          "amountMax": 212000,
          "amountMin": 212000,
          "currency": "USD",
          "dateSeen": [
            "2016-06-06T00:00:00Z"
          ],
          "isSale": "false",
          "sourceURLs": [
            "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970"
          ]
        }
      ],
      "propertyTaxes": [
        {
          "amount": 3195,
          "currency": "USD",
          "dateSeen": [
            "2016-06-06T18:09:28Z"
          ],
          "sourceURLs": [
            "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970"
          ]
        }
      ],
      "propertyType": "Single Family Dwelling",
      "province": "MD",
      "sourceURLs": [
        "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970"
      ],
      "statuses": [
        {
          "dateSeen": [
            "2016-08-09T09:16:10Z"
          ],
          "isUnderContract": "false",
          "sourceURLs": [
            "https://www.homepath.com/listing/711-kent-ave-catonsville-md-21228-46273970"
          ],
          "type": "For Sale"
        }
      ],
      "websiteIDs": [
        "homepath.com-46273970"
      ],
      "id": "AV9WzHyO_RWkykBuv11F"
    }
  ]
]

Let's break down what the API call is all about:

Now let's dive through the response the API returned:

Within the records field, you'll see a single business returned with multiple fields and their values associated with that business. The JSON response will show all fields that have a value. It won't show any fields that don't have a value.

Each property record will have multiple fields associated with it. You can see a full list of available fields in our Property Data Schema.

4. Refine your search

If you think about the original query we made, you'll realize we didn't really specify we only wanted homes for sale. There are several other types of properties (e.g., commercial, rentals) that may also be in the data. Since we only want homes for sale, we should narrow our search appropriately.

We'll need to refine our search to make sure we're only getting US homes for sale. To do that, we can add additional filters to the q parameter to narrow down the results. For example:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/data/properties?view=properties_all&format=JSON&q=country:US AND propertyType:"Single Family Dwelling"&records=10&download=false

This API call is different in a couple ways:

1. It adds AND propertyType:"Single Family Dwelling" to narrow down results to just US hotels.
2. It changes records=1 to records=10 so we can look at more sample matches.

Datafiniti lets you construct very refined boolean queries. If you wanted to do more complicated searches, you could OR operations, negation, and more.

Update your API call in Postman and hit enter to see the updated results.

5. Initiate a full download of the data

Once we like what we see from the sample matches, it's time to download the entire data set! To do this, we're going to make three changes to our API call:

1. Change format=JSON to format=CSV. We'll want to view the files in CSV format, so it will be easier to look at.
2. Remove &records=10.
3. Change properties_all to properties_all_nested. This will make each unique price for each property record its own row while nesting all other multi-valued fields (like propertyTaxes).
4. Change download=false to download=true.

After those changes, the API call looks like:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/data/properties?view=properties_all_nested&format=CSV&q=country:US AND propertyType:"Single Family Dwelling"&download=true

When you make this API call, you'll see a response similar to:

[
  {
    "id": 6073,
    "records": "",
    "query": "country:US AND propertyType:\"Single Family Dwelling\"",
    "dataType": "properties",
    "dataFormat": "csv",
    "accountId": "AAAXXXXXXXXXXXX",
    "planId": "free",
    "view": "properties_all_nested",
    "async": 1,
    "status": "STARTED",
    "error": "",
    "numDownloaded": 0,
    "numFound": 95378,
    "numRequested": 95378,
    "date_started": "2017-1-11 15:52:42",
    "date_completed": "2017-1-11 15:52:43"
  }
]

Here's what these fields mean:

6. Monitor the status of the download

As the download request runs, you can check on its status by using this API call:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/requests/XXXX

You'll want to replace XXXX with the id value for your request. If you keep making the above API call, you'll see numDownloaded gradually increase.

7. Download the result file(s)

Once the download finishes, the /requests/ call will show COMPLETED for status. You can then get links to the data files by making this API call:

https://AAAXXXXXXXXXXXX:@api.datafiniti.co/v3/results/XXXX

(Again, replace XXXX with the actual request id.) The response will look similar to:

[
  {
    "id": 120138,
    "url": "http://datafiniti-downloads.s3.amazonaws.com/AAAXXXXXXXXXXXX/6073_1.csv?AWSAccessKeyId=AKIAIXQMCWHOZB3O35SA&Signature=2YtBsW9xY8CZrWDECcdLzyx4Jlk%3D&Expires=1484754763",
    "request_id": 6073,
    "downloaded": 0,
    "imported": 0,
    "date_created": "2017-1-11"
  }
]

If you've requested a lot of records (i.e., over 10,000), you may see more than 1 result object shown.

To download the result files, copy each url value and paste it into your browser. Your browser will initiate a download to your computer.

8. Open the result file(s) in Excel

Navigate to the file you downloaded and open it. Since it's a CSV file, it should open in Excel automatically. It will look something like:

9. Analyze the results

Using Excel, we can easily do some analysis on the number of properties for each state. The province column gives us where each property is located, so we can use it to tally up the numbers.