Property Data with Postman and CSV

For this guide, we're going to assume you're interested in using Datafiniti's property data to do some marketing analysis on homes in the US. Let's say you're a data scientist that's been tasked with the following:

1. Collect data on homes.
2. Sort the data by state.
3. Find which states have the most expensive homes.

Your environment and data needs:

1. You're working with Postman.
2. You want to work with CSV data.

Here are the steps we'll take:

1. Download Postman

We recommend using Postman when accessing the Datafiniti API. It's a great client for using any sort of RESTful API and works particularly well with Datafiniti. You can download Postman here.

2. Download Postman collection

You can download a Postman collection directly from our web portal. Login at https://portal.datafiniti.co/login and then go to your settings page. Click on the "Get Postman Collection" to download your collection.

The collection will be pre-populated with several API calls that use your API token.

Import this collection into Postman. You can do this by opening Postman and clicking on "Import" at the top-left of your screen.

📘

For the rest of this document, we'll use AAAXXXXXXXXXXXX as a substitute example for your actual API token when showing example API calls.

3. Run your first search

The first thing we'll do is do a test search that will give us a sense for what sort of data might be available. Eventually we'll refine our search so that we get back the most relevant data.

Since we want homes in the US, let's try a simple search that will just give us online listings for US-based properties.

With Postman open, click on the Datafiniti API V4 collection. Then navigate to Secured Endpoints > Search > Search Property Data. Click on "Body" in the main screen.

Enter the following in the text area:

{
  "query": "country:US",
  "num_records": 1
}

Click the blue "Send" button. You should get a response similar to this:

{
  "num_found": 7983205,
  "total_cost": 1,
  "records": [
    {
      "address": "711 Kent Ave",
      "brokers": [
        {
          "agent": "Raj Singh",
          "company": "YOUR REALTY INC.",
          "dateSeen": [
            "2016-06-06T18:09:28Z"
          ],
        }
      ],
      "city": "Catonsville",
      "country": "US",
      "dateAdded": "2016-06-06T18:09:28Z",
      "features": [
        {
          "key": "Air Conditioning",
          "value": [
            "Heat Pumps"
          ]
        },
        {
          "key": "Sewer Type",
          "value": [
            "Public"
          ]
        }
      ],
      "latitude": "39.284462",
      "listingName": "711 Kent Ave, Catonsville, Md 21228",
      "longitude": "-76.734069",
      "lotSizeValue": 0.16,
      "lotSizeUnit": "Acres",
      "mlsNumber": "BC9677283",
      "numBathroom": 2,
      "numBedroom": 4,
      "postalCode": "21228",
      "prices": [
        {
          "amountMax": 199900,
          "amountMin": 199900,
          "currency": "USD",
          "dateSeen": [
            "2016-08-08T00:00:00Z",
            "2016-08-03T00:00:00Z"
          ],
          "isSale": "false",
        },
        {
          "amountMax": 212000,
          "amountMin": 212000,
          "currency": "USD",
          "dateSeen": [
            "2016-06-06T00:00:00Z"
          ],
          "isSale": "false",
        }
      ],
      "propertyTaxes": [
        {
          "amount": 3195,
          "currency": "USD",
          "dateSeen": [
            "2016-06-06T18:09:28Z"
          ],
        }
      ],
      "propertyType": "Single Family Dwelling",
      "province": "MD",
      "statuses": [
        {
          "dateSeen": [
            "2016-08-09T09:16:10Z"
          ],
          "isUnderContract": "false",
          "type": "For Sale"
        }
      ],
      "id": "AV9WzHyO_RWkykBuv11F"
    }
  ]
]

Let's break down each of the parameters we sent in our request:

Now let's dive through the response the API returned:

Within the records field, you'll see a single property returned with multiple fields and the values associated with that property. The JSON response will show all fields that have a value. It won't show any fields that don't have a value.

Each property record will have multiple fields associated with it. You can see a full list of available fields in our Property Data Schema.

4. Refine your search

If you think about the original query we made, you'll realize we didn't really specify we only wanted homes for sale. There are several other types of properties (e.g., commercial, rentals) that may also be in the data. Since we only want homes for sale, we should narrow our search appropriately. Modify your request body to look like this:

{
  "query": "country:US AND propertyType:\"Single Family Dwelling\"",
  "num_records": 10
}

This API call is different in a couple ways:

1. It adds AND propertyType:"Single Family Dwelling" to narrow down results to just US hotels.
2. It changes records=1 to records=10 so we can look at more sample matches.

Datafiniti lets you construct very refined boolean queries. If you wanted to do more complicated searches, you could OR operations, negation, and more.

Hit "Send" to see the updated results.

If you would like to narrow your search to just exact matches you can place the search term in quotation marks.

5. Initiate a download of the data

Once we like what we see from the sample matches, it's time to download a larger data set! To do this, we're going to further modify our request body to look like this:

{
  "query": "country:US AND propertyType:\"Single Family Dwelling\"",
  "num_records": 50,
  "format": "csv",
  "view": "property_flat_prices",
  "download": true
}

Here's what we changed:

1. We added "format": "csv. If you don't specify format, it will default to json. Using the CSV format will make analyzing our data easier for this example.
2. We changed "num_records": 10 to "num_records": 50. This will download the first 50 matching records. If we wanted to download all matching records, we would remove num_records. num_records will tell the API to default to all available records.
3. We added "view": "property_flat_prices". Using property_flat_prices will nest fields like features into a single cell, while splitting each price for every home into multiple rows.
4. We added "download": true. This tells the API to issue a download request instead of a search request.

🚧

If num_records is not specified, ALL of the records matching the query will be downloaded.

When you make this API call, you'll see a response similar to:

{
    "id": 7,
    "results": [],
    "user_id": 15,
    "status": "running",
    "date_started": "2017-11-16 17:46:06.0",
    "num_downloaded": 0,
    "data_type": "property",
    "query": "country:US AND propertyType:\"Single Family Dwelling\"",
    "format": "csv",
    "num_records": 50,
  	"total_cost": 50
}

We'll explain each of these fields in the next section.

🚧

When using the API, you will not receive any warning if you are going past your monthly record limit. Keep a track on how many records you have left by checking your account. You are responsible for any overage fees if you go past your monthly limit.

6. Monitor the status of the download

As the download request runs, you can check on its status by using the "Get Download" call in your Postman collection. Just replace :id with the id value you saw in the previous step.

If you keep running this call, you'll see some of the values update. Once the download completes, it will look something like this:

{
    "id": 7,
    "results": [
        "https://datafiniti-downloads.s3.amazonaws.com/15/7_1.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20171116T174607Z&X-Amz-SignedHeaders=host&X-Amz-Expires=604800&X-Amz-Credential=AKIAJYCTIF46QVBTXWYA%2F2017xxxx%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=ecf13f1bb4b7adfdde1a99143541afd1d12347292eb9ec3f6ed1316c64d4eekf"
    ],
    "user_id": 15,
    "status": "completed",
    "date_started": "2017-11-16 17:46:06.0",
    "date_updated": "2017-11-16 17:46:07.0",
    "num_downloaded": 1,
    "data_type": "property",
    "query": "country:US AND propertyType:\"Single Family Dwelling\"",
    "format": "csv",
    "num_records": 50,
  	"total_cost": 50
}

Here's what these fields mean:

7. Download the result file(s)

Once the download response shows "status": "completed", you can download the data using the URLs in the results field.

If you've requested a lot of records (i.e., over 10,000), you may see more than 1 result object shown.

To download the result files, copy each url value and paste it into your browser. Your browser will initiate a download to your computer.

8. Open the result file(s) in Excel

Navigate to the file you downloaded and open it. Since it's a CSV file, it should open in Excel automatically. It will look something like:

9. Analyze the results

Using Excel, we can easily count the total number of hotels on a state-by-state basis. The province column gives us where each hotel is located, so we can use it to tally up the numbers.