V2 Design

[AnthonyBloomer]

I have been looking into how the internal Daft API works and how we can filter data using a JSON payload. I've figured out most things except how to filter by location. This is an example class:

import requests
from pprint import pprint


class Daft:
    
    def __init__(self):
        self._min_beds = 1
        self._max_beds = 2
        self._min_price = 1500
        self._max_price = 2500
        

    @property
    def min_beds(self):
        return self._min_beds

    @min_beds.setter
    def min_beds(self, min_beds):
        self._min_beds = min_beds

    @property
    def max_beds(self):
        return self._max_beds

    @max_beds.setter
    def max_beds(self, max_beds):
        self._max_beds = max_beds

    @property
    def min_price(self):
        return self._min_price

    @min_price.setter
    def min_price(self, min_price):
        self._min_price = min_price

    @property
    def max_price(self):
        return self._max_price

    @max_price.setter
    def max_price(self, max_price):
        self._max_price = max_price

    def search(self):
        r = requests.post("https://search-gateway.dsch.ie/v1/listings", headers={
            "Content-Type": "application/json",
            "brand": "daft",
            "platform": "web"
        }, json={
            "section": "residential-to-rent",
            "filters":
                [
                    {
                        "name": "adState",
                        "values":
                            [
                                "published"
                            ]
                    },
                    {
                        "values":
                            [
                                "unfurnished"
                            ],
                        "name": "furnishing"
                    }
                ],

            "andFilters":
                [
                    {
                        "values":
                            [
                                "alarm"
                            ],
                        "name": "facilities"
                    }
                ],

            "ranges":
                [
                    {
                        "from": self.min_price,
                        "to": self.max_price,
                        "name": "rentalPrice"
                    },
                    {
                        "from": self.min_beds,
                        "to": self.max_beds,
                        "name": "numBeds"
                    },
                    {
                        "from": "12",
                        "to": "",
                        "name": "leaseLength"
                    }
                ],
            "paging":
                {
                    "from": "0",
                    "pageSize": "20"
                },
            "terms": ""
        }, )
        return r.json()


if __name__ == '__main__':
    daft = Daft()
    daft.min_beds = 1
    daft.max_beds = 2
    daft.min_price = 1500
    daft.max_price = 2000
    listings = daft.search()
    for listing in listings['listings']:
        pprint(listing)

What I was thinking is that the Daft class would be responsible for building the JSON payload based on what the user sets which is similar to how this library worked previously except no scraping would be necessary.

Still a little unsure so open to suggestions on ways to approach this.

[sdfordham]

For location you add e.g.:

"geoFilter":{"storedShapeIds":["4090"],"geoSearchType":"STORED_SHAPES"}

To get the id, the dictionary is embedded in the page source (!), it is too large for pastebin so I upload the dict: [EDIT: there was trash in the middle, updated]

Pastebin

The question would be its practical use, if the location id dict is integrated with class, then how does a user query since the user would need to select say from an enum. Maybe we let the user do daft.set_location(foo) and then do a search in the enum for foo the appropriate location id-pair. Otherwise the user has to do e.g. daft.set_location(LOCATION.howth-dublin).

[AnthonyBloomer]

Great find! I like this approach. We could just search that dict for the given query and append the appropiate object to the request payload.

[sdfordham]

Also I really like the simplicity of this approach as compared to e.g. trying to parse things together to form a URL

[sdfordham]

Can keep this as a reference and edit as necessary

Parameters:

• section: str, (not mandatory (not sure how it figures this out when not present, possibly from propertyType) if a list is given then an error is returned)

  • residential-for-sale
  • residential-to-rent
  • commercial-for-sale
  • commercial-to-rent
  • sharing
  • student-accommodation-to-share
  • new-homes

• filters: list[{"name": str, "values": list[str]}] (multiple lists elements with the same name key gives weird results, do really need to join them)

  • mediaTypes, [virtual-tour, video]
  • propertyType, [houses, detached-houses, semi-detached-houses, terraced-houses, end-of-terrace-houses, townhouses, apartments, studio-apartments, duplexes, bungalows, sites, office-spaces, retail-units, industrial-units, restaurants-bars-hotels, commercial-sites, agricultural-land, development-land, industrial-sites, investment-properties]
  • saleType, [auction]
  • adState, [published, sale-agreed] (here published means 'Available')
  • furnishing, [unfurnished]
  • location, [galway, cork-city, ...?] (probably any "displayValue" value from the location-id dictionary)

• andFilters: list[{"name": str, "values": list[str]}]

  • facilities, [alarm, gas-fired-central-heating, oil-fired-central-heating, parking, wheelchair-access, wired-for-cable-television, cat-5-cabling, cat-6-data-cabling, kitchen-area, meeting-rooms, reception, parking, phone-lines, toilets]

• ranges: list[{"name": str, "from": str, "to": str}] (won't give an error and returns the right results when list has multiple elements with the same name key)

  • rentalPrice, ("" or integer value up-to "5500000")
  • salePrice, ("" or integer value up-to "5500000")
  • floorSize, ("" or positive integer up-to "100000" (measured in sq m2))
  • numBaths, (any non-negative integer between double-quotes (?) )
  • numBeds, (any non-negative integer between double-quotes (?) )
  • salePrice, (any non-negative integer between double-quotes (?) )
  • ber, (A-H as integer between double-quotes (?) )
  • firstPublishDate, "from" and "to" are "" or "now-$d/d" where $ is 1/3/7/14/30
  • leaseLength (any integer 1-12 between double-quotes (?) )
  • viewingTimes, (date ranges)

• paging: {"from": str, "pagesize": str} (pagesize anything larger than 50 is taken as 50)

• geoFilter: {"storedShapeIds": list[str], "geoSearchType":"STORED_SHAPES"} ("id" from the location-id dictionary)

• terms: str (/list?), (keywords entered in textbox)

• sort: str,

  • priceAsc

[EricNeville]

Where did you get the list of facilities given here within andFilters? I'm working on adding a facilities filter but when comparing with the available facilities in the search on the daft site there are a number of facilities given in the list above which don't appear on daft, and there are also a number of the ones from daft not present in the list above.

[sdfordham]

By accident... if you go to daft and do a search in the sale section with some facilities added then inspect the page-source, at the bottom there is a very very long line containing a big dict, the extra options can be seen in the middle (they seem to be sandwiched between two copies of the location-id dict). Whether the values in this dict that are not available in the search filter are legitimate options in the POST request remains to be checked. I originally added them to that enum for completeness sake.

[sdfordham]

Here is bare-bones outline for search class.